Behavioural Computer Science: an agenda for combining modelling of human and system behaviours

The marriage of ubiquitous computing and AI opens up for an environment where humans will interact with autonomous systems that will be indistinguishable from other humans or other systems directly controlled by humans. Not only must humans relate to intelligent machines, the same machines must relate to humans and to other intelligent machines.

For humans, it could be a strange experience to interact with intelligent machines that might have contradicting traits of human behaviour. For example, a human normally reacts defensively (or aggressively) when physically attacked, but a human-looking robot might not react at all, or exhibit behaviour that seams strange to a human, but which is only intended at achieving the job that it was programmed for. The implications of this disconnection between traits that are usually connected (i.e. human-looking and self-defence) is something humans will have to get used to.

Our ethical compass should guide us to build intelligent machines that have desirable traits, whatever that might be. In order to achieve this goal it is essential that we understand haw humans actually behave in interaction with intelligent machines, and this is a largely unexplored field. For example, what are the criteria for trusting an intelligent machine for which the intelligent behaviour a priori is unknown. Also, how can an intelligent machine trust humans with whom it interacts. Finally, how can intelligent machines trust each other. From a security point of view, the most serious vulnerabilities are no longer found in the systems but in the humans who operate the systems. In a sense, it is no longer a question of whether people can trust their systems, but whether systems can trust the humans operating them. These are daunting challenges in the envisaged intelligent ubiquitous computing and cyberphysical infrastructure.

Three important fields of scientific study are fundamental to understanding and designing the future smart-* infrastructures:

• Behavioural sciences: working with systematic analysis and investigation of human behaviour through controlled and naturalistic observation and disciplined scientific experimentation. It attempts to accomplish legitimate, objective conclusions through rigorous formulations and observation. Examples of behavioural sciences include psychology, psychobiology, criminology and cognitive science. In contrast to traditional, rational and normative approaches to how people should ideally behave (we use behaviour as a general concept that includes the subcategories judgement and decision making), behavioural sciences give scientific, empirical, evidence-based, and descriptive approaches to how people actually make judgements and decisions. Thus, these two approaches are complementary: the rational agent model prescribes the ideal behaviour, whereas the behavioural model describes the actual behaviour.

• Ubiquitous Computing and IoT: is a new paradigm in software engineering and computer science where computing is made to appear anytime and everywhere. In contrast to desktop computing, ubiquitous computing can occur using any device, in any location, and in any format. A user interacts with the computer, which can exist in many different forms, including laptop computers, tablets and terminals, but mostly everyday objects such as a fridge or a pair of glasses. The underlying technologies to support ubiquitous computing include Internet, advanced middleware, operating system, mobile code, sensors, microprocessors, new I/O and user interfaces, networks, mobile protocols, location and positioning and new materials. The IoT (Internet of Things) is the connected aspect of ubiquitous computing.

• Artificial intelligence: abbreviated AI, is the intelligence exhibited by machines or software. It is also the name of the academic field which studies how to create computers and computer software that are capable of intelligent behaviour. Major AI researchers and textbooks [1] define this field as “the study and design of intelligent agents”, in which an intelligent agent is a system that perceives its environment and takes actions that maximise its chances of success according to some criteria.

We organize these areas under a new, broader concept which we call “Behavioural Computer Science” (abbreviated BCS). Any outcome of integrating models from these three areas would be called a BCS-model, which will always include behaviour aspects in some way or another. We would like to encourage research focus on the interactions between these three areas. This is illustrated in Fig. 1.

The intersections between any two of these areas represent existing or new research disciplines.

Human–Computer interaction (HCI) and interaction design [2] studies how a technology product should be developed having the user in focus at all stages. One focus of HCI is on the user interface (UI) of a technology, which allows the user to interact with the machine. Traditionally we have seen a two way communication, where the user volunteerly provides input through standard means such as button presses, and receives information e.g., through displays or lights. With the advent of ubiquitous computing, the Internet of Things (IoT) and advanced AI, the distinction and interface between computers and human becomes very blurred. In a smart-home environment the IoT constantly receives information from the human occupant of the house, without her even knowing it.

Artificial behaviour is an emerging discipline which focuses on understanding how intelligent systems behave from a macro-perspective, and not from a computer-program perspective. When intelligent systems become intelligent enough they will have psychological traits that can be studied. As soon as the machine has actuators it can also change its surroundings, which when humans are involved it can turn into artificial influence on human behaviour. Smart home and smart infrastructures, as well as web services based on big data, are emerging technologies highly susceptible to artificial influence. Computational trust [3, 4] becomes an aspect of machine learning or heuristics, that in turn will be part of IoT systems and other (semi-)autonomous (self-*) systems. For such systems that can perform complex computations and communications, we need to study notions of trust, like trust of the user in the system, or of another interacting system or component. Machine ethics is that part of the Ethics of Artificial Intelligence concerned with the moral behaviour of artificially intelligent beings. The field of Roboethics [5, 6] is concerned with the moral behaviour of humans as they design, construct, use and treat such beings.

In domains where humans interact with technology it is necessary to understand human behaviour in order to capture or foresee possible actions taken by humans in interaction with the technology. We refer here to an understanding that can be used by machines, thus through models that can be used in some forms of computations. If technology and their designers understand the typical tendencies of human cognition, emotion and action, it is easier for the resulting system to take into consideration how people actually behave, and adapt to this, instead of relying on assumptions about how they may behave.

As this implies, there are two primary approaches to including human models: one would follow the Rational Agent Model, prescribing the ideal behaviour (e.g., [22–24]) and another the Behavioural Model of Human Agency, describing the actual behaviour (e.g., [25, 26]). The rationalist approach to explaining human behaviour from a priori assumptions, is traditionally widespread—in academia, in professional practice, as well as in society in general. The rationalist tradition generally adheres to the view that people are rational agents that seek to maximize utility. Inherent in this approach lie the assumptions that (i) people know their ultimate goal, (ii) have the means to select the courses of action that are the most likely to lead to goal achievement, and (iii) have the capability to carry out the appropriate courses of action. To do arrive at this end-state, people would need (i) to have unlimited access to all information, (ii) the ability to discriminate relevant from non-relevant information, (iii) the cognitive capability to handle and analyse the interaction between the relevant informational components inherent in the possible courses of action, (iv) to calculate how the courses of action would lead to the possible end-states, and (v) to foresee implications of the end-states. We shall soon see that these assumptions are seldom fulfilled, which leads us to focusing on the Behavioural Model of Human Agency.

One of the first proponents of the behavioural model was Herbert Simon, the 1978 Nobel Laureate in Economics. Simon found that people, when making real judgements and decisions, did not comply with the ideal that was assumed by the rationalist traditions. He was the first to coin the term Bounded Rationality [24] to describe the concept of non-ideal adherence to the rationalist assumptions and thus complemented the traditional rationalistic approach in the field of economics.

Prominent scholars in behavioural Science, after Herbert Simon, are Daniel Kahneman, the 2002 Nobel Laureate in Economics, and his colleague Amos Tversky. Notable findings in Kahneman and Tversky’s research [26, 27] are that people often rely on intuitive thinking when making judgements under conditions of uncertainty. Intuitive thinking, when employed inappropriately in conditions when instead analytic thinking would have been the correct cognitive strategy, often leads to biased (and consequently incorrect) judgements. Although the rationalistic approach is valuable in explaining how people should ideally make judgements under conditions of certainty, the behavioural approach is the better in explaining how people actually make judgements in uncertain conditions, and also identifying the bias-inducing psychological mechanisms people employ (mostly without conscious awareness) [28].

To anchor our thoughts we will use a model introduced in [33], which we call “the Bella–Coles–Kemp model” and abbreviate as BCK model. This is a rather general and abstract model for any forms of human involvement in computer systems, thus providing a good common basis for defining Behavioural Computer Science concepts.

The BCK model, pictured in Fig. 2, provides abstractions and separations, while allowing for more details to be given for each of the vertical layers individually. We can see the behaviour sciences (like psychology and cognitive sciences) as a good ground for finding specific models for the layers V and IV; whereas for layer III we already have a good start in works from Human-Automation Interaction (HAI). Traditional computer science studies layers I, II, and also partly layer III.

When explaining the BCK model it is good to make correlations with existing established concepts; and we choose here models for security protocols, usually based on the Dolev–Yao assumptions [34], and using specification languages like the applied pi-calculus [35, 36]. Usually, security protocols are formed of the parties (or players) and the interaction medium they use for communication (or for any other exchange of information). The parties are usually honest, whereas the intruder (attacker) controls the interaction medium. More than two parties can be involved in a protocol, but for our example purposes here let us consider only two honest parties, Alice and Bob. Third parties, usually dishonest, appear due to the ability of the intruder to disguise as a party in any number of protocol runs. The Dolev–Yao model defines the powers that the attacker has over the interaction medium, like power to delete, change, or insert messages, to and from any other party.

In the BCK model the parties form the light boxes, whereas the interaction medium forms the dark boxes. The parties appear at different layers of the BCK model and in different abstractions; i.e., the light boxes represent the players in the respective layer, which are abstractions of the parties or are controlled by the parties. In the layer I (also called “Informational”) we encounter the processes \(p_{A}\) and \(p_{B}\) controlled by Alice respectively Bob, which are running the computers of Alice and Bob, communicating through the network, i.e., the dark box. Layer I would thus be studied by communicating systems and networking researchers, and for security it could be subject to the Dolev–Yao assumptions. However, this layer has also aspects not related to security, like properties of the transmission medium (e.g., messages can be lost or not, or delayed and for how long).

In BCK other players appear at the other layers: at layer II (also called “Operating System”) the user interface \(UI _{A}\) associated to Alice, which interacts with the computer process \(p_{A}\), e.g., by sending information taken from the user required by the protocol run by \(p_{A}\), like a password or biometrics. The same \(UI _{A}\) interacts at layer III (also called “Human–Computer Interaction”) with a “persona” \(P_{A}\) of Alice for some particular system. We adopt the notion of persona to characterize a user in a socio-technical interaction situation. We define a persona to be a set of social and cognitive attributes of a human, including emotions, senses, or memory [37]. The persona has interaction at layer IV (also called “Personal”) with the self \(S_{A}\) of Alice, which in turn is influenced by the Society through various social interaction protocols at layer V.

Players may interact only as part of a layer, and one layer may involve players pertaining to different users. Important to note is that in BCK one player usually is involved in two adjacent layers.

Research in computer science has mostly focused on layers I and II, and largely ignored layers III–V. We see layer II as pertaining also to the technological community, whereas layer V would pertain to the social sciences. Layer IV would be investigated more by the psychology researchers. Layer III on the other hand is at the interaction between technological and social sciences, with a rapidly evolving field, having terminology s.a.: HAI [38], user-centred design [39], interaction design [2].

One usefulness of BCK is to make explicit the need for collaboration between the fields of sociology, psychology, and technology, in order to tackle the complexities of current systems s.a. security ceremonies or IoT. One can very well focus on individual layers, but the BCK model brings the isolated results into the general picture which eventually needs to be handled in order to claim results and designs of practical use.

We expect BCS research to detail all the layers III–V, the same as has been done until now in CS with the layers I and II. The interaction medium, the dark box, can be split into more fine-grained divisions, and each division would have its interaction protocol and assumptions. For layer II it is easy for computer scientists to bring their knowledge of operating systems design and see that a UI could consist of a screen and its driver, a display client, like a browser displaying an input form, a keyboard with its drivers, and the many other components that transport the information between these many UI components and the end process \(p_{A}\). However, for the social protocols of layer V, completely different concepts and models should be used. One could study various means of social manipulation, and quantitative and qualitative measures could be devised for analysing their usefulness in terms of power to influence, e.g., depending on the social scale or training level of the users, i.e., the self \(S_{A}\).

The BCK model has been applied to study security ceremonies in [40] where the focus is on layer III and introduces probabilities as opposed to classic (rationalist view) non-deterministic models. The concepts discussed in this paper are concentrating on layers V and IV, and how these could be modelled and combined with the methods used in CS in layers I and II. Works on layer III constitute a good middle ground [41–43]. We particularly wish to focus our modelling efforts for V and IV on behavioural research, and depart from the rational view and assumptions on human.

In many domains, academic as well as professional and political, it is a generally held view that people make rational judgements and thus are assumed to think, act, decide and behave according to the rational agent model [24]. The rational agent model implies that people always strive to maximize utility.

Another problem with the rational agent model regards the concept of rationality. In this view it is assumed that people act strictly logical and rational in the pursuit of maximized utility. Inherent in this view, conditions are assumed to be certain, meaning that every individual is assumed to have unlimited access to all information and is also capable of analysing the relevant information needed to make a judgement, as well as calculating the outcome of every combination of informational components, so that the best decision can be made. Of course, no single individual is able to adhere strictly to this model of rational behaviour, but the point here is that the rational agent model assumes rationality as a general principle, and does not concern itself with empirical evidence about actual behaviour (i.e., as opposed to inferred behaviour) as to whether this assumption is actually valid.

In summary, some errors in human behaviour often stem from the differences between predicted, experienced and remembered utility; e.g. when making judgements at time \(t_0\) about some consumption related moment in the future at time \(t_1\), one often disregards the fact that their current experiences will be different from their expectations. Errors may also occur as a consequence of making judgements in conditions under uncertainty, i.e., when the requirements of the rational agent model cannot be fulfilled.

Although many proponents of the rational agent model acknowledge that the model is ideal and that it not necessarily realistically reflects actual human behaviour, they nevertheless assume that individuals’ errors of judgement or of informational processing (i.e., their failure to comply with the model and thus make perfect judgements), are non-systematic. Non-systematic errors mean that each single individual does not exhibit the same types of judgement errors as other individuals and, furthermore, these differ over time, i.e., errors made today are different than those made yesterday. Although it is acknowledged that people are not perfect judgement machines and thus make errors, it is assumed that mistakes are non-systematic and thus random [28, 29] (classically modelled through non-determinism).

Kahneman [29], and other behavioural scientists, questioned the explanatory powers of the rational agent model, because they could not make their empirical data fit the rational agent model. As psychologists—or behavioural scientists—they studied how people actually behave, as opposed to how they are assumed to behave according to the rational agent model. Thereby they provided empirical data that supported a new view—namely that people’s judgement errors were not at all non-systematic and thus random, but in fact systematic; people tended to make the same kinds of misjudgments as others did, and misjudgments made today are the same as those made yesterday. Thus, people’s mistakes were more or less universal. Findings like these paved the way for an extension of Bounded Rationality as a concept of understanding human behaviour [24].

For the spear phishing example, where one receives a malicious email from an address that resembles that of a known colleague. This is an attack that is difficult to counter because it activates both the availability heuristic and the representative heuristic; the user may or may not have no easily accessible information stored in the mind that may suggest that this is an hostile attack (susceptibility to the availability heuristic) and, furthermore, the user recognizes the email address as being from a near colleague (the representative heuristic). Additionally, when considering that malicious attackers could also employ mechanisms of social influence [48], such as the six principles of persuasion, wherein e.g., the concept of Authority (people have a tendency to obey instructions from authority figures) or Liking (people have a tendency to be more easily persuaded by people they like), they have access to a versatile tool-kit of psychological manipulations and deceptions, which they could use with malevolent intent. Thus, no alertness or caution is prompted.

Human choices and human prediction power are very important for interactions with computer systems, e.g. security can be influenced by poor predictions about the possibilities of attacks and attack surface can be wrongly diminished in the mind of the human, whereas wrong choices can incur safety problems. In [26] it is argued that it is difficult for a human to make accurate predictions about a situation or an experience (e.g., sentiment, preference, disposition) when the future forecasting time point \(t_{0}\) is rather distant from the current time point \(t_0\) on which the same experience is evaluated. The more distant this time point is, the more inaccurate the prediction (and thus the choice) will be.

We are interested in how behavioural concepts could be mathematically modelled, and more importantly, how these behavioural models can be coupled and integrated with existing models from computer science. Thus, our study here pertains to the layers V and IV of BCK. We start discussing a very simple model, building on the work done in [40] for layer III, based on works from HCI [49–51] or from cognitive theories [52, 53].

We will thus work with a notion of “State” and changes between states (which we usually call “Transitions”). How exactly to model an emotional or motivational state is not trivial, and we discuss these in more detail later. When we model emotions (as needed for affective forecasting, as well as for many aspect of the Self) we can start from the following concepts related to the impact bias [46]: the strength (or intensity) of an emotion and the duration [54]. Both of these can be quantified and included in a quantified model of emotions.

Events is another concept that we identify as influencing the Self, in the sense that emotions are relative to events. Events can be considered instantaneous and are sometimes modelled as transitions labelled by the respective event. The reason is that the event changes the state in some way, e.g., changes the memory of the Self, or attributes of various variables of the context, as well as of the Self.

A cognitive explanation for people’s biased retrieval of past experiences appears when we relate them to emotions. Whereas currently experienced emotions (related to a currently experienced event) are stored in the episodic memory, past experienced emotions (related to previously experienced events) are stored in the semantic memory. The semantic memory is largely susceptible to biases due to the influence of current beliefs about previously experienced emotions on the retrieval of memories [58]. Thus, memory may make people behave in ways unexplainable by the rational model. These concepts contribute to defining models for the predicted and the remembered utilities, as well as how these models correlate with that of the experienced utility.

For modelling a State we can start by including the aspects that are of interest for the situation under study. Aspects could be modelled as logical variable that are true or false in some state, because they are either considered or not considered (i.e., observable/salient or not). The expressiveness of the logic to be used would be dependent on what aspects we are interested in; but we can start by working with predicate logic. Depending on the system being developed, we encourage to choose the most suited logic, e.g.: the SAL languages and tools which have been nicely used to describe the cognitive architecture of [10], Sec.2]; or one can use higher-order dynamic logic [59], Chap.3] and the tools around it like the KeY system [60].

Modelling memory and especially how can memory be manipulated and how the memory influences choices and thus transitions between states is not easy. Quite a few studies can be considered [61, 62], some of which are more close to models and to logics [63–65]. We can also use models from computer science and logics like dynamic logic [59], used to talk about programming data structures, but also logics of knowledge and belief [66] which have well developed models for how beliefs can be updated over time due to various kinds of changes [67].

The relation between the Self and the Persona can be seen as a simplification relation (called projection in more formal terms). The projection operation is done on a subset of the variables that make up the State of the Self, thus resulting in the state of the Persona. This projection would retain only those aspects that are relevant in the respective context, i.e., in the context of the computer system being studied. This means that the projection operation should also be related to the model of the UI (i.e., the one between layers II and III).

For the second direction we see more the actions of expression (e.g., described by [10, 50]) that the Self makes out of the thoughts, reasoning, intuition, past experiences and memory models, into something relevant to this BCS-system and to the UI that the human interacts with. In consequence, we may say that the Self interacting with the UI is filtered by the Persona we designated. But this Self is aware of more than just the UI: maybe she is aware of computer networking aspects (which pertain to layer I) or operating systems aspects of layer II (like how browsers work or how the operating system can be protected from bugs and viruses, whether an antivirus is installed or a firewall, etc.). All these examples are outside the direct visual interaction of the Persona with the UI, which is captured through the layer III, thus part of layer IV. The few studies on modelling human interacting with a user interface using formal methods [33, 71] are based on a non-deterministic approach, whereas [40] develops a probabilistic model in a more general setting.

Interactions at layers V and IV would be studied empirically, including also models of the Self and of Personas. A model would start from general assumptions, incorporated as prior information/probabilities. For a specific system, with a specific Persona defined, the model would need to be updated, in a continuous fashion, by gradually learning from each empirical study and new evidences, thus updating the priors.

Because we use empirical evidences we need to introduce a notion of uncertainty about the probabilities that the studies reveal. Therefore, models of subjective logic [72] could be useful for expressing things like: “The level of uncertainty about this value given by this empirical study is the following.”.

Modelling frameworks of use for BCS exist in various maturity degrees, and BCS research can consider these as a starting point. One line of research follows the so called Actor Network Procedures (ANP) introduced in [74] and the alternative works of [75, 76]. These have the intention of modelling security ceremonies [7, 8] by drawing inspiration from the Actor Network Theory in sociology, where the book [77] gives a good overview. ANP is meant to be a graphical modelling language, yet with formal logical reasoning underpinnings. For BCS modelling we encourage the use of graphical languages, where prime examples that are also formally grounded include the statecharts [78, 79] and the live sequence charts [80], which were intended for describing concurrent and reactive systems.

Another line of work comes from human factors engineering [81, 82] that use task analytic models [83–87] to capture the human behaviour involved in (often for controlling) the system. In this direction, efforts into modelling human behaviour combined with models of the system have been used for doing model checking in [88, 89]. Modelling has also been related to the development [90] and evaluation [91] of user interfaces. In particular, the area of modelling user experience is largely unexplored [92].

We have argued that concepts and findings from behavioural sciences can be translated into models useful for computer science. Such models could be used for analysing the BCS-systems using techniques such as automated model checking [57]. Moreover, behavioural models and related modelling languages can be used by system developers when making new BCS-systems to also consider the human interacting with the system. We can already see promising results in this direction from using formal methods to analyse HAI systems [38] or human related security breaches [10].

As we have shown, there is now an abundance of research arguing that people behave and act in other ways than those assumed under the rationality paradigm. In consequence we have proposed that computer science incorporates in the design of systems that interact with humans knowledge about actual behaviour. Psychology and Behavioural Sciences have by now provided a large amount of empirical evidence showing how human behavioural tendencies in many instances depart from strict rational assumptions.

Knowledge about human behaviour and tendencies have often been employed by private interests for commercial purposes, for example with the aim of influencing or convincing people to purchase a specific product or service [93]. Such knowledge has also been employed by political interests in order to convince people to endorse a particular political view. Even if some would argue that such approaches may have been employed in people’s own interests, in order to have people make choices that are actually good for them, others would argue that the main purpose of such approaches is to serve the initiator—whether this is a private company interest or a political agenda.

Thus, some may fear that if, by following our proposal, models of human behaviour are made such that machines can work with them, then more easy it can be for a totalitarian regime to control people by using computers for mass-surveillance/-manipulation. One could even fear such models also in a democratic society because large corporations that control information, like those involved in search engines, social media, network corporations, or device and software producers, could be tempted to use such models in a negative manner, trying to gain control over their users. Indeed, any large corporation could possibly gain access to data from the previously mentioned providers and use behaviour models to extort their users, but this is a matter that should be regulatory controlled in a context of legislation.

Contrary to such possible uses of behavioural-machine models (BCS-models), our main intention with the BCS proposal is to take the perspective of the individual when building HAI-interfaces that take into consideration human behavioural tendencies (in the interest of humans instead of political or commercial interests) with the aim of designing systems that empower individuals to make more correct judgements when interacting with an automated system.

Our proposal would serve the individual and thereby the society, as well as system designers and owners, due an increased knowledge about human behavioural tendencies. Additionally, when taking into consideration that society and all its functions rely both on situational awareness and on resilience –in infrastructure as well as in individual citizens– in order to maintain structure and daily life in all societal domains, it is imperative that disruptions of vital societal functions can be avoided or countered [94]. It is thus not an overstatement to claim that the new conceptual domain of Behavioural Computer Science would also have valuable and positive implications for institutions such as National Security Agencies (seeking to secure society and to make society resilient) and for National Intelligence Agencies (seeking to identify threats to society and prompt situational awareness). Future research would explore implications of the concept of Behavioural Computer Science in contexts where individuals and systems interact, ranging from smart living and IoT to national security and intelligence domains.