Intention awareness: improving upon situation awareness in human-centric environments

As computer systems continue to improve in their information processing capability and become more integrated into the everyday lives of end-users, the potential scope of their roles increases as well. For instance, recent attempts to augment the human decision-making process, especially in dynamic and time-sensitive scenarios such as military command and control, game theory, home automation, and swarm robotics, have focused primarily on environmental details such as positions, orientations, and other characteristics of objects and actors of an operating environment. However, a significant factor in such environments is the intentions of the actors involved. While creating systems that can shoulder a greater portion of this decision-making burden is a computationally intensive task, performance advances in modern computer hardware bring us closer to this goal.

This paper discusses Intention Awareness (IA) as the process of integrating actors’ intentions into a unified view of the surrounding environment. IA includes many of the basic principles of Situation Awareness (SA), such as consistent tracking and extrapolation of objects in the user’s environment, but also exploits circumstantial semantics and sentics [1], that is, the conceptual and affective information associated with objects and actors of the operating environment. Consider the everyday example discussed below.

Everyday tasks, from handling fragile objects to navigating a highway to parking one's car at work, require a high degree of SA and spatial aptitude. Such tasks require the human actor to quickly adapt to new stationary and moving objects, as well as unpredictable moves that these may make, such as a pedestrian suddenly crossing the road. In the case of parking a car, the driver must account for the overall distribution of objects in the space of the parking lot at the time he/she plans to park, as well as the actions those objects may take based on the intentions of other human actors, hence the Parking Dilemma [2]. In order to properly understand and predict the actions of others in this space, the driver must predict what others will do, or their actions will otherwise appear random. For instance, a car may suddenly change course to move to a parking spot, which a driver considering the intentions of other drivers is more likely to detect and account for in his subsequent driving maneuvers.

While cars and other motor vehicles possess functions, such as turn signals, to assist in the conveyance of the driver's intention to other drivers, there still remains a significant gap between a driver's intentions and other drivers' awareness of them. Due to finite time and resources, routine activities such as this require integrating not only SA, but IA as well, in order to optimize the exchange of those resources for some other reward (in the above example, a parking space).

In this paper, we discuss and deconstruct the SA paradigm from four distinct perspectives. First, we review relevant theoretical work in SA and organizational theory fields in order to assess their strengths, weaknesses, and their alternatives. We apply the results of this assessment in the second section, where we provide a working definition of SA, and in the third section, we demonstrate the additional utility that IA offers in enhancing the predictive analytical capabilities of SA. In the fourth and final section we summarize these findings and point to specific areas that would benefit from future development and integration of IA.

SA has grown to include a large scope of environmental and informational attributes, which depend primarily on the nature of the situation. In a maritime navigational scenario, for example, the situational picture would include other ships, weather, wind, depth, buoy locations and heading. On the other hand, the parking scenario, which generally entails more direct association with other actors over limited resources, would include a greater emphasis on other driver’s intentions. Thus, employing SA properly requires a significant devotion of cognitive (and, more recently, computational) resources as well as recognition of the appropriate level of focus for new input and ongoing analysis. That is, staying in the bounds of a sea channel takes on a lower priority in an emergency collision avoidance maneuver at sea.

In order to achieve this level of focus effectively, both human actors and their computer system counterparts need to be able to analyze the situation both with existing data and knowledge about other actors’ goals. Data-driven analysis is based on tracking the status of existing goals as new data about the situation become available, and goal-driven analysis is the basis for the formation of new goals (if necessary) based on the results of data-driven analysis. Because each mode of analysis depends on the other, the situation-aware entity, human, or computer, must be able to switch dynamically between them.

Because warfare condenses and renders more urgency to the requirements of SA, many of its original concepts can be traced to scholarship on military operations, such as Clausewitz’s writings on the “frictions” of war, one of the most important being uncertainty, or the “fog of war [3].” Apart from warfare, SA’s innovations have proven useful in all scenarios, including any in which high-stakes, short-term variations are present.

As a result, study of aviation and air traffic control has also yielded significant insight into augmenting human SA [4]. This has resulted in a proliferation of definitions for SA, some more domain-specific than others. However, the most cited definition is that provided by Mica Endsley in 1988:

The perception of the elements in the environment within a volume of space and time, the comprehension of their meaning, the projection of their status into the near future, and the prediction of how various actions will affect the fulfillment of one's goals[5].

Vidulich et al. [6] provide some elaboration on Endsley’s previous definition describing SA as “the continuous extraction of environmental information, the integration of this information with previous knowledge to form a coherent mental picture, and the use of that picture in directing further perception and anticipating future events”. Both Endsley and Viludich appear to agree that SA, which occurs in the present, has the primary goal of forming a coherent narrative between past, ongoing events, and likely future outcomes. While each of their definitions is technically compatible with the IA concept, the lack of implementation of IA at the time they published their theories on SA, and in the present to some degree, has been the primary cause of the conceptual gap between IA and SA.

As an outcome of situation assessment, SA can be viewed as a quadripartite end product of an actor’s existence in a given scenario. Since SA occurs in four critical stages (per Endsley’s definition), they can each be mapped to their own “level” in the SA hierarchy. Perception corresponds to level 1 SA, comprehension to level 2, and projection to level 3, respectively. Demonstrating the utility of this perspective, Jones and Endsley [5] were able to isolate 76% of SA failure in pilots could as problems in level 1 SA (perception of needed information) due to failures and/or shortcomings in the system or in cognitive processes themselves. The study also found that 20% of SA failures were attributable to level 2 SA (projection), and the remaining 4% involve problems with level 3 (projection).

While this study demonstrates a rough analytical approach to intentions and perceptions, it also shows the necessity of a more precise understanding of the interaction between intentions of human actors and the situations in which they find themselves. Endsley and Jones demonstrated significant correlation between these phenomena in their study, suggesting a need for further study. Of particular importance is the question of whether the limits of the human brain’s capabilities impose a limit on computationally augmented SA. If so, a more cognitive-based approach to the SA problem is warranted.

Situation assessment is an intensive cognitive process that serves as a junction between the physical properties of objects (position, heading, etc.) and the human mind’s portrayal of such objects. In order to successfully complete this process, the actor conducting situational assessment must be able to properly account for the intentions of the other actors that share the same situation. This is a shortcoming in many contemporary SA models, which give less priority to intentionality as a driver of human-dominant situations [2, 5, 7]. Due to the inherent structural differences in situational data and intentional information, not all systems that address the latter can also address the former, which produces a significant implementation gap despite the conceptual and potential application overlap between SA and IA.

It is also known that attention plays a significant part in successful SA [8–10]. Up to 35% of all SA failures in Endsley’s study were traceable to attention; “all needed information was present, but not attended by the operator” [5]. Although it may appear to be explicitly tied to perception, attention is also affected by intentions and goals. “[A] gents deployed their attention in ways that are consistent with [their] operational goals” [4].

Each of these precepts is representative of a classical interpretation of intentions in which cognitive states constantly “evolve” into hierarchical forms based on the demands of the environment.

Bruner and several other scholars of cognition and SA suggested that goal-driven planning is a subconscious cognitive process. Thus, the action resulting from this process can be viewed as hierarchical in nature [ibid]. Similarly, intention itself is comprised of several discrete layers, each having a unique relationship to consciousness; there are high-, medium-, and low-level intentions: “A hierarchy is formed by these three levels of intention that give us a relation between the means and ends” [12].

High-level intentions are the behaviors, as well as the beliefs and emotions that drive them, that one actor will use to influence another actor. Mid-level intentions are related to high-level intentions in that they are used to achieve the goals defined by high-level intentions, and low-level intentions provide a means to achieve goals defined by intentions at either of the higher levels. This hierarchy serves to simplify spatio-temporal reality by reducing it to states and actions.

In very general terms, we can describe all human activities as a series of proactive and reactive states and behaviors that cause transitions between states. Actors themselves have internal states, i.e., “states of mind,” and they use these mind states to perceive the interactive environment they inhabit. Because human beings are themselves cognitive systems, and operate in a greater environment that contains other humans, any given situation can be described as a system of systems. Humans in this system are inherently intentional, exhibiting certain consistently rational behavior based on their intentions [13].

When we refer to rationality within this paper, we are specifically discussing “acting as best to satisfy [one’s] goals overall, given what [one’s] know and can tell about [one’s] situation” [14]. In this sense “rationality is the mother of intention” [ibid]. According to this definition, intention can be described as a relationship between some object (primarily physical such as an artifact) and an actor’s internal mind state - desire, belief, goal, purpose, etc. Intentionality^a is thus “the character of one thing being “of” or “about” something else, for instance by representing it, describing it, refereeing to it, aiming at it, and so on” [ibid].

At the lowest level, we can view intention as the relationship between an environmental state and the objects with which it interacts and which act upon it. Intentional stance is an abstract tool that enables the predictive analysis of complex systems such as human actors, but does not account for the cognitive nature of those processes and the way that they operate [15].

Prior research has, for the most part, followed a work template that concentrates primarily on one of these attitudes and/or notions and investigates others in relation to it. Research investigating the relationships among beliefs, desires, and intentions (BDI) assumes that these systems project inherently desired states (and that the nature of these desired states does not change with the environment) [16].

While BDI and classification systems for attitudes present researchers with a straightforward abstraction of environmental systems, there are a number of alternative, abstract views for the analysis of interacting systems such as human actors situated in a given environment. These types of systems can be regarded as exceptional cases, or intentional systems, because they involve the interaction of human actors and thus depend heavily on modeling intentionality. If the human actor must choose how to act in both a spatial and temporal frame, then we can model the system and its inputs by addressing the factors that contribute to those choices. In these systems, observers often note a consistent, rational behavior pattern that can be mapped to intuitive intentions on the part of the human actors in the system [14].

Intentional systems differ from conventional ones in that they possess several additional explanatory characteristics and capacities. First, they tend to view infrequent conditions as similar to more frequent ones because classification depends more on relevance to intentions than empirical frequency. This gives intentional systems the ability to create and analyze “multidimensional” states. This allows a simple, straightforward means to solve the problem of scaling and context.

We can also specify two schemes, or classes, to simplify the definition of choice types in system characterization. The first defines the various types of intentional states, and the second defines types of choices to be made that will ultimately affect those states. Similarly, intention schemes (I-schemes) and choice, or rationality, schemes (R-schemes) can be used to map intentions to their logical behavioral conclusion.

All systems that include some degree of intentionality can thus be analyzed as instances of intentional systems with different levels of I-schemes and R-schemes. Systems composed of animal actors are equipped with this intentionality and rationality, but in a way that is distinct from their expression in human agents. In addition to humans, information systems such as computers and computer networks possess a similar rationality/intentionality scheme. Specifically, their behavior is guided by transitions from initial states to non-initial states by way of intrinsically motivated behavior, or intervention from without.

Distributing SA among human actors

Artman et al. [17] argue that systems driven by teams of actors need an additional dimension of analysis: “it is necessary to shift the unit of analysis from the individual to the whole cognitive system comprising a team of people as well as the artefacts which they use.” This is due to the fact that, while examples of SA are often single-user centric, team-driven action multiplies the complexity of the SA picture by the number of actors involved, because the difference between each of their perceptions will create distinct outcomes. Thus, Artman et al. touch on a SA concept that is not entirely unrelated to IA, which is the multiplicity of actors. The primary distinction is that in IA, we address all actors that impact the environment, not just those that form a loose unit that functions as a larger actor (Figure 1).

IA is the process of integrating actors’ intentions into a unified view of the surrounding environment. IA includes many of the basic principles of SA, such as consistent tracking and extrapolation of objects in the user’s environment, but also exploits circumstantial semantics and sentics associated with objects and actors of the operating environment. Semantics and sentics, in fact, are key for sensemaking, in which they extend the meaningfulness that the system attributes to every single object/actor of the environment by considering it as an entity carrying specific conceptual and affective information, rather than simply a point in a space or time window. In the context of home automation, for example, a greater deal of intentions can be inferred, if the semantics and sentics associated with the object ‘coffee machine’ are known. Such semantics and sentics can be retrieved from external resources, e.g., affective common-sense knowledge bases for concept-level sentiment analysis [18]. For example, if the system has access to pieces of information such as ‘a coffee machine is used for making coffee’ and ‘people generally like coffee’, the repeated proximity of a human actor to the object ‘coffee machine’ can be exploited to infer the intention of the human actor to make coffee.

Intention awareness: a new role for intent in computation

Linking intent to SA-driven pictures of the environment requires more than simply creating an “intent” category in data analysis. From a computational perspective, intentions are causal inferences made from a series of events and a series of prior, linked intentions. Thus, in order for a system to possess IA, it must not simply infer based on available (i.e., situational) data, but must incorporate prior intentional analysis as one of its information sources. Thus, one of the fundamental distinctions between SA as we now know it and IA is that, while the former concerns data analysis that humans probably cannot process in the amount of time required, the latter concerns information analysis, a task that humans perform on a regular basis, but that can be enhanced by artificial systems (Figure 2).

One promising example of the extension of intention into SA is computational “sensemaking,” or the process by which humans (or other reasoners) attach semantics and sentics to their observations of the operating environment in a parallel and dynamic way [35]. IA, in fact, is a viable means of improving synchronization in sensemaking between the human reasoner and associate system interfaces. This is due to two primary factors. The first is that IA is becoming increasingly viable thanks to advances in computer hardware performance. Second, since IA simultaneously frees some of the analytical burden from human operators and uses similar structures as are found in human cognition, interface-based latency is minimized.

In popular applications, SA is seen as a proactive, purposeful behavior that is directed towards achieving a goal in a specific task environment. This view has been readily adopted in many different fields and contexts, where situations are by nature dynamic and subject to rapid, unexpected variation. Because commander’s intent drives the mission, regardless of operational size or scope, it is tempting to cite military science as the primary beneficiary of advances in SA and IA systems. However, expansion upon some of the notions that drive the process of military organization and action reveals that we can apply the same concepts to any system requiring SA, e.g., game theory, home automation, cyber-security, and swarm robotics. This paper has argued that the next logical extension of SA is IA, or the independent analysis and awareness of intentions by computational systems that augment the human actor.

This is for two primary reasons. First, our understanding of cognitive processes has allowed a more computationally feasible problem of intent to be defined as the basis for IA models. Second, computational resources have made significant progress since the notion of SA was first proposed, so we currently experience an implementation gap where we have the resources but not the immediate capability to execute IA in field applications.

^a For the purposes of this paper, the concept of intentionality is the quality of having intentions, not to be confused with a quality of actions, as sometimes mentioned in the literature.

Newton Howard (nhmit@mit.edu) is the director of the Synthetic Intelligence Lab at MIT and former Director of the Mind Machine Project. He received his Doctoral degree in Cognitive Informatics and Mathematics from La Sorbonne, France where he was also awarded the Habilitation a Diriger des Recherches for his leading work on the Physics of Cognition (PoC) and its applications to complex medical, economical, and security equilibriums. While a graduate member of the Faculty of Mathematical Sciences at the University of Oxford, England, he proposed the Theory of IA, which made a significant impact on the design of command and control systems and information exchange systems at tactical operational and strategic levels. He has served as the Chairman of the Center for Advanced Defense Studies (CADS), the leading Washington, D.C, National Security Group and is currently its board director. He is a national security advisor to several U.S. Government organizations. He recently published The Mood State Indicators (MSI) algorithm, which modeled and explained the mental processes involved in human speech and writing, to predict emotional states. Dr. Howard works with multi-disciplinary teams of physicists, chemists, biologists, brain scientists, computer scientists, and engineers to reach a deeper understanding of the brain. Dr. Howard’s most recent work focuses on the development of functional brain and neuron interfacing abilities. To better understand the structure and character of this information transfer, he concentrated on theoretical mathematical models to represent the exchange of information inside the human brain. This work has proven applicable in the diagnosis and study of brain disorders and has aided in developing and implementing necessary pharmacological and therapeutic tools for physicians. He has also developed individualized strategies to incorporate solutions for psychiatric and brain prosthetics.

Erik Cambria (cambria@media.mit.edu) received his BEng and MEng with honors in Electronic Engineering from the University of Genoa, in 2005 and 2008 respectively. In 2011, he has been awarded his PhD in Computing Science and Mathematics, following the completion of an industrial Cooperative Awards in Science and Engineering (CASE) research project, which was born from the collaboration between the University of Stirling, Sitekit Solutions Ltd., and the MIT Media Laboratory, where he currently works as associate researcher (Synthetic Intelligence Project). His interests include AI, Semantic Web, KR, NLP, big social data analysis, affective and cognitive modeling, intention awareness, HCI, and e-health. Dr. Cambria is invited speaker/tutor in many international venues, e.g., IEEE SSCI, MICAI, and WWW, associate editor of Springer Cognitive Computation, and guest editor of leading AI journals, e.g., IEEE Computational Intelligence Magazine, Elsevier Neural Networks, and IEEE Intelligent Systems. He is also chair of several international conferences, e.g., Brain Inspired Cognitive Systems (BICS) and Extreme Learning Machines (ELM), and workshop series, e.g., ICDM SENTIRE, KDD WISDOM, and WWW MABSDA.