Artificial neuro fuzzy logic system for detecting human emotions

The problem of emotion detection based on the measured physiological changes in the human body had received a significant attention lately (Nie et. al. 2011)[1]. However, the instant detection of each human’s emotions had not been thoroughly studied. The problem is that each person manifests emotions in a manner different from others. In social networks, for example, a person sending a message across the net may experience certain emotional status which need to be transmitted to the other party in the same manner voice or image are transmitted.

The human emotional status is rather intangible[2], and therefore cannot be directly measured. However, these emotions can be correlated to external and/or internal factors, which are rather tangible things, and hence they can be measured and analyzed. The internal factors come from different parts of the body in several forms such as electroencephalography (EEG), heart rate (HR), heart rate variability (HRV), pre-ejection period (PEP), stroke volume (SV), systolic blood pressure (SBP), diastolic blood pressure (DBP), skin conductance response (SCR), tidal volume (Vt), oscillatory resistance (Ros), respiration rate (RR), nonspecific skin conductance response rate (nSRR), skin conductance level (SCL), finger temperature (FT), and others (Kreibig 2010)[3].

These factors’ measurements are provided in wide ranges and often their impacts vary from a person to a person and for different postures for the same person. For example, a given measurement of some factors may relate to a person being happy, while the same measurements may reveal a rather “sad” status for another person. This kind of behavior lends itself naturally to fuzzy sets and fuzzy logic (zero and one, true and false or black and white cannot present this kind of data)[4].

In this paper, we will use fuzzy operations[5] to represent the knowledge about each factor. This will enable us to detect the emotion of a person using fuzzy inputs of the various factors. For example, we can use a fuzzy rule such as “IF (Temperature is High) AND (Heart Rate is High) THEN (Person is Excited).” Although fuzzy sets and operations are useful for representing the knowledge base, they fail to model the individual behavior of each and every person. Obviously, a model that is able to adapt to various categories of human responses would be preferred. Consequently, an adaptive learning mechanism is required to adjust the model if we were to cater for the differences in emotions between various humans. This requirement calls for the use of an adaptive learning system such as artificial neural networks (ANN) (Abraham 2005)[6]. However, the ANN model does not allow the use of fuzzy sets or rules, which is the more natural way of representing the relation between human emotions and human physical and physiological parameters. ANN uses exact and crisp values for representing the model’s input.

In order to utilize the benefits of both fuzzy logic and artificial neural networks, we will use the hybrid approach, which combines fuzzy logic and artificial neural networks in a single model.

The analysis and detection of human emotions using an expert system has a direct impact on several fields of the human life such as health, security, social networks, gaming, entertainment, commercials and others[3, 7]. The system will enable social networks (SN) participants to exchange emotions in addition to text, images, and videos.

In health related applications, for example, the interaction between a patient and doctor (in some critical cases) may become difficult or impossible[3], where the patient cannot explain his/her feelings to the doctor (e.g. coma infants, autism). The proposed system would enable the doctor to analyze and detect the patient’s emotions, even when the patient is unable to correctly define his emotional status.

In social networks people exchange all types of information such as text, video, images, and audios. The proposed model would enable communicating parties to detect the emotional status of their partners in a seamless automatic manner. In essence, a person chatting with a friend on the social network would be able to tell whether the other partner is sad, angry, embarrassed, afraid or happy without the need for the partner to explicitly state the emotional status.

Security is another area where the proposed model can be of significant impact. The model can be used to predict a crime before it occurs by detecting a criminal behavior based on the emotional status of the person attempting to commit a crime or breach the security at given facilities. This is based on the psychological status of the criminal before committing a crime. At an airport facility, for example, the system can identify individuals with certain emotional postures based on perceived measures of the individual’s heart rate, EEG frequencies, body temperatures and other measurable factors.

Gaming and entertainment is yet another area where the prediction of a person’s current emotion status is very useful. The system can detect the modes of customers based on the various factors studied and analyzed in this paper.

The rest of this paper is organized as follows. Related work is presented in Section 2. Section 3 provides an overview of the various factors which impact the human emotions. Section 4 presents the ANFIS model, used to build the neuro/fuzzy model. Section 5 presents and analyzes the model results. Conclusions are presented in Section 6.

Human emotion detection and analysis is an important field of study. Some scientists have focused on external effects on human emotions for commercial objectives, such as the use of Electroencephalogram (EEG) measurements for determining the level of attention of a subject to a visual stimulus such as a television commercial displayed on a screen[7].

Timmons at al.[8] introduced a medical instrument that allows doctors to monitor their patients using sensors like insulin and blood pressure sensors. Ohtaki et al.[9] developed wearable instruments capable of indoor movements tracking and monitoring of concurrent psycho–physiologically indicated mental activity. The instrument used electro dermal activity (EDA), heart rate, and vascular change sensors for emotion detection. EDA can be used to detect the human emotional response by measuring the skin humidity, which reflects the activity of the Eccrine sweat glands[9].

In another related study, Petrushin & Grove[10] provided a method for detecting emotional state using statistical analysis of perceived measurements. In their study, they use parameters extracted from a voice speech as an input to an artificial neural network (ANN) to get the related emotion; ANN was used as an adaptive classifier which is taught to recognize one emotional state from a finite number of states[10]. Affectivea[11] introduced a wearable sensor which is capable of quantifying human emotions such as fear, excitement, stress, boredom etc. The devise can be used by doctors to analyze the emotions of autism patients by monitoring their motion and temperature.

Santosh and Scott[12] proposed the use of wearable wireless sensor system for continuous assessment of personal exposures to addictive substances and psychosocial stress as experienced by human participants in their natural environments. It was observed that physiological stress and response vary from person to person, and for the same person with respect to postures and physical activity; it has also been observed that human emotions can be correlated to behavioral patterns such as smoking and speech[12, 13].

Nie et al.[1] evaluated the relationship between the Electroencephalography (EEG) and the human emotions and concluded that EEG can be used to classify two kinds of emotions: negative and positive. Yuen et al.[13] believe that the states of the brain change as feelings change, therefore, EEG is suitable for the task of recording the changes in brain waves, which vary according to feelings or emotions; a neural network was used to train the model. Leupoldt A et al.[14] observed the emotion influence on respiration sensation, skin conductance response and EEG. Kreibig[3] conducted a survey of several research studies to find the relationship between autonomic nervous system (ANS) and the human emotions. ANS includes the cardiovascular, the electrodermal, and the respiratory responses. The survey shows that the ANS response appears in negative emotions clearer than in positive emotions. Kreibig summarized the results of the survey in one table which shows the impact of several measurable factors on both negative and positive emotions. We will rely on this data for the construction of our model and we will choose fourteen factors out of the factors listed by Kriberg. The factors are selected on the basis of their measurability and the availability of sensors for these factors. The model will include all 22 different emotions (11 positive and 11 negative emotions).

In this study, using NFIS editor in MATLAB which supports building a hybrid neuro-fuzzy systems, six models are built with three input membership functions, namely the Gaussian membership function (gaussmf), a combination of two Gaussian functions (gauss2mf), and a product of two sigmoid shaped member functions (psigmodmf). We use the Sugeno constant and linear output functions. The rule in Sugeno fuzzy model has the form

If (input 1 = x) and (input 2 = y)

THEN output z = ax + by + c.

For the constant Sugeno model, the output level z is constant c, where a = b = 0. The output level z_i of each rule is weighted by firing strength w_i of the rule.

For each of the models shown in Table 1: Model Characteristics, we build a neuro/fuzzy structure with 5 layers as shown in Figure4. The general structure of the ANFIS model is the same for all models. The models differ in the specifications of the membership functions and the output specifications. However, the general structure remains the same for all.

We noticed similar behavior for all factors using the three different functions and the different arrays; although the magnitude of change is different for each experiment. We also noticed that the output values of the emotions had changed from the initial values. Figures5 and6 show the Gaussian membership function before and after training respectively.

Training under CTA data arrays produced smaller error values than those for NTA. The STA training data produced the lowest error values. The choice of training data set has an impact on the outcome of the emotions. Using different training data sets may very well produce different emotions output. This is consistent, however, with the fact that different categories of people may respond differently to emotions stimuli. Table 3 and Figure7 show the human emotions response to 10 different trials with different input values for each trial under a model that had been trained for 10,000 epochs with CTA, STA, and NTA using psigmf/Linear functions. The table demonstrates how the models behave under different training sets.

HE-STA	HE-NTA	HE-CTA	HE-BT	Output
8.3	15.4	6.48	22	1
Sadness	Happiness	Fear	Suspense
Crying		Imminent threat
11.1	16.1	11.3	20	2
Sadness	Joy	Sadness	Relief
Acute		Acute
13.8	16.1	15.4	16	3
	Joy	Happiness	Joy
10.7	16	11.8	17	4
Sadness	Joy	Affection	Antic Pleasure
Acute			visual
10	16	4.52	21.1	5
Sadness anticipat	Joy	Embarrass.	Surprise
15.9	16.1	15.7	16.3	6
Joy	Joy	Joy	Joy
12.7	16	15.2	15.8	7
Amusem.	Joy	Happiness	Joy
10.9	10.6	3.91	13.7	8
Sadness	Sadness	Disgust	Contentment
acute	Acute	mutilation
8.29	9.54	7.13	12.8	9
Sadness	Sadness	Fear	Amusement
crying	anticipat	imminent
21	16.1	20.7	9.01	10
Surprise	Joy	Surprise	Sadness Non-cry

The first experiment (column 2 HE-BT) indicates that the human emotion was 22 (suspense) before training and after training it became 15.4 (happiness), 6.48 (fear) and 8.3 (sadness crying) for NTA, CTA and STA respectively. Note that different training sets produce different models. In reality, there could be different training sets representing different human behavior and different human responses. So it is essential to know which category (or training set) an individual belongs to before attempting to define his/her current emotion.

In this paper, we presented a neuro/fuzzy model for the detection of human emotions using fourteen measurable human factors which are known to impact human emotions in varying degrees. The factors are converted into fuzzy variables and used in a set of rules to detect one of twenty two different emotions. The model is trained and the output parameters representing emotions can are adjusted using a set of training data. The developed models can be used in social networks such as Facebook and Twitter, health organizations especially for coma, infant or autism patients, security systems like airports and critical places, and gaming industry. The experiments show that the model is sensitive to the choice of input membership functions as well as the output function.

In this research, we developed a neuro-fuzzy system that deals with 14 human factors that impact human emotions. These human factors are used as input data for the system. These input factors are entered into some specific rules which correlate human emotions to one or more of these factors. A training mechanism is developed, which allows the neuro-fuzzy system to be trained in a manner to detect current human emotions. The developed system benefits the advancement of social networks, security systems, gaming industry and others.