Emotion classification based on brain wave: a survey

Abstract

Brain wave emotion analysis is the most novel method of emotion analysis at present. With the progress of brain science, it is found that human emotions are produced by the brain. As a result, many brain-wave emotion related applications appear. However, the analysis of brain wave emotion improves the difficulty of analysis because of the complexity of human emotion. Many researchers used different classification methods and proposed methods for the classification of brain wave emotions. In this paper, we investigate the existing methods of brain wave emotion classification and describe various classification methods.

Introduction

Brain science has shown that human emotions are controlled by the brain [1]. The brain produces brain waves as it transmits messages. Brain wave data is one of the biological messages, and biological messages usually have emotion features. The feature of emotion can be extracted through the analysis of brain wave messages. But because of the environmental background and cultural differences of human growth, the complexity of human emotion is caused. Therefore, in the analysis of brain wave emotion, classification algorithm is very important. In this article, we will focus on the classification of brain wave emotions.

Emotion classification is one of the most important topics in the field of brain Wave Research [2]. One of the main problems in the analysis of brain wave emotion is how to accurately classify the types of emotion. However, the uniqueness and particularity of Brain Potter [3], resulting in the inability to accurately distinguish between the diversity of human emotions. Although the types of human diversity emotions cannot be classified [4]. However, the twentieth century psychologist “Pail Ekmean” divided emotion into basic emotions and complex emotions and basic emotions are closely related to human physiological responses. Therefore, we can classify the emotion types in the basic emotions through the brain Wave Emotion classification method [5].

Background

Brain waves are the current reactions that occur during neurological activity within the brain. O’Regan et al. and Lan et al. [6, 7] studies have confirmed that brain waves reflect the most real human body reactions. Brain wave emotion analysis compared with other emotion analysis. The authenticity and non-immutable credibility of brain waves are high. The Federation of societies electroencephalography and clinical neurophysiology distinguishes brain waves using different frequency sizes [8]. These distinguished brain waves have different emotion features, which combine to be human emotions. Brain waves are divided into five different kinds of waves, such as Table 1. Brain waves are more likely to obtain emotion features in biological messages when they are differentiated.

Table 1 Introduction to brain wave

The human mood is very complicated. It is the coordinated response of physiology, behavior and neurological mechanisms. Pail Ekmean confirmed that basic emotions are human physiological responses [9]. Basic emotions can be divided into six categories, namely, happiness, anger, fear, surprise, sadness and disgust [10]. Psychologists have the following views on basic emotions.

• Fear: The instinctive behavior of a common creature or person in the face of danger in life. Fear can cause changes in the heart rate, elevated blood pressure, night sweats, tremors and other physiological phenomena, and even the symptoms of cardiac arrest shock.

• Anger: Emotional agitation, being violated, disrespected, or wrongly treated, can lead to instinctive self-preparedness for its combat response. Emotional anger, micro-lukewarm, resentment, inequality, irritability, hostility, and, more extreme, hatred and violence.

• Sadness: It is usually the psychological frustration of failure, the mood is lower meaning. Emotions are sad, depressed, self-pity, loneliness, depression, despair, and morbid severe melancholy.

• Joy: Emotion is the psychological state of pleasure, with the meaning of joy, contentment, self-satisfaction, pride, and excitement in the senses.

• Surprise: By unexpected stimulation in the living environment, resulting in temporary action to stop.

• Disgust: Facing negative stimuli in the environment.

Do not agree with the source of the stimulus is not accepted.

Evaluation metrics

In order to evaluate the emotion classification of brain waves, accuracy is usually used as a pointer. Accuracy as a pointer has been widely used in the literature. In the problem of brain wave emotion classification, how to classify emotion features accurately is the key and the quality of classification method is judged by accuracy.

Classifier of brain wave emotion classification

In this paper, the classification of brain wave emotion is divided into two categories: the use of linear classification methods and nonlinear classification methods. Nowadays, most of the study of brain wave emotion is based on sound source, short film to do the source of stimulation. These stimuli are used to stimulate the subjects to work better [11].

Linear classifier

Linear classifiers are determined by the linear combination of features to make classification decisions. Viewed from the perspective of two-dimensional space. If it is to categorize the feature of the two categories. A linear classifier is a straight line. This line can leave the two types of points as shown in the Fig. 1. The expression is defined as ${\text{y}} = {\text{ax}} + {\text{b}}$. Common linear classifiers are linear regression, linear discriminant analysis (LDA), linear support vector machine, single-layer Perceptron network and simple Bayesian classifier (Naive Bayes). The following sections describe the use of different linear classifiers for brain Wave affective classification. A comparison of the different scenarios is shown in the Table 2.

Fig. 1

Linear classifier

Table 2 Comparison of different schemes of linear classifier

Method by Li et al. [15]

Li et al. for brain wave emotion analysis is the use of 62-channel brain wave instrument. The aim of the experiment was to classify the two emotions of human happiness and sadness. In order to balance the differences in human physiological factors. The average age of the subjects selected was 25 years. In order to classify the two emotions of happiness and sadness. The authors use common Spatial Patterns (CSP) to extract brain wave features [16] and classify features into linear support vector machines (LINER-SVM).

First, the EEG signal to remove the pseudo-image is filtered with fast Fourier first. Then the covariance matrix is calculated by CSP method [17]. The main purpose of CSP method is to extract the size reduction method of brain wave feature supervision. CSP Search direction to maximize the variance projected to two emotion signals. Where $D_{{i_{k} }}^{(1)}$ and $D_{{i_{k} }}^{(2)}$ represent two signals. $i_{1} = 1, \ldots ,n_{1} ,$ $n_{1}$ and $n_{2}$ to indicate the number of tests for each emotion brain wave signal. The average covariance of each type of brain wave emotion is:

$$\sum^{(k)} = \frac{1}{{n_{k} }}\sum\limits_{i = 1}^{{n_{k} }} {\sum_{i}^{(k)} } .$$

(1)

After the derivation size of each brain wave emotion is obtained by CSP method, the logarithmic variance of the dimension push test is used as a feature to introduce the brain wave emotion classification by using linear support vector machine (LINER-SVM). D is characterized by F and calculates $f = \log \left( {Var\left( {DW} \right)} \right) = \log \left( {diag\left( {W^{T} \sum W} \right)} \right).$ Where $Var\left( \cdot \right)$ is calculated for each column method, ${\text{diag}}\left( \cdot \right)$ is the diagonal of the matrix.

Method by Murugappan et al. [11]

Murugappan et al. used 64-channel brain wave instrument to collect brain wave signals. The aim of the experiment was to classify five kinds of human emotions. The five emotions were disgust, joy, surprise, fear, neutrality. There are two kinds of classification methods used in experiments, namely linear discriminant analysis and K nearest neighbor method. This scheme first decomposes the brain wave signal through wavelet transform into five kinds of waves. The feature extraction of these five kinds of waves is then carried out. The preprocessing EEG signal is calculated by Formulae 2 and 3 to achieve the purpose of feature extraction. Finally, two linear classifications are used to classify the features emotionally. The experimental flow is shown in Fig. 2. Experimental results The KNN classifier has good results. KNN has an average emotion recognition rate of 79.174%.

$$\varPsi_{a,b} \left( t \right) = \frac{1}{\sqrt a }\varPsi \left( {\frac{t - b}{a}} \right).$$

(2)

$$C_{\psi } = \mathop \int \limits_{ - \infty }^{\infty } \frac{{\left| {\varPsi \left( \omega \right)} \right|^{2} }}{\omega }d\omega < \infty .$$

(3)

Fig. 2

Method flowchart by Murugappan et al.

Method by Wang et al. [18]

Wang et al. use 62 channel brain wave instrument for brain wave signal acquisition. The goal of the experiment is to classify two kinds of human emotions. The two emotions are positive emotions and negative emotions, respectively. In order to achieve emotion classification, by using the support vector Machine (SVM) classifier. The experimental process is shown in Fig. 3. This scheme first reduces the computational complexity by preprocessing the brain wave signal. Then power spectrum, wavelet and nonlinear dynamic analysis are used for feature extraction. Then through the linear dynamical system (LDS) for the extracted features of pseudo-counterfeiting. After the pseudo-counterfeiting through LDS, the emotion-independent information in the feature will be filtered out. Brain wave is characterized by high dimensional data, so the reduction of feature dimension is a very important step in EEG data analysis. Wang et al. used three dimensional reduction methods to achieve the reduction of feature dimensions. The main component analysis (PCA), linear discriminant analysis (LDA) and related feature selector (CFS) Three known dimension reduction methods are respectively.

Fig. 3

Method flowchart by Wang et al.

The purpose of SVM is to maximize the distance between two kinds of emotion features in hyperspace space. A super plane of two different classes is separated by Formula 4. The training and test mode order of SVM is accomplished by the kernel function in SVM. The kernel function is defined as Formula 5. This scheme compares the performance of three different types of cores, namely linear, polynomial and RBF. Through the 10 cross-verification method, it is found that the SVM classification accuracy of linear kernel function is better. The average classification accuracy of SVM with linear nuclei is 87.53%.

$$y\left( x \right) = w^{T} \varphi \left( x \right) + b = 0\mathop \sum \limits_{i = 1}^{{n_{k} }} \sum_{i}^{\left( k \right)} .$$

(4)

$$k\left( {x_{i} ,x_{j} } \right) = \varphi^{T} \left( {x_{i} } \right) \cdot \varphi \left( {x_{j} } \right)\mathop \sum \limits_{i = 1}^{{n_{k} }} \sum_{i}^{\left( k \right)} .$$

(5)

Method by Petrantonakis et al. [19]

The method of brain wave instrument used by Petrantonakis et al. is different from other studies. Most of the electrical poles used in the study of brain waves are 64 or channels. The electrode points used in this programme are 3 channels, namely FP1, FP2, and a bipolar channel of F3 and F4 positions according to 10–20 system. In this scheme, the number of electric poles in brain wave instrument For other studies less. That is, the computational complexity of this scenario is low. The purpose of this scheme is to classify six kinds of human emotions. Six kinds of emotions are: happiness, surprise, anger, fear, disgust and sadness. The programme uses four classification methods, namely quadratic discriminant analysis (QDA), k-nearest neighbor (KNN), Mahalanobis distance (MD) and the Vector Machines (SVMs). The experimental flow is shown in Fig. 4. In this scheme, three methods were obtained to obtain the brain wave eigenvectors (FVS), which were statistical values [20], wavelet transform [21] and higher order crossings (HOC). The FVs classifies six emotions through four classifiers. The experimental results show that SVM has 83.33% average classification rate. The best results obtained by SVM in four classifiers.

Fig. 4

Method flowchart by Petrantonakis et al.

Method by Duan et al. [22]

Duan et al. have proposed a new EEG characteristic-differential entropy (DE) for brain wave signals. De to denote emotion state-related features. For the proof de is an emotion correlation characteristic. This scheme is compared with the combination of its symmetric electrodes (differential asymmetry, dasm; and rational asymmetry, rasm) with traditional frequency domain features (energy spectrum, ES). Brain wave data is high dimensional data, in order to improve the speed and stability of computation, the reduction of dimension is necessary. The experimental flow of this scheme is shown in Fig. 5.

Fig. 5

Method flowchart by Duan et al.

The feature extraction of this scheme adopts the frequency domain feature and its combination, namely ES, DE, dasm and Rasm. ES is the average energy of the EEG signal. The de is defined as Formula 6. DASM is defined as Formula 7. RASM is defined as Formula 8. This scheme removes emotion-independent features through the LDS method [23]. The classifier adopts two kinds of classifiers, namely linear kernel support vector machine (SVM) and K near neighbor (KNN) algorithm. The aim of the experiment was to classify two emotions of negative and positive. This scheme uses principal component analysis (PCA) algorithm and Minimal-redundancy-maximalrelevance (MRMR) [24] algorithm two methods to achieve the reduction of dimension. Experimental results show that the average accuracy of SVM classifier reaches 74.1%, and the MRMR algorithm can effectively improve the accuracy of classifier.

$$h\left( x \right) = - \mathop \int \limits_{ - \infty }^{\infty } \frac{1}{{\sqrt {2\pi \sigma^{2} } }}e^{{ - \frac{{\left( {x - \mu } \right)^{2} }}{{2\sigma^{2} }}}} \log \left( {\frac{1}{{\sqrt {2\pi \sigma^{2} } }}e^{{ - \frac{{\left( {x - \mu } \right)^{2} }}{{2\sigma^{2} }}}} } \right){\text{dx}} = \frac{1}{2}\log \left( {2\pi e\sigma^{2} } \right).$$

(6)

$$DASM = h\left( {x_{i}^{left} } \right) - h\left( {x_{i}^{right} } \right).$$

(7)

$$RASM = \frac{{h\left( {x_{i}^{left} } \right)}}{{h\left( {x_{i}^{right} } \right)}}.$$

(8)

Nonlinear classifier

There is no limit to the classification boundary of nonlinear classifiers. Nonlinear classifiers can be divided into two broad categories. The first large class is based on discriminant functions. The classification boundary of this type of nonlinear classifier may be a surface, or it may be a combination of multiple curves (Fig. 6). Viewed from the perspective of two-dimensional space. If it is to categorize the feature of the two categories. A nonlinear classifier is a curve. The second largest class is not based on discriminant function, that is, the classification boundary of nonlinear classifier cannot be described by linear relation.

Fig. 6

Nonlinear classifier

Common nonlinear classifiers are k-nearest neighbor (KNN), support vector machines, multilayer sensors, and decision tree (DT). The following sections describe the use of different nonlinear classifiers for brain Wave affective classification. The comparison of the different options is shown in Table 3.

Table 3 Comparison of different schemes of nonlinear classifiers

Method by Liu et al. [25]

Liu et al. use 14 channels brain wave instrument. The purpose of this scheme is to classify six kinds of human emotions. Six emotions were sad, frustrated, fear, satisfied, pleasant and happy. This scheme has different affective classification system for emotion classification. The two-dimensional classification of arousal and valence size is adopted in this scheme. Liu et al. use awakening and price size to design two-dimensional emotion models, as shown in Fig. 7. The experimental flow of this scheme is shown in Fig. 8. The programme proposes fractal dimension based approach to emotion recognition [26,27,28], through the fractal analysis of Fractal Dimension (FD) [29] can be defined with the calculated FD value of the mood awakening and price level.

Fig. 7

Arousal and valence two-dimensional emotion model

Fig. 8

Method flowchart by Liu et al.

Method by Liu et al. [30]

Liu et al. use two different schemes for the selection of electric poles of brain wave instrument. One is the use of 32 channel EEG and the extraction of features through the residual network (resnets). The other is to select 2 channel EEG, through the linear frequency inverted spectrum coefficient (LFCC) to extract the feature. The affective classification method is classified by valence-arousal (VA) model. VA model is a two-dimensional model, which can classify four kinds of emotions as shown in Fig. 9. Seven different classifiers are used in this scheme are: K-nearest neighbor (KNN), Machines (SVM), logical regression (LR), Random Forest (RF), naive Bayesian (NB), decision Tree (DT) and a fully-connected neural network (FC) with 3 dense layers and 2 dropout layers. The experimental process is shown in Fig. 10.

Fig. 9

VA model-four kinds of emotions

Fig. 10

Method flowchart by Liu et al.

There are three test tasks for this scenario: (1) low/high valence. (2) Low/high arousal. (3) Low arousal low valence/high arousal low valence/low arousal high valence/high arousal high valence. The purpose of the first task was to classify positive and negative emotions. The result of the second task represents the degree of emotion stimulation. The result of the third task can also be said to synthesize the results of the first two tasks. The third task results in a variety of emotions.

Method by Zheng et al. [31]

Zheng et al. use 62 channels brain wave instrument to collect brain wave signals. The purpose of this programme is to distinguish between three different human emotions. Three emotions were positive, neutral and negative. The programme uses the deep belief networks (four classifiers). Consider the complexity of the continuous random variable [32] of emotions. In this scheme, differential entropy (DE) [33, 34], is used for feature extraction. In addition, relevant studies have shown that [35, 36], in asymmetric brain activity, seems to be effective in the treatment of emotions. Feature extraction also adds asymmetry (dasm), reasonable asymmetry (rasm) function [34]. In order to compare the above three features. The scheme also extracts the conventional power spectral density (PSD) as the baseline. The experimental flow is shown in Fig. 11. The average accuracy of DBN, SVM, LR and KNN in this experiment is 86.08%, 83.99%, 82.70%, and 72.6%.

Fig. 11

Method flowchart by Zheng et al.

Method by Nie et al. [37]

Nie et al. use 62 channels brain wave instrument. The purpose of this scheme is to classify positive and negative two kinds of emotions. Use the support vector machine classifier to achieve emotion classification. Brain wave data is extracted by Fast Fourier (FFT). Because this scheme uses the channels brain wave instrument, in order to reduce the influence of unrelated emotion. So the feature of the most relevant emotions are found through linear dynamic systems (LDS). The experimental process is shown in Fig. 12.

Fig. 12

Method flowchart by Nie et al.

Method by Zheng et al. [5]

Zheng et al. used the channels brain wave instrument to collect brain wave signals. The aim of the experiment was to classify three kinds of brain wave affective types, namely negative, positive and neutral. Feature extraction using power spectral density (PSD), differential entropy (DE), differential asymmetry (dasm), rational asymmetry (rasm), a symmetry (ASM) and differential caudality (DCAU) six different features and electrode combinations. In order to reduce the feature dimension, the calculation speed is improved. The programme uses minimal redundancy maximal relevance (MRMR) algorithm [24]. The programme has adopted four classifiers for K nearest Neighbor (KNN) [38], logistic regression (LR) [39], and Graph vector Machine (SVM) [40] and regularized Extreme Learning (machine) [41]. The experimental process is shown in Fig. 13. Experimental results The Gelm classifier is superior to other classifiers.

Fig. 13

Method flowchart by Zheng et al.

Method by Lu et al. [42]

Lu et al. collated the previous study of brain wave emotion, and compared the efficiency of 10 kinds of classifiers. The 10 classifiers are: regularized logistic regression with (LL1) [43], linear discriminant analysis (LDA), quadratic discriminant Ana Lysis (QDA), k-nearest neighbor classifier (KNN) algorithm using the Euclidean distance, Vector Machine (SVM) [40, 44], Gaussian Naive Bayes classifier (GNB), decision Tree (DT), regularized linear models with stochastic gradient descent (SGD) learning [45, 46], Random Forests (RF) [47], gradient boosting (GB). The experimental process is shown in Fig. 14. Differential entropy (DE) is used for feature extraction of this scheme. The aim of the experiment was to classify negative, positive, neutral three kinds of brain wave emotions. The experimental results show that the LR with Norm L2 Classifier has a good average accuracy of 81.26%.

Fig. 14

Method flowchart by Lu et al.

Conclusion

Human emotions are very complex. In order to be able to classify the feature of brain wave emotions, many studies have used different techniques to extract these features. This paper first introduces the type of brain wave and describes the emotion features of different types of brain waves. Then the existing classification scheme of the emotion features of brain waves is classified and described. We introduce and compare these classification schemes based on linear and nonlinear methods. Brain wave sentiment analysis is one of the most popular studies. In order to be able to more accurately classify the feature of brain wave emotions. We need to find more ways, such as deep learning, to solve the challenge of classifying the emotion features of brain waves.