Comparative analysis of affective and physiological responses to emotional movies

Abstract

In this research, we investigated on user’s affective and physiological responses to the emotional movies. The emotional movies provide continuous feedback such as changes in saturation, brightness and contrast of the movie in response to users’ emotional states. In the user study, the subjects watched the emotional movies, edited by fearful and joyful scenes, and presented in 2D and 3D formats. The subject’s self-reported emotion responses and their physiological signals were analyzed. The results highlight the importance of scenes (such as, the color, tone, and brightness of the scene) than 2D or 3D format in understanding the impact of users’ emotional responses. More physiological emotional changes, positive user responses and higher correlation rate between physiological and subjective responses were evident for joyful scene than fearful scene when presented in 3D format.

Introduction

The three-dimensional movies are popular among today’s generation. Following the success of the blockbuster movie Avatar, interest in 3D movies has increased significantly, leading to the production of more 3D movies. 3D televisions, 3D broadcasting and 3D Blu-ray movies also hit the market. More recently, virtual reality becomes a popular trend for consumers who want even more immersive 3D experiences. A few months ago Paramount Pictures launches the first virtual reality movie theater.

Along with the rising popularity in 3D technologies, the biggest concern of creating 3D contents is to offer deeper and more meaningful experiences. Many works have shown that 3D images help users experience excitement, a feeling of presence, and enjoyment, and even performance advantages for depth-related tasks [1]. Despite the growing interest in 3D contents and technologies, few are concerned about viewers’ perceptual and emotional responses to 3D stimuli.

Much 3D related research is focused on technical factors, such as the fidelity of stereoscopic depth and reduction of visual discomforts while the viewer is watching 3D [2,3,4]. For example, subjective evaluation was used to compare visual fatigue induced by 2D and 3D high-definition television (HDTV) [5], but subjective evaluations often show considerable variation between individuals.

On the other hand, physiological measurements provide a more objective evaluation for studying visual fatigue induced by a 3D display [6]. For example, the study on influence of 3D TV based on physiological signals measured the viewers’ emotional states [7, 8]. Results showed a correlation between the effect of physiological responses and emotional states. This indicates that physiological responses can be potentially ample indicators of emotionally engaging 3D video.

Human emotion is a complex phenomenon with both affective and physiological components. The affective component refers to the users’ subjective feeling while the physiological component refers to the physiological responses to user emotion. In affective computing, many researchers strive to understand how human emotions are evoked in response to multimedia contents. Some studies have shown that specific audio-visual stimuli affect changes in user emotion [9, 10]. More attention has been devoted to measuring physiological responses to 2D or 3D movie [11]. However, these are rather passive and one-sided, and the contents do not reflect users’ affective states.

In this research, we investigated on the effects of the emotional movie. The emotional movie means the movie that dynamically adjusts its visual appearances, such as saturation, brightness and contrast of the scene according to users’ emotional states in real-time. This approach is essential for creating more engaging and affective environments. Previous research has revealed that such emotional movies had significantly more influence on the subject’s emotional states than the original movies [12].

Despite the need for understanding the interactive nature of user emotions elicited by the emotional movie, little is known about the impact of 3D emotional movie and the role that different colors or tones of light play in the scene. This is challenging because emotional response is complex, and arises not just from the video itself, but from the context of the video, such as the storyline, the colors or tones of the scene, and even from the 3D effects.

This research aims to provide the basis for objective and subjective measurements of users’ emotional responses to emotional movie. In a comparative user study, the participants viewed the emotional movie, edited by fearful and joyful scenes and shown in 2D and 3D format. While viewing the emotional movie, the degree of user attention, emotional sensitivity, memory, interest and fatigue were measured using objective physiological signals and subjective responses.

The overall paper is structured into seven sections. “Background” section reviews the related background information about studies on emotional contents. “System design and implementation” section presents the system design and implementation of the emotional movie. “Methods” section presents the experimental method for assessment of affective and physiological responses in regards to 2D and 3D emotional movie. “Results” and “Discussion” sections describe the results and discussions. The conclusion and directions for future research are discussed in “Conclusion” section.

Background

There is a great deal of research on how color influences emotional changes in users. For example, the international affective picture system (IAPS) research extracted emotion images from pictures, showed them to users and then measured their emotional changes [13]. There is a study that adjusted colors to fit an individual’s environment in order to find the color that reduces stress the most and creates the most calming environment for the individual [14].

Still other research looked at how the color of light affects cognitive functions, such as attentiveness, memory and emotional change. For example, there is a study that evaluated the impact of color hue, brightness and saturation of e-commerce websites on customer emotions and trust. Results showed that more saturated and brighter colors can motivate consumers to have positive reactions to the site [15].

On the other hand, there have been studies conducted to compare 2D and 3D technologies. For example, there has been a study looking at whether changes in presence (a sense of being there) exist for different genres of 3D TV programs [16] and a study investigating the effect of genre on 3D movies [17]. It also studied whether 3D images offer the same amount of user satisfaction [18], how 3D images influence the users’ biological signals and awakening [19], and how 3D games affect the level of user engagement as compared to 2D games [20].

These works have studied emotional changes by adjusting the original content before or after showing them to users and studying whether there is a difference in sense of realism, user satisfaction, and level of engagement between 2D and 3D contents. In contrast to existing studies, this research attempts to evaluate user’s responses to the emotional movie in 2D and 3D format.

The emotional movie is one of the emotionally intelligent contents (EICs). EICs refer to the contents that recognize human emotions in real-time and provide visual and aural feedback by offering appropriate stimuli so that users feel like they are communicating with the contents [21]. In other words, the color, contrast, brightness or sound tempo of the contents are changed dynamically based on users’ current emotional states to provide more memorable or emotionally engaging experience to the users.

System design and implementation

The emotional movie system is built on top of the emotionally intelligent content framework [12] to enable user preference specification of emotion. This framework allows easy creation of emotional expressions so that the contents can be customized for users depending on their own unique emotion rules. The emotion rules describe how the contents are rendered visually and aurally in accordance with the individual’s emotional state.

The emotional movie system consists of the emotion recognition, the emotional movie controller and the emotional movie player module. The emotion recognition detects the users’ current emotional states in real-time by analyzing users’ autonomic nervous system (ANS) physiological data. The emotional movie controller specifies the users’ emotion rules and selects the elements of the movie to change. It then responds to users’ current emotional states and controls the emotional movie player according to the emotion rules. The emotional movie player is responsible for changing the visual and aural effects of the movie according to the emotion rules.

Emotion recognition

Figure 1 shows the real-time user’s emotion recognition process by collecting, normalizing and analyzing photoplethysmography (PPG), galvanic skin response (GSR) and skin temperature (SKT) data. According to the range of the sensor values mapped to the emotion rule base as shown in Table 1, users’ emotions are classified into nine categories (i.e., excited, afraid, fatigued, calm, pleasant, aroused, unpleasant, relaxed, and neutral) based on Russell’s two-dimensional emotion response model.

Fig. 1

Process of emotion recognition from PPG, GSR, SKT sensors

Table 1 Simulated emotion rule base

The PPG frequency and amplitude and the average values for the GSR and SKT signals are used to extract features for emotion estimation, as studied in [22]. Pre-processing signal features are followed to compensate for noises and artefacts, consisting of 0.5–3.0 Hz band-pass filters for signals, and for absolute measures, a DC filter is used including a 60 Hz notch filter for noise cancelling. A sliding window technique and time dependent parameter (TDP) are also used for the real-time signal processing [23, 24].

Normalization is adopted to minimize any individual difference in physiological data readings by using the average value of the corresponding physiological data collected from the neutral state to the current emotional state [24, 25].The normalization method is described by the following Eq. (1):

$$\mathop {\text{E}}\nolimits_{PPG,GSR,SKT} = \frac{{\left( {\mathop {\text{C}}\nolimits_{PPG,GSR,SKT} - \mathop {\text{N}}\nolimits_{PPG,GSR,SKT} } \right)}}{{\mathop {\text{N}}\nolimits_{PPG,GSR,SKT} }}$$

(1)

In Eq. (1), E_PPG,GSR,SKT denotes the percentage increase or decrease of the PPG, GSR, and SKT signals from the neural state (N_PPG,GSR,SKT) to the current state (C_PPG,GSR,SKT). The threshold band is designed as a neutral range.

$$E[x] = \mathop {\lim }\limits_{N \to \infty } \frac{1}{N}\sum\limits_{k = 1}^{N} {X_{k} (t)}$$

(2)

$$S[x] = \mathop {\lim }\limits_{N \to \infty } \frac{1}{N}\sqrt {\sum\limits_{k = 1}^{N} {(X_{k} (t) - E[x])^{2} } }$$

(3)

In Eq. (2) and Eq. (3), E[x] denotes the mean values of the normalized PPG, SKT, and GSR data and S[x] denotes the standard deviation of the normalized PPG, SKT, and GSR data. In Eq. (4) and Eq. (5), the maximum and minimum range of the designed threshold, T_Max[X_n] and T_Min[X_n], are calculated with a sliding window method.

$$T_{Max} [X_{n} ] = E[X_{n} ] + S[X_{n} ]$$

(4)

$$T_{Min} [X_{n} ] = E[X_{n} ] - S[X_{n} ]$$

(5)

The designed threshold band can classify the increase or decrease of physiological signal patterns. The threshold range of normalized physiological signal patterns occurs at the state of the following condition.

“+” symbol: Xk(t) > TMax[Xn]

“−” symbol: Xk(t) < TMin[Xn]

“0” symbol: TMin[Xn] ≤ Xk(t) ≤ TMax[Xn]

Finally, the normalized data are used to set the emotion rule base as shown in Table 1. As a reference, the emotion rule base is defined in terms of nine emotional states. When the physiological signal values are plotted as threshold range values, “+” and “−” denote an “increase” or “decrease” in the normalized physiological signal values, respectively. The “0” symbol stands for “no fluctuation”; in other words, the normalized physiological signal values are affiliated with the threshold range.

Emotional movie controller

The emotional movie controller specifies the users’ emotion rules and the visual elements (i.e., color, brightness and contrast) of the movie in order to effect users’ emotion. It then responds to users’ current emotional states and controls the emotional movie player using the emotion rules. Emotional contents allow users to specify personalized emotion rule sets, which is stored in and retrieved from a database system [12, 21].

Figure 2 shows the emotion rules describing how the emotional movies are rendered visually depending on the users’ emotional state. The emotion rules were constructed using a color theory [15, 26]. In this theory, users feel smooth and lightweight when there is high brightness and low contrast. They feel a heavy calm feeling when there is low brightness and weak contrast.

Fig. 2

Nine domains of emotional states and corresponding video controls

In this research, the emotion rules cause the brightness to increase or decrease gradually as the arousal state increases or decreases. The color saturation and contrast increases or decreases as the valance state increases or decreases. The value of saturation, brightness and contrast is adjusted by 3% level and used by the video rendering filter for calculating the YUV values of the images. YUV is a color model in terms of one luminance (Y) and two chrominance (UV).

Emotional states are used for the corresponding emotional movie controls mapped to the preference of visual effects. For example, when the user’s emotional state is at “Excited”, the brightness, contrast and saturation are increased. If the user’s emotion is “Afraid”, the contrast and saturation are decreased while the brightness is increased.

Emotional movie player

The emotional movie player receives emotion control messages from the emotional movie controller over a network and then makes filtering adjustments to the movie settings as defined in the emotion rules. In this research, the color properties are used to set the saturation (the intensity of a hue), brightness (the relative lightness) and contrast (the difference between the darkest and lightest areas of a video) of the emotional movie. For example, if users begin to be relaxed, the brightness is decreased, and if they feel aroused, the brightness is increased. In addition, depending on the persistence of pleasant and unpleasant emotion, color saturation and contrast effects were also decreased and increased accordingly.

Methods

This experiment evaluates if there is a change in user emotion elicited by the interactive natures of the emotional movie. In this experiment, the subjects watched the fearful and joyful emotional movie clips shown in 2D and 3D format. The subject’s emotional responses using physiological signals and their subjective feelings through a post-test survey were gathered to evaluate if they felt more involved with the storyline and showed more emotional responses.

Participants

A total of 12 college students majoring in computer science-related fields volunteered to take part in this study. The average age of the subjects was 23.9 and of these 12, seven were female and five were male. All subjects reported being familiar with the 3D, either through playing 3D video games or watching 3D movies. However, most of them were not familiar with the physiological sensors. Although none of them had ever seen the actual movie, A Christmas Carol, they were familiar with the storyline and the main character, Scrooge, through stories, books and other sources they had come across as young children.

Movie clips

Figure 3 shows the emotional movie clips used in the experiment. The scenes were chosen where a 3D effect would be most effective. Comparatively dark, frightening and thrilling scenes were selected from the movie, A Christmas Carol, to create a 12 min fearful clip (a–d). Similarly, comparatively light, warm and pleasant scenes were selected to create a joyful clip of similar length (e–h).

Fig. 3

A Christmas Carol edited by fearful (a–d) and joyful (e–h) video clips

• a. Death of Scrooge’s partner, “Marley” (130 s).

• b. Visit from Marley’s ghost (440 s).

• c. Scrooge being chased by ghost hunters (110 s).

• d. Scrooge slipping on a roof (40 s).

• e. The first opening scene (130 s).

• f. Journey with Christmas ghost (210 s).

• g. Scrooge on Christmas morning (160 s).

• h. Scrooge finally learning the joy of giving on Christmas (230 s).

Table 2 shows the average hue saturation value (HSV) color model values for each clip. HSV is a color model that describes colors (hue) in terms of their shade (saturation or amount of gray) and brightness (value). As shown in Table 2, the average brightness (V) of the joyful scenes appears to be two times brighter than the fearful scenes.

Table 2 Average HSV of fearful and joyful movie clips

Apparatus

Figure 4 shows the experimental setting. An active stereoscopic 46 inch Samsung TV was used to show the 3D movie. The 2D clip was shown in full HD, while the 3D clip was shown in a side-by-side stereoscopic image format and subjects wore shutter glasses to receive the full 3D effect. The physiological signals were used to input emotion contents.

Fig. 4

A snapshot of a subject during the experiment

The PPG, GSR, SKT sensor signals were acquired at 500 Hz using the BioPac’s MP100 system during the test. The PPG signals were recorded with TSD203 sensors placed on the index and ring fingers. The GSR signals were measured using two electrodes which were attached to the index and middle fingers of the subject’s right hand with Ag/AgCl gel materials. The SKT signals were measured with TSD202 sensors placed on the thumb. Also, two channel electroencephalography (EEG) signals were measured at 512 Hz.

In addition, one camera was used to record the subjects’ posture and movements and one observer took notes on subjects’ behaviors during the test.

Procedure

Figure 5 shows the experimental procedure. The total experimental procedure took about 80 min. The subject was first given a brief explanation of the study and then they consented to participate in the study. The pre-test survey asked for personal information (i.e. age, gender, etc.) as well as questions regarding familiarity with 3D movies, and familiarity with the movie, A Christmas Carol, and current emotional state. Then, the physiological sensors were attached to the subject’s ear and fingers and initialized to the subject.

Fig. 5

The experimental procedure

The subjects’ physiological signals were measured for 2 min in a relaxed state to be used as reference. The subjects were randomly assigned to view the 2D and 3D emotional movie clips. Six of the 12 subjects were shown the 3D clips first and then 2D clips. The remaining six subjects were shown the 2D clips first followed by the 3D clips. Finally, the subjects were asked to fill out the post-test subjective evaluation questionnaires.

Measurements

Based on two major categories of classical emotion theories: cognitive and somatic [27], the subject’s emotional responses were measured by physiological signals and subjective feeling. In the experiment, the subject’s physiological responses were measured by PPI VLF/HF ratio, GSR startles, and EEG beta/alpha index. The subject’s affective responses were measured by asking subjective feeling in post-test surveys.

The peak-to-peak interval (PPI) was detected from the PPG signals using peak detection [28]. Contiguous PPI values were converted to a time series, and then linearly interpolated to evaluate the true spectrum using an Fast Fourier transform (FFT) containing the Hamming window. very-low frequency (VLF, in the range of 0.0033–0.04 Hz) and high frequency (HF, in the range of 0.15–0.4 Hz) bands were then categorized. VLF/HF ratio was used to identify the activity ratio of the sympathetic and parasympathetic nervous system [29, 30].

The GSR startles were used to measure the subject’s level of excitement. The GSR startles were detected by smoothing the raw signal with a low pass filter, and a threshold is applied to detect the steep slope associated with the rising edge of the startle response by finding the local minimum preceding that point (onset) and the local maximum following that point (peak).

The EEG beta/alpha indices were analyzed to measure the subject’s level of concentration. The 2-channel brain waves were collected at the Fp1 and Fp2 position (i.e., prefrontal) according to the International 10–20 electrode placement system with a 512 Hz sampling rate. A fast Fourier transform was employed to transform a raw signal to the frequency domain. The alpha waves are the frequency range from 7.5 to 12.5 Hz and the beta waves are the frequency range from 12.5 to 30 Hz.

In the post-test survey, subjects were asked to give their response on a scale from 1 to 5 for the questions regarding the overall feeling about the interactive emotional movies and the effect of 3D emotional movie clips. On a more subjective level, the subjects were asked to mark how they felt (out of one of the nine emotional states) as they watched each clip. The subjects were also allowed to freely describe the scene that was most memorable to them. Finally, the correlation rate was measured at which the emotional responses of the subject’s physiological signals (objective measurement) were exactly mapped onto the self-reported subjective feelings (subjective measurement).

Results

In this experiment, the differences in 2D and 3D emotional movies composed of fearful and joyful scenes are compared and analyzed. We evaluated the subject’s emotional responses using their physiological responses as well as their self-reported affective responses.

Physiological responses

Figure 6 shows the average VLF/HF ratio of PPG signals while viewing the fearful and joyful 2D and 3D emotional movie. An increase in high frequency components is associated with activation of the parasympathetic nerve and emotional relaxation. However, the paired t test results showed no significant mean differences between 2D and 3D for fearful and joyful video clips. The VLF/HF ratio was higher for the 2D fearful clips (M = 0.99) than the 2D joyful clips (M = 0.59). It was higher for the 3D joyful clips (M = 0.68) than the 3D fearful clips (M = 0.49).

Fig. 6

Average of the VLF/HF ratio of PPG

Figure 7 shows the average GSR startles for each clip. The GSR startles indicates how the given emotional movie induced user’s emotional responses. More number of GSR startles means more arousals of excitement. Interestingly, greater number of GSR startles was observed on 3D than 2D emotional movie, especially during the screening of “E”, “G”, and “H” joyful clips (p < 0.05) and “C” and “D” fearful clips (p < 0.1).

Fig. 7

Average of the number of GSR startles

The paired t test results showed a significant mean difference in GSR startles between 2D (M = 11.92) and 3D (M = 17.25) during the screening of joyful clips where the t value was − 2.2921 with p = 0.021. There was also a significant mean difference in GSR startles between 3D fearful (M = 9.25) and 3D joyful clips (M = 17.25) where the t value was − 2.9008 with p = 0.00721.

Figure 8 show the average EEG beta/alpha index for each clip. The ratio of beta to alpha index indicates the subject’s level of concentration. The average of EEG beta/alpha index was slightly higher on 3D than 2D. However, the paired t-test showed no significant mean difference between 2D and 3D for both fearful and joyful clips. Interestingly, the statistically significant mean difference in EEG beta/alpha index between 2D and 3D was found for parts of C, E and F clips.

Fig. 8

Average of the EEG beta/alpha index

The paired t-test revealed a significant mean difference in beta/alpha index between 2D (M = 2.429) and 3D (M = 3.552) with “being actively chased by ghost hunters” in clip C (p < 0.05). Similarly, there was a significant difference between 2D and 3D with “ballroom” and “the dog biting the fish and getting away” scenes in clip E (p < 0.05) and “the appearance of the candle ghost”, “flying with the ghost to the past time”, and “talking with friends in childhood” scenes in clip F (p < 0.05).

Subjective responses

The post-test subjective evaluation results showed that ten subjects found the viewing of the emotional movie to be more interesting than watching a regular movie. Eight subjects thought the emotional movie to be more fun and two thought the emotional movie was rather boring and tired. Three subjects (all of them experienced 3D first) reported the visual fatigue, dizziness and discomfort of the 3D stereoscopic glasses. On the other hand, five subjects (four of them experienced 2D first) reported the discomfort with the emotional movie because they had difficulty to adapt to dynamic changes in color of the video.

The most frequently self-reported emotional state was “excited” or “aroused” after the 2D fearful clips and “excited” or “pleasant” after the 2D joyful clips. After the 3D fearful clips, most subjects reported “excited” followed by a mixture of “aroused”, “unpleasant” and “pleasant” state. On the other hand, most subjects reported “pleasant” state after the 3D joyful clips. As aforementioned, it is shown the increased occurrences of “pleasant” state after the viewing of the joyful clips (than fearful clips) and 3D (than 2D).

Interestingly, the subjects recalled more scenes after the 3D emotional movie than 2D emotional movie. Overall, the most memorable scenes were 3D ones with a very perceptible depth, i.e. when Marley throws his chains, or very dynamic 3D ones, i.e. when Scrooge flies to Christmas past with the candle ghost and when Scrooge is chased by ghost hunters. These results matched with the cut-scenes where the EEG beta/alpha index were the highest during the screening of the 3D emotional movie.

Correlation between subject’s physiological and subjective emotional responses

Figure 9 shows the average of the correlation between the subject’s emotional states estimated by using physiological signals and the self-reported subjective feeling. The correlation between the subject’s physiological responses and the subjective feeling was, on average, about 45.83% match (2D fearful), 62.5% (2D joyful), 39.58% (3D fearful), and 68.75% (3D joyful). The paired t-test results showed a significant mean difference in the correlation rate between 3D fearful video clips (M = 39.58%) and 3D joyful video clips (M = 68.75%) where the t value was − 3.189 with p = 0.0043. We believed an accuracy of 62% to be high enough for the intuitive subject responses and the untrained physiological responses since most subjects had never used physiological sensors before and none of them were familiar with the emotional video system.

Fig. 9

Average of the correlation between the emotion responses using physiological signals and subjective feeling

Discussion

The emotionally intelligent contents tried to lead users to specific emotions by using refined color masks, brightness, contrast, and other aural/visual effects. For example, a first-person shooting game dynamically changed the color of the blood from a vivid red to a more soft bright green in real-time when it detected a player being too focused on the game [31]. Changing the color scheme of game artefacts is also found in a few modern games to make them appear less violent for younger game players.

In prior work, it is known that 3D convey a higher sense of presence or realism but also increases visual fatigue. A recent study found that 3D movies have little effect on viewers’ emotional responses as compared to 2D movies [11], except for a thrilling scene from The Polar Express. This particular scene had a great influence on user emotion, with greater 3D effects than used in other films. While there are many studies comparing 2D and 3D on user experience, such as visual fatigue, presence, and emotion [5,6,7], little is known about the effect of emotional movie.

This research focused on studying user’s affective (subjective) and physiological (objective) responses to the 3D emotional movie. Among various films, A Christmas Carol (a 3D computer animation film released in 2009) was chosen because it delivers the remarkable 3D effects that are similar to those in a theatrical 3D exhibition. Viewers may feel like they can reach out and touch Marley’s heavy chains or fly over London or travel through time to the past.

The movie clips are edited by fearful and joyful scenes in terms of both the storyline and the color scheme, which also has an effect on user emotion. We expected that the 3D emotional movie provides a greater level of user experience than the 2D emotional movie due to 3D effects. We also expected there to be more user affective and physiological responses from the joyful clips than the fearful clips because a greater degree of brightness creates a light-hearted feel.

The study results revealed that there was a little difference between 2D and 3D in terms of physiological responses. A statistically significant mean difference was found in number of GSR startles between 2D and 3D joyful clips. Also, a statistically significant mean difference was found in the EEG beta/alpha index between 2D and 3D from moment to moment, such as when Scrooge is being actively chased by ghost hunters, when the candle ghost appears, and when Scrooge flies to Christmas past with the candle ghost.

Interestingly, the results revealed a difference between fearful and joyful clips during the screening of the 3D emotional movie. The VLF/HF ratios of PPG and the beta/alpha index of EEG were slightly increased with the 3D joyful clips as compared to the 3D fearful clips. The number of GSR startles showed statistically significant increase with the 3D joyful clips. The self-reported subjective feeling also showed more “pleasant” with the joyful clips. This is somewhat similar to the findings from a study evaluating the impact of color hue, brightness and saturation of websites on consumer emotions [15]. Moreover, the correlation between subject’s emotional states estimated by physiological signals and self-reported subjective feeling was significantly higher in the 3D joyful clips than the 3D fearful clips.

Conclusions

In recent years, there has been a rapid increase in the development of 3D contents and technologies, such as 3D movies and virtual/augmented reality [32]. As interest in the use of 3D technology increases, researchers have paid more attention to the effect of 3D contents on user experiences. There have been many research efforts showing that the 3D contents can improve visual realism, presence, immersion and even performance gains on some tasks, but at the same time, increase visual fatigue as compared to 2D contents.

Over the past two decades, affective computing research has intensively studied methods for emotion recognition [33], emotion expression [21,22,23,24] and emotion response [27]. However, only a few studies have looked at 3D emotional contents. Many existing studies have strived to evaluate stimuli that evokes human affects, using the presentation of emotional images and short film clips [8,9,10,11, 34, 35]. Typically, affective images or short movie clips that are 1–3 min in length are selected to elicit target emotions in the viewer.

However, these approaches are rather passive since they only measure user emotions elicited by the contents and do not reflect users’ affective states into the contents. In this research, we considered the emotionally intelligent contents that respond interactively to users’ current affective states and the visual effects of the contents that are dynamically changed depending on the users’ emotional changes.

We have conducted a user study on the emotional movies where the saturation, contrast and brightness of the movie were increased or decreased by user’s emotional states. The objective was to study users’ affective (subjective) and physiological (objective) responses to the interactive natures of the emotional movies to find out if users feel more involved with the storyline and/or if they evoke more emotional responses. The edited footage of Disney’s A Christmas Carol, going from fearful and joyful scenes was presented in both 2D and 3D formats.

Overall, there was no statistically significant mean difference between 2D and 3D emotional movies. However, a statistically significant difference was found in the GSR startles between the 2D and 3D joyful clips as well as in the EEG beta/alpha index between some short 2D and 3D scenes. As aforementioned, users showed more responses to 3D bright joyful clips than to 3D dark fearful clips.

User emotion is a complex phenomenon. The factors that affect user emotions are varied. This research is the first attempt to analyze what would affect users’ emotional states in the emotional movie, i.e., the effect of 2D or 3D, and the role of fearful or joyful clips (such as colors/tones of light in the scenes). However, we need to further compare and study regular 3D contents with 3D emotional contents to determine whether the users’ emotional states are elicited by real-time changes of visual effects in the emotional contents or by the colors or tones of the scenes.