Affective Computing and Emotions: A Multimodal Analysis on Oscar-Nominated Films

Thumbnail Image
Lui, Jeffrey
Affective computing , Multimodal , Oscars , Emotion
The field of Affective Computing (AC), or Emotion AI, has rapidly grown in popularity over the last three decades since its inception in 1995. Independent researchers, and academic and industrial organizations have all been discovering new ways to utilize automatic affect recognition to uncover new efficiencies, maximize net profits, and even design new media from a subject’s emotive state. To date, most AC research has primarily focused on the use of one single sensorial modality (unimodal) to extract emotions. However, as the field evolves, the use of multimodal sensory inputs (multimodality) is rising and provides researchers with more rich, accurate, and reliable insights from affect recognition and identification. In this thesis dissertation, a literature review of Affective Computing is included. It defines the main constructs of AC and the main modalities used to detect emotions. I draw from the most significant literature produced in the field, beginning from Rosalind Picard’s originating literature in 1995. I conduct a systematic review of the most notable modalities used in AC research and then how emotion appraisal is foundational to this field. Following this, I conducted an analytical study where affect data is extracted from all Oscar-nominated films for "Best Picture" since 2010 using three AC modalities: (1) Facial Features, (2) Language & Text, and (3) Prosody. Using a novel and proprietary extraction method, followed by descriptive statistical analyses, I answered the research question, "What is the volume and variety of emotions portrayed by actors in Best Picture Oscar-nominated films, and how might they impact the film’s potential for success?" It was discovered that there are moderately strong positive correlations between a film’s critic ratings and the volume of emotional language used. Similarly, results showed that there was a significant negative correlation between audience reviews and the volume of emotions detected from an actor’s face. And when comparing audience reviews to the co-efficient of variation from the variety of emotions detected, it was found that films with a more balanced and equal variety of emotions significantly produced higher audience film review scores.
External DOI