Contextual modulation of affect: Comparing humans and deep neural networks

Soomin Shin, Doo Yon Kim, Christian Wallraven

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)


When inferring emotions, humans rely on a number of cues, including not only facial expressions, body posture, but also expressor-external, contextual information. The goal of the present study was to compare the impact of such contextual information on emotion processing in humans and two deep neural network (DNN) models. We used results from a human experiment in which two types of pictures were rated for valence and arousal: the first type depicted people expressing an emotion in a social context including other people; the second was a context-reduced version in which all information except for the target expressor was blurred out. The resulting human ratings of valence and arousal were systematically decreased in the context-reduced version, highlighting the importance of context. We then compared human ratings with those of two DNN models (one trained on face images only, and the other trained also on contextual information). Analyses of both categorical and the valence/arousal ratings showed that although there were some superficial similarities, both models failed to capture human rating patterns both in context-rich and context-reduced conditions. Our study emphasizes the importance of a more holistic, multi-modal training regime with richer human data to build better emotion-understanding systems in the area of affective computing.

Original languageEnglish
Title of host publicationICMI 2022 - Companion Publication of the 2022 International Conference on Multimodal Interaction
PublisherAssociation for Computing Machinery
Number of pages7
ISBN (Electronic)9781450393898
Publication statusPublished - 2022 Nov 7
Event24th ACM International Conference on Multimodal Interaction, ICMI 2022 - Bangalore, India
Duration: 2022 Nov 72022 Nov 11

Publication series

NameACM International Conference Proceeding Series


Conference24th ACM International Conference on Multimodal Interaction, ICMI 2022

Bibliographical note

Funding Information:
This study was supported by the National Research Foundation of Korea under project BK21 FOUR, grants NRF-2017M3C7A1041824, NRF-2019R1A2C2007612, as well as by three Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning; 2019-0-00079, Department of Artifcial Intelligence, Korea University; 2021-0-02068, Artifcial Intelligence Inovation Hub).

Publisher Copyright:
© 2022 ACM.


  • DNNs
  • affective computing
  • contextual information
  • emotion recognition
  • user study

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications


Dive into the research topics of 'Contextual modulation of affect: Comparing humans and deep neural networks'. Together they form a unique fingerprint.

Cite this