Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation

Serin Park; Christian Wallraven

doi:10.1007/978-3-031-02375-0_30

Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation

Serin Park, Christian Wallraven

Department of Brain and Cognitive Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

Facial expression recognition (FER) is a topic attracting significant research in both psychology and machine learning with a wide range of applications. Despite a wealth of research on human FER and considerable progress in computational FER made possible by deep neural networks (DNNs), comparatively less work has been done on comparing the degree to which DNNs may be comparable to human performance. In this work, we compared the recognition performance and attention patterns of humans and machines during a two-alternative forced-choice FER task. Human attention was here gathered through click data that progressively uncovered a face, whereas model attention was obtained using three different popular techniques from explainable AI: CAM, GradCAM and Extremal Perturbation. In both cases, performance was gathered as percent correct. For this task, we found that humans outperformed machines quite significantly. In terms of attention patterns, we found that Extremal Perturbation had the best overall fit with the human attention map during the task.

Original language	English
Title of host publication	Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers
Editors	Christian Wallraven, Qingshan Liu, Hajime Nagahara
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	403-416
Number of pages	14
ISBN (Print)	9783031023743
DOIs	https://doi.org/10.1007/978-3-031-02375-0_30
Publication status	Published - 2022
Event	6th Asian Conference on Pattern Recognition, ACPR 2021 - Virtual, Online Duration: 2021 Nov 9 → 2021 Nov 12

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13188 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	6th Asian Conference on Pattern Recognition, ACPR 2021
City	Virtual, Online
Period	21/11/9 → 21/11/12

Bibliographical note

Funding Information:
Acknowledgments. This work was supported by Institute of Information Communications Technology Planning Evaluation (IITP; No. 2019-0-00079, Department of Artificial Intelligence, Korea University) and National Research Foundation of Korea (NRF; NRF-2017M3C7A1041824) grant funded by the Korean government (MSIT).

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

Keywords

AffectNet
Facial expression recognition
Human-in-the-loop
Humans versus machines

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-031-02375-0_30

Cite this

Park, S., & Wallraven, C. (2022). Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation. In C. Wallraven, Q. Liu, & H. Nagahara (Eds.), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers (pp. 403-416). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13188 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-02375-0_30

Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation. / Park, Serin; Wallraven, Christian.
Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. ed. / Christian Wallraven; Qingshan Liu; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. p. 403-416 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13188 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Park, S & Wallraven, C 2022, Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation. in C Wallraven, Q Liu & H Nagahara (eds), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13188 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 403-416, 6th Asian Conference on Pattern Recognition, ACPR 2021, Virtual, Online, 21/11/9. https://doi.org/10.1007/978-3-031-02375-0_30

Park S, Wallraven C. Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation. In Wallraven C, Liu Q, Nagahara H, editors, Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2022. p. 403-416. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-02375-0_30

Park, Serin ; Wallraven, Christian. / Comparing Facial Expression Recognition in Humans and Machines : Using CAM, GradCAM, and Extremal Perturbation. Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. editor / Christian Wallraven ; Qingshan Liu ; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. pp. 403-416 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{6f468fadf359454399902bca8e0e20c3,

title = "Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation",

abstract = "Facial expression recognition (FER) is a topic attracting significant research in both psychology and machine learning with a wide range of applications. Despite a wealth of research on human FER and considerable progress in computational FER made possible by deep neural networks (DNNs), comparatively less work has been done on comparing the degree to which DNNs may be comparable to human performance. In this work, we compared the recognition performance and attention patterns of humans and machines during a two-alternative forced-choice FER task. Human attention was here gathered through click data that progressively uncovered a face, whereas model attention was obtained using three different popular techniques from explainable AI: CAM, GradCAM and Extremal Perturbation. In both cases, performance was gathered as percent correct. For this task, we found that humans outperformed machines quite significantly. In terms of attention patterns, we found that Extremal Perturbation had the best overall fit with the human attention map during the task.",

keywords = "AffectNet, Facial expression recognition, Human-in-the-loop, Humans versus machines",

author = "Serin Park and Christian Wallraven",

note = "Funding Information: Acknowledgments. This work was supported by Institute of Information Communications Technology Planning Evaluation (IITP; No. 2019-0-00079, Department of Artificial Intelligence, Korea University) and National Research Foundation of Korea (NRF; NRF-2017M3C7A1041824) grant funded by the Korean government (MSIT). Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.; 6th Asian Conference on Pattern Recognition, ACPR 2021 ; Conference date: 09-11-2021 Through 12-11-2021",

year = "2022",

doi = "10.1007/978-3-031-02375-0_30",

language = "English",

isbn = "9783031023743",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "403--416",

editor = "Christian Wallraven and Qingshan Liu and Hajime Nagahara",

booktitle = "Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers",

address = "Germany",

}

TY - GEN

T1 - Comparing Facial Expression Recognition in Humans and Machines

T2 - 6th Asian Conference on Pattern Recognition, ACPR 2021

AU - Park, Serin

AU - Wallraven, Christian

N1 - Funding Information: Acknowledgments. This work was supported by Institute of Information Communications Technology Planning Evaluation (IITP; No. 2019-0-00079, Department of Artificial Intelligence, Korea University) and National Research Foundation of Korea (NRF; NRF-2017M3C7A1041824) grant funded by the Korean government (MSIT). Publisher Copyright: © 2022, Springer Nature Switzerland AG.

PY - 2022

Y1 - 2022

N2 - Facial expression recognition (FER) is a topic attracting significant research in both psychology and machine learning with a wide range of applications. Despite a wealth of research on human FER and considerable progress in computational FER made possible by deep neural networks (DNNs), comparatively less work has been done on comparing the degree to which DNNs may be comparable to human performance. In this work, we compared the recognition performance and attention patterns of humans and machines during a two-alternative forced-choice FER task. Human attention was here gathered through click data that progressively uncovered a face, whereas model attention was obtained using three different popular techniques from explainable AI: CAM, GradCAM and Extremal Perturbation. In both cases, performance was gathered as percent correct. For this task, we found that humans outperformed machines quite significantly. In terms of attention patterns, we found that Extremal Perturbation had the best overall fit with the human attention map during the task.

AB - Facial expression recognition (FER) is a topic attracting significant research in both psychology and machine learning with a wide range of applications. Despite a wealth of research on human FER and considerable progress in computational FER made possible by deep neural networks (DNNs), comparatively less work has been done on comparing the degree to which DNNs may be comparable to human performance. In this work, we compared the recognition performance and attention patterns of humans and machines during a two-alternative forced-choice FER task. Human attention was here gathered through click data that progressively uncovered a face, whereas model attention was obtained using three different popular techniques from explainable AI: CAM, GradCAM and Extremal Perturbation. In both cases, performance was gathered as percent correct. For this task, we found that humans outperformed machines quite significantly. In terms of attention patterns, we found that Extremal Perturbation had the best overall fit with the human attention map during the task.

KW - AffectNet

KW - Facial expression recognition

KW - Human-in-the-loop

KW - Humans versus machines

UR - http://www.scopus.com/inward/record.url?scp=85130341946&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-02375-0_30

DO - 10.1007/978-3-031-02375-0_30

M3 - Conference contribution

AN - SCOPUS:85130341946

SN - 9783031023743

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 403

EP - 416

BT - Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers

A2 - Wallraven, Christian

A2 - Liu, Qingshan

A2 - Nagahara, Hajime

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 9 November 2021 through 12 November 2021

ER -

Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this