Label Quality in AffectNet: Results of Crowd-Based Re-annotation

Doo Yon Kim; Christian Wallraven

doi:10.1007/978-3-031-02444-3_39

Label Quality in AffectNet: Results of Crowd-Based Re-annotation

Doo Yon Kim
, Christian Wallraven^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

AffectNet is one of the most popular resources for facial expression recognition (FER) on relatively unconstrained in-the-wild images. Given that images were annotated by only one annotator with limited consistency checks on the data, however, label quality and consistency may be limited. Here, we take a similar approach to a study that re-labeled another, smaller dataset (FER2013) with crowd-based annotations, and report results from a re-labeling and re-annotation of a subset of difficult AffectNet faces with 13 people on both expression label, and valence and arousal ratings. Our results show that human labels overall have medium to good consistency, whereas human ratings especially for valence are in excellent agreement. Importantly, however, crowd-based labels are significantly shifting towards neutral and happy categories and crowd-based affective ratings form a consistent pattern different from the original ratings. ResNets fully trained on the original AffectNet dataset do not predict human voting patterns, but when weakly-trained do so much better, particularly for valence. Our results have important ramifications for label quality in affective computing.

Original language	English
Title of host publication	Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers
Editors	Christian Wallraven, Qingshan Liu, Hajime Nagahara
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	518-531
Number of pages	14
ISBN (Print)	9783031024436
DOIs	https://doi.org/10.1007/978-3-031-02444-3_39
Publication status	Published - 2022
Event	6th Asian Conference on Pattern Recognition, ACPR 2021 - Virtual, Online Duration: 2021 Nov 9 → 2021 Nov 12

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13189 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	6th Asian Conference on Pattern Recognition, ACPR 2021
City	Virtual, Online
Period	21/11/9 → 21/11/12

Bibliographical note

Funding Information:
Acknowledgments. This work was supported by Institute of Information Communications Technology Planning Evaluation (IITP; No. 2019-0-00079, Department of Artificial Intelligence, Korea University) and National Research Foundation of Korea (NRF; NRF-2017M3C7A1041824) grant funded by the Korean government (MSIT).

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

Keywords

AffectNet
Affective computing
Crowd annotation
Facial expression recognition

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-02444-3_39

Cite this

Kim, D. Y., & Wallraven, C. (2022). Label Quality in AffectNet: Results of Crowd-Based Re-annotation. In C. Wallraven, Q. Liu, & H. Nagahara (Eds.), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers (pp. 518-531). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13189 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-02444-3_39

Label Quality in AffectNet: Results of Crowd-Based Re-annotation. / Kim, Doo Yon; Wallraven, Christian.
Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. ed. / Christian Wallraven; Qingshan Liu; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. p. 518-531 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13189 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kim, DY & Wallraven, C 2022, Label Quality in AffectNet: Results of Crowd-Based Re-annotation. in C Wallraven, Q Liu & H Nagahara (eds), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13189 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 518-531, 6th Asian Conference on Pattern Recognition, ACPR 2021, Virtual, Online, 21/11/9. https://doi.org/10.1007/978-3-031-02444-3_39

Kim DY, Wallraven C. Label Quality in AffectNet: Results of Crowd-Based Re-annotation. In Wallraven C, Liu Q, Nagahara H, editors, Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2022. p. 518-531. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-02444-3_39

Kim, Doo Yon ; Wallraven, Christian. / Label Quality in AffectNet : Results of Crowd-Based Re-annotation. Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. editor / Christian Wallraven ; Qingshan Liu ; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. pp. 518-531 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{f71076b9891f4730b5205302d63a74f3,

title = "Label Quality in AffectNet: Results of Crowd-Based Re-annotation",

abstract = "AffectNet is one of the most popular resources for facial expression recognition (FER) on relatively unconstrained in-the-wild images. Given that images were annotated by only one annotator with limited consistency checks on the data, however, label quality and consistency may be limited. Here, we take a similar approach to a study that re-labeled another, smaller dataset (FER2013) with crowd-based annotations, and report results from a re-labeling and re-annotation of a subset of difficult AffectNet faces with 13 people on both expression label, and valence and arousal ratings. Our results show that human labels overall have medium to good consistency, whereas human ratings especially for valence are in excellent agreement. Importantly, however, crowd-based labels are significantly shifting towards neutral and happy categories and crowd-based affective ratings form a consistent pattern different from the original ratings. ResNets fully trained on the original AffectNet dataset do not predict human voting patterns, but when weakly-trained do so much better, particularly for valence. Our results have important ramifications for label quality in affective computing.",

keywords = "AffectNet, Affective computing, Crowd annotation, Facial expression recognition",

author = "Kim, \{Doo Yon\} and Christian Wallraven",

note = "Funding Information: Acknowledgments. This work was supported by Institute of Information Communications Technology Planning Evaluation (IITP; No. 2019-0-00079, Department of Artificial Intelligence, Korea University) and National Research Foundation of Korea (NRF; NRF-2017M3C7A1041824) grant funded by the Korean government (MSIT). Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.; 6th Asian Conference on Pattern Recognition, ACPR 2021 ; Conference date: 09-11-2021 Through 12-11-2021",

year = "2022",

doi = "10.1007/978-3-031-02444-3\_39",

language = "English",

isbn = "9783031024436",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "518--531",

editor = "Christian Wallraven and Qingshan Liu and Hajime Nagahara",

booktitle = "Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers",

address = "Germany",

}

TY - GEN

T1 - Label Quality in AffectNet

T2 - 6th Asian Conference on Pattern Recognition, ACPR 2021

AU - Kim, Doo Yon

AU - Wallraven, Christian

N1 - Funding Information: Acknowledgments. This work was supported by Institute of Information Communications Technology Planning Evaluation (IITP; No. 2019-0-00079, Department of Artificial Intelligence, Korea University) and National Research Foundation of Korea (NRF; NRF-2017M3C7A1041824) grant funded by the Korean government (MSIT). Publisher Copyright: © 2022, Springer Nature Switzerland AG.

PY - 2022

Y1 - 2022

N2 - AffectNet is one of the most popular resources for facial expression recognition (FER) on relatively unconstrained in-the-wild images. Given that images were annotated by only one annotator with limited consistency checks on the data, however, label quality and consistency may be limited. Here, we take a similar approach to a study that re-labeled another, smaller dataset (FER2013) with crowd-based annotations, and report results from a re-labeling and re-annotation of a subset of difficult AffectNet faces with 13 people on both expression label, and valence and arousal ratings. Our results show that human labels overall have medium to good consistency, whereas human ratings especially for valence are in excellent agreement. Importantly, however, crowd-based labels are significantly shifting towards neutral and happy categories and crowd-based affective ratings form a consistent pattern different from the original ratings. ResNets fully trained on the original AffectNet dataset do not predict human voting patterns, but when weakly-trained do so much better, particularly for valence. Our results have important ramifications for label quality in affective computing.

AB - AffectNet is one of the most popular resources for facial expression recognition (FER) on relatively unconstrained in-the-wild images. Given that images were annotated by only one annotator with limited consistency checks on the data, however, label quality and consistency may be limited. Here, we take a similar approach to a study that re-labeled another, smaller dataset (FER2013) with crowd-based annotations, and report results from a re-labeling and re-annotation of a subset of difficult AffectNet faces with 13 people on both expression label, and valence and arousal ratings. Our results show that human labels overall have medium to good consistency, whereas human ratings especially for valence are in excellent agreement. Importantly, however, crowd-based labels are significantly shifting towards neutral and happy categories and crowd-based affective ratings form a consistent pattern different from the original ratings. ResNets fully trained on the original AffectNet dataset do not predict human voting patterns, but when weakly-trained do so much better, particularly for valence. Our results have important ramifications for label quality in affective computing.

KW - AffectNet

KW - Affective computing

KW - Crowd annotation

KW - Facial expression recognition

UR - https://www.scopus.com/pages/publications/85130309381

U2 - 10.1007/978-3-031-02444-3_39

DO - 10.1007/978-3-031-02444-3_39

M3 - Conference contribution

AN - SCOPUS:85130309381

SN - 9783031024436

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 518

EP - 531

BT - Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers

A2 - Wallraven, Christian

A2 - Liu, Qingshan

A2 - Nagahara, Hajime

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 9 November 2021 through 12 November 2021

ER -

Label Quality in AffectNet: Results of Crowd-Based Re-annotation

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this