FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model

Soo Ryeon Lee; Dohyun Kim; Mingyu Lee; Sangkeun Lee

doi:10.1109/ICCICC53683.2021.9811331

FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model

Soo Ryeon Lee, Dohyun Kim, Mingyu Lee, Sangkeun Lee

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Visual cognition requires analyzing actions, intentions, and emotions of persons in a given image. Visual Commonsense Reasoning (VCR) is a task that selects rationales and answers to questions for given images. In VCR, facial expressions are important nonverbal signals because they convey emotions and intentions in human interactions. However, ERNIE-ViL and UNITER, which are vision-And-language models to get image and text representations, do not learn them. We find that ERNIE-ViL and UNITER are vulnerable to the problem of identifying emotions. In this paper, therefore, we propose facial expression recognition FERNIE-ViL, which adapts a facial expression recognition module to the existing vision-And-language model. Experimental results (2.4% point improvement on VCR Q?A and 0.3% point improvement on VCR QA?R) demonstrate that our method can enhance visual commonsense reasoning by understanding human interactions.

Original language	English
Title of host publication	Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021
Editors	Yingxu Wang, Jane Z. Wang, Henry Leung, Newton Howard, Paolo Soda, Bernard Widrow, Jerome Feldman
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	228-234
Number of pages	7
ISBN (Electronic)	9781665421195
DOIs	https://doi.org/10.1109/ICCICC53683.2021.9811331
Publication status	Published - 2021
Event	20th IEEE International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021 - Banff, Canada Duration: 2021 Oct 29 → 2021 Oct 31

Publication series

Name	Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021

Conference

Conference	20th IEEE International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021
Country/Territory	Canada
City	Banff
Period	21/10/29 → 21/10/31

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Keywords

Artificial Intelligence
Commonsense Reasoning
Facial Expression
Machine Commonsense
Multi-modal
Natural Language Processing
Visual Recognition

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Information Systems
Cognitive Neuroscience

Access to Document

10.1109/ICCICC53683.2021.9811331

Cite this

Lee, S. R., Kim, D., Lee, M., & Lee, S. (2021). FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model. In Y. Wang, J. Z. Wang, H. Leung, N. Howard, P. Soda, B. Widrow, & J. Feldman (Eds.), Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021 (pp. 228-234). (Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCICC53683.2021.9811331

FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model. / Lee, Soo Ryeon; Kim, Dohyun; Lee, Mingyu et al.
Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021. ed. / Yingxu Wang; Jane Z. Wang; Henry Leung; Newton Howard; Paolo Soda; Bernard Widrow; Jerome Feldman. Institute of Electrical and Electronics Engineers Inc., 2021. p. 228-234 (Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, SR, Kim, D, Lee, M & Lee, S 2021, FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model. in Y Wang, JZ Wang, H Leung, N Howard, P Soda, B Widrow & J Feldman (eds), Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021. Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021, Institute of Electrical and Electronics Engineers Inc., pp. 228-234, 20th IEEE International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021, Banff, Canada, 21/10/29. https://doi.org/10.1109/ICCICC53683.2021.9811331

Lee SR, Kim D, Lee M, Lee S. FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model. In Wang Y, Wang JZ, Leung H, Howard N, Soda P, Widrow B, Feldman J, editors, Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 228-234. (Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021). doi: 10.1109/ICCICC53683.2021.9811331

Lee, Soo Ryeon ; Kim, Dohyun ; Lee, Mingyu et al. / FERNIE-ViL : Facial Expression Enhanced Vision-And-Language Model. Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021. editor / Yingxu Wang ; Jane Z. Wang ; Henry Leung ; Newton Howard ; Paolo Soda ; Bernard Widrow ; Jerome Feldman. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 228-234 (Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021).

@inproceedings{b0fb7ac675774b7789f81a89176e21e1,

title = "FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model",

abstract = "Visual cognition requires analyzing actions, intentions, and emotions of persons in a given image. Visual Commonsense Reasoning (VCR) is a task that selects rationales and answers to questions for given images. In VCR, facial expressions are important nonverbal signals because they convey emotions and intentions in human interactions. However, ERNIE-ViL and UNITER, which are vision-And-language models to get image and text representations, do not learn them. We find that ERNIE-ViL and UNITER are vulnerable to the problem of identifying emotions. In this paper, therefore, we propose facial expression recognition FERNIE-ViL, which adapts a facial expression recognition module to the existing vision-And-language model. Experimental results (2.4% point improvement on VCR Q?A and 0.3% point improvement on VCR QA?R) demonstrate that our method can enhance visual commonsense reasoning by understanding human interactions.",

keywords = "Artificial Intelligence, Commonsense Reasoning, Facial Expression, Machine Commonsense, Multi-modal, Natural Language Processing, Visual Recognition",

author = "Lee, {Soo Ryeon} and Dohyun Kim and Mingyu Lee and Sangkeun Lee",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 20th IEEE International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021 ; Conference date: 29-10-2021 Through 31-10-2021",

year = "2021",

doi = "10.1109/ICCICC53683.2021.9811331",

language = "English",

series = "Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "228--234",

editor = "Yingxu Wang and Wang, {Jane Z.} and Henry Leung and Newton Howard and Paolo Soda and Bernard Widrow and Jerome Feldman",

booktitle = "Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021",

}

TY - GEN

T1 - FERNIE-ViL

T2 - 20th IEEE International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021

AU - Lee, Soo Ryeon

AU - Kim, Dohyun

AU - Lee, Mingyu

AU - Lee, Sangkeun

PY - 2021

Y1 - 2021

N2 - Visual cognition requires analyzing actions, intentions, and emotions of persons in a given image. Visual Commonsense Reasoning (VCR) is a task that selects rationales and answers to questions for given images. In VCR, facial expressions are important nonverbal signals because they convey emotions and intentions in human interactions. However, ERNIE-ViL and UNITER, which are vision-And-language models to get image and text representations, do not learn them. We find that ERNIE-ViL and UNITER are vulnerable to the problem of identifying emotions. In this paper, therefore, we propose facial expression recognition FERNIE-ViL, which adapts a facial expression recognition module to the existing vision-And-language model. Experimental results (2.4% point improvement on VCR Q?A and 0.3% point improvement on VCR QA?R) demonstrate that our method can enhance visual commonsense reasoning by understanding human interactions.

AB - Visual cognition requires analyzing actions, intentions, and emotions of persons in a given image. Visual Commonsense Reasoning (VCR) is a task that selects rationales and answers to questions for given images. In VCR, facial expressions are important nonverbal signals because they convey emotions and intentions in human interactions. However, ERNIE-ViL and UNITER, which are vision-And-language models to get image and text representations, do not learn them. We find that ERNIE-ViL and UNITER are vulnerable to the problem of identifying emotions. In this paper, therefore, we propose facial expression recognition FERNIE-ViL, which adapts a facial expression recognition module to the existing vision-And-language model. Experimental results (2.4% point improvement on VCR Q?A and 0.3% point improvement on VCR QA?R) demonstrate that our method can enhance visual commonsense reasoning by understanding human interactions.

KW - Artificial Intelligence

KW - Commonsense Reasoning

KW - Facial Expression

KW - Machine Commonsense

KW - Multi-modal

KW - Natural Language Processing

KW - Visual Recognition

UR - http://www.scopus.com/inward/record.url?scp=85135372884&partnerID=8YFLogxK

U2 - 10.1109/ICCICC53683.2021.9811331

DO - 10.1109/ICCICC53683.2021.9811331

M3 - Conference contribution

AN - SCOPUS:85135372884

T3 - Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021

SP - 228

EP - 234

BT - Proceedings of 2021 IEEE 20th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2021

A2 - Wang, Yingxu

A2 - Wang, Jane Z.

A2 - Leung, Henry

A2 - Howard, Newton

A2 - Soda, Paolo

A2 - Widrow, Bernard

A2 - Feldman, Jerome

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 29 October 2021 through 31 October 2021

ER -

FERNIE-ViL: Facial Expression Enhanced Vision-And-Language Model

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this