Auto-generating virtual human behavior by understanding user contexts

Hanseob Kim; Ghazanfar Ali; Seungwon Kim; Gerard J. Kim; Jae In Hwang

doi:10.1109/VRW52623.2021.00178

Auto-generating virtual human behavior by understanding user contexts

Hanseob Kim, Ghazanfar Ali, Seungwon Kim, Gerard J. Kim, Jae In Hwang

Department of Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

Virtual humans are most natural and effective when it can act out and animate verbal/gestural actions. One popular method to realize this is to infer the actions from predefined phrases. This research aims to provide a more flexible method to activate various behaviors straight from natural conversations. Our approach uses BERT as the backbone for natural language understanding and, on top of it, a jointly learned sentence classifier (SC) and entity classifier (EC). The SC classifies the input into conversation or action, and EC extracts the entities for the action. The pilot study has shown promising results with high perceived naturalness and positive experiences.

Original language	English
Title of host publication	Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	591-592
Number of pages	2
ISBN (Electronic)	9780738113678
DOIs	https://doi.org/10.1109/VRW52623.2021.00178
Publication status	Published - 2021 Mar
Event	2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021 - Virtual, Lisbon, Portugal Duration: 2021 Mar 27 → 2021 Apr 3

Publication series

Name	Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021

Conference

Conference	2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021
Country/Territory	Portugal
City	Virtual, Lisbon
Period	21/3/27 → 21/4/3

Bibliographical note

Funding Information:
H. Kim and G. Ali contributed equally to this paper. This work has supported by the National Research Council of Science & Technology grant by the Korea Government (MSIP CRC-20-02-KIST).

Publisher Copyright:
© 2021 IEEE.

Keywords

Computing methodologies
Human-centered computing
Interaction design process and methods
Natural language processing
Virtual reality

ASJC Scopus subject areas

Human-Computer Interaction
Media Technology
Modelling and Simulation

Access to Document

10.1109/VRW52623.2021.00178

Cite this

Kim, H., Ali, G., Kim, S., Kim, G. J., & Hwang, J. I. (2021). Auto-generating virtual human behavior by understanding user contexts. In Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021 (pp. 591-592). Article 9419331 (Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/VRW52623.2021.00178

Auto-generating virtual human behavior by understanding user contexts. / Kim, Hanseob; Ali, Ghazanfar; Kim, Seungwon et al.
Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021. Institute of Electrical and Electronics Engineers Inc., 2021. p. 591-592 9419331 (Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kim, H, Ali, G, Kim, S, Kim, GJ & Hwang, JI 2021, Auto-generating virtual human behavior by understanding user contexts. in Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021., 9419331, Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021, Institute of Electrical and Electronics Engineers Inc., pp. 591-592, 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021, Virtual, Lisbon, Portugal, 21/3/27. https://doi.org/10.1109/VRW52623.2021.00178

Kim H, Ali G, Kim S, Kim GJ, Hwang JI. Auto-generating virtual human behavior by understanding user contexts. In Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 591-592. 9419331. (Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021). doi: 10.1109/VRW52623.2021.00178

Kim, Hanseob ; Ali, Ghazanfar ; Kim, Seungwon et al. / Auto-generating virtual human behavior by understanding user contexts. Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 591-592 (Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021).

@inproceedings{69bed9fc6c64483abad7f72bdc2973ee,

title = "Auto-generating virtual human behavior by understanding user contexts",

abstract = "Virtual humans are most natural and effective when it can act out and animate verbal/gestural actions. One popular method to realize this is to infer the actions from predefined phrases. This research aims to provide a more flexible method to activate various behaviors straight from natural conversations. Our approach uses BERT as the backbone for natural language understanding and, on top of it, a jointly learned sentence classifier (SC) and entity classifier (EC). The SC classifies the input into conversation or action, and EC extracts the entities for the action. The pilot study has shown promising results with high perceived naturalness and positive experiences.",

keywords = "Computing methodologies, Human-centered computing, Interaction design process and methods, Natural language processing, Virtual reality",

author = "Hanseob Kim and Ghazanfar Ali and Seungwon Kim and Kim, {Gerard J.} and Hwang, {Jae In}",

note = "Funding Information: H. Kim and G. Ali contributed equally to this paper. This work has supported by the National Research Council of Science & Technology grant by the Korea Government (MSIP CRC-20-02-KIST). Publisher Copyright: {\textcopyright} 2021 IEEE.; 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021 ; Conference date: 27-03-2021 Through 03-04-2021",

year = "2021",

month = mar,

doi = "10.1109/VRW52623.2021.00178",

language = "English",

series = "Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "591--592",

booktitle = "Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021",

}

TY - GEN

T1 - Auto-generating virtual human behavior by understanding user contexts

AU - Kim, Hanseob

AU - Ali, Ghazanfar

AU - Kim, Seungwon

AU - Kim, Gerard J.

AU - Hwang, Jae In

N1 - Funding Information: H. Kim and G. Ali contributed equally to this paper. This work has supported by the National Research Council of Science & Technology grant by the Korea Government (MSIP CRC-20-02-KIST). Publisher Copyright: © 2021 IEEE.

PY - 2021/3

Y1 - 2021/3

N2 - Virtual humans are most natural and effective when it can act out and animate verbal/gestural actions. One popular method to realize this is to infer the actions from predefined phrases. This research aims to provide a more flexible method to activate various behaviors straight from natural conversations. Our approach uses BERT as the backbone for natural language understanding and, on top of it, a jointly learned sentence classifier (SC) and entity classifier (EC). The SC classifies the input into conversation or action, and EC extracts the entities for the action. The pilot study has shown promising results with high perceived naturalness and positive experiences.

AB - Virtual humans are most natural and effective when it can act out and animate verbal/gestural actions. One popular method to realize this is to infer the actions from predefined phrases. This research aims to provide a more flexible method to activate various behaviors straight from natural conversations. Our approach uses BERT as the backbone for natural language understanding and, on top of it, a jointly learned sentence classifier (SC) and entity classifier (EC). The SC classifies the input into conversation or action, and EC extracts the entities for the action. The pilot study has shown promising results with high perceived naturalness and positive experiences.

KW - Computing methodologies

KW - Human-centered computing

KW - Interaction design process and methods

KW - Natural language processing

KW - Virtual reality

UR - http://www.scopus.com/inward/record.url?scp=85105966792&partnerID=8YFLogxK

U2 - 10.1109/VRW52623.2021.00178

DO - 10.1109/VRW52623.2021.00178

M3 - Conference contribution

AN - SCOPUS:85105966792

T3 - Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021

SP - 591

EP - 592

BT - Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021

Y2 - 27 March 2021 through 3 April 2021

ER -

Auto-generating virtual human behavior by understanding user contexts

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this