The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation

Gwantae Kim; Yuanming Li; Hanseok Ko

doi:10.1145/3610661.3616551

The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation

Gwantae Kim, Yuanming Li, Hanseok Ko

School of Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

This paper describes a diffusion model for co-speech gesture generation presented by KU-ISPL entry of the GENEA Challenge 2023. We formulate the gesture generation problem as a co-speech gesture generation problem and a semantic gesture generation problem, and we focus on solving the co-speech gesture generation problem by denoising diffusion probabilistic model with text, audio, and pre-pose conditions. We use the U-Net with cross-attention architecture as a denoising model, and we propose a gesture autoencoder as a mapping function from the gesture domain to the latent domain. The collective evaluation released by GENEA Challenge 2023 shows that our model successfully generates co-speech gestures. Our system receives a mean human-likeness score of 32.0, a preference-matched score of appropriateness for the main agent speech of 53.6%, and an interlocutor speech appropriateness score of 53.5%. We also conduct an ablation study to measure the effects of the pre-pose. By the results, our system contributes to the co-speech gesture generation for natural interaction.

Original language	English
Title of host publication	ICMI 2023 Companion - Companion Publication of the 25th International Conference on Multimodal Interaction
Publisher	Association for Computing Machinery
Pages	220-227
Number of pages	8
ISBN (Electronic)	9798400703218
DOIs	https://doi.org/10.1145/3610661.3616551
Publication status	Published - 2023 Oct 9
Event	25th International Conference on Multimodal Interaction, ICMI 2023 Companion - Paris, France Duration: 2023 Oct 9 → 2023 Oct 13

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	25th International Conference on Multimodal Interaction, ICMI 2023 Companion
Country/Territory	France
City	Paris
Period	23/10/9 → 23/10/13

Bibliographical note

Publisher Copyright:
© 2023 ACM.

Keywords

GENEA Challenge
co-speech gesture generation
diffusion
generative models
neural networks

ASJC Scopus subject areas

Human-Computer Interaction
Computer Networks and Communications
Computer Vision and Pattern Recognition
Software

Access to Document

10.1145/3610661.3616551

Cite this

Kim, G., Li, Y., & Ko, H. (2023). The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation. In ICMI 2023 Companion - Companion Publication of the 25th International Conference on Multimodal Interaction (pp. 220-227). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3610661.3616551

The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation. / Kim, Gwantae; Li, Yuanming; Ko, Hanseok.
ICMI 2023 Companion - Companion Publication of the 25th International Conference on Multimodal Interaction. Association for Computing Machinery, 2023. p. 220-227 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kim, G, Li, Y & Ko, H 2023, The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation. in ICMI 2023 Companion - Companion Publication of the 25th International Conference on Multimodal Interaction. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 220-227, 25th International Conference on Multimodal Interaction, ICMI 2023 Companion, Paris, France, 23/10/9. https://doi.org/10.1145/3610661.3616551

@inproceedings{5e62f39fcea54607bc33cb25c4161084,

title = "The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation",

abstract = "This paper describes a diffusion model for co-speech gesture generation presented by KU-ISPL entry of the GENEA Challenge 2023. We formulate the gesture generation problem as a co-speech gesture generation problem and a semantic gesture generation problem, and we focus on solving the co-speech gesture generation problem by denoising diffusion probabilistic model with text, audio, and pre-pose conditions. We use the U-Net with cross-attention architecture as a denoising model, and we propose a gesture autoencoder as a mapping function from the gesture domain to the latent domain. The collective evaluation released by GENEA Challenge 2023 shows that our model successfully generates co-speech gestures. Our system receives a mean human-likeness score of 32.0, a preference-matched score of appropriateness for the main agent speech of 53.6%, and an interlocutor speech appropriateness score of 53.5%. We also conduct an ablation study to measure the effects of the pre-pose. By the results, our system contributes to the co-speech gesture generation for natural interaction.",

keywords = "GENEA Challenge, co-speech gesture generation, diffusion, generative models, neural networks",

author = "Gwantae Kim and Yuanming Li and Hanseok Ko",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 25th International Conference on Multimodal Interaction, ICMI 2023 Companion ; Conference date: 09-10-2023 Through 13-10-2023",

year = "2023",

month = oct,

day = "9",

doi = "10.1145/3610661.3616551",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "220--227",

booktitle = "ICMI 2023 Companion - Companion Publication of the 25th International Conference on Multimodal Interaction",

}

TY - GEN

T1 - The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation

AU - Kim, Gwantae

AU - Li, Yuanming

AU - Ko, Hanseok

PY - 2023/10/9

Y1 - 2023/10/9

N2 - This paper describes a diffusion model for co-speech gesture generation presented by KU-ISPL entry of the GENEA Challenge 2023. We formulate the gesture generation problem as a co-speech gesture generation problem and a semantic gesture generation problem, and we focus on solving the co-speech gesture generation problem by denoising diffusion probabilistic model with text, audio, and pre-pose conditions. We use the U-Net with cross-attention architecture as a denoising model, and we propose a gesture autoencoder as a mapping function from the gesture domain to the latent domain. The collective evaluation released by GENEA Challenge 2023 shows that our model successfully generates co-speech gestures. Our system receives a mean human-likeness score of 32.0, a preference-matched score of appropriateness for the main agent speech of 53.6%, and an interlocutor speech appropriateness score of 53.5%. We also conduct an ablation study to measure the effects of the pre-pose. By the results, our system contributes to the co-speech gesture generation for natural interaction.

AB - This paper describes a diffusion model for co-speech gesture generation presented by KU-ISPL entry of the GENEA Challenge 2023. We formulate the gesture generation problem as a co-speech gesture generation problem and a semantic gesture generation problem, and we focus on solving the co-speech gesture generation problem by denoising diffusion probabilistic model with text, audio, and pre-pose conditions. We use the U-Net with cross-attention architecture as a denoising model, and we propose a gesture autoencoder as a mapping function from the gesture domain to the latent domain. The collective evaluation released by GENEA Challenge 2023 shows that our model successfully generates co-speech gestures. Our system receives a mean human-likeness score of 32.0, a preference-matched score of appropriateness for the main agent speech of 53.6%, and an interlocutor speech appropriateness score of 53.5%. We also conduct an ablation study to measure the effects of the pre-pose. By the results, our system contributes to the co-speech gesture generation for natural interaction.

KW - GENEA Challenge

KW - co-speech gesture generation

KW - diffusion

KW - generative models

KW - neural networks

UR - http://www.scopus.com/inward/record.url?scp=85170497813&partnerID=8YFLogxK

U2 - 10.1145/3610661.3616551

DO - 10.1145/3610661.3616551

M3 - Conference contribution

AN - SCOPUS:85170497813

T3 - ACM International Conference Proceeding Series

SP - 220

EP - 227

BT - ICMI 2023 Companion - Companion Publication of the 25th International Conference on Multimodal Interaction

PB - Association for Computing Machinery

T2 - 25th International Conference on Multimodal Interaction, ICMI 2023 Companion

Y2 - 9 October 2023 through 13 October 2023

ER -

The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this