3D HUMAN MOTION GENERATION FROM THE TEXT VIA GESTURE ACTION CLASSIFICATION AND THE AUTOREGRESSIVE MODEL

Gwantae Kim, Youngsuk Ryu, Junyeop Lee, David K. Han, Jeongmin Bae, Hanseok Ko

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

In this paper, a deep learning-based model for 3D human motion generation from the text is proposed via gesture action classification and an autoregressive model. The model focuses on generating special gestures that express human thinking, such as waving and nodding. To achieve the goal, the proposed method predicts expression from the sentences using a text classification model based on a pretrained language model and generates gestures using the gate recurrent unit-based autoregressive model. Especially, we proposed the loss for the embedding space for restoring raw motions and generating intermediate motions well. Moreover, the novel data augmentation method and stop token are proposed to generate variable length motions. To evaluate the text classification model and 3D human motion generation model, a gesture action classification dataset and action-based gesture dataset are collected. With several experiments, the proposed method successfully generates perceptually natural and realistic 3D human motion from the text. Moreover, we verified the effectiveness of the proposed method using a public-available action recognition dataset to evaluate cross-dataset generalization performance.

Original languageEnglish
Title of host publication2022 IEEE International Conference on Image Processing, ICIP 2022 - Proceedings
PublisherIEEE Computer Society
Pages1036-1040
Number of pages5
ISBN (Electronic)9781665496209
DOIs
Publication statusPublished - 2022
Event29th IEEE International Conference on Image Processing, ICIP 2022 - Bordeaux, France
Duration: 2022 Oct 162022 Oct 19

Publication series

NameProceedings - International Conference on Image Processing, ICIP
ISSN (Print)1522-4880

Conference

Conference29th IEEE International Conference on Image Processing, ICIP 2022
Country/TerritoryFrance
CityBordeaux
Period22/10/1622/10/19

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

Keywords

  • autoregressive model
  • gesture action classification
  • gesture generation
  • pretrained language model
  • recurrent neural networks

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Signal Processing

Fingerprint

Dive into the research topics of '3D HUMAN MOTION GENERATION FROM THE TEXT VIA GESTURE ACTION CLASSIFICATION AND THE AUTOREGRESSIVE MODEL'. Together they form a unique fingerprint.

Cite this