Real-Time Interaction for 3D Pixel Human in Virtual Environment

Haoke Deng, Qimeng Zhang, Hongyu Jin, Chang Hun Kim

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)

Abstract

Conducting realistic interactions while communicating efficiently in online conferences is highly desired but challenging. In this work, we propose a novel pixel-style virtual avatar for interacting with virtual objects in virtual conferences that can be generated in real-time. It consists of a 2D segmented head video stream for real-time facial expressions and a 3D point cloud body for realistic interactions, both of which are generated from RGB video input of a monocular webcam. We obtain a human-only video stream with a human matting method and generate the 3D avatar’s arms with a 3D pose estimation method, which improves the stereoscopic realism and sense of interaction of conference participants while interacting with virtual objects. Our approach fills the gap between 2D video conferences and 3D virtual avatars and combines the advantages of both. We evaluated our pixel-style avatar by conducting a user study; the result proved that the efficiency of our method is superior to other various existing avatar types.

Original languageEnglish
Article number966
JournalApplied Sciences (Switzerland)
Volume13
Issue number2
DOIs
Publication statusPublished - 2023 Jan

Bibliographical note

Funding Information:
This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. IITP2022-0-002880101002) and National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF2022R1A4A101886911).

Publisher Copyright:
© 2023 by the authors.

Keywords

  • pixel-style art
  • virtual avatar
  • virtual meeting

ASJC Scopus subject areas

  • General Materials Science
  • Instrumentation
  • General Engineering
  • Process Chemistry and Technology
  • Computer Science Applications
  • Fluid Flow and Transfer Processes

Fingerprint

Dive into the research topics of 'Real-Time Interaction for 3D Pixel Human in Virtual Environment'. Together they form a unique fingerprint.

Cite this