Silhouettes from real objects enable realistic interactions with a virtual human in mobile augmented reality

Hanseob Kim, Ghazanfar Ali, Andréas Pastor, Myungho Lee, Gerard J. Kim, Jae In Hwang

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)


Realistic interactions with real objects (e.g., animals, toys, robots) in an augmented reality (AR) environment enhances the user experience. The common AR apps on the market achieve realistic interactions by superimposing pre-modeled virtual proxies on the real objects in the AR environment. This way user perceives the interaction with virtual proxies as interaction with real objects. However, catering to environment change, shape deformation, and view update is not a trivial task. Our proposed method uses the dynamic silhouette of a real object to enable realistic interactions. Our approach is practical, lightweight, and requires no additional hardware besides the device camera. For a case study, we designed a mobile AR application to interact with real animal dolls. Our scenario included a virtual human performing four types of realistic interactions. Results demonstrated our method’s stability that does not require pre-modeled virtual proxies in case of shape deformation and view update. We also conducted a pilot study using our approach and reported significant improvements in user perception of spatial awareness and presence for realistic interactions with a virtual human.

Original languageEnglish
Article number2763
JournalApplied Sciences (Switzerland)
Issue number6
Publication statusPublished - 2021 Mar 2

Bibliographical note

Funding Information:
Funding: This work was supported by the National Research Council of Science and Technology (NST) Grant by the Korean Government through the Ministry of Science and ICT (MSIT) (CRC-20-02-KIST).

Publisher Copyright:
© 2021 by the authors. Licensee MDPI, Basel, Switzerland.


  • Augmented reality
  • Deep learning
  • Mobile AR
  • Occlusion, multi-modal system
  • Perceptual issue
  • Realistic interaction
  • Segmentation
  • User experience
  • Virtual human

ASJC Scopus subject areas

  • Materials Science(all)
  • Instrumentation
  • Engineering(all)
  • Process Chemistry and Technology
  • Computer Science Applications
  • Fluid Flow and Transfer Processes


Dive into the research topics of 'Silhouettes from real objects enable realistic interactions with a virtual human in mobile augmented reality'. Together they form a unique fingerprint.

Cite this