Real-time and on-line removal of moving human figures in hand-held mobile augmented reality

Taehyung Kim, Gerard J. Kim

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)


In this paper, we present a real time on-line augmented/diminished reality system that runs entirely on the hand-held moving mobile device. Specifically, we introduce an improved inpainting algorithm that is designed for the on-line usage (i.e., live streaming video) with the moving view point. Unlike other previous approaches, the proposed algorithm produces a reasonably high-quality inpainting imagery by taking advantage of the 3D camera tracking and scene information from the SLAM in selecting the proper source frame from which to extract the parts to replace the masked region, and robustly applying homography to fill it in from the source. The algorithm is evaluated and compared with the state-of-the-art video inpainting methods in terms of the execution time, and objective and subjective inpainted image quality. Although the algorithm is applicable to any dynamic objects, the current implementation is limited to removing and filling in for human figures only. The evaluation results have shown the quality of the inpainted image quality was on par with those by the off-line state-of-the-art systems and yet ran at an interactive rate on a mobile device. A user study was also conducted to assess the user perception and experience in an outdoor interactive AR application using the proposed algorithm to remove the interfering pedestrians. The algorithm significantly reduced the level of distraction and improved the AR user experience by lowering the visual inconsistency and artifacts, when compared to other nominal test conditions.

Original languageEnglish
Pages (from-to)2571-2582
Number of pages12
JournalVisual Computer
Issue number7
Publication statusPublished - 2023 Jul

Bibliographical note

Publisher Copyright:
© 2022, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.


  • Diminished reality
  • Dynamic object removal
  • Handheld augmented reality
  • Perceptual issue

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Real-time and on-line removal of moving human figures in hand-held mobile augmented reality'. Together they form a unique fingerprint.

Cite this