Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses

Seongwook Yoon; Hee Jeong Lim; Jae Hyun Kim; Hong Seok Lee; Yun Tae Kim; Sanghoon Sull

doi:10.1109/ICCE53296.2022.9730327

Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses

Seongwook Yoon, Hee Jeong Lim, Jae Hyun Kim, Hong Seok Lee, Yun Tae Kim, Sanghoon Sull

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

5 Citations (Scopus)

Abstract

Computationally expensive rendering of virtual 3D objects in mixed reality applications of lightweight AR glasses can be performed by a remotely connected external server. However, nonnegligible 6DOF pose error caused by the remote rendering latency results in 3D visual inconsistency which can be hardly removed by 2D image correction using IMU. In this paper, we propose a novel 6DOF pose prediction algorithm based on learnable combination of consistent motion model and deep prediction. We formulate the combination of both as controlled residual learning and model ensemble. We build a dataset and demonstrate that our algorithm provides accurate prediction under 200ms.

Original language	English
Title of host publication	2022 IEEE International Conference on Consumer Electronics, ICCE 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665441544
DOIs	https://doi.org/10.1109/ICCE53296.2022.9730327
Publication status	Published - 2022
Event	2022 IEEE International Conference on Consumer Electronics, ICCE 2022 - Virtual, Online, United States Duration: 2022 Jan 7 → 2022 Jan 9

Publication series

Name	Digest of Technical Papers - IEEE International Conference on Consumer Electronics
Volume	2022-January
ISSN (Print)	0747-668X

Conference

Conference	2022 IEEE International Conference on Consumer Electronics, ICCE 2022
Country/Territory	United States
City	Virtual, Online
Period	22/1/7 → 22/1/9

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

Keywords

6-DOF pose prediction
AR glasses
augmented reality
deep learning
mixed reality

ASJC Scopus subject areas

Industrial and Manufacturing Engineering
Electrical and Electronic Engineering

Access to Document

10.1109/ICCE53296.2022.9730327

Cite this

Yoon, S., Lim, H. J., Kim, J. H., Lee, H. S., Kim, Y. T., & Sull, S. (2022). Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses. In 2022 IEEE International Conference on Consumer Electronics, ICCE 2022 (Digest of Technical Papers - IEEE International Conference on Consumer Electronics; Vol. 2022-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCE53296.2022.9730327

Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses. / Yoon, Seongwook; Lim, Hee Jeong; Kim, Jae Hyun et al.
2022 IEEE International Conference on Consumer Electronics, ICCE 2022. Institute of Electrical and Electronics Engineers Inc., 2022. (Digest of Technical Papers - IEEE International Conference on Consumer Electronics; Vol. 2022-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yoon, S, Lim, HJ, Kim, JH, Lee, HS, Kim, YT & Sull, S 2022, Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses. in 2022 IEEE International Conference on Consumer Electronics, ICCE 2022. Digest of Technical Papers - IEEE International Conference on Consumer Electronics, vol. 2022-January, Institute of Electrical and Electronics Engineers Inc., 2022 IEEE International Conference on Consumer Electronics, ICCE 2022, Virtual, Online, United States, 22/1/7. https://doi.org/10.1109/ICCE53296.2022.9730327

Yoon S, Lim HJ, Kim JH, Lee HS, Kim YT, Sull S. Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses. In 2022 IEEE International Conference on Consumer Electronics, ICCE 2022. Institute of Electrical and Electronics Engineers Inc. 2022. (Digest of Technical Papers - IEEE International Conference on Consumer Electronics). doi: 10.1109/ICCE53296.2022.9730327

@inproceedings{591d15aa2508468cacb9986784c7d9e2,

title = "Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses",

abstract = "Computationally expensive rendering of virtual 3D objects in mixed reality applications of lightweight AR glasses can be performed by a remotely connected external server. However, nonnegligible 6DOF pose error caused by the remote rendering latency results in 3D visual inconsistency which can be hardly removed by 2D image correction using IMU. In this paper, we propose a novel 6DOF pose prediction algorithm based on learnable combination of consistent motion model and deep prediction. We formulate the combination of both as controlled residual learning and model ensemble. We build a dataset and demonstrate that our algorithm provides accurate prediction under 200ms.",

keywords = "6-DOF pose prediction, AR glasses, augmented reality, deep learning, mixed reality",

author = "Seongwook Yoon and Lim, \{Hee Jeong\} and Kim, \{Jae Hyun\} and Lee, \{Hong Seok\} and Kim, \{Yun Tae\} and Sanghoon Sull",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE International Conference on Consumer Electronics, ICCE 2022 ; Conference date: 07-01-2022 Through 09-01-2022",

year = "2022",

doi = "10.1109/ICCE53296.2022.9730327",

language = "English",

series = "Digest of Technical Papers - IEEE International Conference on Consumer Electronics",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2022 IEEE International Conference on Consumer Electronics, ICCE 2022",

}

TY - GEN

T1 - Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses

AU - Yoon, Seongwook

AU - Lim, Hee Jeong

AU - Kim, Jae Hyun

AU - Lee, Hong Seok

AU - Kim, Yun Tae

AU - Sull, Sanghoon

PY - 2022

Y1 - 2022

N2 - Computationally expensive rendering of virtual 3D objects in mixed reality applications of lightweight AR glasses can be performed by a remotely connected external server. However, nonnegligible 6DOF pose error caused by the remote rendering latency results in 3D visual inconsistency which can be hardly removed by 2D image correction using IMU. In this paper, we propose a novel 6DOF pose prediction algorithm based on learnable combination of consistent motion model and deep prediction. We formulate the combination of both as controlled residual learning and model ensemble. We build a dataset and demonstrate that our algorithm provides accurate prediction under 200ms.

AB - Computationally expensive rendering of virtual 3D objects in mixed reality applications of lightweight AR glasses can be performed by a remotely connected external server. However, nonnegligible 6DOF pose error caused by the remote rendering latency results in 3D visual inconsistency which can be hardly removed by 2D image correction using IMU. In this paper, we propose a novel 6DOF pose prediction algorithm based on learnable combination of consistent motion model and deep prediction. We formulate the combination of both as controlled residual learning and model ensemble. We build a dataset and demonstrate that our algorithm provides accurate prediction under 200ms.

KW - 6-DOF pose prediction

KW - AR glasses

KW - augmented reality

KW - deep learning

KW - mixed reality

UR - http://www.scopus.com/inward/record.url?scp=85127023840&partnerID=8YFLogxK

U2 - 10.1109/ICCE53296.2022.9730327

DO - 10.1109/ICCE53296.2022.9730327

M3 - Conference contribution

AN - SCOPUS:85127023840

T3 - Digest of Technical Papers - IEEE International Conference on Consumer Electronics

BT - 2022 IEEE International Conference on Consumer Electronics, ICCE 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2022 IEEE International Conference on Consumer Electronics, ICCE 2022

Y2 - 7 January 2022 through 9 January 2022

ER -

Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this