3-D virtual studio for natural inter-"acting"

Namgyu Kim; Woontack Woo; Gerard J. Kim; Chan Mo Park

doi:10.1109/TSMCA.2005.855752

3-D virtual studio for natural inter-"acting"

Namgyu Kim, Woontack Woo, Gerard J. Kim, Chan Mo Park

Research output: Contribution to journal › Article › peer-review

12 Citations (Scopus)

Abstract

Virtual studios have long been used in commercial broadcasting. However, most virtual studios are based on "blue screen" technology, and its two-dimensional (2-D) nature restricts the user from making natural three-dimensional (3-D) interactions. Actors have to follow prewritten scripts and pretend as if directly interacting with the synthetic objects. This often creates an unnatural and seemingly uncoordinated output. In this paper, we introduce an improved virtual-studio framework to enable actors/users to interact in 3-D more naturally with the synthetic environment and objects. The proposed system uses a stereo camera to first construct a 3-D environment (for the actor to act in), a multiview camera to extract the image and 3-D information about the actor, and a real-time registration and rendering software for generating the final output. Synthetic 3-D objects can be easily inserted and rendered, in real time, together with the 3-D environment and video actor for natural 3-D interaction. The enabling of natural 3-D interaction would make more cinematic techniques possible including live and spontaneous acting. The proposed system is not limited to broadcast production, but can also be used for creating virtual/augmented-reality environments for training and entertainment.

Original language	English
Pages (from-to)	758-773
Number of pages	16
Journal	IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans
Volume	36
Issue number	4
DOIs	https://doi.org/10.1109/TSMCA.2005.855752
Publication status	Published - 2006 Jul
Externally published	Yes

Bibliographical note

Funding Information:
Manuscript received February 10, 2004; revised July 27, 2004, November 14, 2004, and December 23, 2004. The work described in this paper was supported in part and carried out at Advanced Telecommunications Research (ATR) Media Integration and Communications (MIC) Laboratory, Japan, from 2000 to 2001, and also by the Korean Ministry of Education’s Brain Korea (BK) 21 program and the Korean Ministry of the Information and Communication’s Information Technology Research Center (ITRC) program, and continued at the Pohang University of Science and Technology (POSTECH) Virtual Reality (VR) Laboratory. This paper was recommended by Associate Editor I. Gu.

Keywords

Augmented reality
Calibration
Image segmentation
Multiview and stereo cameras
Registration
Three-dimensional (3-D) interaction
Three-dimensional (3-D) model reconstruction
Virtual environment (VE)
Virtual studio
Z-keying

ASJC Scopus subject areas

Software
Control and Systems Engineering
Human-Computer Interaction
Computer Science Applications
Electrical and Electronic Engineering

Access to Document

10.1109/TSMCA.2005.855752

Cite this

@article{5c0989d042b34c988fd06f99f3d44648,

title = "3-D virtual studio for natural inter-{"}acting{"}",

abstract = "Virtual studios have long been used in commercial broadcasting. However, most virtual studios are based on {"}blue screen{"} technology, and its two-dimensional (2-D) nature restricts the user from making natural three-dimensional (3-D) interactions. Actors have to follow prewritten scripts and pretend as if directly interacting with the synthetic objects. This often creates an unnatural and seemingly uncoordinated output. In this paper, we introduce an improved virtual-studio framework to enable actors/users to interact in 3-D more naturally with the synthetic environment and objects. The proposed system uses a stereo camera to first construct a 3-D environment (for the actor to act in), a multiview camera to extract the image and 3-D information about the actor, and a real-time registration and rendering software for generating the final output. Synthetic 3-D objects can be easily inserted and rendered, in real time, together with the 3-D environment and video actor for natural 3-D interaction. The enabling of natural 3-D interaction would make more cinematic techniques possible including live and spontaneous acting. The proposed system is not limited to broadcast production, but can also be used for creating virtual/augmented-reality environments for training and entertainment.",

keywords = "Augmented reality, Calibration, Image segmentation, Multiview and stereo cameras, Registration, Three-dimensional (3-D) interaction, Three-dimensional (3-D) model reconstruction, Virtual environment (VE), Virtual studio, Z-keying",

author = "Namgyu Kim and Woontack Woo and Kim, {Gerard J.} and Park, {Chan Mo}",

note = "Funding Information: Manuscript received February 10, 2004; revised July 27, 2004, November 14, 2004, and December 23, 2004. The work described in this paper was supported in part and carried out at Advanced Telecommunications Research (ATR) Media Integration and Communications (MIC) Laboratory, Japan, from 2000 to 2001, and also by the Korean Ministry of Education{\textquoteright}s Brain Korea (BK) 21 program and the Korean Ministry of the Information and Communication{\textquoteright}s Information Technology Research Center (ITRC) program, and continued at the Pohang University of Science and Technology (POSTECH) Virtual Reality (VR) Laboratory. This paper was recommended by Associate Editor I. Gu.",

year = "2006",

month = jul,

doi = "10.1109/TSMCA.2005.855752",

language = "English",

volume = "36",

pages = "758--773",

journal = "IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans",

issn = "1083-4427",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "4",

}

TY - JOUR

T1 - 3-D virtual studio for natural inter-"acting"

AU - Kim, Namgyu

AU - Woo, Woontack

AU - Kim, Gerard J.

AU - Park, Chan Mo

N1 - Funding Information: Manuscript received February 10, 2004; revised July 27, 2004, November 14, 2004, and December 23, 2004. The work described in this paper was supported in part and carried out at Advanced Telecommunications Research (ATR) Media Integration and Communications (MIC) Laboratory, Japan, from 2000 to 2001, and also by the Korean Ministry of Education’s Brain Korea (BK) 21 program and the Korean Ministry of the Information and Communication’s Information Technology Research Center (ITRC) program, and continued at the Pohang University of Science and Technology (POSTECH) Virtual Reality (VR) Laboratory. This paper was recommended by Associate Editor I. Gu.

PY - 2006/7

Y1 - 2006/7

N2 - Virtual studios have long been used in commercial broadcasting. However, most virtual studios are based on "blue screen" technology, and its two-dimensional (2-D) nature restricts the user from making natural three-dimensional (3-D) interactions. Actors have to follow prewritten scripts and pretend as if directly interacting with the synthetic objects. This often creates an unnatural and seemingly uncoordinated output. In this paper, we introduce an improved virtual-studio framework to enable actors/users to interact in 3-D more naturally with the synthetic environment and objects. The proposed system uses a stereo camera to first construct a 3-D environment (for the actor to act in), a multiview camera to extract the image and 3-D information about the actor, and a real-time registration and rendering software for generating the final output. Synthetic 3-D objects can be easily inserted and rendered, in real time, together with the 3-D environment and video actor for natural 3-D interaction. The enabling of natural 3-D interaction would make more cinematic techniques possible including live and spontaneous acting. The proposed system is not limited to broadcast production, but can also be used for creating virtual/augmented-reality environments for training and entertainment.

AB - Virtual studios have long been used in commercial broadcasting. However, most virtual studios are based on "blue screen" technology, and its two-dimensional (2-D) nature restricts the user from making natural three-dimensional (3-D) interactions. Actors have to follow prewritten scripts and pretend as if directly interacting with the synthetic objects. This often creates an unnatural and seemingly uncoordinated output. In this paper, we introduce an improved virtual-studio framework to enable actors/users to interact in 3-D more naturally with the synthetic environment and objects. The proposed system uses a stereo camera to first construct a 3-D environment (for the actor to act in), a multiview camera to extract the image and 3-D information about the actor, and a real-time registration and rendering software for generating the final output. Synthetic 3-D objects can be easily inserted and rendered, in real time, together with the 3-D environment and video actor for natural 3-D interaction. The enabling of natural 3-D interaction would make more cinematic techniques possible including live and spontaneous acting. The proposed system is not limited to broadcast production, but can also be used for creating virtual/augmented-reality environments for training and entertainment.

KW - Augmented reality

KW - Calibration

KW - Image segmentation

KW - Multiview and stereo cameras

KW - Registration

KW - Three-dimensional (3-D) interaction

KW - Three-dimensional (3-D) model reconstruction

KW - Virtual environment (VE)

KW - Virtual studio

KW - Z-keying

UR - http://www.scopus.com/inward/record.url?scp=33845439774&partnerID=8YFLogxK

U2 - 10.1109/TSMCA.2005.855752

DO - 10.1109/TSMCA.2005.855752

M3 - Article

AN - SCOPUS:33845439774

SN - 1083-4427

VL - 36

SP - 758

EP - 773

JO - IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans

JF - IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans

IS - 4

ER -

3-D virtual studio for natural inter-"acting"

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this