Achieving real-time lip-synch via SVM-based phoneme classification and lip shape refinement

Taeyoon Kim; Yongsung Kang; Hanseok Ko

doi:10.1109/ICMI.2002.1167010

Achieving real-time lip-synch via SVM-based phoneme classification and lip shape refinement

Taeyoon Kim
, Yongsung Kang
, Hanseok Ko

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

In this paper, we develop a real time lip-synch system that activates a 2D avatar's lip motion in synch with incoming speech utterance. To realize "real time" operation of the system, we contain the processing time by invoking a merge and split procedure performing coarse-to-fine phoneme classification. At each stage of phoneme classification, we apply a support vector machine (SVM) to constrain the computational load while attaining desirable accuracy. Coarse-to-fine phoneme classification is accomplished via 2 stages of feature extraction, where each speech frame is acoustically analyzed first for 3 classes of lip opening using MFCC as the feature and then a further refined classification for detailed lip shape using formant information. We implemented the system with 2D lip animation that shows the effectiveness of the proposed 2-stage procedure accomplishing the real-time lip-synch task.

Original language	English
Title of host publication	Proceedings - 4th IEEE International Conference on Multimodal Interfaces, ICMI 2002
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	299-304
Number of pages	6
ISBN (Print)	0769518346, 9780769518343
DOIs	https://doi.org/10.1109/ICMI.2002.1167010
Publication status	Published - 2002
Event	4th IEEE International Conference on Multimodal Interfaces, ICMI 2002 - Pittsburgh, United States Duration: 2002 Oct 14 → 2002 Oct 16

Other

Other	4th IEEE International Conference on Multimodal Interfaces, ICMI 2002
Country/Territory	United States
City	Pittsburgh
Period	02/10/14 → 02/10/16

ASJC Scopus subject areas

Artificial Intelligence
Computer Graphics and Computer-Aided Design
Computer Vision and Pattern Recognition
Hardware and Architecture

Access to Document

10.1109/ICMI.2002.1167010

Cite this

Achieving real-time lip-synch via SVM-based phoneme classification and lip shape refinement. / Kim, Taeyoon; Kang, Yongsung; Ko, Hanseok.
Proceedings - 4th IEEE International Conference on Multimodal Interfaces, ICMI 2002. Institute of Electrical and Electronics Engineers Inc., 2002. p. 299-304 1167010.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kim, T, Kang, Y & Ko, H 2002, Achieving real-time lip-synch via SVM-based phoneme classification and lip shape refinement. in Proceedings - 4th IEEE International Conference on Multimodal Interfaces, ICMI 2002., 1167010, Institute of Electrical and Electronics Engineers Inc., pp. 299-304, 4th IEEE International Conference on Multimodal Interfaces, ICMI 2002, Pittsburgh, United States, 02/10/14. https://doi.org/10.1109/ICMI.2002.1167010

@inproceedings{d16576078a604d1e99cb3b0a703bf184,

title = "Achieving real-time lip-synch via SVM-based phoneme classification and lip shape refinement",

abstract = "In this paper, we develop a real time lip-synch system that activates a 2D avatar's lip motion in synch with incoming speech utterance. To realize {"}real time{"} operation of the system, we contain the processing time by invoking a merge and split procedure performing coarse-to-fine phoneme classification. At each stage of phoneme classification, we apply a support vector machine (SVM) to constrain the computational load while attaining desirable accuracy. Coarse-to-fine phoneme classification is accomplished via 2 stages of feature extraction, where each speech frame is acoustically analyzed first for 3 classes of lip opening using MFCC as the feature and then a further refined classification for detailed lip shape using formant information. We implemented the system with 2D lip animation that shows the effectiveness of the proposed 2-stage procedure accomplishing the real-time lip-synch task.",

author = "Taeyoon Kim and Yongsung Kang and Hanseok Ko",

year = "2002",

doi = "10.1109/ICMI.2002.1167010",

language = "English",

isbn = "0769518346",

pages = "299--304",

booktitle = "Proceedings - 4th IEEE International Conference on Multimodal Interfaces, ICMI 2002",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

note = "4th IEEE International Conference on Multimodal Interfaces, ICMI 2002 ; Conference date: 14-10-2002 Through 16-10-2002",

}

TY - GEN

T1 - Achieving real-time lip-synch via SVM-based phoneme classification and lip shape refinement

AU - Kim, Taeyoon

AU - Kang, Yongsung

AU - Ko, Hanseok

PY - 2002

Y1 - 2002

N2 - In this paper, we develop a real time lip-synch system that activates a 2D avatar's lip motion in synch with incoming speech utterance. To realize "real time" operation of the system, we contain the processing time by invoking a merge and split procedure performing coarse-to-fine phoneme classification. At each stage of phoneme classification, we apply a support vector machine (SVM) to constrain the computational load while attaining desirable accuracy. Coarse-to-fine phoneme classification is accomplished via 2 stages of feature extraction, where each speech frame is acoustically analyzed first for 3 classes of lip opening using MFCC as the feature and then a further refined classification for detailed lip shape using formant information. We implemented the system with 2D lip animation that shows the effectiveness of the proposed 2-stage procedure accomplishing the real-time lip-synch task.

AB - In this paper, we develop a real time lip-synch system that activates a 2D avatar's lip motion in synch with incoming speech utterance. To realize "real time" operation of the system, we contain the processing time by invoking a merge and split procedure performing coarse-to-fine phoneme classification. At each stage of phoneme classification, we apply a support vector machine (SVM) to constrain the computational load while attaining desirable accuracy. Coarse-to-fine phoneme classification is accomplished via 2 stages of feature extraction, where each speech frame is acoustically analyzed first for 3 classes of lip opening using MFCC as the feature and then a further refined classification for detailed lip shape using formant information. We implemented the system with 2D lip animation that shows the effectiveness of the proposed 2-stage procedure accomplishing the real-time lip-synch task.

UR - https://www.scopus.com/pages/publications/84963799348

UR - https://www.scopus.com/pages/publications/84963799348#tab=citedBy

U2 - 10.1109/ICMI.2002.1167010

DO - 10.1109/ICMI.2002.1167010

M3 - Conference contribution

AN - SCOPUS:84963799348

SN - 0769518346

SN - 9780769518343

SP - 299

EP - 304

BT - Proceedings - 4th IEEE International Conference on Multimodal Interfaces, ICMI 2002

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 4th IEEE International Conference on Multimodal Interfaces, ICMI 2002

Y2 - 14 October 2002 through 16 October 2002

ER -

Achieving real-time lip-synch via SVM-based phoneme classification and lip shape refinement

Abstract

Other

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this