The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

Yujin Jeong; Wonjeong Ryoo; Seunghyun Lee; Dabin Seo; Wonmin Byeon; Sangpil Kim; Jinkyu Kim

doi:10.1109/ICCV51070.2023.00719

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Seo, Wonmin Byeon, Sangpil Kim, Jinkyu Kim

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/

Original language	English
Title of host publication	Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	7788-7798
Number of pages	11
ISBN (Electronic)	9798350307184
DOIs	https://doi.org/10.1109/ICCV51070.2023.00719
Publication status	Published - 2023
Event	2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Paris, France Duration: 2023 Oct 2 → 2023 Oct 6

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
ISSN (Print)	1550-5499

Conference

Conference	2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
Country/Territory	France
City	Paris
Period	23/10/2 → 23/10/6

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/ICCV51070.2023.00719

Cite this

Jeong, Y., Ryoo, W., Lee, S., Seo, D., Byeon, W., Kim, S., & Kim, J. (2023). The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion. In Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 (pp. 7788-7798). (Proceedings of the IEEE International Conference on Computer Vision). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV51070.2023.00719

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion. / Jeong, Yujin; Ryoo, Wonjeong; Lee, Seunghyun et al.
Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 7788-7798 (Proceedings of the IEEE International Conference on Computer Vision).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Jeong, Y, Ryoo, W, Lee, S, Seo, D, Byeon, W, Kim, S & Kim, J 2023, The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion. in Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023. Proceedings of the IEEE International Conference on Computer Vision, Institute of Electrical and Electronics Engineers Inc., pp. 7788-7798, 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, 23/10/2. https://doi.org/10.1109/ICCV51070.2023.00719

Jeong Y, Ryoo W, Lee S, Seo D, Byeon W, Kim S et al. The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion. In Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 7788-7798. (Proceedings of the IEEE International Conference on Computer Vision). doi: 10.1109/ICCV51070.2023.00719

@inproceedings{ab8cf26c39e94c1499d711feb9130c49,

title = "The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion",

abstract = "In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/",

author = "Yujin Jeong and Wonjeong Ryoo and Seunghyun Lee and Dabin Seo and Wonmin Byeon and Sangpil Kim and Jinkyu Kim",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 ; Conference date: 02-10-2023 Through 06-10-2023",

year = "2023",

doi = "10.1109/ICCV51070.2023.00719",

language = "English",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "7788--7798",

booktitle = "Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023",

}

TY - GEN

T1 - The Power of Sound (TPoS)

T2 - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023

AU - Jeong, Yujin

AU - Ryoo, Wonjeong

AU - Lee, Seunghyun

AU - Seo, Dabin

AU - Byeon, Wonmin

AU - Kim, Sangpil

AU - Kim, Jinkyu

PY - 2023

Y1 - 2023

N2 - In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/

AB - In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/

UR - http://www.scopus.com/inward/record.url?scp=85180263122&partnerID=8YFLogxK

U2 - 10.1109/ICCV51070.2023.00719

DO - 10.1109/ICCV51070.2023.00719

M3 - Conference contribution

AN - SCOPUS:85180263122

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 7788

EP - 7798

BT - Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 2 October 2023 through 6 October 2023

ER -

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this