VoiceMixer: Adversarial Voice Style Mixup

Sang Hoon Lee; Ji Hoon Kim; Hyunseung Chung; Seong Whan Lee

VoiceMixer: Adversarial Voice Style Mixup

Sang Hoon Lee
, Ji Hoon Kim
, Hyunseung Chung
, Seong Whan Lee^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

34 Citations (Scopus)

Abstract

Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. A key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads to the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through a novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only a small loss of content information. Also, for adversarial feedback of each information, the discriminator is decomposed into content and style discriminator with self-supervision, which enable our model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of our model in disentanglement and transfer performance, and improve audio quality by preserving content information.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Publisher	Neural information processing systems foundation
Pages	294-308
Number of pages	15
ISBN (Electronic)	9781713845393
Publication status	Published - 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: 2021 Dec 6 → 2021 Dec 14

Publication series

Name	Advances in Neural Information Processing Systems
Volume	1
ISSN (Print)	1049-5258

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	21/12/6 → 21/12/14

Bibliographical note

Funding Information:
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University) and No. 2021-0-02068, Artificial Intelligence Innovation Hub) and Microsoft Research Asia (MSRA).

Publisher Copyright:
© 2021 Neural information processing systems foundation. All rights reserved.

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

Lee, S. H., Kim, J. H., Chung, H., & Lee, S. W. (2021). VoiceMixer: Adversarial Voice Style Mixup. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 294-308). (Advances in Neural Information Processing Systems; Vol. 1). Neural information processing systems foundation.

VoiceMixer: Adversarial Voice Style Mixup. / Lee, Sang Hoon; Kim, Ji Hoon; Chung, Hyunseung et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. p. 294-308 (Advances in Neural Information Processing Systems; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, SH, Kim, JH, Chung, H & Lee, SW 2021, VoiceMixer: Adversarial Voice Style Mixup. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 1, Neural information processing systems foundation, pp. 294-308, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 21/12/6.

Lee SH, Kim JH, Chung H, Lee SW. VoiceMixer: Adversarial Voice Style Mixup. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Neural information processing systems foundation. 2021. p. 294-308. (Advances in Neural Information Processing Systems).

Lee, Sang Hoon ; Kim, Ji Hoon ; Chung, Hyunseung et al. / VoiceMixer : Adversarial Voice Style Mixup. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. pp. 294-308 (Advances in Neural Information Processing Systems).

@inproceedings{e44195022f314d9cb907866c14f296d4,

title = "VoiceMixer: Adversarial Voice Style Mixup",

abstract = "Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. A key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads to the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through a novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only a small loss of content information. Also, for adversarial feedback of each information, the discriminator is decomposed into content and style discriminator with self-supervision, which enable our model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of our model in disentanglement and transfer performance, and improve audio quality by preserving content information.",

author = "Lee, \{Sang Hoon\} and Kim, \{Ji Hoon\} and Hyunseung Chung and Lee, \{Seong Whan\}",

note = "Funding Information: This work was partly supported by Institute of Information \& communications Technology Planning \& Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University) and No. 2021-0-02068, Artificial Intelligence Innovation Hub) and Microsoft Research Asia (MSRA). Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

pages = "294--308",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, \{Percy S.\} and \{Wortman Vaughan\}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

TY - GEN

T1 - VoiceMixer

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

AU - Lee, Sang Hoon

AU - Kim, Ji Hoon

AU - Chung, Hyunseung

AU - Lee, Seong Whan

N1 - Funding Information: This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University) and No. 2021-0-02068, Artificial Intelligence Innovation Hub) and Microsoft Research Asia (MSRA). Publisher Copyright: © 2021 Neural information processing systems foundation. All rights reserved.

PY - 2021

Y1 - 2021

N2 - Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. A key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads to the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through a novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only a small loss of content information. Also, for adversarial feedback of each information, the discriminator is decomposed into content and style discriminator with self-supervision, which enable our model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of our model in disentanglement and transfer performance, and improve audio quality by preserving content information.

AB - Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. A key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads to the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through a novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only a small loss of content information. Also, for adversarial feedback of each information, the discriminator is decomposed into content and style discriminator with self-supervision, which enable our model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of our model in disentanglement and transfer performance, and improve audio quality by preserving content information.

UR - https://www.scopus.com/pages/publications/85131716896

M3 - Conference contribution

AN - SCOPUS:85131716896

T3 - Advances in Neural Information Processing Systems

SP - 294

EP - 308

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

PB - Neural information processing systems foundation

Y2 - 6 December 2021 through 14 December 2021

ER -

VoiceMixer: Adversarial Voice Style Mixup

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this