PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling

Ji Sang Hwang; Sang Hoon Lee; Seong Whan Lee

doi:10.1007/978-3-031-47634-1_31

PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling

Ji Sang Hwang
, Sang Hoon Lee
, Seong Whan Lee^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https://jisang93.github.io/pausespeech-demo/.

Original language	English
Title of host publication	Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings
Editors	Huimin Lu, Michael Blumenstein, Sung-Bae Cho, Cheng-Lin Liu, Yasushi Yagi, Tohru Kamiya
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	415-427
Number of pages	13
ISBN (Print)	9783031476334
DOIs	https://doi.org/10.1007/978-3-031-47634-1_31
Publication status	Published - 2023
Event	7th Asian Conference on Pattern Recognition, ACPR 2023 - Kitakyushu, Japan Duration: 2023 Nov 5 → 2023 Nov 8

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14406 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	7th Asian Conference on Pattern Recognition, ACPR 2023
Country/Territory	Japan
City	Kitakyushu
Period	23/11/5 → 23/11/8

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

Pause-based prosody modeling
Pre-trained language model
Text-to-speech

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-47634-1_31

Cite this

Hwang, J. S., Lee, S. H., & Lee, S. W. (2023). PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling. In H. Lu, M. Blumenstein, S.-B. Cho, C.-L. Liu, Y. Yagi, & T. Kamiya (Eds.), Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings (pp. 415-427). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14406 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-47634-1_31

PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling. / Hwang, Ji Sang; Lee, Sang Hoon; Lee, Seong Whan.
Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings. ed. / Huimin Lu; Michael Blumenstein; Sung-Bae Cho; Cheng-Lin Liu; Yasushi Yagi; Tohru Kamiya. Springer Science and Business Media Deutschland GmbH, 2023. p. 415-427 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14406 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Hwang, JS, Lee, SH & Lee, SW 2023, PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling. in H Lu, M Blumenstein, S-B Cho, C-L Liu, Y Yagi & T Kamiya (eds), Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14406 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 415-427, 7th Asian Conference on Pattern Recognition, ACPR 2023, Kitakyushu, Japan, 23/11/5. https://doi.org/10.1007/978-3-031-47634-1_31

Hwang JS, Lee SH, Lee SW. PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling. In Lu H, Blumenstein M, Cho SB, Liu CL, Yagi Y, Kamiya T, editors, Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 415-427. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-47634-1_31

Hwang, Ji Sang ; Lee, Sang Hoon ; Lee, Seong Whan. / PauseSpeech : Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling. Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings. editor / Huimin Lu ; Michael Blumenstein ; Sung-Bae Cho ; Cheng-Lin Liu ; Yasushi Yagi ; Tohru Kamiya. Springer Science and Business Media Deutschland GmbH, 2023. pp. 415-427 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{69924904c18243fbbef28029878a07c9,

title = "PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-Based Prosody Modeling",

abstract = "Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https://jisang93.github.io/pausespeech-demo/.",

keywords = "Pause-based prosody modeling, Pre-trained language model, Text-to-speech",

author = "Hwang, \{Ji Sang\} and Lee, \{Sang Hoon\} and Lee, \{Seong Whan\}",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 7th Asian Conference on Pattern Recognition, ACPR 2023 ; Conference date: 05-11-2023 Through 08-11-2023",

year = "2023",

doi = "10.1007/978-3-031-47634-1\_31",

language = "English",

isbn = "9783031476334",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "415--427",

editor = "Huimin Lu and Michael Blumenstein and Sung-Bae Cho and Cheng-Lin Liu and Yasushi Yagi and Tohru Kamiya",

booktitle = "Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings",

address = "Germany",

}

TY - GEN

T1 - PauseSpeech

T2 - 7th Asian Conference on Pattern Recognition, ACPR 2023

AU - Hwang, Ji Sang

AU - Lee, Sang Hoon

AU - Lee, Seong Whan

PY - 2023

Y1 - 2023

N2 - Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https://jisang93.github.io/pausespeech-demo/.

AB - Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https://jisang93.github.io/pausespeech-demo/.

KW - Pause-based prosody modeling

KW - Pre-trained language model

KW - Text-to-speech

UR - https://www.scopus.com/pages/publications/85177204888

U2 - 10.1007/978-3-031-47634-1_31

DO - 10.1007/978-3-031-47634-1_31

M3 - Conference contribution

AN - SCOPUS:85177204888

SN - 9783031476334

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 415

EP - 427

BT - Pattern Recognition - 7th Asian Conference, ACPR 2023, Proceedings

A2 - Lu, Huimin

A2 - Blumenstein, Michael

A2 - Cho, Sung-Bae

A2 - Liu, Cheng-Lin

A2 - Yagi, Yasushi

A2 - Kamiya, Tohru

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 5 November 2023 through 8 November 2023

ER -