Oh My Mistake! Toward Realistic Dialogue State Tracking including Turnback Utterances

Takyoung Kim, Yukyung Lee, Hoonsang Yoon, Pilsung Kang, Junseong Bang, Misuk Kim

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The primary purpose of dialogue state tracking (DST), a critical component of an end-to-end conversational system, is to build a model that responds well to real-world situations. Although we often change our minds from time to time during ordinary conversations, current benchmark datasets do not adequately reflect such occurrences and instead consist of over-simplified conversations, in which no one changes their mind during a conversation. As the main question inspiring the present study, “Are current benchmark datasets sufficiently diverse to handle casual conversations in which one changes their mind after a certain topic is over?” We found that the answer is “No” because DST models cannot refer to previous user preferences when template-based turnback utterances are injected into the dataset. Even in the the simplest mind-changing (turnback) scenario, the performance of DST models significantly degenerated. However, we found that this performance degeneration can be recovered when the turnback scenarios are explicitly designed in the training set, implying that the problem is not with the DST models but rather with the construction of the benchmark dataset.

Original languageEnglish
Title of host publicationSereTOD 2022 - Towards Semi-Supervised and Reinforced Task-Oriented Dialog Systems, Proceedings of the Workshop
PublisherAssociation for Computational Linguistics (ACL)
Pages1-12
Number of pages12
ISBN (Electronic)9781959429210
Publication statusPublished - 2022
Event2022 Workshop on Towards Semi-Supervised and Reinforced Task-Oriented Dialog Systems, SereTOD 2022, co-located with EMNLP 2022 - Abu Dhabi, United Arab Emirates
Duration: 2022 Dec 7 → …

Publication series

NameSereTOD 2022 - Towards Semi-Supervised and Reinforced Task-Oriented Dialog Systems, Proceedings of the Workshop

Conference

Conference2022 Workshop on Towards Semi-Supervised and Reinforced Task-Oriented Dialog Systems, SereTOD 2022, co-located with EMNLP 2022
Country/TerritoryUnited Arab Emirates
CityAbu Dhabi
Period22/12/7 → …

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language

Fingerprint

Dive into the research topics of 'Oh My Mistake! Toward Realistic Dialogue State Tracking including Turnback Utterances'. Together they form a unique fingerprint.

Cite this