Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking

Takyoung Kim, Hoonsang Yoon, Yukyung Lee, Pilsung Kang, Misuk Kim

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    9 Citations (Scopus)

    Abstract

    Dialogue state tracking (DST) aims to extract essential information from multi-turn dialogue situations and take appropriate actions. A belief state, one of the core pieces of information, refers to the subject and its specific content, and appears in the form of domain-slot-value. The trained model predicts “accumulated” belief states in every turn, and joint goal accuracy and slot accuracy are mainly used to evaluate the prediction; however, we specify that the current evaluation metrics have a critical limitation when evaluating belief states accumulated as the dialogue proceeds, especially in the most used MultiWOZ dataset. Additionally, we propose relative slot accuracy to complement existing metrics. Relative slot accuracy does not depend on the number of predefined slots, and allows intuitive evaluation by assigning relative scores according to the turn of each dialogue. This study also encourages not solely the reporting of joint goal accuracy, but also various complementary metrics in DST tasks for the sake of a realistic evaluation.

    Original languageEnglish
    Title of host publicationACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)
    EditorsSmaranda Muresan, Preslav Nakov, Aline Villavicencio
    PublisherAssociation for Computational Linguistics (ACL)
    Pages297-309
    Number of pages13
    ISBN (Electronic)9781955917223
    Publication statusPublished - 2022
    Event60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - Dublin, Ireland
    Duration: 2022 May 222022 May 27

    Publication series

    NameProceedings of the Annual Meeting of the Association for Computational Linguistics
    Volume2
    ISSN (Print)0736-587X

    Conference

    Conference60th Annual Meeting of the Association for Computational Linguistics, ACL 2022
    Country/TerritoryIreland
    CityDublin
    Period22/5/2222/5/27

    Bibliographical note

    Publisher Copyright:
    © 2022 Association for Computational Linguistics.

    ASJC Scopus subject areas

    • Computer Science Applications
    • Linguistics and Language
    • Language and Linguistics

    Fingerprint

    Dive into the research topics of 'Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking'. Together they form a unique fingerprint.

    Cite this