Look at the first sentence: Position bias in question answering

Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, Jaewoo Kang

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    62 Citations (Scopus)

    Abstract

    Many extractive question answering models are trained to predict start and end positions of answers. The choice of predicting answers as positions is mainly due to its simplicity and effectiveness. In this study, we hypothesize that when the distribution of the answer positions is highly skewed in the training set (e.g., answers lie only in the k-th sentence of each passage), QA models predicting answers as positions can learn spurious positional cues and fail to give answers in different positions. We first illustrate this position bias in popular extractive QA models such as BiDAF and BERT and thoroughly examine how position bias propagates through each layer of BERT. To safely deliver position information without position bias, we train models with various de-biasing methods including entropy regularization and bias ensembling. Among them, we found that using the prior distribution of answer positions as a bias model is very effective at reducing position bias, recovering the performance of BERT from 37.48% to 81.64% when trained on a biased SQuAD dataset.

    Original languageEnglish
    Title of host publicationEMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference
    PublisherAssociation for Computational Linguistics (ACL)
    Pages1109-1121
    Number of pages13
    ISBN (Electronic)9781952148606
    Publication statusPublished - 2020
    Event2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020 - Virtual, Online
    Duration: 2020 Nov 162020 Nov 20

    Publication series

    NameEMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

    Conference

    Conference2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020
    CityVirtual, Online
    Period20/11/1620/11/20

    Bibliographical note

    Publisher Copyright:
    © 2020 Association for Computational Linguistics

    ASJC Scopus subject areas

    • Information Systems
    • Computer Science Applications
    • Computational Theory and Mathematics

    Fingerprint

    Dive into the research topics of 'Look at the first sentence: Position bias in question answering'. Together they form a unique fingerprint.

    Cite this