Contextualized sparse representations for real-time open-domain question answering

Jinhyuk Lee; Minjoon Seo; Hannaneh Hajishirzi; Jaewoo Kang

doi:10.18653/v1/2020.acl-main.85

Contextualized sparse representations for real-time open-domain question answering

Jinhyuk Lee, Minjoon Seo, Hannaneh Hajishirzi, Jaewoo Kang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

23 Citations (Scopus)

Abstract

Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (SPARC). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with SPARC, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.

Original language	English
Title of host publication	ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	912-919
Number of pages	8
ISBN (Electronic)	9781952148255
DOIs	https://doi.org/10.18653/v1/2020.acl-main.85
Publication status	Published - 2020
Event	58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 - Virtual, Online, United States Duration: 2020 Jul 5 → 2020 Jul 10

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	58th Annual Meeting of the Association for Computational Linguistics, ACL 2020
Country/Territory	United States
City	Virtual, Online
Period	20/7/5 → 20/7/10

Bibliographical note

Publisher Copyright:
© 2020 Association for Computational Linguistics

ASJC Scopus subject areas

Computer Science Applications
Linguistics and Language
Language and Linguistics

Access to Document

10.18653/v1/2020.acl-main.85

Cite this

Lee, J., Seo, M., Hajishirzi, H., & Kang, J. (2020). Contextualized sparse representations for real-time open-domain question answering. In ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (pp. 912-919). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2020.acl-main.85

Contextualized sparse representations for real-time open-domain question answering. / Lee, Jinhyuk; Seo, Minjoon; Hajishirzi, Hannaneh et al.
ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2020. p. 912-919 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, J, Seo, M, Hajishirzi, H & Kang, J 2020, Contextualized sparse representations for real-time open-domain question answering. in ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 912-919, 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Virtual, Online, United States, 20/7/5. https://doi.org/10.18653/v1/2020.acl-main.85

Lee J, Seo M, Hajishirzi H, Kang J. Contextualized sparse representations for real-time open-domain question answering. In ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2020. p. 912-919. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2020.acl-main.85

Lee, Jinhyuk ; Seo, Minjoon ; Hajishirzi, Hannaneh et al. / Contextualized sparse representations for real-time open-domain question answering. ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2020. pp. 912-919 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{cb64010d35d646489bc7b33b3f23d38c,

title = "Contextualized sparse representations for real-time open-domain question answering",

abstract = "Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (SPARC). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with SPARC, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.",

author = "Jinhyuk Lee and Minjoon Seo and Hannaneh Hajishirzi and Jaewoo Kang",

note = "Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 ; Conference date: 05-07-2020 Through 10-07-2020",

year = "2020",

doi = "10.18653/v1/2020.acl-main.85",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "912--919",

booktitle = "ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference",

}

TY - GEN

T1 - Contextualized sparse representations for real-time open-domain question answering

AU - Lee, Jinhyuk

AU - Seo, Minjoon

AU - Hajishirzi, Hannaneh

AU - Kang, Jaewoo

PY - 2020

Y1 - 2020

N2 - Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (SPARC). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with SPARC, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.

AB - Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (SPARC). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with SPARC, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.

UR - http://www.scopus.com/inward/record.url?scp=85111436395&partnerID=8YFLogxK

U2 - 10.18653/v1/2020.acl-main.85

DO - 10.18653/v1/2020.acl-main.85

M3 - Conference contribution

AN - SCOPUS:85111436395

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 912

EP - 919

BT - ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

T2 - 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020

Y2 - 5 July 2020 through 10 July 2020

ER -

Contextualized sparse representations for real-time open-domain question answering

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this