TY - GEN
T1 - Contextualized sparse representations for real-time open-domain question answering
AU - Lee, Jinhyuk
AU - Seo, Minjoon
AU - Hajishirzi, Hannaneh
AU - Kang, Jaewoo
N1 - Funding Information:
This research was supported by National Research Foundation of Korea (NRF-2017R1A2A1A17069 645, NRF-2017M3C4A7065887), ONR N00014-18-1-2826, DARPA N66001-19-2-403, Allen Distinguished Investigator Award, and Sloan Fellowship. We thank the members of Korea University, University of Washington, NAVER Clova AI, and the anonymous reviewers for their insightful comments.
Funding Information:
This research was supported by National Research Foundation of Korea (NRF-2017R1A2A1A17069 645, NRF-2017M3C4A7065887), ONR N00014-18-1-2826, DARPA N66001-19-2-403, Allen Distinguished Investigator Award, and Sloan Fellowship. We thank the members of Korea University, University of Washington, NAVER Clova AI, and
Publisher Copyright:
© 2020 Association for Computational Linguistics
PY - 2020
Y1 - 2020
N2 - Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (SPARC). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with SPARC, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.
AB - Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (SPARC). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with SPARC, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.
UR - http://www.scopus.com/inward/record.url?scp=85111436395&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85111436395
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 912
EP - 919
BT - ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
PB - Association for Computational Linguistics (ACL)
T2 - 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020
Y2 - 5 July 2020 through 10 July 2020
ER -