Pre-trained Language Model for Biomedical Question Answering

Wonjin Yoon; Jinhyuk Lee; Donghyeon Kim; Minbyul Jeong; Jaewoo Kang

doi:10.1007/978-3-030-43887-6_64

Pre-trained Language Model for Biomedical Question Answering

Wonjin Yoon, Jinhyuk Lee, Donghyeon Kim, Minbyul Jeong, Jaewoo Kang

Department of Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

33 Citations (Scopus)

Abstract

The recent success of question answering systems is largely attributed to pre-trained language models. However, as language models are mostly pre-trained on general domain corpora such as Wikipedia, they often have difficulty in understanding biomedical questions. In this paper, we investigate the performance of BioBERT, a pre-trained biomedical language model, in answering biomedical questions including factoid, list, and yes/no type questions. BioBERT uses almost the same structure across various question types and achieved the best performance in the 7th BioASQ Challenge (Task 7b, Phase B). BioBERT pre-trained on SQuAD or SQuAD 2.0 easily outperformed previous state-of-the-art models. BioBERT obtains the best performance when it uses the appropriate pre-/post-processing strategies for questions, passages, and answers.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings
Editors	Peggy Cellier, Kurt Driessens
Publisher	Springer
Pages	727-740
Number of pages	14
ISBN (Print)	9783030438869
DOIs	https://doi.org/10.1007/978-3-030-43887-6_64
Publication status	Published - 2020
Event	19th Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019 - Wurzburg, Germany Duration: 2019 Sept 16 → 2019 Sept 20

Publication series

Name	Communications in Computer and Information Science
Volume	1168 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	19th Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019
Country/Territory	Germany
City	Wurzburg
Period	19/9/16 → 19/9/20

Bibliographical note

Funding Information:
We appreciate Susan Kim for editing the manuscript. This work was funded by the National Research Foundation of Korea (NRF-2017R1A2A1A17069645, NRF-2016M3A9A7916996) and the National IT Industry Promotion Agency grant funded by the Ministry of Science and ICT and Ministry of Health and Welfare (NO. C1202-18-1001, Development Project of The Precision Medicine Hospital Information System (P-HIS)).

Funding Information:
Acknowledgements. We appreciate Susan Kim for editing the manuscript. This work was funded by the National Research Foundation of Korea (NRF-2017R1A2A1A17069645, NRF-2016M3A9A7916996) and the National IT Industry Promotion Agency grant funded by the Ministry of Science and ICT and Ministry of Health and Welfare (NO. C1202-18-1001, Development Project of The Precision Medicine Hospital Information System (P-HIS)).

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Keywords

Biomedical question answering
Pre-trained language model
Transfer learning

ASJC Scopus subject areas

Computer Science(all)
Mathematics(all)

Access to Document

10.1007/978-3-030-43887-6_64

Cite this

Yoon, W., Lee, J., Kim, D., Jeong, M., & Kang, J. (2020). Pre-trained Language Model for Biomedical Question Answering. In P. Cellier, & K. Driessens (Eds.), Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings (pp. 727-740). (Communications in Computer and Information Science; Vol. 1168 CCIS). Springer. https://doi.org/10.1007/978-3-030-43887-6_64

Pre-trained Language Model for Biomedical Question Answering. / Yoon, Wonjin; Lee, Jinhyuk; Kim, Donghyeon et al.
Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings. ed. / Peggy Cellier; Kurt Driessens. Springer, 2020. p. 727-740 (Communications in Computer and Information Science; Vol. 1168 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yoon, W, Lee, J, Kim, D, Jeong, M & Kang, J 2020, Pre-trained Language Model for Biomedical Question Answering. in P Cellier & K Driessens (eds), Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings. Communications in Computer and Information Science, vol. 1168 CCIS, Springer, pp. 727-740, 19th Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019, Wurzburg, Germany, 19/9/16. https://doi.org/10.1007/978-3-030-43887-6_64

@inproceedings{9ddbb866c73547a69a01703144330c99,

title = "Pre-trained Language Model for Biomedical Question Answering",

abstract = "The recent success of question answering systems is largely attributed to pre-trained language models. However, as language models are mostly pre-trained on general domain corpora such as Wikipedia, they often have difficulty in understanding biomedical questions. In this paper, we investigate the performance of BioBERT, a pre-trained biomedical language model, in answering biomedical questions including factoid, list, and yes/no type questions. BioBERT uses almost the same structure across various question types and achieved the best performance in the 7th BioASQ Challenge (Task 7b, Phase B). BioBERT pre-trained on SQuAD or SQuAD 2.0 easily outperformed previous state-of-the-art models. BioBERT obtains the best performance when it uses the appropriate pre-/post-processing strategies for questions, passages, and answers.",

keywords = "Biomedical question answering, Pre-trained language model, Transfer learning",

author = "Wonjin Yoon and Jinhyuk Lee and Donghyeon Kim and Minbyul Jeong and Jaewoo Kang",

note = "Funding Information: We appreciate Susan Kim for editing the manuscript. This work was funded by the National Research Foundation of Korea (NRF-2017R1A2A1A17069645, NRF-2016M3A9A7916996) and the National IT Industry Promotion Agency grant funded by the Ministry of Science and ICT and Ministry of Health and Welfare (NO. C1202-18-1001, Development Project of The Precision Medicine Hospital Information System (P-HIS)). Funding Information: Acknowledgements. We appreciate Susan Kim for editing the manuscript. This work was funded by the National Research Foundation of Korea (NRF-2017R1A2A1A17069645, NRF-2016M3A9A7916996) and the National IT Industry Promotion Agency grant funded by the Ministry of Science and ICT and Ministry of Health and Welfare (NO. C1202-18-1001, Development Project of The Precision Medicine Hospital Information System (P-HIS)). Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 19th Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019 ; Conference date: 16-09-2019 Through 20-09-2019",

year = "2020",

doi = "10.1007/978-3-030-43887-6_64",

language = "English",

isbn = "9783030438869",

series = "Communications in Computer and Information Science",

publisher = "Springer",

pages = "727--740",

editor = "Peggy Cellier and Kurt Driessens",

booktitle = "Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings",

}

TY - GEN

T1 - Pre-trained Language Model for Biomedical Question Answering

AU - Yoon, Wonjin

AU - Lee, Jinhyuk

AU - Kim, Donghyeon

AU - Jeong, Minbyul

AU - Kang, Jaewoo

N1 - Funding Information: We appreciate Susan Kim for editing the manuscript. This work was funded by the National Research Foundation of Korea (NRF-2017R1A2A1A17069645, NRF-2016M3A9A7916996) and the National IT Industry Promotion Agency grant funded by the Ministry of Science and ICT and Ministry of Health and Welfare (NO. C1202-18-1001, Development Project of The Precision Medicine Hospital Information System (P-HIS)). Funding Information: Acknowledgements. We appreciate Susan Kim for editing the manuscript. This work was funded by the National Research Foundation of Korea (NRF-2017R1A2A1A17069645, NRF-2016M3A9A7916996) and the National IT Industry Promotion Agency grant funded by the Ministry of Science and ICT and Ministry of Health and Welfare (NO. C1202-18-1001, Development Project of The Precision Medicine Hospital Information System (P-HIS)). Publisher Copyright: © 2020, Springer Nature Switzerland AG.

PY - 2020

Y1 - 2020

N2 - The recent success of question answering systems is largely attributed to pre-trained language models. However, as language models are mostly pre-trained on general domain corpora such as Wikipedia, they often have difficulty in understanding biomedical questions. In this paper, we investigate the performance of BioBERT, a pre-trained biomedical language model, in answering biomedical questions including factoid, list, and yes/no type questions. BioBERT uses almost the same structure across various question types and achieved the best performance in the 7th BioASQ Challenge (Task 7b, Phase B). BioBERT pre-trained on SQuAD or SQuAD 2.0 easily outperformed previous state-of-the-art models. BioBERT obtains the best performance when it uses the appropriate pre-/post-processing strategies for questions, passages, and answers.

AB - The recent success of question answering systems is largely attributed to pre-trained language models. However, as language models are mostly pre-trained on general domain corpora such as Wikipedia, they often have difficulty in understanding biomedical questions. In this paper, we investigate the performance of BioBERT, a pre-trained biomedical language model, in answering biomedical questions including factoid, list, and yes/no type questions. BioBERT uses almost the same structure across various question types and achieved the best performance in the 7th BioASQ Challenge (Task 7b, Phase B). BioBERT pre-trained on SQuAD or SQuAD 2.0 easily outperformed previous state-of-the-art models. BioBERT obtains the best performance when it uses the appropriate pre-/post-processing strategies for questions, passages, and answers.

KW - Biomedical question answering

KW - Pre-trained language model

KW - Transfer learning

UR - http://www.scopus.com/inward/record.url?scp=85083635942&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-43887-6_64

DO - 10.1007/978-3-030-43887-6_64

M3 - Conference contribution

AN - SCOPUS:85083635942

SN - 9783030438869

T3 - Communications in Computer and Information Science

SP - 727

EP - 740

BT - Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings

A2 - Cellier, Peggy

A2 - Driessens, Kurt

PB - Springer

T2 - 19th Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019

Y2 - 16 September 2019 through 20 September 2019

ER -

Pre-trained Language Model for Biomedical Question Answering

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this