Lifelong Language Learning with the Most Forgotten Knowledge

Heejeong Choi, Pilsung Kang

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

Lifelong language learning enables a language model to accumulate knowledge through training on a stream of text data. Recent research on lifelong language learning is based on samples of previous tasks from an episodic memory or generative model. LAMOL, a representative generative model-based lifelong language learning model, preserves the previous information with the generated pseudo-old samples, which are suboptimal. In this paper, we propose an improved version of LAMOL, MFK-LAMOL, which constructs a generative replay using a more effective method. When a new task is received, MFK-LAMOL replays sufficient previous data and retrieves important examples for training alongside the new task. Specifically, it selects the examples with the most forgotten knowledge learned from previous tasks based on the extent to which they include knowledge that has been forgotten after learning new information. We showed that the proposed method outperforms LAMOL on a stream of three different natural language processing tasks.

Original languageEnglish
Article number9399079
Pages (from-to)57941-57948
Number of pages8
JournalIEEE Access
Volume9
DOIs
Publication statusPublished - 2021

Bibliographical note

Funding Information:
This work was supported in part by the National Research Foundation of Korea (NRF) funded by the Korean Government (MSIT) under Grant NRF-2019R1F1A1060338 and Grant NRF-2019R1A4A1024732.

Publisher Copyright:
© 2013 IEEE.

Keywords

  • Lifelong language learning
  • a stream of text data
  • catastrophic forgetting
  • generative replay
  • natural language processing

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering

Fingerprint

Dive into the research topics of 'Lifelong Language Learning with the Most Forgotten Knowledge'. Together they form a unique fingerprint.

Cite this