Text Embedding Augmentation Based on Retraining With Pseudo-Labeled Adversarial Embedding

Myeongsup Kim, Pilsung Kang

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)


Pre-trained language models (LMs) have been shown to achieve outstanding performance in various natural language processing tasks; however, these models have a significantly large number of parameters to handle large-scale text corpora during the pre-training process, and thus, they entail the risk of overfitting when fine-tuning for small task-oriented datasets is conducted. In this paper, we propose a text embedding augmentation method to prevent such overfitting. The proposed method applies augmentation to a text embedding by generating an adversarial embedding, which is not identical to original input embedding but maintaining the characteristics of the original input embedding, using PGD-based adversarial training for input text data. A pseudo-label that is identical to the label of the input text is then assigned to adversarial embedding to conduct retraining by using adversarial embedding and pseudo-label as input embedding and label pair for a separate LM. Experimental results on several text classification benchmark datasets demonstrated that the proposed method effectively prevented overfitting, which commonly occurs when adjusting a large-scale pre-trained LM to a specific task.

Original languageEnglish
Pages (from-to)8363-8376
Number of pages14
JournalIEEE Access
Publication statusPublished - 2022

Bibliographical note

Publisher Copyright:
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/


  • Data models
  • Extrapolation
  • Interpolation
  • Semantics
  • Task analysis
  • Training
  • Transformers

ASJC Scopus subject areas

  • General Engineering
  • General Materials Science
  • General Computer Science


Dive into the research topics of 'Text Embedding Augmentation Based on Retraining With Pseudo-Labeled Adversarial Embedding'. Together they form a unique fingerprint.

Cite this