Comparison of the Evaluation Metrics for Neural Grammatical Error Correction with Overcorrection

Chanjun Park, Yeongwook Yang, Chanhee Lee, Heuiseok Lim

Research output: Contribution to journalArticlepeer-review

11 Citations (Scopus)

Abstract

Grammar error correction (GEC) refers to the proper correction of grammatical errors in a given sentence. Important factors to consider in GEC are not only the grammatical correction of the sentence, but also the recognition of a correct sentence in which no changes are required. However, GEC approaches in which deep learning recently started being used consider only the former aspect, which leads to overcorrection, whereby changes are made to a correct sentence unnecessarily. Because this bias is also reflected in performance metrics, conventional performance metrics consider only part of the important factors in GEC. This study proposes a new metric to consider both important aspects in GEC and to provide a new viewpoint for the GEC task. To the best of the authors knowledge, this study is the first to deal with comprehensively considering the correction performance and overcorrection problem in GEC. The experimental results demonstrate that the model performance ranking was reversed when evaluating the performance with the proposed metric compared to the General Language Understanding Evaluation benchmark [21], which only considers the correction performance. This indicates that the high performance of the correction does not result in less problems with the overcorrection and that the overcorrection problem should also be considered when evaluating the model performance. Moreover, we found that the copy mechanism [14] helps to alleviate the problem of overcorrection.

Original languageEnglish
Article number9102992
Pages (from-to)106264-106272
Number of pages9
JournalIEEE Access
Volume8
DOIs
Publication statusPublished - 2020

Bibliographical note

Funding Information:
This work was supported in part by the Ministry of Science and ICT (MSIT), South Korea, through the Information Technology Research Center (ITRC) Support Program supervised by the Institute for Information and Communications Technology Planning and Evaluation (IITP) under Grant IITP-2020-2018-0-01405, and in part by the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (MSIP), Korea Government under Grant NRF-2017M3C4A7068189.

Publisher Copyright:
© 2013 IEEE.

Keywords

  • Grammar error correction
  • copy mechanism
  • metric
  • neural machine translation
  • overcorrection

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Comparison of the Evaluation Metrics for Neural Grammatical Error Correction with Overcorrection'. Together they form a unique fingerprint.

Cite this