Comparative analysis of current approaches to quality estimation for neural machine translation

Sugyeong Eo; Chanjun Park; Hyeonseok Moon; Jaehyung Seo; Heuiseok Lim

doi:10.3390/app11146584

Comparative analysis of current approaches to quality estimation for neural machine translation

Sugyeong Eo, Chanjun Park, Hyeonseok Moon, Jaehyung Seo, Heuiseok Lim

Research output: Contribution to journal › Article › peer-review

6 Citations (Scopus)

Abstract

Quality estimation (QE) has recently gained increasing interest as it can predict the quality of machine translation results without a reference translation. QE is an annual shared task at the Conference on Machine Translation (WMT), and most recent studies have applied the multilingual pretrained language model (mPLM) to address this task. Recent studies have focused on the performance improvement of this task using data augmentation with finetuning based on a large-scale mPLM. In this study, we eliminate the effects of data augmentation and conduct a pure performance comparison between various mPLMs. Separate from the recent performance-driven QE research involved in competitions addressing a shared task, we utilize the comparison for sub-tasks from WMT20 and identify an optimal mPLM. Moreover, we demonstrate QE using the multilingual BART model, which has not yet been utilized, and conduct comparative experiments and analyses with cross-lingual language models (XLMs), multilingual BERT, and XLM-RoBERTa.

Original language	English
Article number	6584
Journal	Applied Sciences (Switzerland)
Volume	11
Issue number	14
DOIs	https://doi.org/10.3390/app11146584
Publication status	Published - 2021 Jul 2

Bibliographical note

Funding Information:
Funding: This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2018-0-01405) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation) and the MSIT, Korea, under the ICT Creative Consilience program (IITP-2021-2020-0-01819) supervised by the IITP. Additionally, this work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques).

Publisher Copyright:
© 2021 by the authors. Licensee MDPI, Basel, Switzerland.

Keywords

Multilingual pre-trained language model
Neural machine translation
Pretrained language model
Quality estimation
WMT

ASJC Scopus subject areas

Materials Science(all)
Instrumentation
Engineering(all)
Process Chemistry and Technology
Computer Science Applications
Fluid Flow and Transfer Processes

Access to Document

10.3390/app11146584

Cite this

@article{2e920c9e9ba84aac9f7c787fabde036c,

title = "Comparative analysis of current approaches to quality estimation for neural machine translation",

abstract = "Quality estimation (QE) has recently gained increasing interest as it can predict the quality of machine translation results without a reference translation. QE is an annual shared task at the Conference on Machine Translation (WMT), and most recent studies have applied the multilingual pretrained language model (mPLM) to address this task. Recent studies have focused on the performance improvement of this task using data augmentation with finetuning based on a large-scale mPLM. In this study, we eliminate the effects of data augmentation and conduct a pure performance comparison between various mPLMs. Separate from the recent performance-driven QE research involved in competitions addressing a shared task, we utilize the comparison for sub-tasks from WMT20 and identify an optimal mPLM. Moreover, we demonstrate QE using the multilingual BART model, which has not yet been utilized, and conduct comparative experiments and analyses with cross-lingual language models (XLMs), multilingual BERT, and XLM-RoBERTa.",

keywords = "Multilingual pre-trained language model, Neural machine translation, Pretrained language model, Quality estimation, WMT",

author = "Sugyeong Eo and Chanjun Park and Hyeonseok Moon and Jaehyung Seo and Heuiseok Lim",

note = "Funding Information: Funding: This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2018-0-01405) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation) and the MSIT, Korea, under the ICT Creative Consilience program (IITP-2021-2020-0-01819) supervised by the IITP. Additionally, this work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques). Publisher Copyright: {\textcopyright} 2021 by the authors. Licensee MDPI, Basel, Switzerland.",

year = "2021",

month = jul,

day = "2",

doi = "10.3390/app11146584",

language = "English",

volume = "11",

journal = "Applied Sciences (Switzerland)",

issn = "2076-3417",

publisher = "Multidisciplinary Digital Publishing Institute",

number = "14",

}

TY - JOUR

T1 - Comparative analysis of current approaches to quality estimation for neural machine translation

AU - Eo, Sugyeong

AU - Park, Chanjun

AU - Moon, Hyeonseok

AU - Seo, Jaehyung

AU - Lim, Heuiseok

N1 - Funding Information: Funding: This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2018-0-01405) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation) and the MSIT, Korea, under the ICT Creative Consilience program (IITP-2021-2020-0-01819) supervised by the IITP. Additionally, this work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques). Publisher Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland.

PY - 2021/7/2

Y1 - 2021/7/2

N2 - Quality estimation (QE) has recently gained increasing interest as it can predict the quality of machine translation results without a reference translation. QE is an annual shared task at the Conference on Machine Translation (WMT), and most recent studies have applied the multilingual pretrained language model (mPLM) to address this task. Recent studies have focused on the performance improvement of this task using data augmentation with finetuning based on a large-scale mPLM. In this study, we eliminate the effects of data augmentation and conduct a pure performance comparison between various mPLMs. Separate from the recent performance-driven QE research involved in competitions addressing a shared task, we utilize the comparison for sub-tasks from WMT20 and identify an optimal mPLM. Moreover, we demonstrate QE using the multilingual BART model, which has not yet been utilized, and conduct comparative experiments and analyses with cross-lingual language models (XLMs), multilingual BERT, and XLM-RoBERTa.

AB - Quality estimation (QE) has recently gained increasing interest as it can predict the quality of machine translation results without a reference translation. QE is an annual shared task at the Conference on Machine Translation (WMT), and most recent studies have applied the multilingual pretrained language model (mPLM) to address this task. Recent studies have focused on the performance improvement of this task using data augmentation with finetuning based on a large-scale mPLM. In this study, we eliminate the effects of data augmentation and conduct a pure performance comparison between various mPLMs. Separate from the recent performance-driven QE research involved in competitions addressing a shared task, we utilize the comparison for sub-tasks from WMT20 and identify an optimal mPLM. Moreover, we demonstrate QE using the multilingual BART model, which has not yet been utilized, and conduct comparative experiments and analyses with cross-lingual language models (XLMs), multilingual BERT, and XLM-RoBERTa.

KW - Multilingual pre-trained language model

KW - Neural machine translation

KW - Pretrained language model

KW - Quality estimation

KW - WMT

UR - http://www.scopus.com/inward/record.url?scp=85111315296&partnerID=8YFLogxK

U2 - 10.3390/app11146584

DO - 10.3390/app11146584

M3 - Article

AN - SCOPUS:85111315296

SN - 2076-3417

VL - 11

JO - Applied Sciences (Switzerland)

JF - Applied Sciences (Switzerland)

IS - 14

M1 - 6584

ER -

Comparative analysis of current approaches to quality estimation for neural machine translation

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this