Findings of the WMT 2022 Shared Task on Quality Estimation
Zerva,Chrysoula Blain,Frédéric Rei,Ricardo Lertvittayakumjorn,Piyawat de Souza,José G.C. Eger,Steffen Kanojia,Diptesh Alves,Duarte Orăsan,Constantin Fomicheva,Marina
Zerva,Chrysoula
Blain,Frédéric
Rei,Ricardo
Lertvittayakumjorn,Piyawat
de Souza,José G.C.
Eger,Steffen
Kanojia,Diptesh
Alves,Duarte
Orăsan,Constantin
Fomicheva,Marina
Abstract
We report the results of the WMT 2022 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the Direct Assessments and post-edit data (MLQE-PE) to new language pairs: we present a novel and large dataset on English-Marathi, as well as a zero-shot test-set on English-Yoruba. Further, we include an explainability sub-task for all language pairs and present a new format of a critical error detection task for two new language pairs. Participants from 11 different teams submitted altogether 991 systems to different task variants and language pairs.
Description
Funding Information: Ricardo Rei and José G. C. de Souza are supported by the P2020 program (MAIA: contract 045909) and by European Union’s Horizon Europe Research and Innovation Actions (UTTER: contract 101070631) André Martins and Chrysoula Zerva are supported by the P2020 program (MAIA: contract 045909), by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. Funding Information: Ricardo Rei and José G. C. de Souza are supported by the P2020 program (MAIA: contract 045909) and by European Union's Horizon Europe Research and Innovation Actions (UTTER: contract 101070631) André Martins and Chrysoula Zerva are supported by the P2020 program (MAIA: contract 045909), by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. Marina Fomicheva and Lucia Specia were supported by funding from the Bergamot project (EU H2020 Grant No. 825303). Funding Information: Marina Fomicheva and Lucia Specia were supported by funding from the Bergamot project (EU H2020 Grant No. 825303). Publisher Copyright: © 2022 Association for Computational Linguistics.
Date
2022
Journal Title
Journal ISSN
Volume Title
Publisher
Association for Computational Linguistics
Research Projects
Organizational Units
Journal Issue
Keywords
neural machine translation systems
Citation
Zerva, C, Blain, F, Rei, R, Lertvittayakumjorn, P, de Souza, J G C, Eger, S, Kanojia, D, Alves, D, Orăsan, C, Fomicheva, M, Martins, A F T & Specia, L 2022, Findings of the WMT 2022 Shared Task on Quality Estimation. in WMT 2022 - 7th Conference on Machine Translation, Proceedings of the Conference. Conference on Machine Translation - Proceedings, Association for Computational Linguistics, pp. 69-99, 7th Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates, 7/12/22. < https://aclanthology.org/2022.wmt-1.3/ >