Evaluating and Improving the Coreference Capabilities of Machine Translation Models

Asaf Yehudai; Arie Cattan; Omri Abend; Gabriel Stanovsky

doi:10.18653/v1/2023.eacl-main.69

Evaluating and Improving the Coreference Capabilities of Machine Translation Models

Asaf Yehudai
, Arie Cattan
, Omri Abend
, Gabriel Stanovsky

Department of Computer Science

Hebrew University of Jerusalem

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we ask: How well do MT models learn coreference resolution from implicit signal? To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. We further evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the-art coreference resolvers on three challenging benchmarks. Our results show that the monolingual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, showing improvement over strong baselines.

Original language	English
Title of host publication	EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	980-992
Number of pages	13
ISBN (Electronic)	9781959429449
DOIs	https://doi.org/10.18653/v1/2023.eacl-main.69
State	Published - 2023
Event	17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023 - Dubrovnik, Croatia, Croatia Duration: 2 May 2023 → 6 May 2023

Publication series

Name	EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

Conference

Conference	17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023
Country/Territory	Croatia
City	Dubrovnik, Croatia
Period	2/05/23 → 6/05/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Funding

We thank the reviewers for their insightful comments and suggestions. This work was partially supported by the Israeli Ministry of Science and Technology (grant no. 2088). Arie Cattan is partially supported by the PBC fellowship for outstanding PhD candidates in data science.

Funders	Funder number
Ministry of science and technology, Israel	2088
Planning and Budgeting Committee of the Council for Higher Education of Israel

Access to Document

10.18653/v1/2023.eacl-main.69

Cite this

Yehudai, A., Cattan, A., Abend, O., & Stanovsky, G. (2023). Evaluating and Improving the Coreference Capabilities of Machine Translation Models. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference (pp. 980-992). (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.eacl-main.69

Yehudai, Asaf ; Cattan, Arie ; Abend, Omri et al. / Evaluating and Improving the Coreference Capabilities of Machine Translation Models. EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2023. pp. 980-992 (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference).

@inproceedings{1a46aedced1e493f8a3a905cbc81b18c,

title = "Evaluating and Improving the Coreference Capabilities of Machine Translation Models",

abstract = "Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we ask: How well do MT models learn coreference resolution from implicit signal? To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. We further evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the-art coreference resolvers on three challenging benchmarks. Our results show that the monolingual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, showing improvement over strong baselines.",

author = "Asaf Yehudai and Arie Cattan and Omri Abend and Gabriel Stanovsky",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023 ; Conference date: 02-05-2023 Through 06-05-2023",

year = "2023",

doi = "10.18653/v1/2023.eacl-main.69",

language = "אנגלית",

series = "EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "980--992",

booktitle = "EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference",

address = "ארצות הברית",

}

Yehudai, A, Cattan, A, Abend, O & Stanovsky, G 2023, Evaluating and Improving the Coreference Capabilities of Machine Translation Models. in EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 980-992, 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, Croatia, 2/05/23. https://doi.org/10.18653/v1/2023.eacl-main.69

Evaluating and Improving the Coreference Capabilities of Machine Translation Models. / Yehudai, Asaf; Cattan, Arie; Abend, Omri et al.
EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2023. p. 980-992 (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Evaluating and Improving the Coreference Capabilities of Machine Translation Models

AU - Yehudai, Asaf

AU - Cattan, Arie

AU - Abend, Omri

AU - Stanovsky, Gabriel

PY - 2023

Y1 - 2023

N2 - Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we ask: How well do MT models learn coreference resolution from implicit signal? To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. We further evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the-art coreference resolvers on three challenging benchmarks. Our results show that the monolingual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, showing improvement over strong baselines.

AB - Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we ask: How well do MT models learn coreference resolution from implicit signal? To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. We further evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the-art coreference resolvers on three challenging benchmarks. Our results show that the monolingual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, showing improvement over strong baselines.

UR - https://www.scopus.com/pages/publications/85159856325

U2 - 10.18653/v1/2023.eacl-main.69

DO - 10.18653/v1/2023.eacl-main.69

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85159856325

T3 - EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

SP - 980

EP - 992

BT - EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

T2 - 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023

Y2 - 2 May 2023 through 6 May 2023

ER -

Yehudai A, Cattan A, Abend O, Stanovsky G. Evaluating and Improving the Coreference Capabilities of Machine Translation Models. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2023. p. 980-992. (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference). doi: 10.18653/v1/2023.eacl-main.69

Evaluating and Improving the Coreference Capabilities of Machine Translation Models

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this