SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation

Elizabeth Clark; Shruti Rijhwani; Sebastian Gehrmann; Joshua Maynez; Roee Aharoni; Vitaly Nikolaev; Thibault Sellam; Aditya Siddhant; Dipanjan Das; Ankur P. Parikh

SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation

Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, Ankur P. Parikh

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness. SEAHORSE covers 6 languages, 9 systems (including the reference text), and 4 summarization datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on two out-of-domain meta-evaluation benchmarks: TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2023). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.

Original language	English
Title of host publication	EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings
Editors	Houda Bouamor, Juan Pino, Kalika Bali
Publisher	Association for Computational Linguistics (ACL)
Pages	9397-9413
Number of pages	17
ISBN (Electronic)	9798891760608
State	Published - 2023
Externally published	Yes
Event	2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - Hybrid, Singapore, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Publication series

Name	EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

Conference

Conference	2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023
Country/Territory	Singapore
City	Hybrid, Singapore
Period	6/12/23 → 10/12/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Funding

We would like to thank Ashwin Kakarla and his team for help with the annotations, as well as Slav Petrov, Hannah Rashkin, and our EMNLP reviewers for their feedback on the paper.

Funders	Funder number
Ashwin Kakarla

Cite this

Clark, E., Rijhwani, S., Gehrmann, S., Maynez, J., Aharoni, R., Nikolaev, V., Sellam, T., Siddhant, A., Das, D., & Parikh, A. P. (2023). SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation. In H. Bouamor, J. Pino, & K. Bali (Eds.), EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 9397-9413). (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings). Association for Computational Linguistics (ACL).

Clark, Elizabeth ; Rijhwani, Shruti ; Gehrmann, Sebastian et al. / SEAHORSE : A Multilingual, Multifaceted Dataset for Summarization Evaluation. EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. editor / Houda Bouamor ; Juan Pino ; Kalika Bali. Association for Computational Linguistics (ACL), 2023. pp. 9397-9413 (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

@inproceedings{f08da46255704e998d7fdc911f26e39f,

title = "SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation",

abstract = "Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness. SEAHORSE covers 6 languages, 9 systems (including the reference text), and 4 summarization datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on two out-of-domain meta-evaluation benchmarks: TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2023). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.",

author = "Elizabeth Clark and Shruti Rijhwani and Sebastian Gehrmann and Joshua Maynez and Roee Aharoni and Vitaly Nikolaev and Thibault Sellam and Aditya Siddhant and Dipanjan Das and Parikh, {Ankur P.}",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

language = "אנגלית",

series = "EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "9397--9413",

editor = "Houda Bouamor and Juan Pino and Kalika Bali",

booktitle = "EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings",

address = "ארצות הברית",

}

Clark, E, Rijhwani, S, Gehrmann, S, Maynez, J, Aharoni, R, Nikolaev, V, Sellam, T, Siddhant, A, Das, D & Parikh, AP 2023, SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation. in H Bouamor, J Pino & K Bali (eds), EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings, Association for Computational Linguistics (ACL), pp. 9397-9413, 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Hybrid, Singapore, Singapore, 6/12/23.

SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation. / Clark, Elizabeth; Rijhwani, Shruti; Gehrmann, Sebastian et al.
EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. ed. / Houda Bouamor; Juan Pino; Kalika Bali. Association for Computational Linguistics (ACL), 2023. p. 9397-9413 (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - SEAHORSE

T2 - 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023

AU - Clark, Elizabeth

AU - Rijhwani, Shruti

AU - Gehrmann, Sebastian

AU - Maynez, Joshua

AU - Aharoni, Roee

AU - Nikolaev, Vitaly

AU - Sellam, Thibault

AU - Siddhant, Aditya

AU - Das, Dipanjan

AU - Parikh, Ankur P.

PY - 2023

Y1 - 2023

N2 - Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness. SEAHORSE covers 6 languages, 9 systems (including the reference text), and 4 summarization datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on two out-of-domain meta-evaluation benchmarks: TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2023). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.

AB - Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness. SEAHORSE covers 6 languages, 9 systems (including the reference text), and 4 summarization datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on two out-of-domain meta-evaluation benchmarks: TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2023). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.

UR - http://www.scopus.com/inward/record.url?scp=85184800920&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85184800920

T3 - EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

SP - 9397

EP - 9413

BT - EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

A2 - Bouamor, Houda

A2 - Pino, Juan

A2 - Bali, Kalika

PB - Association for Computational Linguistics (ACL)

Y2 - 6 December 2023 through 10 December 2023

ER -

Clark E, Rijhwani S, Gehrmann S, Maynez J, Aharoni R, Nikolaev V et al. SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation. In Bouamor H, Pino J, Bali K, editors, EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL). 2023. p. 9397-9413. (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation

Abstract

Publication series

Conference

Bibliographical note

Funding

Other files and links

Fingerprint

Cite this