Abstract
Practical summarization systems are expected to produce summaries of varying lengths, per user needs. While a couple of early summarization benchmarks tested systems across multiple summary lengths, this practice was mostly abandoned due to the assumed cost of producing reference summaries of multiple lengths. In this paper, we raise the research question of whether reference summaries of a single length can be used to reliably evaluate system summaries of multiple lengths. For that, we have analyzed a couple of datasets as a case study, using several variants of the ROUGE metric that are standard in summarization evaluation. Our findings indicate that the evaluation protocol in question is indeed competitive. This result paves the way to practically evaluating varying-length summaries with simple, possibly existing, summarization benchmarks.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 |
| Editors | Ellen Riloff, David Chiang, Julia Hockenmaier, Jun'ichi Tsujii |
| Publisher | Association for Computational Linguistics |
| Pages | 774-778 |
| Number of pages | 5 |
| ISBN (Electronic) | 9781948087841 |
| State | Published - 2018 |
| Event | 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 - Brussels, Belgium Duration: 31 Oct 2018 → 4 Nov 2018 |
Publication series
| Name | Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 |
|---|
Conference
| Conference | 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 |
|---|---|
| Country/Territory | Belgium |
| City | Brussels |
| Period | 31/10/18 → 4/11/18 |
Bibliographical note
Publisher Copyright:© 2018 Association for Computational Linguistics
Funding
We would like to thank the anonymous reviewers for their constructive comments. We thank Yin-fei Yang for his assistance in producing the IC-SISumm summaries that we utilized in our analysis. This work was supported in part by grants from the MAGNET program of the Israel Innovation Authority; by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1); by the BIU Center for Research in Applied Cryptography and Cyber Security in conjunction with the Israel National Cyber Bureau in the Prime Ministers Office; and by the Israel Science Foundation (grants 1157/16 and 1951/17).
| Funders | Funder number |
|---|---|
| DIP | DA 1600/1-1 |
| German-Israeli Project Cooperation | |
| Israel Innovation Authority | |
| Deutsche Forschungsgemeinschaft | |
| Israel Science Foundation | 1157/16, 1951/17 |