Abstract
We introduce a new pretraining approach geared for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective. First, instead of considering documents in isolation, we pretrain over sets of multiple related documents, encouraging the model to learn cross-document relationships. Second, we improve over recent long-range transformers by introducing dynamic global attention that has access to the entire input to predict masked tokens. We release CDLM (Cross-Document Language Model), a new general language model for multi-document setting that can be easily applied to downstream tasks. Our extensive analysis shows that both ideas are essential for the success of CDLM, and work in synergy to set new state-of-the-art results for several multi-text tasks.
| Original language | English |
|---|---|
| Title of host publication | Findings of the Association for Computational Linguistics, Findings of ACL |
| Subtitle of host publication | EMNLP 2021 |
| Editors | Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-Tau Yih |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 2648-2662 |
| Number of pages | 15 |
| ISBN (Electronic) | 9781955917100 |
| State | Published - 2021 |
| Event | 2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 - Punta Cana, Dominican Republic Duration: 7 Nov 2021 → 11 Nov 2021 |
Publication series
| Name | Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 |
|---|
Conference
| Conference | 2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 |
|---|---|
| Country/Territory | Dominican Republic |
| City | Punta Cana |
| Period | 7/11/21 → 11/11/21 |
Bibliographical note
Publisher Copyright:© 2021 Association for Computational Linguistics.
Funding
The use of cross-document attention has been We thank Doug Downey and Luke Zettlemoyer for recently explored by the Cross-Document Atten-fruitful discussions and helpful feedback, and Yoav tion (CDA) (Zhou et al., 2020). CDA specifi-Goldberg for helping us connect with collaborators cally encodes two documents, using hierarchical on this project. The work described herein was attention networks, with the addition of cross at-supported in part by grants from Intel Labs, the tention between documents, and makes similar-Israel Science Foundation grant 1951/17, the Israeli ity decision between them. Similarly, the recent Ministry of Science and Technology, and the NSF DCS model (Ginzburg et al., 2021) suggested a Grant OIA-2033558. cross-document finetuning scheme for unsuper- vised document-pair matching method (processing only two documents at once). Our CDLM, by contrast, is a general pretrained language model that can be applied to a variety of multi-document downstream tasks, without restrictions on the number of input documents, as long as they fit the input length of the Longformer.
| Funders | Funder number |
|---|---|
| Intel Labs | |
| National Science Foundation | OIA-2033558 |
| Ministry of science and technology, Israel | |
| makes similar-Israel Science Foundation | 1951/17 |