Lexical Generalization Improves with Larger Models and Longer Training

Elron Bandel; Yoav Goldberg; Yanai Elazar

doi:10.18653/v1/2022.findings-emnlp.426

Lexical Generalization Improves with Larger Models and Longer Training

Elron Bandel, Yoav Goldberg, Yanai Elazar

Department of Computer Science

Research output: Contribution to conference › Paper › peer-review

6 Scopus citations

Abstract

While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.

Original language	English
Pages	4427-4439
Number of pages	13
DOIs	https://doi.org/10.18653/v1/2022.findings-emnlp.426
State	Published - 2022
Event	2022 Findings of the Association for Computational Linguistics: EMNLP 2022 - Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022

Conference

Conference	2022 Findings of the Association for Computational Linguistics: EMNLP 2022
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	7/12/22 → 11/12/22

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Funding

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT). Yanai Elazar is grateful to have been supported by the PBC fellowship for outstanding PhD candidates in Data Science and the Google PhD fellowship for his PhD, where he spent most of his time on this project. This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEX-TRACT). Yanai Elazar is grateful to have been supported by the PBC fellowship for outstanding PhD candidates in Data Science and the Google PhD fellowship for his PhD, where he spent most of his time on this project.

Funders	Funder number
Horizon 2020 Framework Programme
European Commission
Horizon 2020	802774
Planning and Budgeting Committee of the Council for Higher Education of Israel

Access to Document

10.18653/v1/2022.findings-emnlp.426

Cite this

@conference{350b5a1f168d4dad90c1f887c61ac6a4,

title = "Lexical Generalization Improves with Larger Models and Longer Training",

abstract = "While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.",

author = "Elron Bandel and Yoav Goldberg and Yanai Elazar",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Findings of the Association for Computational Linguistics: EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

year = "2022",

doi = "10.18653/v1/2022.findings-emnlp.426",

language = "אנגלית",

pages = "4427--4439",

}

TY - CONF

T1 - Lexical Generalization Improves with Larger Models and Longer Training

AU - Bandel, Elron

AU - Goldberg, Yoav

AU - Elazar, Yanai

PY - 2022

Y1 - 2022

N2 - While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.

AB - While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.

UR - http://www.scopus.com/inward/record.url?scp=85149869602&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.findings-emnlp.426

DO - 10.18653/v1/2022.findings-emnlp.426

M3 - ???researchoutput.researchoutputtypes.contributiontoconference.paper???

AN - SCOPUS:85149869602

SP - 4427

EP - 4439

T2 - 2022 Findings of the Association for Computational Linguistics: EMNLP 2022

Y2 - 7 December 2022 through 11 December 2022

ER -

Lexical Generalization Improves with Larger Models and Longer Training

Abstract

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this