BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Elad Ben-Zaken; Shauli Ravfogel; Yoav Goldberg

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg

Allen Institute for Artificial Intelligence

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

199 Scopus citations

Abstract

We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

Original language	English
Title of host publication	ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)
Editors	Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Publisher	Association for Computational Linguistics (ACL)
Pages	1-9
Number of pages	9
ISBN (Electronic)	9781955917223
State	Published - 2022
Event	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - Dublin, Ireland Duration: 22 May 2022 → 27 May 2022

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
Volume	2
ISSN (Print)	0736-587X

Conference

Conference	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022
Country/Territory	Ireland
City	Dublin
Period	22/05/22 → 27/05/22

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Funding

Funders	Funder number
Horizon 2020 Framework Programme	802774

Cite this

Ben-Zaken, E., Ravfogel, S., & Goldberg, Y. (2022). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers) (pp. 1-9). (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 2). Association for Computational Linguistics (ACL).

Ben-Zaken, Elad ; Ravfogel, Shauli ; Goldberg, Yoav. / BitFit : Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). editor / Smaranda Muresan ; Preslav Nakov ; Aline Villavicencio. Association for Computational Linguistics (ACL), 2022. pp. 1-9 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{7fce4d3bb99d43ddaa9b97636a10da44,

title = "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models",

abstract = "We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.",

author = "Elad Ben-Zaken and Shauli Ravfogel and Yoav Goldberg",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 ; Conference date: 22-05-2022 Through 27-05-2022",

year = "2022",

language = "אנגלית",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1--9",

editor = "Smaranda Muresan and Preslav Nakov and Aline Villavicencio",

booktitle = "ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)",

address = "ארצות הברית",

}

Ben-Zaken, E, Ravfogel, S & Goldberg, Y 2022, BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. in S Muresan, P Nakov & A Villavicencio (eds), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 2, Association for Computational Linguistics (ACL), pp. 1-9, 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, Dublin, Ireland, 22/05/22.

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. / Ben-Zaken, Elad; Ravfogel, Shauli; Goldberg, Yoav.
ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). ed. / Smaranda Muresan; Preslav Nakov; Aline Villavicencio. Association for Computational Linguistics (ACL), 2022. p. 1-9 (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - BitFit

T2 - 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022

AU - Ben-Zaken, Elad

AU - Ravfogel, Shauli

AU - Goldberg, Yoav

PY - 2022

Y1 - 2022

N2 - We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

AB - We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

UR - http://www.scopus.com/inward/record.url?scp=85142911444&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85142911444

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 1

EP - 9

BT - ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)

A2 - Muresan, Smaranda

A2 - Nakov, Preslav

A2 - Villavicencio, Aline

PB - Association for Computational Linguistics (ACL)

Y2 - 22 May 2022 through 27 May 2022

ER -

Ben-Zaken E, Ravfogel S, Goldberg Y. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In Muresan S, Nakov P, Villavicencio A, editors, ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). Association for Computational Linguistics (ACL). 2022. p. 1-9. (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Abstract

Publication series

Conference

Bibliographical note

Funding

Other files and links

Fingerprint

Cite this