You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models

Tomasz Limisiewicz; Dan Malkin; Gabriel Stanovsky

You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models

Tomasz Limisiewicz, Dan Malkin, Gabriel Stanovsky

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Multilingual models have been widely used for cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their underrepresentation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacherstudent knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers knowledge into a single multilingual student. Our method outperforms standard training methods in lowresource languages and retains performance on high-resource languages.

Original language	English
Title of host publication	SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop
Editors	Lisa Beinborn, Koustava Goswami, Saliha Muradoglu, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Edoardo M. Ponti, Ryan Cotterell, Ekaterina Vylomova
Publisher	Association for Computational Linguistics
Pages	1-11
Number of pages	11
ISBN (Electronic)	9781959429562
State	Published - 2023
Externally published	Yes
Event	5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, SIGTYP 2023, co-located with the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023 - Hybrid, Dubrovnik, Croatia Duration: 6 May 2023 → …

Publication series

Name	SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop

Conference

Conference	5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, SIGTYP 2023, co-located with the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023
Country/Territory	Croatia
City	Hybrid, Dubrovnik
Period	6/05/23 → …

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Funding

We thank anonymous reviewers for their valuable comments on the previous versions of this article. This work was supported in part by a research gift from the Allen Institute for AI, and a research grant 2336 from the Israeli Ministry of Science and Technology. Tomasz Limisiewicz’s visit to the Hebrew University has been supported by grant 338521 of the Charles University Grant Agency and the Mobility Fund of Charles University.

Funders	Funder number
Mobility Fund of Charles University
Grantová Agentura, Univerzita Karlova
Ministry of science and technology, Israel	338521

Cite this

Limisiewicz, T., Malkin, D., & Stanovsky, G. (2023). You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models. In L. Beinborn, K. Goswami, S. Muradoglu, A. Sorokin, R. Kumar, A. Shcherbakov, E. M. Ponti, R. Cotterell, & E. Vylomova (Eds.), SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop (pp. 1-11). (SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop). Association for Computational Linguistics.

Limisiewicz, Tomasz ; Malkin, Dan ; Stanovsky, Gabriel. / You Can Have Your Data and Balance It Too : Towards Balanced and Efficient Multilingual Models. SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop. editor / Lisa Beinborn ; Koustava Goswami ; Saliha Muradoglu ; Alexey Sorokin ; Ritesh Kumar ; Andreas Shcherbakov ; Edoardo M. Ponti ; Ryan Cotterell ; Ekaterina Vylomova. Association for Computational Linguistics, 2023. pp. 1-11 (SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop).

@inproceedings{80d90e6ca7f24bec9c205359f042bb43,

title = "You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models",

abstract = "Multilingual models have been widely used for cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their underrepresentation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacherstudent knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers knowledge into a single multilingual student. Our method outperforms standard training methods in lowresource languages and retains performance on high-resource languages.",

author = "Tomasz Limisiewicz and Dan Malkin and Gabriel Stanovsky",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, SIGTYP 2023, co-located with the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023 ; Conference date: 06-05-2023",

year = "2023",

language = "אנגלית",

series = "SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics",

pages = "1--11",

editor = "Lisa Beinborn and Koustava Goswami and Saliha Muradoglu and Alexey Sorokin and Ritesh Kumar and Andreas Shcherbakov and Ponti, {Edoardo M.} and Ryan Cotterell and Ekaterina Vylomova",

booktitle = "SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop",

}

Limisiewicz, T, Malkin, D & Stanovsky, G 2023, You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models. in L Beinborn, K Goswami, S Muradoglu, A Sorokin, R Kumar, A Shcherbakov, EM Ponti, R Cotterell & E Vylomova (eds), SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop. SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop, Association for Computational Linguistics, pp. 1-11, 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, SIGTYP 2023, co-located with the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Hybrid, Dubrovnik, Croatia, 6/05/23.

You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models. / Limisiewicz, Tomasz; Malkin, Dan; Stanovsky, Gabriel.
SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop. ed. / Lisa Beinborn; Koustava Goswami; Saliha Muradoglu; Alexey Sorokin; Ritesh Kumar; Andreas Shcherbakov; Edoardo M. Ponti; Ryan Cotterell; Ekaterina Vylomova. Association for Computational Linguistics, 2023. p. 1-11 (SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - You Can Have Your Data and Balance It Too

T2 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, SIGTYP 2023, co-located with the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023

AU - Limisiewicz, Tomasz

AU - Malkin, Dan

AU - Stanovsky, Gabriel

PY - 2023

Y1 - 2023

N2 - Multilingual models have been widely used for cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their underrepresentation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacherstudent knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers knowledge into a single multilingual student. Our method outperforms standard training methods in lowresource languages and retains performance on high-resource languages.

AB - Multilingual models have been widely used for cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their underrepresentation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacherstudent knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers knowledge into a single multilingual student. Our method outperforms standard training methods in lowresource languages and retains performance on high-resource languages.

UR - http://www.scopus.com/inward/record.url?scp=85174726360&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85174726360

T3 - SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop

SP - 1

EP - 11

BT - SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop

A2 - Beinborn, Lisa

A2 - Goswami, Koustava

A2 - Muradoglu, Saliha

A2 - Sorokin, Alexey

A2 - Kumar, Ritesh

A2 - Shcherbakov, Andreas

A2 - Ponti, Edoardo M.

A2 - Cotterell, Ryan

A2 - Vylomova, Ekaterina

PB - Association for Computational Linguistics

Y2 - 6 May 2023

ER -

Limisiewicz T, Malkin D, Stanovsky G. You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models. In Beinborn L, Goswami K, Muradoglu S, Sorokin A, Kumar R, Shcherbakov A, Ponti EM, Cotterell R, Vylomova E, editors, SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop. Association for Computational Linguistics. 2023. p. 1-11. (SIGTYP 2023 - 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Proceedings of the Workshop).

You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models

Abstract

Publication series

Conference

Bibliographical note

Funding

Other files and links

Fingerprint

Cite this