DenoisingWord Embeddings by Averaging in a Shared Space

Avi Caciularu; Ido Dagan; Jacob Goldberger

DenoisingWord Embeddings by Averaging in a Shared Space

Avi Caciularu, Ido Dagan, Jacob Goldberger

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.

Original language	English
Title of host publication	*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference
Editors	Lun-Wei Ku, Vivi Nastase, Ivan Vulic
Publisher	Association for Computational Linguistics (ACL)
Pages	294-301
Number of pages	8
ISBN (Electronic)	9781954085770
State	Published - 2021
Event	10th Conference on Lexical and Computational Semantics, *SEM 2021 - Virtual, Bangkok, Thailand Duration: 5 Aug 2021 → 6 Aug 2021

Publication series

Name	*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference

Conference

Conference	10th Conference on Lexical and Computational Semantics, *SEM 2021
Country/Territory	Thailand
City	Virtual, Bangkok
Period	5/08/21 → 6/08/21

Bibliographical note

Publisher Copyright:
© 2021 Lexical and Computational Semantics

Funding

The authors would like to thank the anonymous reviewers for their comments and suggestions. The work described herein was supported in part by grants from Intel Labs, Facebook, the Israel Science Foundation grant 1951/17 and the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1).

Funders	Funder number
DIP	DA 1600/1-1
German-Israeli Project Cooperation
Intel Labs
Deutsche Forschungsgemeinschaft
Israel Science Foundation	1951/17

Cite this

Caciularu, A., Dagan, I., & Goldberger, J. (2021). DenoisingWord Embeddings by Averaging in a Shared Space. In L.-W. Ku, V. Nastase, & I. Vulic (Eds.), *SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference (pp. 294-301). (*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference). Association for Computational Linguistics (ACL).

Caciularu, Avi ; Dagan, Ido ; Goldberger, Jacob. / DenoisingWord Embeddings by Averaging in a Shared Space. *SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference. editor / Lun-Wei Ku ; Vivi Nastase ; Ivan Vulic. Association for Computational Linguistics (ACL), 2021. pp. 294-301 (*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference).

@inproceedings{74be636fafd94be58e3fc8388f691d25,

title = "DenoisingWord Embeddings by Averaging in a Shared Space",

abstract = "We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.",

author = "Avi Caciularu and Ido Dagan and Jacob Goldberger",

note = "Publisher Copyright: {\textcopyright} 2021 Lexical and Computational Semantics; 10th Conference on Lexical and Computational Semantics, *SEM 2021 ; Conference date: 05-08-2021 Through 06-08-2021",

year = "2021",

language = "אנגלית",

series = "*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "294--301",

editor = "Lun-Wei Ku and Vivi Nastase and Ivan Vulic",

booktitle = "*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference",

address = "ארצות הברית",

}

Caciularu, A, Dagan, I & Goldberger, J 2021, DenoisingWord Embeddings by Averaging in a Shared Space. in L-W Ku, V Nastase & I Vulic (eds), *SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference. *SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 294-301, 10th Conference on Lexical and Computational Semantics, *SEM 2021, Virtual, Bangkok, Thailand, 5/08/21.

DenoisingWord Embeddings by Averaging in a Shared Space. / Caciularu, Avi; Dagan, Ido ; Goldberger, Jacob.
*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference. ed. / Lun-Wei Ku; Vivi Nastase; Ivan Vulic. Association for Computational Linguistics (ACL), 2021. p. 294-301 (*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - DenoisingWord Embeddings by Averaging in a Shared Space

AU - Caciularu, Avi

AU - Dagan, Ido

AU - Goldberger, Jacob

PY - 2021

Y1 - 2021

N2 - We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.

AB - We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.

UR - http://www.scopus.com/inward/record.url?scp=85119199482&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85119199482

T3 - *SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference

SP - 294

EP - 301

BT - *SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference

A2 - Ku, Lun-Wei

A2 - Nastase, Vivi

A2 - Vulic, Ivan

PB - Association for Computational Linguistics (ACL)

T2 - 10th Conference on Lexical and Computational Semantics, *SEM 2021

Y2 - 5 August 2021 through 6 August 2021

ER -

Caciularu A, Dagan I , Goldberger J. DenoisingWord Embeddings by Averaging in a Shared Space. In Ku LW, Nastase V, Vulic I, editors, *SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2021. p. 294-301. (*SEM 2021 - 10th Conference on Lexical and Computational Semantics, Proceedings of the Conference).

DenoisingWord Embeddings by Averaging in a Shared Space

Abstract

Publication series

Conference

Bibliographical note

Funding

Other files and links

Fingerprint

Cite this