The Spectral Underpinning of word2vec

Ariel Jaffe; Yuval Kluger; Ofir Lindenbaum; Jonathan Patsenker; Erez Peterfreund; Stefan Steinerberger

doi:10.3389/fams.2020.593406

The Spectral Underpinning of word2vec

Ariel Jaffe, Yuval Kluger, Ofir Lindenbaum, Jonathan Patsenker, Erez Peterfreund, Stefan Steinerberger

Research output: Contribution to journal › Article › peer-review

8 Scopus citations

Abstract

Word2vec introduced by Mikolov et al. is a word embedding method that is widely used in natural language processing. Despite its success and frequent use, a strong theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We support these findings by numerical simulations. One fascinating open question is whether the nonlinear properties of word2vec that are not captured by the spectral method are beneficial and, if so, by what mechanism.

Original language	English
Article number	593406
Journal	Frontiers in Applied Mathematics and Statistics
Volume	6
DOIs	https://doi.org/10.3389/fams.2020.593406
State	Published - 3 Dec 2020
Externally published	Yes

Bibliographical note

Publisher Copyright:
© Copyright © 2020 Jaffe, Kluger, Lindenbaum, Patsenker, Peterfreund and Steinerberger.

Funding

YK is supported in part by NIH grants UM1DA051410, R01GM131642, P50CA121974 and R61DA047037. SS was funded by NSF-DMS 1763179 and the Alfred P. Sloan Foundation. EP has been partially supported by the Blavatnik Interdisciplinary Research Center (ICRC), the Federmann Research Center (Hebrew University) Israeli Science Foundation research grant no. 1523/16, and by the DARPA PAI program (Agreement No. HR00111890032, Dr. T. Senator). The authors thank James Garritano and the anonymous reviewers for their helpful feedback.

Funders	Funder number
Blavatnik Interdisciplinary Research Center
Federmann Research Center
Hebrew University) Israeli Science Foundation	1523/16
ICRC
James Garritano
NSF-DMS	1763179
National Institutes of Health	UM1DA051410, P50CA121974, R61DA047037, R01GM131642
Defense Advanced Research Projects Agency	HR00111890032
Alfred P. Sloan Foundation

Keywords

dimensionality reduction
nonlinear functional
skip-gram model
spectral method
word embedding
word2vec

Access to Document

10.3389/fams.2020.593406

Cite this

@article{bb0d8b7bb1f54e22bb6cc47d60dcd222,

title = "The Spectral Underpinning of word2vec",

abstract = "Word2vec introduced by Mikolov et al. is a word embedding method that is widely used in natural language processing. Despite its success and frequent use, a strong theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We support these findings by numerical simulations. One fascinating open question is whether the nonlinear properties of word2vec that are not captured by the spectral method are beneficial and, if so, by what mechanism.",

keywords = "dimensionality reduction, nonlinear functional, skip-gram model, spectral method, word embedding, word2vec",

author = "Ariel Jaffe and Yuval Kluger and Ofir Lindenbaum and Jonathan Patsenker and Erez Peterfreund and Stefan Steinerberger",

note = "Publisher Copyright: {\textcopyright} Copyright {\textcopyright} 2020 Jaffe, Kluger, Lindenbaum, Patsenker, Peterfreund and Steinerberger.",

year = "2020",

month = dec,

day = "3",

doi = "10.3389/fams.2020.593406",

language = "אנגלית",

volume = "6",

journal = "Frontiers in Applied Mathematics and Statistics",

issn = "2297-4687",

publisher = "Frontiers Media SA",

}

TY - JOUR

T1 - The Spectral Underpinning of word2vec

AU - Jaffe, Ariel

AU - Kluger, Yuval

AU - Lindenbaum, Ofir

AU - Patsenker, Jonathan

AU - Peterfreund, Erez

AU - Steinerberger, Stefan

PY - 2020/12/3

Y1 - 2020/12/3

N2 - Word2vec introduced by Mikolov et al. is a word embedding method that is widely used in natural language processing. Despite its success and frequent use, a strong theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We support these findings by numerical simulations. One fascinating open question is whether the nonlinear properties of word2vec that are not captured by the spectral method are beneficial and, if so, by what mechanism.

AB - Word2vec introduced by Mikolov et al. is a word embedding method that is widely used in natural language processing. Despite its success and frequent use, a strong theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We support these findings by numerical simulations. One fascinating open question is whether the nonlinear properties of word2vec that are not captured by the spectral method are beneficial and, if so, by what mechanism.

KW - dimensionality reduction

KW - nonlinear functional

KW - skip-gram model

KW - spectral method

KW - word embedding

KW - word2vec

UR - http://www.scopus.com/inward/record.url?scp=85098129182&partnerID=8YFLogxK

U2 - 10.3389/fams.2020.593406

DO - 10.3389/fams.2020.593406

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

C2 - 34504892

AN - SCOPUS:85098129182

SN - 2297-4687

VL - 6

JO - Frontiers in Applied Mathematics and Statistics

JF - Frontiers in Applied Mathematics and Statistics

M1 - 593406

ER -

The Spectral Underpinning of word2vec

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this