A simple language model based on PMI matrix approximations

Oren Melamud; Ido Dagan; Jacob Goldberger

doi:10.18653/v1/d17-1198

A simple language model based on PMI matrix approximations

Oren Melamud, Ido Dagan, Jacob Goldberger

IBM

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then deriving the desired conditional probabilities from PMI at test time. Specifically, we show that with minor modifications to word2vec’s algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same simple negative sampling objective function that is commonly used in word2vec to learn word embeddings.

Original language	English
Title of host publication	EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings
Publisher	Association for Computational Linguistics (ACL)
Pages	1860-1865
Number of pages	6
ISBN (Electronic)	9781945626838
DOIs	https://doi.org/10.18653/v1/d17-1198
State	Published - 2017
Event	2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017 - Copenhagen, Denmark Duration: 9 Sep 2017 → 11 Sep 2017

Publication series

Name	EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings

Conference

Conference	2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017
Country/Territory	Denmark
City	Copenhagen
Period	9/09/17 → 11/09/17

Bibliographical note

Publisher Copyright:
© 2017 Association for Computational Linguistics.

Funding

This work is supported by the Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI).

Funders	Funder number
Intel Collaboration Research Institute for Computational Intelligence

Access to Document

10.18653/v1/d17-1198

Cite this

Melamud, O., Dagan, I., & Goldberger, J. (2017). A simple language model based on PMI matrix approximations. In EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 1860-1865). (EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/d17-1198

@inproceedings{debd64ddebf1458e966d363e866ebd69,

title = "A simple language model based on PMI matrix approximations",

abstract = "In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then deriving the desired conditional probabilities from PMI at test time. Specifically, we show that with minor modifications to word2vec{\textquoteright}s algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same simple negative sampling objective function that is commonly used in word2vec to learn word embeddings.",

author = "Oren Melamud and Ido Dagan and Jacob Goldberger",

note = "Publisher Copyright: {\textcopyright} 2017 Association for Computational Linguistics.; 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017 ; Conference date: 09-09-2017 Through 11-09-2017",

year = "2017",

doi = "10.18653/v1/d17-1198",

language = "אנגלית",

series = "EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1860--1865",

booktitle = "EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings",

address = "ארצות הברית",

}

Melamud, O, Dagan, I & Goldberger, J 2017, A simple language model based on PMI matrix approximations. in EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings. EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings, Association for Computational Linguistics (ACL), pp. 1860-1865, 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, 9/09/17. https://doi.org/10.18653/v1/d17-1198

A simple language model based on PMI matrix approximations. / Melamud, Oren; Dagan, Ido ; Goldberger, Jacob.
EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL), 2017. p. 1860-1865 (EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A simple language model based on PMI matrix approximations

AU - Melamud, Oren

AU - Dagan, Ido

AU - Goldberger, Jacob

PY - 2017

Y1 - 2017

N2 - In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then deriving the desired conditional probabilities from PMI at test time. Specifically, we show that with minor modifications to word2vec’s algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same simple negative sampling objective function that is commonly used in word2vec to learn word embeddings.

AB - In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then deriving the desired conditional probabilities from PMI at test time. Specifically, we show that with minor modifications to word2vec’s algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same simple negative sampling objective function that is commonly used in word2vec to learn word embeddings.

UR - http://www.scopus.com/inward/record.url?scp=85070386835&partnerID=8YFLogxK

U2 - 10.18653/v1/d17-1198

DO - 10.18653/v1/d17-1198

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85070386835

T3 - EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings

SP - 1860

EP - 1865

BT - EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings

PB - Association for Computational Linguistics (ACL)

T2 - 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017

Y2 - 9 September 2017 through 11 September 2017

ER -

Melamud O, Dagan I , Goldberger J. A simple language model based on PMI matrix approximations. In EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL). 2017. p. 1860-1865. (EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings). doi: 10.18653/v1/d17-1198

A simple language model based on PMI matrix approximations

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this