Do language embeddings capture scales?

Xikun Zhang; Deepak Ramachandran; Ian Tenney; Yanai Elazar; Dan Roth

doi:10.18653/v1/2020.findings-emnlp.439

Do language embeddings capture scales?

Xikun Zhang
, Deepak Ramachandran
, Ian Tenney
, Yanai Elazar
, Dan Roth

Department of Computer Science and Artificial Intelligence

Stanford University
Alphabet Inc.
University of Pennsylvania

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

44 Scopus citations

Abstract

Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key factors affecting their performance, and show that a simple method of canonicalizing numbers can have a significant effect on the results.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics Findings of ACL
Subtitle of host publication	EMNLP 2020
Publisher	Association for Computational Linguistics (ACL)
Pages	4889-4896
Number of pages	8
ISBN (Electronic)	9781952148903
DOIs	https://doi.org/10.18653/v1/2020.findings-emnlp.439
State	Published - 2020
Event	Findings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020 - Virtual, Online Duration: 16 Nov 2020 → 20 Nov 2020

Publication series

Name	Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020

Conference

Conference	Findings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020
City	Virtual, Online
Period	16/11/20 → 20/11/20

Bibliographical note

Publisher Copyright:
© 2020 Association for Computational Linguistics

Access to Document

10.18653/v1/2020.findings-emnlp.439

Cite this

Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020). Do language embeddings capture scales? In Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020 (pp. 4889-4896). (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2020.findings-emnlp.439

@inproceedings{507408d6a75f4a21b3beedc7c472f5f9,

title = "Do language embeddings capture scales?",

abstract = "Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key factors affecting their performance, and show that a simple method of canonicalizing numbers can have a significant effect on the results.",

author = "Xikun Zhang and Deepak Ramachandran and Ian Tenney and Yanai Elazar and Dan Roth",

note = "Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; Findings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020 ; Conference date: 16-11-2020 Through 20-11-2020",

year = "2020",

doi = "10.18653/v1/2020.findings-emnlp.439",

language = "אנגלית",

series = "Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020",

publisher = "Association for Computational Linguistics (ACL)",

pages = "4889--4896",

booktitle = "Findings of the Association for Computational Linguistics Findings of ACL",

address = "ארצות הברית",

}

Zhang, X, Ramachandran, D, Tenney, I, Elazar, Y & Roth, D 2020, Do language embeddings capture scales? in Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020, Association for Computational Linguistics (ACL), pp. 4889-4896, Findings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020, Virtual, Online, 16/11/20. https://doi.org/10.18653/v1/2020.findings-emnlp.439

Do language embeddings capture scales? / Zhang, Xikun; Ramachandran, Deepak; Tenney, Ian et al.
Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. Association for Computational Linguistics (ACL), 2020. p. 4889-4896 (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review