Evaluating text gans as language models

Guy Tevet; Gavriel Habib; Vered Shwartz; Jonathan Berant

Evaluating text gans as language models

Guy Tevet, Gavriel Habib, Vered Shwartz, Jonathan Berant

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of “exposure bias”. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.

Original language	English
Title of host publication	Long and Short Papers
Publisher	Association for Computational Linguistics (ACL)
Pages	2241-2247
Number of pages	7
ISBN (Electronic)	9781950737130
State	Published - 2019
Event	2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019 - Minneapolis, United States Duration: 2 Jun 2019 → 7 Jun 2019

Publication series

Name	NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume	1

Conference

Conference	2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019
Country/Territory	United States
City	Minneapolis
Period	2/06/19 → 7/06/19

Bibliographical note

Publisher Copyright:
© 2019 Association for Computational Linguistics

Funding

We would like to thank Shimi Salant for his comments and suggestions. This research was partially supported by The Israel Science Foundation grant 942/16, the Blavatnik Computer Science Research Fund, and The Yandex Initiative for Machine Learning.

Funders	Funder number
Blavatnik Computer Science Research Fund
Yandex Initiative for Machine Learning
Israel Science Foundation	942/16

Cite this

Tevet, G., Habib, G., Shwartz, V., & Berant, J. (2019). Evaluating text gans as language models. In Long and Short Papers (pp. 2241-2247). (NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1). Association for Computational Linguistics (ACL).

@inproceedings{8ba52969856b4ca2baf407aec2734e8e,

title = "Evaluating text gans as language models",

abstract = "Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of “exposure bias”. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.",

author = "Guy Tevet and Gavriel Habib and Vered Shwartz and Jonathan Berant",

note = "Publisher Copyright: {\textcopyright} 2019 Association for Computational Linguistics; 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019 ; Conference date: 02-06-2019 Through 07-06-2019",

year = "2019",

language = "אנגלית",

series = "NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "2241--2247",

booktitle = "Long and Short Papers",

address = "ארצות הברית",

}

Tevet, G, Habib, G, Shwartz, V & Berant, J 2019, Evaluating text gans as language models. in Long and Short Papers. NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, Association for Computational Linguistics (ACL), pp. 2241-2247, 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019, Minneapolis, United States, 2/06/19.

Evaluating text gans as language models. / Tevet, Guy; Habib, Gavriel; Shwartz, Vered et al.
Long and Short Papers. Association for Computational Linguistics (ACL), 2019. p. 2241-2247 (NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Evaluating text gans as language models

AU - Tevet, Guy

AU - Habib, Gavriel

AU - Shwartz, Vered

AU - Berant, Jonathan

PY - 2019

Y1 - 2019

N2 - Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of “exposure bias”. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.

AB - Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of “exposure bias”. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.

UR - http://www.scopus.com/inward/record.url?scp=85085547602&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85085547602

T3 - NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference

SP - 2241

EP - 2247

BT - Long and Short Papers

PB - Association for Computational Linguistics (ACL)

T2 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019

Y2 - 2 June 2019 through 7 June 2019

ER -

Evaluating text gans as language models

Abstract

Publication series

Conference

Bibliographical note

Funding

Other files and links

Fingerprint

Cite this