Demystifying Prompts in Language Models via Perplexity Estimation

Hila Gonen; Srini Iyer; Terra Blevins; Noah A. Smith; Luke Zettlemoyer

doi:10.18653/v1/2023.findings-emnlp.679

Demystifying Prompts in Language Models via Perplexity Estimation

Hila Gonen
, Srini Iyer
, Terra Blevins
, Noah A. Smith
, Luke Zettlemoyer

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

58 Scopus citations

Abstract

Language models can be prompted to perform a wide variety of tasks with zero- and few-shot in-context learning. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens. In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it. As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EMNLP 2023
Publisher	Association for Computational Linguistics (ACL)
Pages	10136-10148
Number of pages	13
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.679
State	Published - 2023
Externally published	Yes
Event	2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Hybrid, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Publication series

Name	Findings of the Association for Computational Linguistics: EMNLP 2023

Conference

Conference	2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/Territory	Singapore
City	Hybrid
Period	6/12/23 → 10/12/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Funding

We thank Alisa Liu and Orevaoghene Ahia for their help in annotating noisy prompts. We also thank the reviewers for their valuable comments on the paper.

Access to Document

10.18653/v1/2023.findings-emnlp.679

Cite this

Gonen, H., Iyer, S., Blevins, T., Smith, N. A., & Zettlemoyer, L. (2023). Demystifying Prompts in Language Models via Perplexity Estimation. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 10136-10148). (Findings of the Association for Computational Linguistics: EMNLP 2023). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-emnlp.679

@inproceedings{f593ba9d760242a9a5e6270963c8e9b9,

title = "Demystifying Prompts in Language Models via Perplexity Estimation",

abstract = "Language models can be prompted to perform a wide variety of tasks with zero- and few-shot in-context learning. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens. In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it. As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.",

author = "Hila Gonen and Srini Iyer and Terra Blevins and Smith, \{Noah A.\} and Luke Zettlemoyer",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Findings of the Association for Computational Linguistics: EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.679",

language = "אנגלית",

series = "Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

pages = "10136--10148",

booktitle = "Findings of the Association for Computational Linguistics",

address = "ארצות הברית",

}

Gonen, H, Iyer, S, Blevins, T, Smith, NA & Zettlemoyer, L 2023, Demystifying Prompts in Language Models via Perplexity Estimation. in Findings of the Association for Computational Linguistics: EMNLP 2023. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics (ACL), pp. 10136-10148, 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, Hybrid, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.679

Demystifying Prompts in Language Models via Perplexity Estimation. / Gonen, Hila; Iyer, Srini; Blevins, Terra et al.
Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), 2023. p. 10136-10148 (Findings of the Association for Computational Linguistics: EMNLP 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Demystifying Prompts in Language Models via Perplexity Estimation

AU - Gonen, Hila

AU - Iyer, Srini

AU - Blevins, Terra

AU - Smith, Noah A.

AU - Zettlemoyer, Luke

PY - 2023

Y1 - 2023

N2 - Language models can be prompted to perform a wide variety of tasks with zero- and few-shot in-context learning. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens. In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it. As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.

AB - Language models can be prompted to perform a wide variety of tasks with zero- and few-shot in-context learning. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens. In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it. As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.

UR - https://www.scopus.com/pages/publications/85183303783

U2 - 10.18653/v1/2023.findings-emnlp.679

DO - 10.18653/v1/2023.findings-emnlp.679

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85183303783

T3 - Findings of the Association for Computational Linguistics: EMNLP 2023

SP - 10136

EP - 10148

BT - Findings of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2023 Findings of the Association for Computational Linguistics: EMNLP 2023

Y2 - 6 December 2023 through 10 December 2023

ER -

Gonen H, Iyer S, Blevins T, Smith NA, Zettlemoyer L. Demystifying Prompts in Language Models via Perplexity Estimation. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL). 2023. p. 10136-10148. (Findings of the Association for Computational Linguistics: EMNLP 2023). doi: 10.18653/v1/2023.findings-emnlp.679

Demystifying Prompts in Language Models via Perplexity Estimation

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this