Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?

Weijia Shi; Xiaochuang Han; Hila Gonen; Ari Holtzman; Yulia Tsvetkov; Luke Zettlemoyer

doi:10.18653/v1/2023.findings-emnlp.733

Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?

Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, Luke Zettlemoyer

University of Washington

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are in natural language. In this paper, we investigate common attributes shared by effective prompts in classification problems. We first propose a human readable prompt tuning method (FLUENTPROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of output labels. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks. We release our code and data in github.com/swj0419/FluentPrompt.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EMNLP 2023
Publisher	Association for Computational Linguistics (ACL)
Pages	10994-11005
Number of pages	12
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.733
State	Published - 2023
Externally published	Yes
Event	2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Hybrid, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Publication series

Name	Findings of the Association for Computational Linguistics: EMNLP 2023

Conference

Conference	2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/Territory	Singapore
City	Hybrid
Period	6/12/23 → 10/12/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Funding

We gratefully acknowledge support from NSF CAREER Grant No. IIS2142739. This material is funded in part by the DARPA Grant under Contract No. HR001120C0124. This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Funders	Funder number
National Science Foundation	IIS2142739
Defense Advanced Research Projects Agency	HR001120C0124
Office of the Director of National Intelligence
Intelligence Advanced Research Projects Activity	2022-22072200004

Access to Document

10.18653/v1/2023.findings-emnlp.733

Cite this

Shi, W., Han, X., Gonen, H., Holtzman, A., Tsvetkov, Y., & Zettlemoyer, L. (2023). Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too? In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 10994-11005). (Findings of the Association for Computational Linguistics: EMNLP 2023). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-emnlp.733

@inproceedings{04dfb51d9b62470e8f37b614afb8e6a9,

title = "Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?",

abstract = "Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are in natural language. In this paper, we investigate common attributes shared by effective prompts in classification problems. We first propose a human readable prompt tuning method (FLUENTPROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of output labels. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0\% accuracy across three tasks. We release our code and data in github.com/swj0419/FluentPrompt.",

author = "Weijia Shi and Xiaochuang Han and Hila Gonen and Ari Holtzman and Yulia Tsvetkov and Luke Zettlemoyer",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Findings of the Association for Computational Linguistics: EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.733",

language = "אנגלית",

series = "Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

pages = "10994--11005",

booktitle = "Findings of the Association for Computational Linguistics",

address = "ארצות הברית",

}

Shi, W, Han, X, Gonen, H, Holtzman, A, Tsvetkov, Y & Zettlemoyer, L 2023, Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too? in Findings of the Association for Computational Linguistics: EMNLP 2023. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics (ACL), pp. 10994-11005, 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, Hybrid, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.733

Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too? / Shi, Weijia; Han, Xiaochuang; Gonen, Hila et al.
Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), 2023. p. 10994-11005 (Findings of the Association for Computational Linguistics: EMNLP 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Toward Human Readable Prompt Tuning

T2 - 2023 Findings of the Association for Computational Linguistics: EMNLP 2023

AU - Shi, Weijia

AU - Han, Xiaochuang

AU - Gonen, Hila

AU - Holtzman, Ari

AU - Tsvetkov, Yulia

AU - Zettlemoyer, Luke

PY - 2023

Y1 - 2023

N2 - Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are in natural language. In this paper, we investigate common attributes shared by effective prompts in classification problems. We first propose a human readable prompt tuning method (FLUENTPROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of output labels. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks. We release our code and data in github.com/swj0419/FluentPrompt.

AB - Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are in natural language. In this paper, we investigate common attributes shared by effective prompts in classification problems. We first propose a human readable prompt tuning method (FLUENTPROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of output labels. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks. We release our code and data in github.com/swj0419/FluentPrompt.

UR - https://www.scopus.com/pages/publications/85183299198

U2 - 10.18653/v1/2023.findings-emnlp.733

DO - 10.18653/v1/2023.findings-emnlp.733

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85183299198

T3 - Findings of the Association for Computational Linguistics: EMNLP 2023

SP - 10994

EP - 11005

BT - Findings of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

Y2 - 6 December 2023 through 10 December 2023

ER -

Shi W, Han X, Gonen H, Holtzman A, Tsvetkov Y, Zettlemoyer L. Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too? In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL). 2023. p. 10994-11005. (Findings of the Association for Computational Linguistics: EMNLP 2023). doi: 10.18653/v1/2023.findings-emnlp.733

Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this