NeuroPrompts: An Adaptive Framework to Optimize Prompts for Text-to-Image Generation

Shachar Rosenman, Vasudev Lal, Phillip Howard

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Despite impressive recent advances in text-to-image diffusion models, obtaining high-quality images often requires prompt engineering by humans who have developed expertise in using them. In this work, we present NeuroPrompts, an adaptive framework that automatically enhances a user’s prompt to improve the quality of generations produced by text-to-image models. Our framework utilizes constrained text decoding with a pre-trained language model that has been adapted to generate prompts similar to those produced by human prompt engineers. This approach enables higher-quality text-to-image generations and provides user control over stylistic features via constraint set specification. We demonstrate the utility of our framework by creating an interactive application for prompt enhancement and image generation using Stable Diffusion. Additionally, we conduct experiments utilizing a large dataset of human-engineered prompts for text-to-image generation and show that our approach automatically produces enhanced prompts that result in superior image quality. We make our code1 and a screencast video demo2 of NeuroPrompts publicly available.

Original languageEnglish
Title of host publicationEACL 2024 - 18th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of System Demonstrations
EditorsNikolaos Aletras, Orphee De Clercq
PublisherAssociation for Computational Linguistics (ACL)
Pages159-167
Number of pages9
ISBN (Electronic)9798891760912
StatePublished - 2024
Externally publishedYes
Event18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - St. Julian's, Malta
Duration: 17 Mar 202422 Mar 2024

Publication series

NameEACL 2024 - 18th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of System Demonstrations

Conference

Conference18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024
Country/TerritoryMalta
CitySt. Julian's
Period17/03/2422/03/24

Bibliographical note

Publisher Copyright:
© 2024 Association for Computational Linguistics.

Fingerprint

Dive into the research topics of 'NeuroPrompts: An Adaptive Framework to Optimize Prompts for Text-to-Image Generation'. Together they form a unique fingerprint.

Cite this