Abstract
Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.
| Original language | English |
|---|---|
| Title of host publication | Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023 |
| Editors | A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, S. Levine |
| Publisher | Neural information processing systems foundation |
| ISBN (Electronic) | 9781713899921 |
| State | Published - 2023 |
| Externally published | Yes |
| Event | 37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States Duration: 10 Dec 2023 → 16 Dec 2023 |
Publication series
| Name | Advances in Neural Information Processing Systems |
|---|---|
| Volume | 36 |
| ISSN (Print) | 1049-5258 |
Conference
| Conference | 37th Conference on Neural Information Processing Systems, NeurIPS 2023 |
|---|---|
| Country/Territory | United States |
| City | New Orleans |
| Period | 10/12/23 → 16/12/23 |
Bibliographical note
Publisher Copyright:© 2023 Neural information processing systems foundation. All rights reserved.
Fingerprint
Dive into the research topics of 'What You See is What You Read? Improving Text-Image Alignment Evaluation'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver