MEMECAP: A Dataset for Captioning and Interpreting Memes

Eun Jeong Hwang; Vered Shwartz

doi:10.18653/v1/2023.emnlp-main.89

MEMECAP: A Dataset for Captioning and Interpreting Memes

Eun Jeong Hwang, Vered Shwartz

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

Memes are a widely popular tool for web users to express their thoughts using visual metaphors. Understanding memes requires recognizing and interpreting visual metaphors with respect to the text inside or around the meme, often while employing background knowledge and reasoning abilities. We present the task of meme captioning and release a new dataset, MEMECAP. Our dataset contains 6.3K memes along with the title of the post containing the meme, the meme captions, the literal image caption, and the visual metaphors. Despite the recent success of vision and language (VL) models on tasks such as image captioning and visual question answering, our extensive experiments using state-of-the-art VL models show that they still struggle with visual metaphors, and perform substantially worse than humans.

Original language	English
Title of host publication	EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings
Editors	Houda Bouamor, Juan Pino, Kalika Bali
Publisher	Association for Computational Linguistics (ACL)
Pages	1433-1445
Number of pages	13
ISBN (Electronic)	9798891760608
DOIs	https://doi.org/10.18653/v1/2023.emnlp-main.89
State	Published - 2023
Externally published	Yes
Event	2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - Hybrid, Singapore, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Publication series

Name	EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

Conference

Conference	2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023
Country/Territory	Singapore
City	Hybrid, Singapore
Period	6/12/23 → 10/12/23

Bibliographical note

Publisher Copyright:
©2023 Association for Computational Linguistics.

Funding

This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chairs program, an NSERC discovery grant, and a research gift from AI2.

Funders	Funder number
Natural Sciences and Engineering Research Council of Canada
Vector Institute

Access to Document

10.18653/v1/2023.emnlp-main.89

Cite this

Hwang, E. J., & Shwartz, V. (2023). MEMECAP: A Dataset for Captioning and Interpreting Memes. In H. Bouamor, J. Pino, & K. Bali (Eds.), EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 1433-1445). (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.emnlp-main.89

Hwang, Eun Jeong ; Shwartz, Vered. / MEMECAP : A Dataset for Captioning and Interpreting Memes. EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. editor / Houda Bouamor ; Juan Pino ; Kalika Bali. Association for Computational Linguistics (ACL), 2023. pp. 1433-1445 (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

@inproceedings{dec3800a91f343b5a1a2390bf61bda74,

title = "MEMECAP: A Dataset for Captioning and Interpreting Memes",

abstract = "Memes are a widely popular tool for web users to express their thoughts using visual metaphors. Understanding memes requires recognizing and interpreting visual metaphors with respect to the text inside or around the meme, often while employing background knowledge and reasoning abilities. We present the task of meme captioning and release a new dataset, MEMECAP. Our dataset contains 6.3K memes along with the title of the post containing the meme, the meme captions, the literal image caption, and the visual metaphors. Despite the recent success of vision and language (VL) models on tasks such as image captioning and visual question answering, our extensive experiments using state-of-the-art VL models show that they still struggle with visual metaphors, and perform substantially worse than humans.",

author = "Hwang, {Eun Jeong} and Vered Shwartz",

note = "Publisher Copyright: {\textcopyright}2023 Association for Computational Linguistics.; 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.emnlp-main.89",

language = "אנגלית",

series = "EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1433--1445",

editor = "Houda Bouamor and Juan Pino and Kalika Bali",

booktitle = "EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings",

address = "ארצות הברית",

}

Hwang, EJ & Shwartz, V 2023, MEMECAP: A Dataset for Captioning and Interpreting Memes. in H Bouamor, J Pino & K Bali (eds), EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings, Association for Computational Linguistics (ACL), pp. 1433-1445, 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Hybrid, Singapore, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.emnlp-main.89

MEMECAP: A Dataset for Captioning and Interpreting Memes. / Hwang, Eun Jeong; Shwartz, Vered.
EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. ed. / Houda Bouamor; Juan Pino; Kalika Bali. Association for Computational Linguistics (ACL), 2023. p. 1433-1445 (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - MEMECAP

T2 - 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023

AU - Hwang, Eun Jeong

AU - Shwartz, Vered

PY - 2023

Y1 - 2023

N2 - Memes are a widely popular tool for web users to express their thoughts using visual metaphors. Understanding memes requires recognizing and interpreting visual metaphors with respect to the text inside or around the meme, often while employing background knowledge and reasoning abilities. We present the task of meme captioning and release a new dataset, MEMECAP. Our dataset contains 6.3K memes along with the title of the post containing the meme, the meme captions, the literal image caption, and the visual metaphors. Despite the recent success of vision and language (VL) models on tasks such as image captioning and visual question answering, our extensive experiments using state-of-the-art VL models show that they still struggle with visual metaphors, and perform substantially worse than humans.

AB - Memes are a widely popular tool for web users to express their thoughts using visual metaphors. Understanding memes requires recognizing and interpreting visual metaphors with respect to the text inside or around the meme, often while employing background knowledge and reasoning abilities. We present the task of meme captioning and release a new dataset, MEMECAP. Our dataset contains 6.3K memes along with the title of the post containing the meme, the meme captions, the literal image caption, and the visual metaphors. Despite the recent success of vision and language (VL) models on tasks such as image captioning and visual question answering, our extensive experiments using state-of-the-art VL models show that they still struggle with visual metaphors, and perform substantially worse than humans.

UR - http://www.scopus.com/inward/record.url?scp=85184812429&partnerID=8YFLogxK

U2 - 10.18653/v1/2023.emnlp-main.89

DO - 10.18653/v1/2023.emnlp-main.89

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85184812429

T3 - EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

SP - 1433

EP - 1445

BT - EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

A2 - Bouamor, Houda

A2 - Pino, Juan

A2 - Bali, Kalika

PB - Association for Computational Linguistics (ACL)

Y2 - 6 December 2023 through 10 December 2023

ER -

Hwang EJ, Shwartz V. MEMECAP: A Dataset for Captioning and Interpreting Memes. In Bouamor H, Pino J, Bali K, editors, EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL). 2023. p. 1433-1445. (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings). doi: 10.18653/v1/2023.emnlp-main.89

MEMECAP: A Dataset for Captioning and Interpreting Memes

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this