VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge

Sahithya Ravi; Aditya Chinchure; Leonid Sigal; Renjie Liao; Vered Shwartz

doi:10.1109/WACV56688.2023.00121

VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge

Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, Vered Shwartz

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

29 Scopus citations

Abstract

There has been a growing interest in solving Visual Question Answering (VQA) tasks that require the model to reason beyond the content present in the image. In this work, we focus on questions that require commonsense reasoning. In contrast to previous methods which inject knowledge from static knowledge bases, we investigate the incorporation of contextualized knowledge using Commonsense Transformer (COMET), an existing knowledge model trained on human-curated knowledge bases. We propose a method to generate, select, and encode external commonsense knowledge alongside visual and textual cues in a new pre-trained Vision-Language-Commonsense transformer model, VLC-BERT. Through our evaluation on the knowledge-intensive OK-VQA and A-OKVQA datasets, we show that VLC-BERT is capable of outperforming existing models that utilize static knowledge bases. Furthermore, through a detailed analysis, we explain which questions benefit, and which don't, from contextualized commonsense knowledge from COMET. Code: https://github.com/aditya10/VLC-BERT

Original language	English
Title of host publication	Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1155-1165
Number of pages	11
ISBN (Electronic)	9781665493468
DOIs	https://doi.org/10.1109/WACV56688.2023.00121
State	Published - 2023
Externally published	Yes
Event	23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023 - Waikoloa, United States Duration: 3 Jan 2023 → 7 Jan 2023

Publication series

Name	Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023

Conference

Conference	23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023
Country/Territory	United States
City	Waikoloa
Period	3/01/23 → 7/01/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Funding

This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC CRC, NSERC DG and Accelerator Grants, and a research gift from AI2. Hardware resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute4. Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition award. Finally, we sincerely thank Prof. Giuseppe Carenini for valuable feedback and discussions.

Funders	Funder number
John R. Evans Leaders Fund CFI
Canadian Institute for Advanced Research
Compute Canada
Government of Ontario
Natural Sciences and Engineering Research Council of Canada
Vector Institute

Keywords

Algorithms: Vision + language and/or other modalities
Image recognition and understanding (object detection, categorization, segmentation, scene modeling, visual reasoning)

Access to Document

10.1109/WACV56688.2023.00121

Cite this

Ravi, S., Chinchure, A., Sigal, L., Liao, R., & Shwartz, V. (2023). VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge. In Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023 (pp. 1155-1165). (Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/WACV56688.2023.00121

Ravi, Sahithya ; Chinchure, Aditya ; Sigal, Leonid et al. / VLC-BERT : Visual Question Answering with Contextualized Commonsense Knowledge. Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 1155-1165 (Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023).

@inproceedings{5029c57b5ef34ab3aa900489a1aa26d7,

title = "VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge",

abstract = "There has been a growing interest in solving Visual Question Answering (VQA) tasks that require the model to reason beyond the content present in the image. In this work, we focus on questions that require commonsense reasoning. In contrast to previous methods which inject knowledge from static knowledge bases, we investigate the incorporation of contextualized knowledge using Commonsense Transformer (COMET), an existing knowledge model trained on human-curated knowledge bases. We propose a method to generate, select, and encode external commonsense knowledge alongside visual and textual cues in a new pre-trained Vision-Language-Commonsense transformer model, VLC-BERT. Through our evaluation on the knowledge-intensive OK-VQA and A-OKVQA datasets, we show that VLC-BERT is capable of outperforming existing models that utilize static knowledge bases. Furthermore, through a detailed analysis, we explain which questions benefit, and which don't, from contextualized commonsense knowledge from COMET. Code: https://github.com/aditya10/VLC-BERT",

keywords = "Algorithms: Vision + language and/or other modalities, Image recognition and understanding (object detection, categorization, segmentation, scene modeling, visual reasoning)",

author = "Sahithya Ravi and Aditya Chinchure and Leonid Sigal and Renjie Liao and Vered Shwartz",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023 ; Conference date: 03-01-2023 Through 07-01-2023",

year = "2023",

doi = "10.1109/WACV56688.2023.00121",

language = "אנגלית",

series = "Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1155--1165",

booktitle = "Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023",

address = "ארצות הברית",

}

Ravi, S, Chinchure, A, Sigal, L, Liao, R & Shwartz, V 2023, VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge. in Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023. Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023, Institute of Electrical and Electronics Engineers Inc., pp. 1155-1165, 23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, United States, 3/01/23. https://doi.org/10.1109/WACV56688.2023.00121

VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge. / Ravi, Sahithya; Chinchure, Aditya; Sigal, Leonid et al.
Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1155-1165 (Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - VLC-BERT

T2 - 23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023

AU - Ravi, Sahithya

AU - Chinchure, Aditya

AU - Sigal, Leonid

AU - Liao, Renjie

AU - Shwartz, Vered

PY - 2023

Y1 - 2023

N2 - There has been a growing interest in solving Visual Question Answering (VQA) tasks that require the model to reason beyond the content present in the image. In this work, we focus on questions that require commonsense reasoning. In contrast to previous methods which inject knowledge from static knowledge bases, we investigate the incorporation of contextualized knowledge using Commonsense Transformer (COMET), an existing knowledge model trained on human-curated knowledge bases. We propose a method to generate, select, and encode external commonsense knowledge alongside visual and textual cues in a new pre-trained Vision-Language-Commonsense transformer model, VLC-BERT. Through our evaluation on the knowledge-intensive OK-VQA and A-OKVQA datasets, we show that VLC-BERT is capable of outperforming existing models that utilize static knowledge bases. Furthermore, through a detailed analysis, we explain which questions benefit, and which don't, from contextualized commonsense knowledge from COMET. Code: https://github.com/aditya10/VLC-BERT

AB - There has been a growing interest in solving Visual Question Answering (VQA) tasks that require the model to reason beyond the content present in the image. In this work, we focus on questions that require commonsense reasoning. In contrast to previous methods which inject knowledge from static knowledge bases, we investigate the incorporation of contextualized knowledge using Commonsense Transformer (COMET), an existing knowledge model trained on human-curated knowledge bases. We propose a method to generate, select, and encode external commonsense knowledge alongside visual and textual cues in a new pre-trained Vision-Language-Commonsense transformer model, VLC-BERT. Through our evaluation on the knowledge-intensive OK-VQA and A-OKVQA datasets, we show that VLC-BERT is capable of outperforming existing models that utilize static knowledge bases. Furthermore, through a detailed analysis, we explain which questions benefit, and which don't, from contextualized commonsense knowledge from COMET. Code: https://github.com/aditya10/VLC-BERT

KW - Algorithms: Vision + language and/or other modalities

KW - Image recognition and understanding (object detection, categorization, segmentation, scene modeling, visual reasoning)

UR - http://www.scopus.com/inward/record.url?scp=85149022891&partnerID=8YFLogxK

U2 - 10.1109/WACV56688.2023.00121

DO - 10.1109/WACV56688.2023.00121

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85149022891

T3 - Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023

SP - 1155

EP - 1165

BT - Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 3 January 2023 through 7 January 2023

ER -

Ravi S, Chinchure A, Sigal L, Liao R, Shwartz V. VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge. In Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1155-1165. (Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023). doi: 10.1109/WACV56688.2023.00121

VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this