Contrastive Learning for Weakly Supervised Phrase Grounding

Tanmay Gupta; Arash Vahdat; Gal Chechik; Xiaodong Yang; Jan Kautz; Derek Hoiem

doi:10.1007/978-3-030-58580-8_44

Contrastive Learning for Weakly Supervised Phrase Grounding

Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, Derek Hoiem

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

65 Scopus citations

Abstract

Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a ∼ 10 % absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of 5.7 % to achieve 76.7 % accuracy on Flickr30K Entities benchmark. Our code and project material will be available at http://tanmaygupta.info/info-ground.

Original language	English
Title of host publication	Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings
Editors	Andrea Vedaldi, Horst Bischof, Thomas Brox, Jan-Michael Frahm
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	752-768
Number of pages	17
ISBN (Print)	9783030585792
DOIs	https://doi.org/10.1007/978-3-030-58580-8_44
State	Published - 2020
Event	16th European Conference on Computer Vision, ECCV 2020 - Glasgow, United Kingdom Duration: 23 Aug 2020 → 28 Aug 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12348 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	16th European Conference on Computer Vision, ECCV 2020
Country/Territory	United Kingdom
City	Glasgow
Period	23/08/20 → 28/08/20

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Funding

Acknowledgement. This work was done partly at NVIDIA and is partly supported by ONR MURI Award N00014-16-1-2007 This work was done partly at NVIDIA and is partly supported by ONR MURI Award N00014-16-1-2007.

Funders	Funder number
ONR MURI
Multidisciplinary University Research Initiative	N00014-16-1-2007

Keywords

Attention
Grounding
InfoNCE
Mutual information

Access to Document

10.1007/978-3-030-58580-8_44

Cite this

Gupta, T., Vahdat, A., Chechik, G., Yang, X., Kautz, J., & Hoiem, D. (2020). Contrastive Learning for Weakly Supervised Phrase Grounding. In A. Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Eds.), Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings (pp. 752-768). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12348 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-58580-8_44

Gupta, Tanmay ; Vahdat, Arash ; Chechik, Gal et al. / Contrastive Learning for Weakly Supervised Phrase Grounding. Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings. editor / Andrea Vedaldi ; Horst Bischof ; Thomas Brox ; Jan-Michael Frahm. Springer Science and Business Media Deutschland GmbH, 2020. pp. 752-768 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{aad6c74fda0e4b90bf91795a4c3ac98e,

title = "Contrastive Learning for Weakly Supervised Phrase Grounding",

abstract = "Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a ∼ 10 % absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of 5.7 % to achieve 76.7 % accuracy on Flickr30K Entities benchmark. Our code and project material will be available at http://tanmaygupta.info/info-ground.",

keywords = "Attention, Grounding, InfoNCE, Mutual information",

author = "Tanmay Gupta and Arash Vahdat and Gal Chechik and Xiaodong Yang and Jan Kautz and Derek Hoiem",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 16th European Conference on Computer Vision, ECCV 2020 ; Conference date: 23-08-2020 Through 28-08-2020",

year = "2020",

doi = "10.1007/978-3-030-58580-8_44",

language = "אנגלית",

isbn = "9783030585792",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "752--768",

editor = "Andrea Vedaldi and Horst Bischof and Thomas Brox and Jan-Michael Frahm",

booktitle = "Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings",

address = "גרמניה",

}

Gupta, T, Vahdat, A, Chechik, G, Yang, X, Kautz, J & Hoiem, D 2020, Contrastive Learning for Weakly Supervised Phrase Grounding. in A Vedaldi, H Bischof, T Brox & J-M Frahm (eds), Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12348 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 752-768, 16th European Conference on Computer Vision, ECCV 2020, Glasgow, United Kingdom, 23/08/20. https://doi.org/10.1007/978-3-030-58580-8_44

Contrastive Learning for Weakly Supervised Phrase Grounding. / Gupta, Tanmay; Vahdat, Arash; Chechik, Gal et al.
Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings. ed. / Andrea Vedaldi; Horst Bischof; Thomas Brox; Jan-Michael Frahm. Springer Science and Business Media Deutschland GmbH, 2020. p. 752-768 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12348 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Contrastive Learning for Weakly Supervised Phrase Grounding

AU - Gupta, Tanmay

AU - Vahdat, Arash

AU - Chechik, Gal

AU - Yang, Xiaodong

AU - Kautz, Jan

AU - Hoiem, Derek

PY - 2020

Y1 - 2020

N2 - Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a ∼ 10 % absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of 5.7 % to achieve 76.7 % accuracy on Flickr30K Entities benchmark. Our code and project material will be available at http://tanmaygupta.info/info-ground.

AB - Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a ∼ 10 % absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of 5.7 % to achieve 76.7 % accuracy on Flickr30K Entities benchmark. Our code and project material will be available at http://tanmaygupta.info/info-ground.

KW - Attention

KW - Grounding

KW - InfoNCE

KW - Mutual information

UR - http://www.scopus.com/inward/record.url?scp=85097849142&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-58580-8_44

DO - 10.1007/978-3-030-58580-8_44

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

SN - 9783030585792

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 752

EP - 768

BT - Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings

A2 - Vedaldi, Andrea

A2 - Bischof, Horst

A2 - Brox, Thomas

A2 - Frahm, Jan-Michael

PB - Springer Science and Business Media Deutschland GmbH

T2 - 16th European Conference on Computer Vision, ECCV 2020

Y2 - 23 August 2020 through 28 August 2020

ER -

Gupta T, Vahdat A, Chechik G, Yang X, Kautz J, Hoiem D. Contrastive Learning for Weakly Supervised Phrase Grounding. In Vedaldi A, Bischof H, Brox T, Frahm JM, editors, Computer Vision – ECCV 2020 - 16th European Conference 2020, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 752-768. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-58580-8_44

Contrastive Learning for Weakly Supervised Phrase Grounding

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this