On the limits of learning to actively learn semantic representations

Omri Koshorek, Gabriel Stanovsky, Yichu Zhou, Vivek Srikumar, Jonathan Berant

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

5 Scopus citations

Abstract

One of the goals of natural language understanding is to develop models that map sentences into meaning representations. However, training such models requires expensive annotation of complex structures, which hinders their adoption. Learning to actively-learn (LTAL) is a recent paradigm for reducing the amount of labeled data by learning a policy that selects which samples should be labeled. In this work, we examine LTAL for learning semantic representations, such as QA-SRL. We show that even an oracle policy that is allowed to pick examples that maximize performance on the test set (and constitutes an upper bound on the potential of LTAL), does not substantially improve performance compared to a random policy. We investigate factors that could explain this finding and show that a distinguishing characteristic of successful applications of LTAL is the interaction between optimization and the oracle policy selection process. In successful applications of LTAL, the examples selected by the oracle policy do not substantially depend on the optimization procedure, while in our setup the stochastic nature of optimization strongly affects the examples selected by the oracle. We conclude that the current applicability of LTAL for improving data efficiency in learning semantic meaning representations is limited.

Original languageEnglish
Title of host publicationCoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference
PublisherAssociation for Computational Linguistics
Pages452-462
Number of pages11
ISBN (Electronic)9781950737727
StatePublished - 2019
Externally publishedYes
Event23rd Conference on Computational Natural Language Learning, CoNLL 2019 - Hong Kong, China
Duration: 3 Nov 20194 Nov 2019

Publication series

NameCoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference

Conference

Conference23rd Conference on Computational Natural Language Learning, CoNLL 2019
Country/TerritoryChina
CityHong Kong
Period3/11/194/11/19

Bibliographical note

Publisher Copyright:
© 2019 Association for Computational Linguistics.

Fingerprint

Dive into the research topics of 'On the limits of learning to actively learn semantic representations'. Together they form a unique fingerprint.

Cite this