Abstract
In natural language generation tasks, a neural language model is used for generating a sequence of words forming a sentence. The topmost weight matrix of the language model, known as the classification layer, can be viewed as a set of vectors, each representing a target word from the target dictionary. The target word vectors, along with the rest of the model parameters, are learned and updated during training. In this paper, we analyze the properties encoded in the target vectors and question the necessity of learning these vectors. We suggest to randomly draw the target vectors and set them as fixed so that no weights updates are being made during training. We show that by excluding the vectors from the optimization, the number of parameters drastically decreases with a marginal effect on the performance. We demonstrate the effectiveness of our method in image-captioning and machine-translation.
Original language | English |
---|---|
Title of host publication | Multimodal Artificial Intelligence, MAI Workshop 2021 - Proceedings of the 3rd Workshop |
Editors | Amir Zadeh, Louis-Philippe Morency, Paul Pu Liang, Candace Ross, Ruslan Salakhutdinov, Soujanya Poria, Erik Cambria, Kelly Shi |
Publisher | Association for Computational Linguistics (ACL) |
Pages | 6-11 |
Number of pages | 6 |
ISBN (Electronic) | 9781954085251 |
DOIs | |
State | Published - 2021 |
Event | 3rd NAACL Workshop on Multimodal Artificial Intelligence, MAI Workshop 2021 - Mexico City, Mexico Duration: 6 Jun 2021 → … |
Publication series
Name | Multimodal Artificial Intelligence, MAI Workshop 2021 - Proceedings of the 3rd Workshop |
---|
Conference
Conference | 3rd NAACL Workshop on Multimodal Artificial Intelligence, MAI Workshop 2021 |
---|---|
Country/Territory | Mexico |
City | Mexico City |
Period | 6/06/21 → … |
Bibliographical note
Publisher Copyright:© 2021 Association for Computational Linguistics