Semantics-aware Attention Improves Neural Machine Translation

Aviv Slobodkin; Leshem Choshen; Omri Abend

doi:10.18653/v1/2022.starsem-1.3

Semantics-aware Attention Improves Neural Machine Translation

Aviv Slobodkin, Leshem Choshen, Omri Abend

Hebrew University of Jerusalem

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.

Original language	English
Title of host publication	*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference
Editors	Vivi Nastase, Ellie Pavlick, Mohammad Taher Pilehvar, Jose Camacho-Collados, Alessandro Raganato
Publisher	Association for Computational Linguistics (ACL)
Pages	28-43
Number of pages	16
ISBN (Electronic)	9781955917988
DOIs	https://doi.org/10.18653/v1/2022.starsem-1.3
State	Published - 2022
Externally published	Yes
Event	11th Joint Conference on Lexical and Computational Semantics, *SEM 2022 - Seattle, United States Duration: 14 Jul 2022 → 15 Jul 2022

Publication series

Name	*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference

Conference

Conference	11th Joint Conference on Lexical and Computational Semantics, *SEM 2022
Country/Territory	United States
City	Seattle
Period	14/07/22 → 15/07/22

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Funding

This work was supported in part by the Israel Science Foundation (grant no. 2424/21), and by the Applied Research in Academia Program of the Israel Innovation Authority.

Funders	Funder number
Israel Innovation Authority
Israel Science Foundation	2424/21

Access to Document

10.18653/v1/2022.starsem-1.3

Cite this

Slobodkin, A., Choshen, L., & Abend, O. (2022). Semantics-aware Attention Improves Neural Machine Translation. In V. Nastase, E. Pavlick, M. T. Pilehvar, J. Camacho-Collados, & A. Raganato (Eds.), *SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference (pp. 28-43). (*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.starsem-1.3

Slobodkin, Aviv ; Choshen, Leshem ; Abend, Omri. / Semantics-aware Attention Improves Neural Machine Translation. *SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference. editor / Vivi Nastase ; Ellie Pavlick ; Mohammad Taher Pilehvar ; Jose Camacho-Collados ; Alessandro Raganato. Association for Computational Linguistics (ACL), 2022. pp. 28-43 (*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference).

@inproceedings{ca2c6ef5f6d34ced8865f7b8fbcc08f3,

title = "Semantics-aware Attention Improves Neural Machine Translation",

abstract = "The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.",

author = "Aviv Slobodkin and Leshem Choshen and Omri Abend",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 11th Joint Conference on Lexical and Computational Semantics, *SEM 2022 ; Conference date: 14-07-2022 Through 15-07-2022",

year = "2022",

doi = "10.18653/v1/2022.starsem-1.3",

language = "אנגלית",

series = "*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "28--43",

editor = "Vivi Nastase and Ellie Pavlick and Pilehvar, {Mohammad Taher} and Jose Camacho-Collados and Alessandro Raganato",

booktitle = "*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference",

address = "ארצות הברית",

}

Slobodkin, A, Choshen, L & Abend, O 2022, Semantics-aware Attention Improves Neural Machine Translation. in V Nastase, E Pavlick, MT Pilehvar, J Camacho-Collados & A Raganato (eds), *SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference. *SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 28-43, 11th Joint Conference on Lexical and Computational Semantics, *SEM 2022, Seattle, United States, 14/07/22. https://doi.org/10.18653/v1/2022.starsem-1.3

Semantics-aware Attention Improves Neural Machine Translation. / Slobodkin, Aviv; Choshen, Leshem; Abend, Omri.
*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference. ed. / Vivi Nastase; Ellie Pavlick; Mohammad Taher Pilehvar; Jose Camacho-Collados; Alessandro Raganato. Association for Computational Linguistics (ACL), 2022. p. 28-43 (*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Semantics-aware Attention Improves Neural Machine Translation

AU - Slobodkin, Aviv

AU - Choshen, Leshem

AU - Abend, Omri

PY - 2022

Y1 - 2022

N2 - The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.

AB - The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.

UR - http://www.scopus.com/inward/record.url?scp=85139097947&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.starsem-1.3

DO - 10.18653/v1/2022.starsem-1.3

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85139097947

T3 - *SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference

SP - 28

EP - 43

BT - *SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference

A2 - Nastase, Vivi

A2 - Pavlick, Ellie

A2 - Pilehvar, Mohammad Taher

A2 - Camacho-Collados, Jose

A2 - Raganato, Alessandro

PB - Association for Computational Linguistics (ACL)

T2 - 11th Joint Conference on Lexical and Computational Semantics, *SEM 2022

Y2 - 14 July 2022 through 15 July 2022

ER -

Slobodkin A, Choshen L, Abend O. Semantics-aware Attention Improves Neural Machine Translation. In Nastase V, Pavlick E, Pilehvar MT, Camacho-Collados J, Raganato A, editors, *SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2022. p. 28-43. (*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference). doi: 10.18653/v1/2022.starsem-1.3

Semantics-aware Attention Improves Neural Machine Translation

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this