An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation

Amit Moryossef; Mathias Müller; Anne Göhring; Zifan Jiang; Yoav Goldberg; Sarah Ebling

An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation

Amit Moryossef, Mathias Müller, Anne Göhring, Zifan Jiang, Yoav Goldberg, Sarah Ebling

Department of Computer Science

University of Zurich

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.

Original language	English
Title of host publication	AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages
Editors	Dimitar Shterionov, Mirella De Sisto, Mathias Muller, Davy Van Landuyt, Rehana Omardeen, Shaun O'boyle, Annelies Braffort, Floris Roelofsen, Frederic Blain, Bram Vanroy, Eleftherios Avramidis
Publisher	European Association for Machine Translation
Pages	25-36
Number of pages	12
ISBN (Electronic)	9789464857184
State	Published - 2023
Event	2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023 - Tampere, Finland Duration: 15 Jun 2023 → …

Publication series

Name	AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages

Conference

Conference	2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023
Country/Territory	Finland
City	Tampere
Period	15/06/23 → …

Bibliographical note

Publisher Copyright:
© 2023 2023 VFS Handling Qualities Technical Meeting. All rights reserved.

Funding

This work was funded by the EU Horizon 2020 project EASIER (grant agreement no. 101016982), the Swiss Innovation Agency (Inno-suisse) flagship IICT (PFFS-21-47), and the EU Horizon 2020 project iEXTRACT (grant agreement no. 802774).

Funders	Funder number
Inno-suisse	802774, PFFS-21-47
Horizon 2020	101016982
Innosuisse - Schweizerische Agentur für Innovationsförderung

Cite this

Moryossef, A., Müller, M., Göhring, A., Jiang, Z., Goldberg, Y., & Ebling, S. (2023). An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation. In D. Shterionov, M. De Sisto, M. Muller, D. Van Landuyt, R. Omardeen, S. O'boyle, A. Braffort, F. Roelofsen, F. Blain, B. Vanroy, & E. Avramidis (Eds.), AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages (pp. 25-36). (AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages). European Association for Machine Translation.

Moryossef, Amit ; Müller, Mathias ; Göhring, Anne et al. / An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation. AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages. editor / Dimitar Shterionov ; Mirella De Sisto ; Mathias Muller ; Davy Van Landuyt ; Rehana Omardeen ; Shaun O'boyle ; Annelies Braffort ; Floris Roelofsen ; Frederic Blain ; Bram Vanroy ; Eleftherios Avramidis. European Association for Machine Translation, 2023. pp. 25-36 (AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages).

@inproceedings{08bec01dc7cf43b8bee9efa6a9f35c3f,

title = "An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation",

abstract = "Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.",

author = "Amit Moryossef and Mathias M{\"u}ller and Anne G{\"o}hring and Zifan Jiang and Yoav Goldberg and Sarah Ebling",

note = "Publisher Copyright: {\textcopyright} 2023 2023 VFS Handling Qualities Technical Meeting. All rights reserved.; 2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023 ; Conference date: 15-06-2023",

year = "2023",

language = "אנגלית",

series = "AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages",

publisher = "European Association for Machine Translation",

pages = "25--36",

editor = "Dimitar Shterionov and {De Sisto}, Mirella and Mathias Muller and {Van Landuyt}, Davy and Rehana Omardeen and Shaun O'boyle and Annelies Braffort and Floris Roelofsen and Frederic Blain and Bram Vanroy and Eleftherios Avramidis",

booktitle = "AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages",

}

Moryossef, A, Müller, M, Göhring, A, Jiang, Z, Goldberg, Y & Ebling, S 2023, An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation. in D Shterionov, M De Sisto, M Muller, D Van Landuyt, R Omardeen, S O'boyle, A Braffort, F Roelofsen, F Blain, B Vanroy & E Avramidis (eds), AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages. AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages, European Association for Machine Translation, pp. 25-36, 2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023, Tampere, Finland, 15/06/23.

An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation. / Moryossef, Amit; Müller, Mathias; Göhring, Anne et al.
AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages. ed. / Dimitar Shterionov; Mirella De Sisto; Mathias Muller; Davy Van Landuyt; Rehana Omardeen; Shaun O'boyle; Annelies Braffort; Floris Roelofsen; Frederic Blain; Bram Vanroy; Eleftherios Avramidis. European Association for Machine Translation, 2023. p. 25-36 (AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation

AU - Moryossef, Amit

AU - Müller, Mathias

AU - Göhring, Anne

AU - Jiang, Zifan

AU - Goldberg, Yoav

AU - Ebling, Sarah

PY - 2023

Y1 - 2023

N2 - Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.

AB - Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.

UR - http://www.scopus.com/inward/record.url?scp=85174784958&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85174784958

T3 - AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages

SP - 25

EP - 36

BT - AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages

A2 - Shterionov, Dimitar

A2 - De Sisto, Mirella

A2 - Muller, Mathias

A2 - Van Landuyt, Davy

A2 - Omardeen, Rehana

A2 - O'boyle, Shaun

A2 - Braffort, Annelies

A2 - Roelofsen, Floris

A2 - Blain, Frederic

A2 - Vanroy, Bram

A2 - Avramidis, Eleftherios

PB - European Association for Machine Translation

T2 - 2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023

Y2 - 15 June 2023

ER -

Moryossef A, Müller M, Göhring A, Jiang Z, Goldberg Y, Ebling S. An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation. In Shterionov D, De Sisto M, Muller M, Van Landuyt D, Omardeen R, O'boyle S, Braffort A, Roelofsen F, Blain F, Vanroy B, Avramidis E, editors, AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages. European Association for Machine Translation. 2023. p. 25-36. (AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages).

An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation

Abstract

Publication series

Conference

Bibliographical note

Funding

Other files and links

Fingerprint

Cite this