An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation

Amit Moryossef, Mathias Müller, Anne Göhring, Zifan Jiang, Yoav Goldberg, Sarah Ebling

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.

Original languageEnglish
Title of host publicationAT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages
EditorsDimitar Shterionov, Mirella De Sisto, Mathias Muller, Davy Van Landuyt, Rehana Omardeen, Shaun O'boyle, Annelies Braffort, Floris Roelofsen, Frederic Blain, Bram Vanroy, Eleftherios Avramidis
PublisherEuropean Association for Machine Translation
Pages25-36
Number of pages12
ISBN (Electronic)9789464857184
StatePublished - 2023
Event2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023 - Tampere, Finland
Duration: 15 Jun 2023 → …

Publication series

NameAT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages

Conference

Conference2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023
Country/TerritoryFinland
CityTampere
Period15/06/23 → …

Bibliographical note

Publisher Copyright:
© 2023 2023 VFS Handling Qualities Technical Meeting. All rights reserved.

Fingerprint

Dive into the research topics of 'An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation'. Together they form a unique fingerprint.

Cite this