Meaningful Pose-Based Sign Language Evaluation

Zifan Jiang; Colin Leong; Amit Moryossef; Oliver Cory; Maksym Ivashechkin; Neha Tarigopula; Biao Zhang; Anne Göhring; Annette Rios; Rico Sennrich; Sarah Ebling

doi:10.18653/v1/2025.wmt-1.4

Meaningful Pose-Based Sign Language Evaluation

Zifan Jiang
, Colin Leong
, Amit Moryossef
, Oliver Cory
, Maksym Ivashechkin
, Neha Tarigopula
, Biao Zhang
, Anne Göhring
, Annette Rios
, Rico Sennrich
, Sarah Ebling

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through (1) automatic meta-evaluation of sign-level retrieval, and (2) a human correlation study of text-to-pose translation across different sign languages. Our findings, along with the open-source pose-evaluation toolkit, provide a practical and reproducible approach for developing and evaluating sign language translation or generation systems.

Original language	English
Title of host publication	WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference
Editors	Barry Haddow, Tom Kocmi, Philipp Koehn, Christof Monz
Publisher	Association for Computational Linguistics
Pages	64-80
Number of pages	17
ISBN (Electronic)	9798891763418
DOIs	https://doi.org/10.18653/v1/2025.wmt-1.4
State	Published - 2025
Externally published	Yes
Event	10th Conference on Machine Translation, WMT 2025 - Suzhou, China Duration: 8 Nov 2025 → 9 Nov 2025

Publication series

Name	Conference on Machine Translation - Proceedings
ISSN (Electronic)	2768-0983

Conference

Conference	10th Conference on Machine Translation, WMT 2025
Country/Territory	China
City	Suzhou
Period	8/11/25 → 9/11/25

Bibliographical note

Publisher Copyright:
© 2025 Association for Computational Linguistics.

Access to Document

10.18653/v1/2025.wmt-1.4

https://aclanthology.org/2025.wmt-1.4.pdfLicense: CC BY

Cite this

Jiang, Z., Leong, C., Moryossef, A., Cory, O., Ivashechkin, M., Tarigopula, N., Zhang, B., Göhring, A., Rios, A., Sennrich, R., & Ebling, S. (2025). Meaningful Pose-Based Sign Language Evaluation. In B. Haddow, T. Kocmi, P. Koehn, & C. Monz (Eds.), WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference (pp. 64-80). (Conference on Machine Translation - Proceedings). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.wmt-1.4

@inproceedings{e07ddf5ffa2d45c485bbb971d3d27a44,

title = "Meaningful Pose-Based Sign Language Evaluation",

abstract = "We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through (1) automatic meta-evaluation of sign-level retrieval, and (2) a human correlation study of text-to-pose translation across different sign languages. Our findings, along with the open-source pose-evaluation toolkit, provide a practical and reproducible approach for developing and evaluating sign language translation or generation systems.",

author = "Zifan Jiang and Colin Leong and Amit Moryossef and Oliver Cory and Maksym Ivashechkin and Neha Tarigopula and Biao Zhang and Anne G{\"o}hring and Annette Rios and Rico Sennrich and Sarah Ebling",

note = "Publisher Copyright: {\textcopyright} 2025 Association for Computational Linguistics.; 10th Conference on Machine Translation, WMT 2025 ; Conference date: 08-11-2025 Through 09-11-2025",

year = "2025",

doi = "10.18653/v1/2025.wmt-1.4",

language = "אנגלית",

series = "Conference on Machine Translation - Proceedings",

publisher = "Association for Computational Linguistics",

pages = "64--80",

editor = "Barry Haddow and Tom Kocmi and Philipp Koehn and Christof Monz",

booktitle = "WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference",

}

Jiang, Z, Leong, C, Moryossef, A, Cory, O, Ivashechkin, M, Tarigopula, N, Zhang, B, Göhring, A, Rios, A, Sennrich, R & Ebling, S 2025, Meaningful Pose-Based Sign Language Evaluation. in B Haddow, T Kocmi, P Koehn & C Monz (eds), WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference. Conference on Machine Translation - Proceedings, Association for Computational Linguistics, pp. 64-80, 10th Conference on Machine Translation, WMT 2025, Suzhou, China, 8/11/25. https://doi.org/10.18653/v1/2025.wmt-1.4

Meaningful Pose-Based Sign Language Evaluation. / Jiang, Zifan; Leong, Colin; Moryossef, Amit et al.
WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference. ed. / Barry Haddow; Tom Kocmi; Philipp Koehn; Christof Monz. Association for Computational Linguistics, 2025. p. 64-80 (Conference on Machine Translation - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Meaningful Pose-Based Sign Language Evaluation

AU - Jiang, Zifan

AU - Leong, Colin

AU - Moryossef, Amit

AU - Cory, Oliver

AU - Ivashechkin, Maksym

AU - Tarigopula, Neha

AU - Zhang, Biao

AU - Göhring, Anne

AU - Rios, Annette

AU - Sennrich, Rico

AU - Ebling, Sarah

PY - 2025

Y1 - 2025

N2 - We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through (1) automatic meta-evaluation of sign-level retrieval, and (2) a human correlation study of text-to-pose translation across different sign languages. Our findings, along with the open-source pose-evaluation toolkit, provide a practical and reproducible approach for developing and evaluating sign language translation or generation systems.

AB - We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through (1) automatic meta-evaluation of sign-level retrieval, and (2) a human correlation study of text-to-pose translation across different sign languages. Our findings, along with the open-source pose-evaluation toolkit, provide a practical and reproducible approach for developing and evaluating sign language translation or generation systems.

UR - https://www.scopus.com/pages/publications/105028851378

U2 - 10.18653/v1/2025.wmt-1.4

DO - 10.18653/v1/2025.wmt-1.4

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:105028851378

T3 - Conference on Machine Translation - Proceedings

SP - 64

EP - 80

BT - WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference

A2 - Haddow, Barry

A2 - Kocmi, Tom

A2 - Koehn, Philipp

A2 - Monz, Christof

PB - Association for Computational Linguistics

T2 - 10th Conference on Machine Translation, WMT 2025

Y2 - 8 November 2025 through 9 November 2025

ER -

Jiang Z, Leong C, Moryossef A, Cory O, Ivashechkin M, Tarigopula N et al. Meaningful Pose-Based Sign Language Evaluation. In Haddow B, Kocmi T, Koehn P, Monz C, editors, WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference. Association for Computational Linguistics. 2025. p. 64-80. (Conference on Machine Translation - Proceedings). doi: 10.18653/v1/2025.wmt-1.4

Meaningful Pose-Based Sign Language Evaluation

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this