Evaluating the immediate applicability of pose estimation for sign language recognition

Amit Moryossef; Ioannis Tsochantaridis; Joe DInn; Necati Cihan Camgoz; Richard Bowden; Tao Jiang; Annette Rios; Mathias Muller; Sarah Ebling

doi:10.1109/CVPRW53098.2021.00382

Evaluating the immediate applicability of pose estimation for sign language recognition

Amit Moryossef, Ioannis Tsochantaridis, Joe DInn, Necati Cihan Camgoz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, Sarah Ebling

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

33 Scopus citations

Abstract

Sign languages are visual languages produced by the movement of the hands, face, and body. In this paper, we evaluate representations based on skeleton poses, as these are explainable, person-independent, privacy-preserving, low-dimensional representations. Basically, skeletal representations generalize over an individual's appearance and background, allowing us to focus on the recognition of motion. But how much information is lost by the skeletal representation? We perform two independent studies using two state-of-the-art pose estimation systems. We analyze the applicability of the pose estimation systems to sign language recognition by evaluating the failure cases of the recognition models. Importantly, this allows us to characterize the current limitations of skeletal pose estimation approaches in sign language recognition.

Original language	English
Title of host publication	Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
Publisher	IEEE Computer Society
Pages	3429-3435
Number of pages	7
ISBN (Electronic)	9781665448994
DOIs	https://doi.org/10.1109/CVPRW53098.2021.00382
State	Published - Jun 2021
Event	2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 - Virtual, Online, United States Duration: 19 Jun 2021 → 25 Jun 2021

Publication series

Name	IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
ISSN (Print)	2160-7508
ISSN (Electronic)	2160-7516

Conference

Conference	2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
Country/Territory	United States
City	Virtual, Online
Period	19/06/21 → 25/06/21

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Access to Document

10.1109/CVPRW53098.2021.00382

Cite this

Moryossef, A., Tsochantaridis, I., DInn, J., Camgoz, N. C., Bowden, R., Jiang, T., Rios, A., Muller, M., & Ebling, S. (2021). Evaluating the immediate applicability of pose estimation for sign language recognition. In Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 (pp. 3429-3435). (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops). IEEE Computer Society. https://doi.org/10.1109/CVPRW53098.2021.00382

Moryossef, Amit ; Tsochantaridis, Ioannis ; DInn, Joe et al. / Evaluating the immediate applicability of pose estimation for sign language recognition. Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society, 2021. pp. 3429-3435 (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops).

@inproceedings{b4fc5a1a1a554f21a82ba8f14d76105b,

title = "Evaluating the immediate applicability of pose estimation for sign language recognition",

abstract = "Sign languages are visual languages produced by the movement of the hands, face, and body. In this paper, we evaluate representations based on skeleton poses, as these are explainable, person-independent, privacy-preserving, low-dimensional representations. Basically, skeletal representations generalize over an individual's appearance and background, allowing us to focus on the recognition of motion. But how much information is lost by the skeletal representation? We perform two independent studies using two state-of-the-art pose estimation systems. We analyze the applicability of the pose estimation systems to sign language recognition by evaluating the failure cases of the recognition models. Importantly, this allows us to characterize the current limitations of skeletal pose estimation approaches in sign language recognition.",

author = "Amit Moryossef and Ioannis Tsochantaridis and Joe DInn and Camgoz, {Necati Cihan} and Richard Bowden and Tao Jiang and Annette Rios and Mathias Muller and Sarah Ebling",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 ; Conference date: 19-06-2021 Through 25-06-2021",

year = "2021",

month = jun,

doi = "10.1109/CVPRW53098.2021.00382",

language = "אנגלית",

series = "IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops",

publisher = "IEEE Computer Society",

pages = "3429--3435",

booktitle = "Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021",

address = "ארצות הברית",

}

Moryossef, A, Tsochantaridis, I, DInn, J, Camgoz, NC, Bowden, R, Jiang, T, Rios, A, Muller, M & Ebling, S 2021, Evaluating the immediate applicability of pose estimation for sign language recognition. in Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, IEEE Computer Society, pp. 3429-3435, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021, Virtual, Online, United States, 19/06/21. https://doi.org/10.1109/CVPRW53098.2021.00382

Evaluating the immediate applicability of pose estimation for sign language recognition. / Moryossef, Amit; Tsochantaridis, Ioannis; DInn, Joe et al.
Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society, 2021. p. 3429-3435 (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Evaluating the immediate applicability of pose estimation for sign language recognition

AU - Moryossef, Amit

AU - Tsochantaridis, Ioannis

AU - DInn, Joe

AU - Camgoz, Necati Cihan

AU - Bowden, Richard

AU - Jiang, Tao

AU - Rios, Annette

AU - Muller, Mathias

AU - Ebling, Sarah

PY - 2021/6

Y1 - 2021/6

N2 - Sign languages are visual languages produced by the movement of the hands, face, and body. In this paper, we evaluate representations based on skeleton poses, as these are explainable, person-independent, privacy-preserving, low-dimensional representations. Basically, skeletal representations generalize over an individual's appearance and background, allowing us to focus on the recognition of motion. But how much information is lost by the skeletal representation? We perform two independent studies using two state-of-the-art pose estimation systems. We analyze the applicability of the pose estimation systems to sign language recognition by evaluating the failure cases of the recognition models. Importantly, this allows us to characterize the current limitations of skeletal pose estimation approaches in sign language recognition.

AB - Sign languages are visual languages produced by the movement of the hands, face, and body. In this paper, we evaluate representations based on skeleton poses, as these are explainable, person-independent, privacy-preserving, low-dimensional representations. Basically, skeletal representations generalize over an individual's appearance and background, allowing us to focus on the recognition of motion. But how much information is lost by the skeletal representation? We perform two independent studies using two state-of-the-art pose estimation systems. We analyze the applicability of the pose estimation systems to sign language recognition by evaluating the failure cases of the recognition models. Importantly, this allows us to characterize the current limitations of skeletal pose estimation approaches in sign language recognition.

UR - http://www.scopus.com/inward/record.url?scp=85115278322&partnerID=8YFLogxK

U2 - 10.1109/CVPRW53098.2021.00382

DO - 10.1109/CVPRW53098.2021.00382

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85115278322

T3 - IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops

SP - 3429

EP - 3435

BT - Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021

PB - IEEE Computer Society

T2 - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021

Y2 - 19 June 2021 through 25 June 2021

ER -

Moryossef A, Tsochantaridis I, DInn J, Camgoz NC, Bowden R, Jiang T et al. Evaluating the immediate applicability of pose estimation for sign language recognition. In Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society. 2021. p. 3429-3435. (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops). doi: 10.1109/CVPRW53098.2021.00382

Evaluating the immediate applicability of pose estimation for sign language recognition

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this