Real-Time Sign Language Detection Using Human Pose Estimation

Amit Moryossef; Ioannis Tsochantaridis; Roee Aharoni; Sarah Ebling; Srini Narayanan

doi:10.1007/978-3-030-66096-3_17

Real-Time Sign Language Detection Using Human Pose Estimation

Amit Moryossef
, Ioannis Tsochantaridis
, Roee Aharoni
, Sarah Ebling
, Srini Narayanan

Department of Computer Science and Artificial Intelligence

Alphabet Inc.
University of Zurich

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

55 Scopus citations

Abstract

We propose a lightweight real-time sign language detection model, as we identify the need for such a case in videoconferencing. We extract optical flow features based on human pose estimation and, using a linear classifier, show these features are meaningful with an accuracy of 80%, evaluated on the Public DGS Corpus. Using a recurrent model directly on the input, we see improvements of up to 91% accuracy, while still working under 4 ms. We describe a demo application to sign language detection in the browser in order to demonstrate its usage possibility in videoconferencing applications.

Original language	English
Title of host publication	Computer Vision – ECCV 2020 Workshops, Proceedings
Editors	Adrien Bartoli, Andrea Fusiello
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	237-248
Number of pages	12
ISBN (Print)	9783030660956
DOIs	https://doi.org/10.1007/978-3-030-66096-3_17
State	Published - 2020
Event	Workshops held at the 16th European Conference on Computer Vision, ECCV 2020 - Glasgow, United Kingdom Duration: 23 Aug 2020 → 28 Aug 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12536 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Workshops held at the 16th European Conference on Computer Vision, ECCV 2020
Country/Territory	United Kingdom
City	Glasgow
Period	23/08/20 → 28/08/20

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Keywords

Sign language detection
Sign language processing

Access to Document

10.1007/978-3-030-66096-3_17

Cite this

Moryossef, A., Tsochantaridis, I., Aharoni, R., Ebling, S., & Narayanan, S. (2020). Real-Time Sign Language Detection Using Human Pose Estimation. In A. Bartoli, & A. Fusiello (Eds.), Computer Vision – ECCV 2020 Workshops, Proceedings (pp. 237-248). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12536 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-66096-3_17

Moryossef, Amit ; Tsochantaridis, Ioannis ; Aharoni, Roee et al. / Real-Time Sign Language Detection Using Human Pose Estimation. Computer Vision – ECCV 2020 Workshops, Proceedings. editor / Adrien Bartoli ; Andrea Fusiello. Springer Science and Business Media Deutschland GmbH, 2020. pp. 237-248 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{8184f3d947244fbf88156b44a7910839,

title = "Real-Time Sign Language Detection Using Human Pose Estimation",

abstract = "We propose a lightweight real-time sign language detection model, as we identify the need for such a case in videoconferencing. We extract optical flow features based on human pose estimation and, using a linear classifier, show these features are meaningful with an accuracy of 80\%, evaluated on the Public DGS Corpus. Using a recurrent model directly on the input, we see improvements of up to 91\% accuracy, while still working under 4 ms. We describe a demo application to sign language detection in the browser in order to demonstrate its usage possibility in videoconferencing applications.",

keywords = "Sign language detection, Sign language processing",

author = "Amit Moryossef and Ioannis Tsochantaridis and Roee Aharoni and Sarah Ebling and Srini Narayanan",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; Workshops held at the 16th European Conference on Computer Vision, ECCV 2020 ; Conference date: 23-08-2020 Through 28-08-2020",

year = "2020",

doi = "10.1007/978-3-030-66096-3\_17",

language = "אנגלית",

isbn = "9783030660956",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "237--248",

editor = "Adrien Bartoli and Andrea Fusiello",

booktitle = "Computer Vision – ECCV 2020 Workshops, Proceedings",

address = "גרמניה",

}

Moryossef, A, Tsochantaridis, I, Aharoni, R, Ebling, S & Narayanan, S 2020, Real-Time Sign Language Detection Using Human Pose Estimation. in A Bartoli & A Fusiello (eds), Computer Vision – ECCV 2020 Workshops, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12536 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 237-248, Workshops held at the 16th European Conference on Computer Vision, ECCV 2020, Glasgow, United Kingdom, 23/08/20. https://doi.org/10.1007/978-3-030-66096-3_17

Real-Time Sign Language Detection Using Human Pose Estimation. / Moryossef, Amit; Tsochantaridis, Ioannis; Aharoni, Roee et al.
Computer Vision – ECCV 2020 Workshops, Proceedings. ed. / Adrien Bartoli; Andrea Fusiello. Springer Science and Business Media Deutschland GmbH, 2020. p. 237-248 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12536 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Real-Time Sign Language Detection Using Human Pose Estimation

AU - Moryossef, Amit

AU - Tsochantaridis, Ioannis

AU - Aharoni, Roee

AU - Ebling, Sarah

AU - Narayanan, Srini

PY - 2020

Y1 - 2020

N2 - We propose a lightweight real-time sign language detection model, as we identify the need for such a case in videoconferencing. We extract optical flow features based on human pose estimation and, using a linear classifier, show these features are meaningful with an accuracy of 80%, evaluated on the Public DGS Corpus. Using a recurrent model directly on the input, we see improvements of up to 91% accuracy, while still working under 4 ms. We describe a demo application to sign language detection in the browser in order to demonstrate its usage possibility in videoconferencing applications.

AB - We propose a lightweight real-time sign language detection model, as we identify the need for such a case in videoconferencing. We extract optical flow features based on human pose estimation and, using a linear classifier, show these features are meaningful with an accuracy of 80%, evaluated on the Public DGS Corpus. Using a recurrent model directly on the input, we see improvements of up to 91% accuracy, while still working under 4 ms. We describe a demo application to sign language detection in the browser in order to demonstrate its usage possibility in videoconferencing applications.

KW - Sign language detection

KW - Sign language processing

UR - https://www.scopus.com/pages/publications/85101737644

U2 - 10.1007/978-3-030-66096-3_17

DO - 10.1007/978-3-030-66096-3_17

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85101737644

SN - 9783030660956

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 237

EP - 248

BT - Computer Vision – ECCV 2020 Workshops, Proceedings

A2 - Bartoli, Adrien

A2 - Fusiello, Andrea

PB - Springer Science and Business Media Deutschland GmbH

T2 - Workshops held at the 16th European Conference on Computer Vision, ECCV 2020

Y2 - 23 August 2020 through 28 August 2020

ER -

Moryossef A, Tsochantaridis I, Aharoni R, Ebling S, Narayanan S. Real-Time Sign Language Detection Using Human Pose Estimation. In Bartoli A, Fusiello A, editors, Computer Vision – ECCV 2020 Workshops, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 237-248. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-66096-3_17

Real-Time Sign Language Detection Using Human Pose Estimation

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this