Using EfficientNet-B7 (CNN), Variational Auto Encoder (VAE) and Siamese Twins’ Networks to Evaluate Human Exercises as Super Objects in a TSSCI Images

Yoram Segal, Ofer Hadar, Lenka Lhotska

Research output: Contribution to journalArticlepeer-review

5 Scopus citations

Abstract

In this article, we introduce a new approach to human movement by defining the movement as a static super object represented by a single two-dimensional image. The described method is applicable in remote healthcare applications, such as physiotherapeutic exercises. It allows researchers to label and describe the entire exercise as a standalone object, isolated from the reference video. This approach allows us to perform various tasks, including detecting similar movements in a video, measuring and comparing movements, generating new similar movements, and defining choreography by controlling specific parameters in the human body skeleton. As a result of the presented approach, we can eliminate the need to label images manually, disregard the problem of finding the start and the end of an exercise, overcome synchronization issues between movements, and perform any deep learning network-based operation that processes super objects in images in general. As part of this article, we will demonstrate two application use cases: one illustrates how to verify and score a fitness exercise. In contrast, the other illustrates how to generate similar movements in the human skeleton space by addressing the challenge of supplying sufficient training data for deep learning applications (DL). A variational auto encoder (VAE) simulator and an EfficientNet-B7 classifier architecture embedded within a Siamese twin neural network are presented in this paper in order to demonstrate the two use cases. These use cases demonstrate the versatility of our innovative concept in measuring, categorizing, inferring human behavior, and generating gestures for other researchers.

Original languageEnglish
Article number874
JournalJournal of Personalized Medicine
Volume13
Issue number5
DOIs
StatePublished - 22 May 2023
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2023 by the authors.

Funding

This research was supported by a grant from the Ministry of Science Technology, Israel and The Ministry of Education, Youth and Sports of the Czech Republic. The described research was supported by the project No. LTAIZ19008 (Czech) and No. 8773451 (Israel) Enhancing Robotic Physiotherapeutic Treatment using Machine Learning awarded in frame of the Czech–Israeli cooperative scientific research program (Inter-Excellence MEYS CR and MOST Israel).

FundersFunder number
Czech–Israeli cooperative scientific research program
Ministry of Science, Technology and Space
Ministerstvo Školství, Mládeže a TělovýchovyLTAIZ19008, 8773451
Ministry of science and technology, Israel

    Keywords

    • MediaPipe (MP)
    • OpenPose (OP)
    • Siamese twin neural network
    • computational creativity
    • computational imagination
    • human body movements
    • human pose estimation (HPE)
    • rehabilitation
    • simulator
    • tree structure skeleton color image (TSSCI)
    • tree structure skeleton image (TSSI)
    • variational auto encoder (VAE)

    Fingerprint

    Dive into the research topics of 'Using EfficientNet-B7 (CNN), Variational Auto Encoder (VAE) and Siamese Twins’ Networks to Evaluate Human Exercises as Super Objects in a TSSCI Images'. Together they form a unique fingerprint.

    Cite this