Multimodal Human Action Recognition for Rehabilitation Exercise of Upper Body for Individuals With Cerebral Palsy

Cerebral palsy (CP) is a neurological disorder that severely limits physical movements in everyday life. For individuals with CP, precise movement patterns are critical to improving their physical abilities. However, because of the time and expense involved, obtaining feedback from qualified rehabilitation professionals could be difficult. To address this problem, we propose a multimodal human action recognition (HAR) framework that integrates time-series data on skeletons and angles for upper body exercise for individuals with CP. Previous, studies typically used skeleton data as input to HAR. However, given the repetitive motion sequence, angles also are a crucial factor. In this study, we create video data of upper body exercises. Using the data, we estimate the skeleton using BlazePose, a human pose estimation (HPE) algorithm, and calculate the critical angles from the estimated skeletons. Our proposed framework incorporates graph-based models for skeletons and recurrent-based models for angles with an attention mechanism, allowing identification of important temporal information. It is shown that our framework performs better than unimodal HAR models. By reducing the costs associated with exercise feedback, we expect our approach to ultimately improve the physical health of individuals with CP.