SWISS: Signal-wise self-supervised learning to learn a multi-signal interaction for human activity recognition

Over the past few years, the growing interest in adapting wearable and mobile devices for smart healthcare and living systems has led to an expansion of studies on human activity recognition (HAR). Recently, many HAR studies have focused on deep supervised learning, which automatically extracts meaningful features through extensive labeled data, is widely employed and demonstrated impressive progress. However, several issues, such as privacy sensitivity, cost intensity, and restricted laboratory conditions in HAR, impede the scalability of human activity labels, and the lack of labeled data degrades the performance of supervised learning. To address these challenges, we aim to extract useful representations without labels through self-supervised learning. Especially, we propose signal-wise self-supervised learning (SWISS) to learn signal interactions, taking into account the characteristics of inertial measurement unit used in sensor-based HAR data. First, we design data-level and feature-level reconstruction tasks for masked signals. These tasks compel the model to learn signal interactions by reconstructing masked signals in both the original data space and the low-dimensional feature space. Additionally, we design a new model architecture called the signal token transformer (STT), which is suitable for encoding both temporal features and signal interactions. We demonstrate the effectiveness of our approach on four publicly available human activity datasets. Furthermore, we show that STT can identify important signals and facilitate the learning of signal interactions through a self-attention mechanism. The code is available at https://github.com/DMQA-SR/HAR_SWISS.