Binary dense sift flow based two stream CNN for human action recognition

Sang Kyoo Park, Jun Ho Chung, Tae Koo Kang, Myo Taeg Lim

    Research output: Contribution to journalArticlepeer-review

    14 Citations (Scopus)

    Abstract

    Two-stream CNN is a widely-used network for human action recognition. Two-stream CNN consists of a spatial stream and a temporal stream. The spatial stream, through which the RGB image passes, extracts the shape features of human motion. The temporal stream, through which the optical flow images pass, extracts the sequence features of the listed motions. However, because of the constraints of the optical flow, such as brightness, constancy, and piecewise smoothness, there are limitations to the performance of two-stream CNN. One of the efficient methods to solve this problem is to expand the network model to a three-stream network, fuse it with LSTM, and add a modified pooling layer. This method improves the performance of the model but it increases the computational cost. Besides, the limitations of the optical flow are still present. In this paper, without extending the network model, a binary dense SIFT flow-based two-stream CNN is used instead of the optical flow. Unlike the optical flow, binary dense SIFT flow, which is a feature-based matching flow field is robust in brightness, constancy and piecewise smoothness. To evaluate the binary dense SIFT flow-based two-stream CNN, the UCF-101 dataset was selected for human action recognition. Furthermore, to evaluate the robustness of its brightness constancy and piecewise smoothness, a custom dataset was made up of classes that were extracted from UCF-101. Finally, the proposed method was compared with the state-of-the-art, which uses an optical flow-based two-stream CNN.

    Original languageEnglish
    Pages (from-to)35697-35720
    Number of pages24
    JournalMultimedia Tools and Applications
    Volume80
    Issue number28-29
    DOIs
    Publication statusPublished - 2021 Nov

    Bibliographical note

    Publisher Copyright:
    © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.

    Keywords

    • Action recognition
    • Binary dense SIFT flow
    • Binary descriptor
    • Two-Stream CNN

    ASJC Scopus subject areas

    • Software
    • Media Technology
    • Hardware and Architecture
    • Computer Networks and Communications

    Fingerprint

    Dive into the research topics of 'Binary dense sift flow based two stream CNN for human action recognition'. Together they form a unique fingerprint.

    Cite this