Binary dense sift flow based two stream CNN for human action recognition

Sang Kyoo Park, Jun Ho Chung, Tae Koo Kang, Myo Taeg Lim

Research output: Contribution to journalArticlepeer-review

6 Citations (Scopus)

Abstract

Two-stream CNN is a widely-used network for human action recognition. Two-stream CNN consists of a spatial stream and a temporal stream. The spatial stream, through which the RGB image passes, extracts the shape features of human motion. The temporal stream, through which the optical flow images pass, extracts the sequence features of the listed motions. However, because of the constraints of the optical flow, such as brightness, constancy, and piecewise smoothness, there are limitations to the performance of two-stream CNN. One of the efficient methods to solve this problem is to expand the network model to a three-stream network, fuse it with LSTM, and add a modified pooling layer. This method improves the performance of the model but it increases the computational cost. Besides, the limitations of the optical flow are still present. In this paper, without extending the network model, a binary dense SIFT flow-based two-stream CNN is used instead of the optical flow. Unlike the optical flow, binary dense SIFT flow, which is a feature-based matching flow field is robust in brightness, constancy and piecewise smoothness. To evaluate the binary dense SIFT flow-based two-stream CNN, the UCF-101 dataset was selected for human action recognition. Furthermore, to evaluate the robustness of its brightness constancy and piecewise smoothness, a custom dataset was made up of classes that were extracted from UCF-101. Finally, the proposed method was compared with the state-of-the-art, which uses an optical flow-based two-stream CNN.

Original languageEnglish
Pages (from-to)35697-35720
Number of pages24
JournalMultimedia Tools and Applications
Volume80
Issue number28-29
DOIs
Publication statusPublished - 2021 Nov

Keywords

  • Action recognition
  • Binary dense SIFT flow
  • Binary descriptor
  • Two-Stream CNN

ASJC Scopus subject areas

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Binary dense sift flow based two stream CNN for human action recognition'. Together they form a unique fingerprint.

Cite this