Interpretable human action recognition in compressed domain

Vignesh Srinivasan, Sebastian Lapuschkin, Cornelius Hellge, Klaus Robert Muller, Wojciech Samek

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    25 Citations (Scopus)

    Abstract

    Compressed domain human action recognition algorithms are extremely efficient, because they only require a partial decoding of the video bit stream. However, the question what exactly makes these algorithms decide for a particular action is still a mystery. In this paper, we present a general method, Layer-wise Relevance Propagation (LRP), to understand and interpret action recognition algorithms and apply it to a state-of-the-art compressed domain method based on Fisher vector encoding and SVM classification. By using LRP, the classifiers decisions are propagated back every step in the action recognition pipeline until the input is reached. This methodology allows to identify where and when the important (from the classifier's perspective) action happens in the video. To our knowledge, this is the first work to interpret a compressed domain action recognition algorithm. We evaluate our method on the HMDB51 dataset and show that in many cases a few significant frames contribute most towards the prediction of the video to a particular class.

    Original languageEnglish
    Title of host publication2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages1692-1696
    Number of pages5
    ISBN (Electronic)9781509041176
    DOIs
    Publication statusPublished - 2017 Jun 16
    Event2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - New Orleans, United States
    Duration: 2017 Mar 52017 Mar 9

    Publication series

    NameICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
    ISSN (Print)1520-6149

    Other

    Other2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017
    Country/TerritoryUnited States
    CityNew Orleans
    Period17/3/517/3/9

    Bibliographical note

    Publisher Copyright:
    © 2017 IEEE.

    Keywords

    • Action recognition
    • compressed domain
    • fisher vector encoding
    • interpretable classification
    • motion vectors

    ASJC Scopus subject areas

    • Software
    • Signal Processing
    • Electrical and Electronic Engineering

    Fingerprint

    Dive into the research topics of 'Interpretable human action recognition in compressed domain'. Together they form a unique fingerprint.

    Cite this