HTNet: Anchor-free Temporal Action Localization with Hierarchical Transformers

Tae Kyung Kang, Gun Hee Lee, Seong Whan Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Citations (Scopus)


Temporal action localization (TAL) is a task of identifying a set of actions in a video, which involves localizing the start and end frames and classifying each action instance. Existing methods have addressed this task by using predefined anchor windows or heuristic bottom-up boundary-matching strategies, which are major bottlenecks in inference time. Additionally, the main challenge is the inability to capture long-range actions due to a lack of global contextual information. In this paper, we present a novel anchor-free framework, referred to as HTNet, which predicts a set of langlestart time, end time, classrangle triplets from a video based on a Transformer architecture. After the prediction of coarse boundaries, we refine it through a background feature sampling (BFS) module and hierarchical Transformers, which enables our model to aggregate global contextual information and effectively exploit the inherent semantic relationships in a video. We demonstrate how our method localizes accurate action instances and achieves state-of-the-art performance on two TAL benchmark datasets: THUMOS14 and ActivityNet 1.3.

Original languageEnglish
Title of host publication2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages6
ISBN (Electronic)9781665452588
Publication statusPublished - 2022
Event2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Prague, Czech Republic
Duration: 2022 Oct 92022 Oct 12

Publication series

NameConference Proceedings - IEEE International Conference on Systems, Man and Cybernetics
ISSN (Print)1062-922X


Conference2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022
Country/TerritoryCzech Republic

Bibliographical note

Funding Information:
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. B0101-15-0266, Development of High Performance Visual BigData Discovery Platform for Large-Scale Realtime Data Analysis, No. 2021-0-02068, Artificial Intelligence Innovation Hub).

Publisher Copyright:
© 2022 IEEE.


  • Context Aggregation
  • Temporal Action Detection
  • Temporal Action Localization
  • Transformer

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Control and Systems Engineering
  • Human-Computer Interaction


Dive into the research topics of 'HTNet: Anchor-free Temporal Action Localization with Hierarchical Transformers'. Together they form a unique fingerprint.

Cite this