Time-series anomaly detection with stacked Transformer representations and 1D convolutional network

Jina Kim, Hyeongwon Kang, Pilsung Kang

Research output: Contribution to journalArticlepeer-review

19 Citations (Scopus)

Abstract

Time-series anomaly detection is a task of detecting data that do not follow normal data distribution among continuously collected data. It is used for system maintenance in various industries; hence, studies on time-series anomaly detection are being carried out actively. Most of the methodologies are based on Long Short-Term Memory (LSTM) and Convolution Neural Network (CNN) to model the temporal structure of time-series data. In this study, we propose an unsupervised prediction-based time-series anomaly detection methodology using Transformer, which shows superior performance to LSTM and CNN in learning dynamic patterns of sequential data through a self-attention mechanism. The prediction model consists of an encoder comprising multiple Transformer encoder layers and a decoder that includes a 1D convolution layer. The output representation of each Transformer layer is accumulated in the encoder to obtain a representation with multi-level, rich information. The decoder fuses this representation through a 1d convolution operation. Consequently, the model can perform predictions considering both the global trend and local variability of the input time-series. The anomaly score is defined as the difference between the predicted and the actual value at the corresponding timestamp, assuming that the trained model produces the predictions that follow the normal data distribution. Finally, the data with an anomaly score above the threshold is detected as an anomaly. Experiments on the benchmark datasets show that the proposed method has performance superior to those of the baselines.

Original languageEnglish
Article number105964
JournalEngineering Applications of Artificial Intelligence
Volume120
DOIs
Publication statusPublished - 2023 Apr

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) ( NRF-2022R1A2C2005455 ) and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-00034 , Clustering technologies of fragmented data for time-based data analysis).

Publisher Copyright:
© 2023 Elsevier Ltd

Keywords

  • Convolution Neural Network
  • Time series anomaly detection
  • Transformer

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Time-series anomaly detection with stacked Transformer representations and 1D convolutional network'. Together they form a unique fingerprint.

Cite this