A Comprehensive Analysis of Datasets for Automotive Intrusion Detection Systems

Seyoung Lee, Wonsuk Choi, Insup Kim, Ganggyu Lee, Dong Hoon Lee

Research output: Contribution to journalArticlepeer-review

Abstract

Recently, automotive intrusion detection systems (IDSs) have emerged as promising defense approaches to counter attacks on in-vehicle networks (IVNs). However, the effectiveness of IDSs relies heavily on the quality of the datasets used for training and evaluation. Despite the availability of several datasets for automotive IDSs, there has been a lack of comprehensive analysis focusing on assessing these datasets. This paper aims to address the need for dataset assessment in the context of automotive IDSs. It proposes qualitative and quantitative metrics that are independent of specific automotive IDSs, to evaluate the quality of datasets. These metrics take into consideration various aspects such as dataset description, collection environment, and attack complexity. This paper evaluates eight commonly used datasets for automotive IDSs using the proposed metrics. The evaluation reveals biases in the datasets, particularly in terms of limited contexts and lack of diversity. Additionally, it highlights that the attacks in the datasets were mostly injected without considering normal behaviors, which poses challenges for training and evaluating machine learning-based IDSs. This paper emphasizes the importance of addressing the identified limitations in existing datasets to improve the performance and adaptability of automotive IDSs. The proposed metrics can serve as valuable guidelines for researchers and practitioners in selecting and constructing high-quality datasets for automotive security applications. Finally, this paper presents the requirements for high-quality datasets, including the need for representativeness, diversity, and balance.

Original languageEnglish
Pages (from-to)3413-3442
Number of pages30
JournalComputers, Materials and Continua
Volume76
Issue number3
DOIs
Publication statusPublished - 2023

Bibliographical note

Publisher Copyright:
© 2023 Tech Science Press. All rights reserved.

Keywords

  • automotive security
  • Controller area network (CAN)
  • dataset
  • intrusion detection system (IDS)
  • machine learning (ML)

ASJC Scopus subject areas

  • Biomaterials
  • Modelling and Simulation
  • Mechanics of Materials
  • Computer Science Applications
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'A Comprehensive Analysis of Datasets for Automotive Intrusion Detection Systems'. Together they form a unique fingerprint.

Cite this