CATS: cache-aware task scheduling for Hadoop-based systems

Byungnam Lim, Jong Wook Kim, Yon Dohn Chung

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)

Abstract

Today with the explosion of big data, data-intensive cluster computing systems have driven to a new data processing paradigm. As Hadoop, one of the most famous data processing frameworks, achieves high performance by running multiple tasks in parallel across nodes in large clusters, task scheduling is considered as one of the most important factors affecting the overall performance. In modern operating systems, caching is used to improve local disk access times, providing data from the main memory without disk accesses. This option, however, is poorly utilized by existing task scheduling methods of Hadoop-based systems, mainly due to the inability of tracking cached data in shared-nothing distributed environments. In this paper, we propose a cache-aware task scheduling method, cache-aware task scheduling (CATS), for Hadoop-based systems which is able to exploit the operating system’s buffer cache and assign tasks to nodes in consideration of the cached data. Through comprehensive experiments, we show that the proposed cache-aware scheduling improves the overall job execution time for various workload types and data sizes.

Original languageEnglish
Pages (from-to)3691-3705
Number of pages15
JournalCluster Computing
Volume20
Issue number4
DOIs
Publication statusPublished - 2017 Dec 1

Keywords

  • Distributed systems
  • Hadoop
  • In-memory
  • Task scheduling

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'CATS: cache-aware task scheduling for Hadoop-based systems'. Together they form a unique fingerprint.

Cite this