Volume 18, No. 2, 2021

Resource Efficient Memory Aware Scheduler For Hadoop Mapreduce Framework


JAGADEVI BAKKA , SANJEEV C LINGAREDDY

Abstract

MapReduce (MR) has been one of the popular computing framework for analyzing semi-structured and unstructured BigData and processing application in last decade; further Hadoop MR (HMR) framework is an open source platform which is the widely used MR framework. Moreover, existing HMR scheduling design faces major issues like I/O overhead and memory overhead. In this research work, we focus on developing resource efficient memory aware scheduler for HMR namely REMAS-HMR for efficient utilization of system resources and data processing in real time. REMAS-HMR is developed for analyzing the Global Memory Management; thus minimizing the Disk I/O seek. Further, modelled makespan model for mitigating intermediate task failures in REMAS-HMR. The REMAS method are evaluated on the Microsoft Azure HDInsight cloud platform in consideration with text mining and clustering applications, also comparative analysis with the existing model is carried out. Further, comparative analysis shows that REMAS model outperforms existing model in terms of makespan, computational cost, memory utilization, and core-resource utilization.


Pages: 3072-3085

Keywords: Cloud computing, I/O optimization, Iterative model, MapReduce, Memory Aware, Performance modelling, Resource utilization, Task scheduling.

Full Text