download0 view154
twitter facebook

공공누리This item is licensed Korea Open Government License

Title
A New Efficient Resource Management Framework for Iterative MapReduce Processing in Large-Scale Data Analysis
Author(s)
홍승태장재우박경석임채덕
Publication Year
2017-04-17
Abstract
To analyze large-scale data efficiently, studies on Hadoop, one of the most popular MapReduce frameworks, have been actively done. Meanwhile, most of the large-scale data analysis applications, e.g., data clustering, are required to do the same map and reduce functions repeatedly. However, Hadoop cannot provide an optimal performance for iterative MapReduce jobs because it derives a result by doing one phase of map and reduce functions. To solve the problems, in this paper, we propose a new efficient resource management framework for iterative MapReduce processing in large-scale data analysis. For this, we first design an iterative job state-machine for managing the iterative MapReduce jobs. Secondly, we propose an invariant data caching mechanism for reducing the I/O costs of data accesses. Thirdly, we propose an iterative resource management technique for efficiently managing the resources of a Hadoop cluster. Fourthly, we devise a stop condition check mechanism for preventing unnecessary computation. Finally, we show the performance superiority of the proposed framework by comparing it with the existing frameworks.
Keyword
large-scale data analysis; iterative data processing framework; MapReduce; Hadoop
Journal Title
IEICE TRANSACTIONS on Information and Systems
Citation Volume
E100-D
ISSN
0916-8532
Files in This Item:
There are no files associated with this item.
Appears in Collections:
7. KISTI 연구성과 > 학술지 발표논문
URI
https://repository.kisti.re.kr/handle/10580/14718
http://www.ndsl.kr/ndsl/search/detail/article/articleSearchResultDetail.do?cn=NART78519451
Export
RIS (EndNote)
XLS (Excel)
XML

Browse