download0 view11
twitter facebook

공공누리This item is licensed Korea Open Government License

Title
FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling
Author(s)
PREETHIKA KASU김영재SCOTT ATCHLEY박경석엄정호김태욱
Publisher
IEEE
Publication Year
2019-03-21
Abstract
The layout-aware data scheduling (LADS) data movement framework optimizes congestion for end-to-end data transfers. During data transfer, LADS can avoid congested storage elements by exploiting the underlying storage layout at each endpoint. This improves the I/O bandwidth and hence the data transfer rate across high-speed networks. However, the absence of fault tolerance (FT) in LADS results in data retransmission overhead and may lead to possible data integrity issues upon faults. In this paper, we propose object-logging FT mechanisms to avoid transmitting the objects that are successfully written into the parallel file system (PFS) at the sink end. Depending on the number of log files created for the whole dataset, we have classified our FT mechanisms into three different categories: file logger, transaction logger, and universal logger. Also, to address the space overhead, we have proposed different methods of populating the log files with the information of the successfully transferred objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object-logging-based FT mechanisms on the LADS data transfer framework. Our experimental results show that FT mechanisms exhibit negligible overhead (<; 1%) with respect to the data transfer time. However, the fault recovery time is 10% higher than the total data transfer time at any fault point.
Keyword
Big data; geo-distributed data centers; fault tolerance; parallel system
Journal Title
IEEE Access;
Citation Volume
7
ISSN
2169-3536
DOI
10.1109/access.2019.2905158
Files in This Item:
There are no files associated with this item.
Appears in Collections:
7. KISTI 연구성과 > 학술지 발표논문
URI
https://repository.kisti.re.kr/handle/10580/16257
Export
RIS (EndNote)
XLS (Excel)
XML

Browse