資源控管與進度平衡之MapReduce排程機制

Resource-Aware and Mismatch Controlling MapReduce Scheduler

伍銘基

指導教授 ： 周志遠

國立清華大學/電機資訊學院/資訊工程學系/碩士(2014年)

https://doi.org/10.6843/NTHU.2014.00475

摘要

隨著網路上日漸增多的資料量,能夠處理新數量級問題的方法也
更加重要。在今日的巨量資料時代,Hadoop mapreduce是其中一種被 廣泛運用來處理大量、成長快速資料的重要工具。許多的研究在傳統 的mapreduce架構上提出他們的策略來適應不同的情況,然而,為了 達到真正使用到叢集裡所有資源,我們仍然需要一個新的架構來分配 工作。在這篇論文中,我們研發了一個資源控管策略來克服傳統架構 上的缺點,並提出進度平衡演算法來調控Mapper與Reducer間的進度 不平衡,藉此達到叢集內資源的高利用率。

關鍵字

資源 ； 不平衡

並列摘要

As more and more data generated every moment over the internet, the requirement of method to solve new scale problems is getting important. Hadoop mapreduce is one of the most important tools which widely used on solving large scale and rapidly growing problems in today’s big data era. Based on traditional mapreduce frame- work, many researches proposed their strategy to adapt different situations. But to actually full use the resource of hadoop cluster, we still require a new framework to allocate tasks. In this thesis, we develop a resource-aware scheduling strategy to overcome the drawbacks of traditional framework, and propose a mismatch control- ling algorithm that coordinates the progress of mapper and reducer to achieve the full usage of resource.

並列關鍵字

mapreduce ； resource ； mismatch

參考文獻

[3] J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51(1):107–113, Jan. 2008.

[4] H. Jin, X. Yang, X.-H. Sun, and I. Raicu. ADAPT: Availability-Aware MapRe- duce Data Placement for Non-dedicated Distributed Computing. In Proceedings of the 2012 IEEE 32Nd International Conference on Distributed Computing Systems, ICDCS ’12, pages 516–525, Washington, DC, USA, 2012. IEEE Com- puter Society.

[7] B. Palanisamy, A. Singh, L. Liu, and B. Jain. Purlieus: Locality-aware Re- source Allocation for MapReduce in a Cloud. In Proceedings of 2011 Interna-

[10] B. T. Rao and L. S. S. Reddy. Survey on Improved Scheduling in Hadoop MapReduce in Cloud Environments. CoRR, abs/1207.0780, 2012.

[12] J. Tan, X. Meng, and L. Zhang. Performance analysis of Coupling Scheduler for MapReduce/Hadoop. In INFOCOM, 2012 Proceedings IEEE, pages 2586–2590, March 2012.