基於Apache Spark異構叢集系統任務排程優化之研究

隨著硬體設備的升級，企業的數據中心的硬體資源不斷的更新換代，新加入的節點使得叢集產生異構性，由於異構的叢集各個節點計算能力不一致，因此當同一個任務分配到不同節點上運算，將會對節點的負載造成不同的影響。另外，對於任務也存在著異構性的問題，如CPU密集型和I/O密集型任務對於計算資源需求不一致，在分配節點時也應考慮，以Apache Spark大數據計算框架為例，在分配任務時，並不考慮到叢集節點的異構性以及節點資源利用情況，因此，導致叢集中各個節點在任務執行時出現負載平衡不平均的問題，導致一部分的資源消耗過載，使得整體效率受限於弱節點，導致整體任務計算效能下降。針對上述問題，本研究提出了一種新的調度策略以優化Spark在異構叢集的表現，提出了新的分層排程調度方法，先透過分群的方式，將相近計算能力的節點組成叢集，而在調度時運用測試任務來進行初步任務執行時間的推估，而後利用歷史數據與機器學習方法更準確的預估任務執行時間，以實現更高效率的任務調度算法。

關鍵字

Apache Spark ；雲端運算；巨量資料；調度策略；任務排程調度策略；異構叢集

並列摘要

As the hardware equipment is continue to be upgraded, the hardware resources of the company data center are constantly renewed and replaced. Consequently, newly added nodes cause the cluster nodes to express heterogeneity. However, due to cluster heterogeneity, the processing abilities of the each node are different. As a result, when a task is assigned to different nodes, it affects the loading of each node differently. In addition, heterogeneity causes issues in tasks themselves as well. For example: When assigning nodes, one should consider the need for different resources of CPU intensives and I/O intensives. Take the Apache Spark data analytic framework as an example, the current Spark does not take heterogeneity nor resource utilization of cluster nodes into consideration. Therefore, the nodes of each cluster demonstrate uneven loading when they are performing tasks. This causes partial system overloading and resource depletion, and limits the overall efficiency to the lesser capable nodes. As a result, overall computational performance drops. In order to counteract the problems discussed above, this study suggests a new scheduling strategy that can optimize Spark’s performance in relation to heterogeneos clusters. This study proposes a new hierarchical scheduling strategy that first divides nodes with similar calculating abilities into groups. During this process, test assignments are used to assess preliminary executing time. Then, historical data and machine learning techniques are used to further accurately estimate the execution time. Finally, with the strategy explained above, a more efficient task scheduling algorithm are proposed and implemented.

並列關鍵字

Apache Spark ； Cloud Computing ； Big Data ； Dispatching Strategy ； Task Scheduling and Dispatching strategy ； Heterogenic Clusters

參考文獻

[1] Jo, M., Maksymyuk, T., Strykhalyuk, B., & Cho, C. H. (2015). Device-to-device-based heterogeneous radio access network architecture for mobile cloud computing. IEEE Wireless Communications, 22(3), 50-58.

[2] Beloglazov, A., Buyya, R., Lee, Y. C., & Zomaya, A. (2011). A taxonomy and survey of energy-efficient data centers and cloud computing systems. Advances in computers, 82(2), 47-111.

[5] Vasile, M. A., Pop, F., Tutueanu, R. I., Cristea, V., & Kołodziej, J. (2015). Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing. Future Generation Computer Systems, 51, 61-71.

[6] Di Nitto, E., Ghezzi, C., Metzger, A., Papazoglou, M., & Pohl, K. (2008). A journey to highly dynamic, self-adaptive service-based applications. Automated Software Engineering, 15(3), 313-341.

[7] Xu, X., Cao, L., & Wang, X. (2016). Adaptive task scheduling strategy based on dynamic workload adjustment for heterogeneous Hadoop clusters. IEEE Systems Journal, 10(2), 471-482.

國際替代計量

基於Apache Spark異構叢集系統任務排程優化之研究

全文下載

主題瀏覽