雲端運算越來越受到歡迎並持續於結構、網路以及軟體上發展。Hadoop-MapReduce使用分散式叢集來平行處理處理大量資料,是一個很常見的軟體架構。它裡面的處理節點可以擴充到一個相當大的數量,因此挾著強大的運算能力的Hadoop-MapReduce可以提供相當好的一個處理平台。而網路流量一直以來都是資料密集性運算最大的瓶頸,在資料平行系統對效能會造成顯著的影響。此網路瓶頸是網路頻寬所導致,使得網路速度比硬碟資料存取還要慢上許多。然而,好的資料局部性可以減少網路流量並使資料密集的HPC(High-performance computing)系統效能增加。不過Hadoop的排程在資源分配上有個缺乏考慮資料局部性的缺點,所以本論文提出了一個Hadoop-MapReduce位置感知排程演算法。首先我們提出了一個Hadoop排程的資料影響權重數學模型,其次,使用資料局部性排程演算法與資料影響權重搭配來提供位置感知的資源分配。最後建立三台安裝Xen Cloud Platform的實體機器,而每台實體機器上運行兩個裝有hadoop的虛擬機並使用模擬來驗證此演算法的效能。
Cloud computing has become more popular, and it has been continuously developed in architecture, software, and network. Hadoop-MapReduce is a common software framework processing parallelizable problem across big datasets using a distributed cluster. Cloud Hadoop-MapReduce can scale incrementally in the number of processing nodes. Hence, the Hadoop-MapReduce is designed to provide a processing platform with powerful computation. Network traffic is always a most important bottleneck in data-intensive computing and network latency decreases significant performance in data parallel systems. Network bottleneck is caused by network bandwidth and the network speed is much slower than disk data access. So that, good data locality can reduces network traffic and increases performance in data-intensive HPC systems. However, Hadoop’s scheduler has a defect of data locality in resource assignment. This paper includes a locality-aware scheduling algorithm for Hadoop-MapReduce scheduler. Firstly, we propose a mathematical model of weight of data interference in Hadoop scheduler. Secondly, we present the algorithm to use weight of data interference to provide data locality-aware resource assignment in Hadoop scheduler. Finally, we build an experimental environment with 3 physical machines which were installed Xen Cloud Platform and 2 virtual machines which are installed hadoop on each physical machine. Then, run simulation to verify the performance of locality-aware scheduling algorithm for Hadoop-MapReduce scheduler.
為了持續優化網站功能與使用者體驗,本網站將Cookies分析技術用於網站營運、分析和個人化服務之目的。
若您繼續瀏覽本網站,即表示您同意本網站使用Cookies。