基於SNMP之Hadoop雲端運算平台監測機制

SNMP-Based Monitoring of Hadoop Cloud Computing Platform

顏晟陞(Cheng-Sheng Yan)

指導教授 ： 陳彥錚

國立暨南國際大學/管理學院/資訊管理學系/碩士(2014年)

https://doi.org/10.6837/NCNU.2014.00025

摘要

隨著雲端運算技術的漸趨成熟及巨量資料時代的來臨，使得許多雲端服務與應用面臨大量資料的承載，進而許多雲端運算平台被陸續開發，用以儲存、處理、分析及管理大量資料，其中尤以Hadoop雲端運算平台被廣泛應用，其HDFS資料儲存檔案系統與MapReduce資料處理運算系統於平台中扮演重要角色，當叢集系統運行、計算的同時，即時監測HDFS運行狀態及MapReduce執行情況，並報告叢集系統可靠度、叢集節點可用性與系統軟硬體資源消耗上相當重要。此外，現有的Hadoop管理工具之監測項目缺乏統一之監控標準，另需佈署多套監測工具，使得叢集系統管理更為困難。因此，本研究提出以SNMP網管協定作為Hadoop雲端運算平台的監測標準，對於叢集系統之軟硬體資訊與運算節點效能之監測，採用標準之MIB-II及HOST-RESOURCES-MIB與企業自定之UCD-SNMP-MIB及LM-SENSORS-MIB模組進行監測。此外，並藉由剖析、處理Hadoop雲端運算平台的登錄檔(Log Files)，予以結構化後，並整合Hadoop API，取得更為詳盡之監測資訊。為符合SNMP標準，本論文另設計Hadoop平台監測之專用MIB模組，並以可擴充代理程式(AgentX)結合登錄檔剖析(Log Parser)工具程式及Hadoop API程式進行MIB模組實作，以提供更為完整的Hadoop雲端運算平台監測資訊，實現以SNMP管理Hadoop雲端運算平台的叢集環境。

關鍵字

Hadoop ； 簡單網路管理協定 ； 雲端運算 ； 巨量資料 ； 網路及系統管理

並列摘要

With the rapid development of cloud computing and the emerging big data environments, large amounts of data are manipulated in cloud services. The Hadoop cloud computing platform has been used widely for its capability of storing, processing, analysis, and management of large-scale data. The most important task of Hadoop monitoring is resource consumption and availability of HDFS and MapReduce. This requires the installation of multiple monitoring systems in each component of the Hadoop cluster. In this paper, the research will propose the use of SNMP in Hadoop monitoring. The proposed monitoring scheme makes use of four standard and enterprise MIBs, including MIB-II, HOST-RESOURCES-MIB, UCD-SNMP-MIB and LM-SENSORS-MIB, to provide essential information for monitoring cluster components. In addition, the research will develop a log analysis system to parse log files of Hadoop, together with the use of Hadoop API, for retrieving valuable data for effective monitoring. An enterprise MIB will be defined to make the above monitoring information available in a standard way and implemented on an extensible agent (AgentX). Finally, a web-based management system will be presented to demonstrate the superiority of SNMP in monitoring the Hadoop system.

並列關鍵字

Hadoop ； SNMP ； Cloud Computing ； Big Data ； Network and System Management

參考文獻

[1] Amazon.com, Inc, “Amazon EC2,” http://aws.amazon.com/, accessed on December 2013.

[2] Google, Inc, “Google App Engine,” https://developers.google.com/appengine/, accessed on December 2013.

[3] Salesforce.com, inc, “force.com,” http://www.salesforce.com/, accessed on December 2013.

[4] Google, Inc, “Google Maps,” https://maps.google.com/, accessed on December 2013.

[5] Google, Inc, “Google Docs,” https://drive.google.com/, accessed on December 2013.