基於Hadoop平台之分散式權重式字尾樹暨天文時序性資料分析系統

Distributed Astronomy Sequential Pattern Analysis System Using Hadoop Platform with Weighted Suffix Tree

蔡昀翰(Yun-Hang Tsai)

指導教授 ： 蔡孟峰

國立中央大學/資訊電機學院/資訊工程學系/碩士(2015年)

摘要

隨著科技的發展，泛星計畫（Panoramic Survey Telescope And Rapid Response System，Pan-STARRS）中所觀測到的資料量也隨之增長，而儲存設備成本降低，也讓天文學家們得以將大量且詳細的觀測資料儲存起來。
由於收集到的天文資料其各個元素間是具有時間順序性的，而傳統的方法卻難以處理此類資料，所以我們選用字尾樹作為其結構的原型，提供天文學家快速而有效率的星體資料查詢功能，並且能夠在分析後提供與查詢相似的星體資訊給天文學家們。
因為字尾樹的資料結構其記憶體使用量驚人，而天文資料的數量又十分龐大，在兩項因素交互影響之下，導致單一機器無法負荷，所以我們選用在開源的OpenStack系統上，建構Hadoop平台的雲端系統來構成分散式環境，將資料分散處理，以提升系統的整體效能。
透過分散式系統處理大量的天文資料，減少了在資料處理上所耗費的人力，在效率上也得到了明顯的提升，提供了研究人員在未來面對大量觀測資料時一個有效的解決方法。在未來我們也期望能利用此系統架構來為所有具有時序性的資料作分析。

關鍵字

泛星計畫 ； 分散式系統 ； 資料探勘 ； 權重字尾樹

並列摘要

Because of the ongoing construction of observatories from Pan-Starrs projects with technological advancements, the size of observation data has exploded. And the storage device cost reduction. Astronomical researchers were able to make a large and detailed observation data stored.
The various elements of collected astronomical data have time sequential features. And the traditional method is difficult to handle such data. So we use the suffix tree as a prototype of system structure to provide astronomical researchers a fast and efficient data query system. And we can provide approximate patterns to astronomical researchers after finish the analysis.
Because the interaction of the amazing memory consumed of suffix tree data structure and the very large number of astronomical data lead to a single machine overload, we use the open source OpenStack system to construct Hadoop platform cloud system to complete a distributed environment. So that we can process astronomical data distributed, and enhance the effectiveness of the system.
To Process large amounts of astronomical data through distributed systems can reduce the cost of manually data processing and the efficiency has been significantly improved. We provided a valid solution when astronomical researchers face a lot of observation data in the future. We hope to use this system architecture to analyze all the time sequential data in the future.

並列關鍵字

Pan-Starrs ； Distributed System ； Data Mining ； Weighted Suffix Tree

參考文獻

〔1〕陳文屏, 「天文觀測的新挑戰─談泛星計畫」, 科儀新知, 第30卷第3期, 2008年.

〔9〕P. Weiner, “Linear Pattern Matching Algorithm,” 14th Annual IEEE Symposium on Switching and Automata Theory, 1973.

〔10〕吳彥慶, “Exploiting Frequent Episodes in Weighted Suffix Tree to Improve Intrusion Detection System,” 國立中央大學, 碩士論文, 2007.

〔11〕沈敬軒, “Mining Similar Astronomical Sequence Pattern with Hierarchical Weighted Suffix Tree,” 國立中央大學, 碩士論文, 2011.

〔13〕劉書宏, Distributed Astronomical Sequence Data Indexing System with Suffix Tree, 國立中央大學, 碩士論文, 2014.