摘要
隨著現今電腦的運算能力愈來愈強大，而且在資料量不斷提升 的情況下有限的I/O頻寬卻無法等比例的提升，兩者間日趨擴大的效 能差異導致傳統的模擬後數據處理方法(post-simulation data processing method)已面臨效能上的瓶頸。因此原位計算(in-situ computing)與查詢驅動數據分析(query-driven data analysis)是 用於縮短資料搬移路徑很重要的技巧。我們實作一個結合了位圖索 引(bitmap indexing)、空間資料結構重組(spatial data reorganization) 、分散式共享內存(distributed shared memory)與 位置感知平行執行(location-aware parallel execution)的索引系 統，並且使用了NERSC的超級電腦作為真實環境對兩個真實科學模擬 資料運行實驗分析。結果顯示對比於傳統依賴平行儲存檔案系統的 查詢系統，我們的系統可以達到10倍以上的效能優化。
並列摘要
The growing gap between compute performance and I/O bandwidth coupled with the increasing data volumes has resulted in a bottleneck to the traditional post- simulation data processing method. Hence in-situ computing and query-driven data analysis are important techniques to minimize data movement. By taking advantage of the growing memory capacity on supercomputers, we developed an in-memory query system for scientific data analysis. Our approach is a combination of bitmap indexing, spatial data layout re-organization, distributed shared memory, and location-aware parallel execution. Our evaluations on a NERSC supercomputer using two real scientific datasets showed that we can aggregate the memory ca- pacity from thousands of computes nodes to analyze a 750GB simulation dataset without transferring data to remote nodes or storage systems. Comparing to the traditional solutions based on out-of-core parallel file systems, we achieve more than x10 speedup. Therefore, our system can support interactive query and serve as a vehicle for steering simulations.
參考文獻
Adding value to the io pipelines of high performance applications with jitstaging.
In Proceedings of the 20th International Symposium on High Performance
Ecient query execution on raw data les. In Proceedings of the 2012 ACM
pages 241{252, 2012.
[4] IPCC Fifth Assessment Report. http://en.wikipedia.org/wiki/IPCCF ifthAssessmentReport:
延伸閱讀
- 陳熾強（2007）。行動電話供應商庫存管理系統之研究〔碩士論文，國立臺北科技大學〕。華藝線上圖書館。https://www.airitilibrary.com/Article/Detail?DocID=U0006-2407200713341200
- Bai, J. W. (2012). 空間資料於分散式資料庫上的查詢與處理 [master's thesis, National Chiao Tung University]. Airiti Library. https://doi.org/10.6842/NCTU.2012.00726
- Chang, Y. J. (2012). 應用於大容量隨機存取記憶體之內建備份元件分析技術 [master's thesis, National Central University]. Airiti Library. https://www.airitilibrary.com/Article/Detail?DocID=U0031-1903201314432837
- Lin, M. Y., & Lee, S. Y. (2005). Fast Discovery of Sequential Patterns through Memory Indexing and Database Partitioning. Journal of Information Science and Engineering, 21(1), 109-128. https://doi.org/10.6688/JISE.2005.21.1.6
- 賴嘉陞（2010）。Design and Implementation of Electronic Database Query System〔碩士論文，國立虎尾科技大學〕。華藝線上圖書館。https://doi.org/10.6827/NFU.2010.00024