校園的大數據與深度學習整合開發環境

An Integrated Development Environment of Big Data and Deep Learning in Campus

唐琮壹 ； 劉威辰 ； 梁廷宇

《TANET2017 臺灣網際網路研討會》 2017年 (2017/10) Pp. 1045-1050

https://doi.org/10.6728/TANET.201710.0182

摘要

近來各大學院校掀起了一股大數據與深度學習的研究熱潮。為了師生的研究與學習需要，建置一個支援大數據與深度學習的程式整合開發環境是必須的。然而，對於很多非資訊相關的學校系所而言，由於缺乏具有專業系統實務經驗的人員，所以要自己在校園網路上建置與管理一個兼具經濟、效率、友善與普及性的程式開發環境，其實是個很大的挑戰。為了因應這個需求，本篇論文發表一個支援大數據與深度學習的程式整合開發環境，稱為EZCOM。在此開發環境中，使用者可以透過網頁介面隨時隨地用Python撰寫與執行Spark與Tensorflow程式，並且監看程式執行的狀態與結果，完全不用再煩惱系統安裝與管理維護的事情，只需專注於程式的開發即可。在考量校園電腦的資源可用度下，此開發環境採用Docker容器取代虛擬機器做為快速提供安全且獨立之使用者程式執行環境的機制。目前我們已經將它佈署在一個以Intel Core i3 PC為主的校園運算叢集上，並提供給本校電機系的師生作為大數據與深度學習的教學與研究之用。根據實際的效能測試數據顯示Docker確實比VM更為有效率。

關鍵字

大數據 ； 深度學習 ； Python ； Spark ； Tensorflow ； Docker 容器

並列摘要

Recently, many departments in universities and colleges open the courses of big data and deep learning. For the demand of teaching and learning, it is necessary to provide teachers and students with an integrated development environment of Spark and Tensorflow in campus. However, because of no expert of system setup and management, it is difficult for many departments which are not information related to build an economic, efficient, friendly and ubiquitous development environment of Spark and Tensorflow programs. To resolve this problem, this paper is aimed at proposing an integrated development environment for big data and deep learning in campus called EZCOM. In this environment, users can write and execute their Spark and Tensorflow programs, and watch the status and result of program execution through the interface of web pages. They can focus only on program development because they have no need to care about the setup and maintenance of system. For considering the resource capability of campus computers, this environment replaces VM by Docker container for fast deploying a safe and independent execution environment for user programs. The proposed environment has been deployed on a cluster of Intel Core i3 550 PCs for the use of teachers and students in the department of electrical engineering. Our experimental result of performance evaluation shows that Docker container indeed is more efficient and effective than VM for the execution performance of user programs.

並列關鍵字

big data ； deep learning ； Python ； Spark ； Tensorflow ； Docker container