Design and Implement a MapReduce Framework for Executing Standalone Software Packages in Hadoop-based Distributed Environments

Chao-Chun Chen ； Min-Hsiung Hung ； Nguyen Huu Tinh Giang ； Hsuan-Chun Lin ； Tzu-Chao Lin

《Smart Science》 1卷2期 (2013/12) Pp. 99-107

https://doi.org/10.6493/SmartSci.2013.161

並列摘要

The Hadoop MapReduce is the programming model of designing the auto scalable distributed computing applications. It provides developer an effective environment to attain automatic parallelization. However, most existing manufacturing systems are arduous and restrictive to migrate to MapReduce private cloud, due to the platform incompatible and tremendous complexity of system reconstruction. For increasing the efficiency of manufacturing systems with minimum modification of existing systems, we design a framework in this thesis, called MC-Framework: Multi-uses-based Cloudizing-Application Framework. It provides the simple interface to users for fairly executing requested tasks worked with traditional standalone software packages in MapReduce-based private cloud environments. Moreover, this thesis focuses on the multiuser workloads, but the default Hadoop scheduling scheme, i.e., FIFO, would increase delay under multiuser scenarios. Hence, we also propose a new scheduling mechanism, called Job-Sharing Scheduling, to explore and fairly share the jobs to machines in the MapReduce-based private cloud. Then, we prototype an experimental virtual-metrology module of a manufacturing system as a case study to verify and analysis the proposed MC-Framework. The results of our experiments indicate that our proposed framework enormously improved the time performance compared with the original package.

並列關鍵字

Mapreduce ； Hadoop ； Cloudization ； Multi-users scheduling

參考文獻

Y. J. Chang, Y. K. , C. L. Hsu, C. T. Chang, and T. Y. Chan, "Virtual Metrology Technique for Semiconductor Manufacturing" International Joint Conference on Neural Networks (IJCNN’06), pp. 5289-5293, July (2006)

M. H. Hung, C. F. Chen, H. C. Huang, H. C. Yang, and F. T. Cheng, "Development of an AVM System Implementation Framework" IEEE Transactions on Semiconductor Manufacturing, 25, 598-613 (2012)

M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica, "Improving MapReduce Performance on Heterogeneous Environments" Proceedings of the 8th USENIX conference on Operating systems design and implementation (OSDI'08), pp. 29-42, December (2008)

Q. Zhang, L. Cheng, and R. Boutaba, "Cloud Computing: State-of-the-Art and Research Challenges" Journal of Internet Services and Applications, 1, 7-18 (2010)

H. Chen, Y. Qiao, "Research of Cloud Computing based on the Hadoop platform" International Conference on Computational and Information Sciences (ICCIS), pp. 181-184, October (2011)