針對圖形化使用者介面之標記式語言生成模型

Im2code - Markup Language Generation Model for Graphical User Interface

林哲賢(Che-Hsien Lin)

指導教授 ： 王勝德

國立臺灣大學/電機資訊學院/電機工程學研究所/碩士(2018年)

https://doi.org/10.6342/NTU201803930

摘要

開發流暢且美觀的應用程式使用者介面仰賴設計師與工程師的合作，然而，設計師創作的設計圖及工程師開發產品介面所使用的標記式語言之間存在著概念及邏輯上的差異，導致根據模式相近的設計圖微調建構標記式邏輯往往成為產品開發過程中冗餘的階段。為加速並自動化此流程，我們提出兩個基於深度殘差網路與注意力機制解碼器的簡化架構，分別代表語言翻譯模型與圖片說明模型；後者的架構節省了10 倍的訓練時間並達到與原技術相近的表現。進一步藉由視覺化與數據方法分析實驗結果，本文詳細分析了圖片說明模型（Image-caption）的架構，如何根據使用者介面提高自動生成標記式語言的效率。

關鍵字

使用者介面 ； 圖片說明 ； 機器翻譯

並列摘要

Successful application user interfaces (UI) depend on well cooperation between designers and developers. However, a conceptual gap exists between UI design and UI development, which relies on a markup language to describe the layout for devices to generate UI. And it often takes a lot of time to bridge the gap during development processes. To speed up the automated generation of UI descriptions in a markup language, we propose simplified schemes based on ResNet feature extractors and attention decoders. The proposed scheme can be further divided into two models, Im2code-MT and Im2code-IC, according to attaching feature encoder or not. Through the Im2code-IC model, we achieve the same level performance but 10 times less training period. We compare our experiments in different architectures with visualization and statistic methods. Based on our achievements, the task of UI-tomarkup-language is reformulated into image caption tasks, which would be recommended for future research.

並列關鍵字

User interface ； image caption ； machine translation

參考文獻

[1] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and vqa. arXiv preprint arXiv:1707.07998, 2017.

[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.

[3] T. Beltramelli. pix2code: Generating code from a graphical user interface screenshot. CoRR, abs/1705.07962, 2017.

[4] C. Chen, T. Su, G. Meng, Z. Xing, and Y. Liu. From ui design image to gui skeleton: A neural machine translator to bootstrap mobile gui implementation. In The 40th International Conference on Software Engineering, Gothenburg, Sweden. ACM, 2018.

[5] Y. Deng, A. Kanervisto, J. Ling, and A. M. Rush. Image-to-Markup Generation with Coarse-to-Fine Attention. ArXiv e-prints, Sept. 2016.