侵入式自動測試化與失敗測試案例叢集化

Intrusive Test Automation and Failed Test Case Clustering

薛健鑫(Hsueh Chien-Hsin)

指導教授 ： 鄭永斌

國立臺灣師範大學/理學院/資訊工程研究所/碩士(2011年)

摘要

在開發軟體的過程中，回歸測試是必不可少的程序，它可以確保開發者修改程式後，程式仍然能夠正常運行。圖形介面運用程式是最受歡迎的運用程式，而目前市面上也已經有許多的圖形介面測試工具。不幸的是，如果程式輸出是動態的圖像，大多的測試工具都很難地去驗證其正確性。在本論文中，我們提出一個測試自動化的解決方案，稱為侵入式測試自動化（Intrusive Test Automation），透過程序探測(Program Instrumentation)的技術方案來解決某些應用程序不能忠實地重放錄製/重播測試。 所謂“侵入式”，代表我們會在原本程式碼中，插入一些側錄程式行為的程式碼，藉此來完成擷取資訊以及驗證資訊。

此外，當自動化測試系統報告有數百個測試案例失敗時，我們透過集群技術找出一群失敗測試案例的代表，開發者只需要從此叢群中開始測試來減輕開發者的負擔。 本論文中提出藉由侵入式測試中所擷取到的資訊，來將所有失敗的測試案例叢集化的方法，進而選出代表測試案例。原則上，若修正了一個代表測試案例，也可以同時修正類似的測試案例。此外，我們以一個商業軟體為個案研究，並實施測試自動化與失敗測試案例群集化，分析我們所提出的方法是否有效。

關鍵字

叢集化 ； 程序探測 ； 圖型化介面錄製與重播 ； 測試自動化 ； 選擇測試案例 ； 減少測試案例

並列摘要

Regression testing is an indispensable process in software development, which ensures stable features have not been adversely broken by new changes. When GUI plays an important role in an application, a popular choice to automate the tests is applying GUI Capture/Replay tools. Unfortunately, in many applications which render images as output, the correctness of a replay run can no longer be straightforwardly verified. In this paper, we propose a test automation solution, called Intrusive Test Automation, which uses program instrumentation technique to collect the runtime internal information of a program. As a result, the correctness of a test run can be verified by the runtime traces.

In addition, when large number of failed test cases are reported by the test automation system, recommending some representative test cases as a start for debugging can be helpful to programmers. This paper proposes a clustering technique based on the information collected from the instrumented code. In principle, fixing bugs in one representative test case can fix its related failed test cases as well. A case study is presented to demonstrate the effectiveness of the approach.

並列關鍵字

Clustering ； Program instrumentation ； GUI Capture/Replay ； Test automation ； Test case selection ； Test-suite reduction

參考文獻

[3] T. S. Chow. Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng., 4:178–187, May 1978.

[7] M. J. Harrold, R. Gupta, and M. L. Soffa. A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol., 2:270–285, July 1993.

[10] J.-M. Kim and A. Porter. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the 24th International Conference on Software Engineering, ICSE ’02, pages 119–129, New York, NY, USA, 2002. ACM.

[11] D. Leon, W. Masri, and A. Podgurski. An empirical evaluation of test case filtering techniques based on exercising complex information flows. In Proceedings of the 27th international conference on Software engineering, ICSE ’05, pages 412–421, New York, NY, USA, 2005. ACM.

[12] S. Needleman and C. Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48(3):443–453, Mar. 1970.