具高效率及成本效益軟體測試方法之設計與分析

Design and Analysis of High-Efficient and Cost-Effective Software Testing Methods

張君儒(Jun-Ru Chang)

指導教授 ： 黃慶育

國立清華大學/電機資訊學院/資訊系統與應用研究所/博士(2011年)

https://doi.org/10.6843/NTHU.2011.00534

摘要

The rapid development of technology provides high performance and reliability for the hardware system; based on this, software engineers can focus their developed software on more convenience and ultra high reliability. To reach this goal, the testing stage of software development life cycle (SDLC) usually takes more time and effort due to the growing complexity of the software. In general, users would like to use the convenience or artistic software, thus command-mode applications are gradually replaced by GUI-based applications in recent years. In today's modern society, we usually use all kinds of GUI applications frequently when using the desktop computer, notebook or mobile phone. However, testing the correctness of a GUI-based application is more complex than the conventional code-based application. In addition to testing the underlying codes of the GUI application, the space of possible combinations of events with a large GUI input sequence also requires creating numerous test cases to confirm the adequacy of the GUI testing.
It is noted that running all GUI test cases and then fixing all found bugs may be time-consuming and delaying the project completion. Hence, it is important to advance the test cases that uncover the most faults as fast as possible in the testing process. Test-case prioritization has been proposed and used in recent years because it can improve the rate of fault detection during the testing phase. However, few studies have discussed the problem of GUI test-case prioritization. To solve this issue, we propose a weighted-event flow graph for solving the non-weighted GUI test case and ranking GUI test cases based on weight scores. The weighted scores can either be ranked from high to low or be ordered by dynamic adjusted scores.
On the other hand, how to build software that can be tested efficiently has become an important topic in addition to enhancing and developing new testing methods. In the past, a dynamic technique for estimating program testability was proposed and called propagation, infection, and execution (PIE) analysis, but it requires a lot of computational overhead in estimating the testability of software components. In this dissertation, we propose an Extended PIE (EPIE) method to accelerate the conventional PIE analysis, based on generating group testability as a substitute for statement testability. Our proposed method can be systematically separated into three steps: breaking a program into blocks, dividing blocks into groups, and marking target statements. These three steps can decrease the number of analyzed statements effectively and the calculated values of testability are still acceptable.
Finally, Modified Condition/Decision Coverage (MC/DC) is a structural coverage measure and it can be used to assess the adequacy and quality of the requirements-based testing (RBT) process. NASA has been proposed a method to select the needed test cases for satisfying this criterion. However, it is noted that using NASA’s method, the selected test cases may not satisfy the original definition of the MC/DC criterion. Additionally, this method is too complex and could take a lot of time to obtain the needed test cases. In this dissertation, we will propose a classification-based algorithm to select the needed test cases. First, test cases will be classified based on the outcome value of expression and the target condition. After classifying all test cases, MC/DC pairs can be found quickly, conveniently and effectively. Also, if there are some missing (unfound) test cases, our proposed classification-based method can also suggest to developers what kinds of test cases have to be generated. Finally, some experiments are performed based upon real programs to evaluate the performance and effectiveness of our proposed classification-based algorithm.

關鍵字

測試個案排序 ； 帶有權重事件流程圖 ； 可測試性 ； PIE分析 ； 修正條件/覆蓋涵蓋率 ； 結構覆蓋率

並列摘要

由於現代科技快速演進，軟體工程師能夠在高效能及高可靠度的硬體系統環境中，開發出更為方便和高可靠度的應用程式。為了能夠生產出高可靠度的程式，在軟體開發週期中的軟體測試階段，必須要投入更多的測試時間及人力來尋找是否有錯誤隱藏在程式之中，特別是對於複雜且龐大的軟體。一般而言，使用者較喜歡使用便利且美觀的軟體，且近幾年，命令模式的程式逐漸被圖形介面的軟體所取代。在現今的社會中，無論我們使用桌上型電腦、筆記型電腦或是手機，軟體執行畫面幾乎都是圖形介面。為了要提高軟體品質，對於圖形介面軟體進行測試是必要的。可是要測試此類軟體比一般傳統程式還來的複雜，且在測試此類軟體時，需要產生各種不同的輸入來確認此程式的正確性。因此在測試過程中，會使用大量空間來暫存中間產生資料。
值得注意的是，在執行測試個案和修復找到的錯誤會花費較長的時間，進而有機會造成計劃進度的延遲。假使能在測試階段，預先執行可能能找出比較多的錯誤的測試個案，使得大部分的錯誤能在比較早的時間被發現及修正，這是一件很重要的研究。最近幾年，有些學者提出測試個案排序的方法來解決以上敘述的問題。可是非常少的研究對於圖形介面軟體的測試個案來進行排序。為了能夠解決此問題，我們提出一帶有權重的事件流程圖來排序要測試圖形介面軟體的測試個案。另外可以根據權重總和，將測試個案由高到低做排序，或是動態調整權重總合來做動態排序。
另一方面，除了發展新的軟體測試技術外，要如何撰寫出一容易測試的程式也是一個重要的議題。過去，有學者提出一方法，用來評估軟體的可測試性。此方法被稱作PIE分析。但是PIE分析需要花費大量的計算時間來評估軟體元件的可測試性。因此在本論文中，我們提出一方法，用計算群體的可測試性取代計算每一行程式碼的可測試性，進而減少計算時間。我們所提出的方法可分做三個步驟：先將程式分成數個區塊，再將每個區塊分成數個群體，最後標記要計算可測試性的程式碼。經過此三步驟，需要計算可測試性的程式碼數量就會有效地減少，並且所計算出來的可測試性的預估值也能夠被接受。
在軟體測試過程中，通常會計算程式涵蓋率來當作是否停止測試的準則。而修正條件/覆蓋涵蓋率(MC/DC)是一個高階涵蓋率準則。NASA過去已提出要如何實做此準則來篩選測試個案。然而經由此方法所篩選出來的測試個案無法符合原本修正條件/覆蓋涵蓋率的定義。另外在執行NASA所提出的方法會花費大量的時間，且此方法在實作上也太過於複雜。因此，本篇論文提出基於測試個案分類的演算法來有效地篩選測試個案，且被篩選出來的測試個案能夠符合修正條件/覆蓋涵蓋率準則的要求。當在對測試個案做分類時，會根據判斷式的值和每個條件的值來做為分類的依據。由於測試個案在前一步驟已經先分類完成，因此在接下來的步驟就能夠快速且有效地尋找成對的測試個案，來符合修正條件/覆蓋涵蓋率的定義。另外，如果有缺少的測試個案能符合修正條件/覆蓋涵蓋率準則，我們也會建議開發者需要產生哪些測試個案來滿足此準則的要求。此外，在實驗部分，我們會實際在真實的程式做實驗，來觀察我們演算法的效率，並且會將我們的方法運用在減少測試個案的技術和測試個案的排序上。

並列關鍵字

Test Case Prioritization ； Weight-based Event-flow Graph ； Testability ； PIE Analysis ； Modified Condition/Decision Coverage ； Structural Coverage

參考文獻

[1] C. Y. Huang and C. T. Lin, “Analysis of Software Reliability Modeling Considering Testing Compression Factor and Failure-to-Fault Relationship,” IEEE Trans. on Computers, Vol. 59, No. 2, pp. 283-288, 2010.

[2] C. J. Hsu, C. Y. Huang, and J. R. Chang, “Enhancing Software Reliability Modeling and Prediction through the Introduction of Time-Variant Fault Reduction Factor,” Applied Mathematical Modelling, Vol. 35, No. 1, pp. 506-521, 2011.

[3] Y. Tamura and S. Yamada, “Optimisation Analysis for Reliability Assessment Based on Stochastic Differential Equation Modelling for Open Source Software,” International Journal of Systems Science, Vol. 40, No. 4, pp. 429-438, 2009.

[9] J. W. Lin and C. Y. Huang, “Analysis of Test Suite Reduction with Enhanced Tie-Breaking Techniques,” Information and Software Technology, Vol. 51, No. 4, pp. 679-690, 2009.

[10] C. Y. Huang, J. R. Chang, and Y. H. Chang, “Design and Analysis of GUI Test Case Prioritization Using Weight-Based Methods,” Journal of Systems and Software, Vol. 83, No. 4, pp. 646-659, 2010.