Lasso迴歸於可詮釋預測分析：強階層與樹狀結構

Lasso Regression for Interpretable Predictive Analytics: Strong Hierarchy and Tree Structure

陳婷文(Chen Ting-Wen)

指導教授 ： 莊皓鈞 ； 周彥君

國立政治大學/商學院/資訊管理學系/碩士(2020年)

https://doi.org/10.6814/NCCU202001103

摘要

有鑒於數據分析被廣泛應用在不同問題領域，且近年來資料筆數與變數數目大幅增加，以機器學習建構的預測模型因而興起，其中隨機森林和梯度提升機運用集成樹演算法，能在模型內納入自變數與依變數間的非線性關係並處理高維度資料，提升預測準確度。然而這類模型缺乏解釋性，在商業領域如金融授信風險評估難以使用，故產業界仍倚賴具高透通性的迴歸模型，但一般而言其預測準確度低於解釋性弱的集成式學習。本研究利用在高維建模相當重要的Lasso迴歸相關技術，探討兩個可大幅改善迴歸模型預測準確度並保留解釋性的方案，一為由Lim and Hastie (2015)提出運用自變數交互項拓展維度，但保留強階層使模型易解釋的Hierarchical group-lasso regularization，二為本研究提出的Cluster-while-regression with tree hierarchy，後者將樣本同步分群與訓練後產出數個迴歸模型，以分群加入非線性關係，結合樹狀結構與各子葉Lasso迴歸，以混合整數規劃進行訓練，達成模型的全域最佳化。接著以不同資料集比較以上所提到的五種演算法後，本研究運用的兩種強化版迴歸模型預測表現皆顯著優於Lasso迴歸，我們所提出的Cluster-while-regression with tree hierarchy預測準確度更不遜於隨機森林與梯度提升機，並保留高可解釋性，對可詮釋人工智慧有所貢獻。

關鍵字

詮釋性 ； Lasso迴歸 ； 機器學習 ； 樹狀結構 ； 強階層

並列摘要

Due to the availability of observational data and variables, predictive machine learning has been widely applied in different fields. Random Forests and Gradient Boosting Machine are two popular machine learning models which use ensemble trees to incorporate the nonlinear relationship between independent and dependent variables and to process high-dimensional data, resulting in improved prediction accuracy. However, these models are lack of interpretability and hence not applicable to business situations like credit risk assessment. As a results, practitioners still rely on the regression model for interpretability. To improve prediction accuracy, Lasso regression is a key technique to include high-dimensional data while avoiding overfitting. In this study, we discuss two Lasso-based models that can greatly improve prediction accuracy while retaining interpretability. One is Hierarchical group-lasso regularization, which was proposed by Lim and Hastie (2015) and uses interaction terms to expand the dimension and further enforces strong hierarchy to make the model easy to interpret. The other is Cluster-while-regression with tree hierarchy, which adds nonlinear relationships by clustering. This model simultaneously considers tree structure for clustering and runs Lasso regression for each cluster. A mixed-integer programming is applied to achieve global optimization of the model. These two enhanced Lasso regression models performs better than the traditional Lasso regression model in different datasets. Cluster-while-regression with tree hierarchy even performs not worse than Random Forests and Gradient Boosting Machine and at the same time retain high interpretability. Our study thus contributes to interpretable artificial intelligence.

並列關鍵字

Interpretability ； Lasso regression ； Machine learning ； Tree hierarchy ； Strong hierarchy

參考文獻

Alpaydin, E. (2020). Introduction to machine learning (4th ed.), America: MIT press.

Baardman, L., Levin, I., Perakis, G., & Singhvi, D. (2018). Leveraging comparables for new product sales forecasting. Production and Operations Management, 27(12), 2339-2449.

Bien, J., Taylor, J., & Tibshirani, R. (2013). A lasso for hierarchical interactions. Annals of statistics, 41(3), 1111.

Cox, D. R. (1984). Interaction. International Statistical Review/Revue Internationale de Statistique, 52(1), 1-24.

DeSarbo, W. S., Oliver, R. L., & Rangaswamy, A. (1989). A simulated annealing methodology for clusterwise linear regression. Psychometrika, 54(4), 707-736.