Luận án Tiến sĩ Bùi Thị Thiện Mỹ: Dữ liệu mất cân bằng phân loại & chấm điểm tín dụng

MINISTRY OF EDUCATION AND TRAINING UNIVERSITY OF ECONOMICS HO CHI MINH CITY BUI THI THIEN MY IMBALANCED DATA IN CLASSIFICATION: A CASE STUDY OF CREDIT SCORING DOCTORAL DISSERTATION Ho Chi Minh City - 2024 MINISTRY OF EDUCATION AND TRAINING UNIVERSITY OF ECONOMICS HO CHI MINH CITY BUI THI THIEN MY IMBALANCED DATA IN CLASSIFICATION: A CASE STUDY OF CREDIT SCORING Major: Statistics Major ID: 946.01 DOCTORAL DISSERTATION ACADEMIC ADVISORS: 1. Le Xuan Truong 2. Ta Quoc Bao Ho Chi Minh City - 2024 i STATEMENT OF AUTHENTICATION I certify that the Ph. dissertation, “Imbalanced data in classification: A case study of credit scoring”, is solely my own research.

This dissertation is only used for the Ph. degree at the University of Eco- nomics Ho Chi Minh City (UEH), and no part of it has been submitted to any other university or organization to obtain any other degree. Any studies of other authors used in this dissertation are properly cited. Ho Chi Minh City, April 2, 2024 ii ACKNOWLEDGMENT First of all, I would like to express my deepest gratitude to my supervisors, Assoc.

Le Xuan Truong and Dr. Ta Quoc Bao, for their scientific direction and dedicated guidance throughout the process of conducting this Ph. I sincerely thank the teachers of the UEH’s doctoral training program for imparting valuable knowledge, and the teachers at the Department of Mathe- matics and Statistics, UEH for their sincere comments on my dissertation. I sincerely thank Dr.

Le Thi Thanh An for her moral and academic support so that I could complete the research. Besides, I really appreciate the interest and help of my colleagues at Ho Chi Minh City University of Banking. Finally, I am grateful for the unconditional support that my mother and my family have given to me on my educational path. Ho Chi Minh City, April 2, 2024 iii TABLE OF CONTENTS STATEMENT OF AUTHENTICATION i ACKNOWLEDGMENT ii TABLE OF CONTENTS iii LIST OF ABBREVIATIONS ix LIST OF FIGURES xii LIST OF TABLES xiii ABSTRACT xv TÓM TẮT xvi 1 INTRODUCTION 1 1.1 Overview of imbalanced data in classification .3 Research gap identifications .1 Gaps in credit scoring .2 Gaps in the approaches to solving imbalanced data .3 Gaps in Logistic regression with imbalanced data .4 Research objectives, research subjects, and research scopes .5 Research data and research methods .6 Contributions of the dissertation.

14 2 LITERATURE REVIEW OF IMBALANCED DATA 16 2.1 Imbalanced data in classification .1 Description of imbalanced data .2 Obstacles in imbalanced classification .3 Categories of imbalanced data .2 Performance measures for imbalanced data .1 Performance measures for labeled outputs .2 Performance measures for scored outputs .1 Area under the Receiver Operating Character- istics Curve .2 Kolmogorov-Smirnov statistic .3 Conclusion of performance measures in imbalanced clas- sification .3 Approaches to imbalanced classification .1 Algorithm-level approach .1 Modifying the current classifier algorithms .2 Cost-sensitive learning .3 Comments on algorithm-level approach .2 Data-level approach .1 Under-sampling method .2 Over-sampling method .4 Comments on data-level approach .3 Ensemble-based approach .1 Integration of algorithm-level method and en- semble classifier algorithm .2 Integration of data-level method and ensemble classifier algorithm .3 Comments on ensemble-based approach .4 Conclusions of approaches to imbalanced data .1 Meaning of credit scoring .2 Inputs for credit scoring models .3 Interpretability of credit scoring models .4 Approaches to imbalanced data in credit scoring .5 Recent credit scoring ensemble models. 55 3 IMBALANCED DATA IN CREDIT SCORING 56 3.1 Classifiers for credit scoring .4 Lasso-Logistic regression .6 Support vector machine .7 Artificial neural network .1 Heterogeneous ensemble classifiers .2 Homogeneous ensemble classifiers .3 Conclusions of statistical models for credit scoring .2 The proposed credit scoring ensemble model base Decision tree 71 3.1 The proposed algorithms .1 Algorithm for balancing data - OUS(B ) algorithm 71 3.2 Algorithm for constructing ensemble classifier - DTE(B ) algorithm .2 Empirical data sets .1 The optimal Decision tree ensemble classifier .2 Performance of the proposed model on the Viet- namese data sets .3 Performance of the proposed model on the pub- lic data sets .5 Conclusions of the proposed credit scoring ensemble model based Decision tree .3 The proposed algorithm for imbalanced and overlapping data .1 The proposed algorithms .1 Algorithm for dealing with noise, overlapping, and imbalanced data .2 Algorithm for constructing ensemble model .2 Empirical data sets .1 Computation protocol of the Lasso Logistic en- semble .2 Computation protocol of the Decision tree en- semble .1 The optimal ensemble classifier .2 Performance of LLE(B ) .3 Performance of DTE(B ) .5 Conclusions of the proposed technique. 92 4 A MODIFICATION OF LOGISTIC REGRESSION WITH IM- BALANCED DATA 93 4.2 Weighted likelihood estimation (WLE) .3 Penalized likelihood regression (PLR) .3 The proposed works .1 The modification of the cross-validation procedure .2 The modification of Logistic regression .1 Empirical data sets .6 Important variables for output .1 Important variables for F-LLR fitted model .2 Important variables of the Vietnamese data set 112 4.5 Discussions and Conclusions .1 Summary of contributions .1 The interpretable credit scoring ensemble classifier .2 The technique for imbalanced data, noise, and overlap- ping samples .3 The modification of Logistic regression .3 Limitations and suggestions for further research. 122 LIST OF PUBLICATION 123 REFERENCES 124 viii Appendices 135 A Distance functions 136 B Pseudo-code of popular ensemble classifiers 138 C Empirical data sets 140 C.1 German credit data set (GER) .2 Vietnamese 1 data set (VN1) .3 Vietnamese 2 data set (VN2) .4 Taiwanese credit data set (TAI) .5 Bank personal loan data set (BANK) .6 Hepatitis C patients data set (HEPA) .7 The Loan schema data from lending club (US) .8 Vietnamese 3 data set (VN3) .9 Australian credit data set (AUS) .10 Credit risk data set (Credit 1) .11 Credit card data set (Credit 2) .12 Credit default data set (Credit 3) .13 Vietnamese 4 data set (VN4).

155 ix LIST OF ABBREVIATIONS ADASYN Adaptive synthetic sampling ANN Artificial neural network AUC Area under the ROC curves AUS Australian credit data set BANK Bank personal loan data set CART Classification and regression tree algorithm CHAID Chi-square automatic interaction detector algorithm CNN Condensed nearest neighbors Credit 1 Credit risk data set Credit 2 Credit card data set Credit 3 Credit default data set CSL Cost-sensitive learning CV Cross-validation procedure DA Discriminant analysis DT Decision tree DTE Decision tree ensemble classifier F-CV F-measure-oriented cross-validation procedure FICO Fair Issac Corporation FLAC Firth’s logistic regression with added covariate FLIC Firth’s logistic regression with intercept-correction F-LLR F-measure-oriented Lasso-Logistic regression FIR Firth-type - a version of Penalized likelihood regression FN, FNR False negative, False negative rate x FP, FPR False positive, False positive rate GER German credit data set HEPA Hepatitis patient data set HEOM Heterogeneous Euclidean-Overlap metric HVDM Heterogeneous value difference metric ID Imbalanced data IR Imbalanced ratio KNN K-nearest neighbor classifier KS Kolmogorov-Smirnov statistic LDA Linear discriminant analysis LLE Lasso-Logistic regression ensemble classifier LR Logistic regression LLR Lasso-Logistic regression MLE Maximum likelihood estimate NCL Neighborhood cleaning rule OSS One-side selection OUS Over-Under sampling - the proposed algorithm for balancing data PLR Penalized likelihood regression QDA Quadratic discriminant analysis RF Random forest ROC Receiver Operating Characteristics Curve ROS Random over-sampling RPART Recursive Partitioning and Regression Tree algorithm RUS Random under-sampling SMOTE Synthetic Minority Over-sampling technique xi SVM Support vector machine TAI Taiwanese credit data set TN, TNR True negative, True negative rate TP, TPR True positive, True positive rate TOUS Tomek-link -Over-Under sampling technique UCI University of California, Irvine US Loan schema data set from lending club VACM Vietnam Asset Management Company VN1 Vietnamese credit 1 data set VN2 Vietnamese credit 2 data set VN3 Vietnamese credit 3 data set VN4 Vietnamese credit 4 data set WLE Weighted likelihood estimation xii LIST OF FIGURES 2.1 Examples of circumstances of imbalanced data.2 Illustration of ROCs .3 Illustration of KS metric .4 Illustration of RUS technique .5 Illustration of CNN rule .6 Illustration of tomek-links .7 Illustration of ROS technique .8 Illustration of SMOTE technique .9 Approaches to imbalanced data in classification .1 Illustration of a Decision tree .2 Illustration of a decision boundary of SVM .3 Illustration of a two-hidden-layer ANN .4 Importance level of features of the Vietnamese data sets .5 Computation protocol of the proposed ensemble classifier .1 Illustration of F-CV .2 Illustration of F-LLR. 102 xiii LIST OF TABLES 1.1 General implementation protocol in the dissertation .2 Representatives employing the algorithm-level approach to ID .3 Cost matrix in Cost-sensitive learning .4 Summary of SMOTE algorithm .5 Representatives employing the data-level approach to ID .6 Representatives employing the ensemble-based approach to ID .1 Representatives of classifiers in credit scoring .4 Description of empirical data sets .5 Computation protocol of empirical study on DTE .6 Performance measures of DTE(B ) on the Vietnamese data sets 76 3.7 Performance of ensemble classifiers on the Vietnamese data sets 78 3.8 Performance of ensemble classifiers on the German data set .9 Performance of ensemble classifiers on the Taiwanese data set .12 Description of empirical data sets .13 Average testing AUC of the proposed ensembles .14 Average testing AUC of the models based LLR .15 Average testing AUC of the ensemble classifiers based tree .1 Cross-validation procedure for Lasso Logistic regression .2 F-measure-oriented Cross-Validation Procedure .3 Algorithm for F-LLR classifier .4 Description of empirical data sets .5 Implementation protocol of empirical study .6 Average testing performance measures of classifiers .7 Average testing performance measures of classifiers (cont.8 The number of wins of F-LLR on empirical data sets .9 Important features of the Vietnamese data set .10 Important features of the Vietnamese data set (cont.1 Algorithm of Bagging classifier .2 Algorithm of Random Forest .3 Algorithm of AdaBoost .1 Summary of the German credit data set .2 Summary of the Vietnamese 1 data set .3 Summary of Vietnamese 2 data set .4 Summary of the Taiwanese credit data set (a) .5 Summary of the Taiwanese credit data set (b) .6 Summary of the Bank personal loan data set .7 Summary of the Hepatitis C patients data set .8 Summary of the Loan schema data from lending club (a) .9 Summary of the Loan schema data from lending club (b) .10 Summary of the Loan schema data from lending club (c) .11 Summary of the Vietnamese 3 data set .12 Summary of the Australian credit data set .13 Summary of the Credit 1 data set .14 Summary of the Credit 2 data set .15 Summary of the Credit 3 data set .16 Summary of the Vietnamese 4 data set. 155 xv ABSTRACT In classification, imbalanced data occurs when there is a great difference in the quantities of classes of the training data set. This problem frequently arises in various fields, for example, credit scoring and medical diagnosis.

With imbalanced data, predictive modeling for real-world applications has posed a challenge because most machine learning algorithms are designed for balanced data sets. Therefore, addressing imbalanced data has attracted much attention from researchers and practitioners. In this dissertation, we propose solutions for imbalanced classification. Fur- thermore, these solutions are applied to a credit scoring case study.

The solu- tions are derived from three papers published in the scientific journals. • The first paper presents an interpretable decision tree ensemble model for imbalanced credit scoring data sets. • The second paper introduces a novel technique for addressing imbalanced data, particularly in the cases of overlapping and noisy samples. • The final paper proposes a modification of Logistic regression focusing on the optimization F-measure, a popular metric in imbalanced classification.

These classifiers have been trained on a range of public and private data sets with highly imbalanced status and overlapping classes. The primary results demonstrate that the proposed works outperform both traditional and some recent models. xvi TÓM TẮT Khi giải quyết các bài toán phân loại, hiện tượng dữ liệu không cân bằng xảy ra nếu các lớp trong tập huấn luyện có sự chênh lệch số phần tử đáng kể.