Luận án: Phân loại mã độc Android sử dụng học sâu - Lê Đức Thuận, ĐH Bách Khoa HN

MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY LE DUC THUAN ANDROID MALWARE CLASSIFICATION USING DEEP LEARNING DOCTORAL DISSERTATION OF COMPUTER ENGINEERING Hanoi−2024 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY LE DUC THUAN ANDROID MALWARE CLASSIFICATION USING DEEP LEARNING Major: Computer Engineering Code: 9480106 DOCTORAL DISSERTATION OF COMPUTER ENGINEERING SUPERVISORS Dr. Nguyen Kim Khanh Dr. Hoang Van Hiep Hanoi−2024 DECLARATION I certify that this is my research work under the guidance of my supervisor and scientists. References used in the Dissertation have been fully cited.

The data and results in the Dissertation are truthful and have never been published by other authors. Hanoi, July, 2024 Supervisors Dissertation Author Dr. Nguyen Kim Khanh Le Duc Thuan Dr. Hoang Van Hiep HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY ON BEHALF OF THE PRESIDENT P.

DIRECTOR OF DEPARTMENT OF ACADEMIC AFFAIRS ASSOCIATE DIRECTOR OF DEPARTMENT OF ACADEMIC AFFAIRS Assoc. Duong Ngoc Khanh ACKNOWLEDGEMENT My dissertation was realized during my doctoral course at the School of Information and Communications Technology (SoICT), Hanoi University of Science and Technology (HUST). HUST is a special place where I accumulated immense knowledge in my Ph. process is not a one-man process.

Therefore, I am heartily thankful to my supervisors, Dr. Nguyen Kim Khanh and Dr. Hoang Van Hiep, whose encouragement, guidance, and support from start to finish enabled me to develop my research skills and understanding of the subject. I have learned countless things from them.

This dissertation would not have been possible without their precious support. I would like to thank the Executive Board and all members of the Computer Engi- neering Department, SoICT, and HUST for their frequent support in my Ph. I thank my colleagues at the Academy of Cryptography Techniques for their help. Last but not least, I would like to thank my family: my parents, my wife, and my friends, who have supported me spiritually throughout my life.

They were always there to cheer me up and stand by me through good and bad times. Hanoi, July, 2024 Dissertation Author LE DUC THUAN CONTENTS CONTENTS i ABBREVIATIONS v LIST OF TABLES vi LIST OF FIGURES viii INTRODUCTION 1 1 OVERVIEW OF ANDROID MALWARE CLASSIFICATION BASED ON MACHINE LEARNING 6 1.2 Overview of Android Malware .2 Android Malware Classification Methods .1 Signature-based Method .2 Anomaly-based Method .3 Android Malware Classification Evaluation Metrics .1 Metrics for the Binary Classification Problem .2 Metrics for Multi-labelled Classification Problem .4 Android Malware Dataset .3 Machine Learning-based Method for Android Malware Classification .1 Related Works on Feature Extraction .1 Features Extraction Methods .2 Feature Augmentation Methods .3 Feature Selection Methods .2 Related Works on Machine Learning-based Methods .1 Random Forest Algorithm .2 Support Vector Machine .3 K-Nearest Neighbor Algorithm .4 Deep Belief Network .5 Convolutional Neural Network .6 Some Other Models. 47 2 PROPOSED METHODS FOR FEATURE EXTRACTION 49 2.1 Feature Augmentation based on Co-occurrence matrix .2 Raw Feature Extraction .3 Co-occurrence Matrix Feature Computation .3 Malware Classification based on CNN Model .4 Summary of Experimental Results .2 Feature Augmentation based on Apriori Algorithm .1 Introduction to Apriori Algorithm .3 Feature Set Creation .1 Raw Android Feature Set .2 The Feature Augmentation Set .3 Input Feature Normalization .4 Feature Augmentation Set .1 Experimental Dataset and Scenario .2 Experiment based on CNN Model .3 Summary of Experimental Results .3 Feature Selection Based on Popularity and Contrast Value in a Multi- objective Approach .2 Popularity and Contrast Computation .3 Pareto Multi-objective Optimization Method .4 Selection Function and Implementation .3 Summary of Experimental Results. 72 ii 3 DEEP LEARNING-BASED ANDROID MALWARE CLASSIFICA- TION 75 3.1 Applying DBN Model .2 Boltzmann Machine and Deep Belief Network .1 Restricted Boltzmann Machine .2 Deep Belief Network .3 Summary of Experimental Results .2 Applying CNN Model .2 Raw Feature Dataset .3 Malware Classification using CNN Model .4 Summary of Experimental Results .3 Proposed Method using WDCNN Model for Android Malware Classifi- cation .2 Building Components in the WDCNN Model .4 Summary of Experimental Results .4 Applying Federated Learning Model .1 Federated Learning Model .2 Implement Federated Learning Model .2 The Process of Synthesizing Weight Set .3 Summary of Experimental Results.

106 CONCLUSIONS 110 PUBLICATIONS 112 BIBLIOGRAPHY 114 iv ABBREVIATIONS No. Abbreviation Meaning 1 Acc Accuracy 2 API Application Programming Interface 3 CNN Convolutional Neural Network 4 DBN Deep Belief Network 5 DNN Deep Neural Network 6 FN False Negative 7 FP False Positive 8 GA Genetic Algorithm 9 GAN Generative Adversarial Network 10 GRB Red-Green-Blue 11 IG Information Gain 12 KNN K-Nearest Neighbors 13 LSTM Long Short-Term Memory 14 PSO Particle Swarm Optimization 15 RF Random Forest 16 RNN Recurrent Neural Network 17 SVM Support Vector Machine 18 TF-IDF Term Frequency – Inverse Document Frequency 19 TN True Negative 20 TP True Positive 21 RBM Restricted Boltzmann Machine 22 WDCNN Wide and Deep CNN 23 XML Androidmanifest.xml 24 DEX Classes.dex v LIST OF TABLES 1.1 Types of malware .2 Summary of Android malware datasets .4 Common API packages .5 Common suspicious API call .6 Some typical traffic flows .1 Details of parameters set in the CNN model .2 Classification with CNN model using accuracy measure (%) .3 Measurements evaluate effectiveness (%) .4 Details of parameters set in the CNN model .5 Classification results by CNN .6 Results of using CNN with measurements (%) .7 Details of parameters set in the CNN model for selection feature .8 Summary of feature evaluation measures selectivity functions (top (10)) – with API set .9 Summary of results with datasets and feature sets .10 Summary of results of proposed feature augmentation methods .1 Result with Acc measure (%) in scenario 1 .2 Result with Acc measure (%) in scenario 2 .3 Results with measures in scenario 3 (%) .4 Experimental results using CNN model .5 The datasets used for the experiment .6 Experimental results of Simple dataset .7 Experimental results of Complex dataset .8 Experimental results when comparing models .9 Accuracy comparison of models Features: Images 128x128 + permission + API .10 Experimental results with scenario 3 (%) .11 Average set of weights (accuracy - %) .12 Set of Weights according to the number of samples (accuracy - %) .13 Our proposed set of weights (accuracy - %) .14 Summary of results of proposed deep learning models and comparison. 107 vi LIST OF FIGURES 1.1 Architecture of Android OS system [37] .2 The increase of malware on Android OS .3 Types of malware on Android OS .4 Anomaly-Based Detection/Classification Technique .5 Overview of the problem of detecting malware on the Android .6 General model of feature extraction methods .7 Statistics of papers using traditional machine learning and deep learning from 2019-2022 on dblp .8 Architecture of the CNN model [118] .1 Evaluation model for Android malware classification using co-occurrence matrix .2 Output matrix with different size .3 Top (10) malware families in Drebin dataset .4 CNN having multi-convolutional networks .5 The process of research and experiment using Apriori .6 Apply the Apriori algorithm to the feature set .7 Architecture of CNN model used in the experiment with Apriori .8 Learning method implementation results .9 Proposing a feature selection model .10 Top (20) family of malware with the most samples in the AMD dataset 67 2.11 Experimental model when applying feature selection algorithm .12 Experimental results when applying feature selection algorithm .1 System development and evaluation process using the DBN .2 Architectural diagram of DBN application in Android malware detection 78 3.3 The overall model of the training and classification of malware using the CNN model .4 Test rate according to the 10-fold .5 WDCNN model operation diagram .6 Structure and parameters of the WDCNN model .7 Top 20 malware family AMD and Drebin .9 Classification of malware depending on the number of labels .10 DEX file size by size in the Drebin dataset .11 Overall model using federated learning .12 Compare the results of the weighted aggregation methods .13 Classification results with influence factor. 106 viii INTRODUCTION In the present day, there is a growing inclination towards the adoption of digital transformation and artificial intelligence in smart device applications across diverse operating systems.

This trend aligns with the advancements of the fourth industrial revolution and is being observed in numerous domains of social and economic activity. According to the statistics [1] in June 2023, Android dominated the market for mobile operating systems with 70. Furthermore, the Android operating system is utilized in a diverse range of smart devices, including but not limited to mobile phones, televi- sions, watches, automobiles, vending machines, and network routers. The rapid growth and variety of devices that use the Android operating system (OS) have contributed to the significant increase in the number, style, and appearance of malware.

Accord- ing to the statistics [2], in 2021, there were a total of 3.36 million malware found in the Android OS market. This situation leads to danger for users of mobile operating systems. Solving the problems of malware detection/classification is, therefore, urgent and necessary. As reported in the DBLP database [3] from 2013 to 2022, there were 1,081 researches on this issue.

Two main approaches are commonly applied to detect Android malware: static and dynamic analysis [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]. Static analysis involves inspecting a program’s executable file structure, characteristics, and source code. The advantage of static analysis is that it does not require that the code be executed (of course, it is pretty dangerous to run a malware file on a natural system). By examining the decompiled code, the static analysis can determine the flows and actions of the execution file and thus identify it as either malware or benign.

The disadvantage, however, is that some sophisticated malware can include malicious runtime behavior that can go undetected. On the other hand, dynamic analysis involves executing potentially malicious code in a real or sandbox environment to monitor its behavior. The sandbox environment helps analysts examine potential threats without putting the system at risk of infection. Although dynamic analysis could detect threats that might be ignored by static analy- sis, this approach requires more time and resources than static analysis.

It may not be able to cover all the possible execution paths of the malware. In summary, static analy- sis is said to help find known threats and vulnerabilities. In contrast, dynamic analysis is suitable for finding new types and uncovering threats not previously documented (i., zero-day threats). For the problem of malware detection/classification, dynamic analysis seems recommended for organizations that need a deeper understanding of malware behavior or impact and have the necessary tools and expertise to perform it.

For the problem of malware classification, static analysis is more popular due to 1 its more straightforward implementation. This dissertation uses static analysis as the main method for feature extraction. Malware classification assigns malware samples into specific malware families, in- cluding benign ones. Signature-based and machine learning-based methods have usu- ally been used for this problem.

Signature-based methods have been traditional and widely used [15, 16, 17]. They rely on matching the "signature" of known malware sam- ples with unknown ones. As mentioned in the previous paragraph, static or dynamic analysis can extract the "signature" from samples. Several limitations of signature- based methods exist as follows: (i) they cannot detect new or unknown malware; (ii) they are vulnerable to obfuscation and encryption techniques used by malware authors to evade detection; and (iii) they require constant updates of the signature database.

Machine-learning-based methods are emerging and promising techniques for malware classification. They use various machine learning algorithms to learn from a large set of labeled malware samples and then classify new ones based on their features. Conversely, machine-learning-based methods can overcome some of the challenges of signature-based methods, such as detecting new or unknown malware, handling com- plex or dynamic code features, and reducing human intervention and manual analysis.