Nghiên cứu và phát triển phương pháp nhận dạng cây đa phương thức - Luận án tiến sĩ

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY NGUYEN THI THANH NHAN INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION SUPERVISORS: 1. Le Thi Lan 2. Hoang Van Sam Hanoi − 2020 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Nguyen Thi Thanh Nhan INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION OF COMPUTER SCIENCE SUPERVISORS: 1. Le Thi Lan 2.

Hoang Van Sam Hanoi − 2020 DECLARATION OF AUTHORSHIP I, Nguyen Thi Thanh Nhan, declare that this dissertation entitled, ”Interactive and multi-organ based plant species identification”, and the work presented in it is my own. I confirm that:  This work was done wholly or mainly while in candidature for a Ph. research degree at Hanoi University of Science and Technology.  Where any part of this dissertation has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated.

 Where I have consulted the published work of others, this is always clearly at- tributed.  Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this dissertation is entirely my own work.  I have acknowledged all main sources of help.

 Where the dissertation is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself. Hanoi, January, 2020 PhD Student Nguyen Thi Thanh Nhan SUPERVISORS i ACKNOWLEDGEMENT First of all, I would like to thank my supervisors Assoc. Le Thi Lan at The International Research Institute MICA - Hanoi University of Science and Technology, Assoc. Hoang Van Sam at Vietnam National University of Forestry for their inspiration, guidance, and advice.

Their guidance helped me all the time of research and writing this dissertation. Besides my advisors, I would like to thank Dr. Vu Hai, Assoc. Tran Thi Thanh Hai for their great discussion.

Special thanks to my friends/colleagues in MICA, Hanoi University of Science and Technology: Hoang Van Nam, Nguyen Hong Quan, Nguyen Van Toi, Duong Nam Duong, Le Van Tuan, Nguyen Huy Hoang, Do Thanh Binh for their technical supports. They have assisted me a lot in my research process as well as they are co-authored in the published papers. Moreover, I would like to thank reviewers of scientific conferences, journals and protection council, reviewers, they help me with many useful comments. I would like to express a since gratitude to the Management Board of MICA In- stitute.

I would like to thank the Thai Nguyen University of Information and Commu- nication Technology, Thai Nguyen over the years both at my career work and outside of the work. student of the 911 program, I would like to thank this program for financial support. I also gratefully acknowledge the financial support for attending the conferences from the Collaborative Research Program for Common Regional Is- sue (CRC) funded by ASEAN University Network (Aun-Seed/Net), under the grant reference HUST/CRC/1501 and NAFOSTED (grant number 106. Special thanks to my family, to my parents-in-law who took care of my family and created favorable conditions for me to study.

I also would like to thank my beloved husband and children for everything they supported and encouraged me for a long time to study. Hanoi, January, 2020 Ph. Student Nguyen Thi Thanh Nhan ii CONTENTS DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT ii CONTENTS v SYMBOLS vi SYMBOLS viii LIST OF TABLES x LIST OF FIGURES xiv INTRODUCTION 1 1 LITERATURE REVIEW 10 1.1 Manual plant identification .2 Plant identification based on semi-automatic graphic tool .3 Automated plant identification .2 Automatic plant identification from images of single organ .1 Introducing the plant organs .2 General model of image-based plant identification .3 Preprocessing techniques for images of plant .1 Hand-designed features .2 Deeply-learned features .3 Plant identification from images of multiple organs .1 Early fusion techniques for plant identification from images of multiple organs .2 Late fusion techniques for plant identification from images of multiple organs .4 Plant identification studies in Vietnam .5 Plant data collection and identification systems. 44 iii 2 LEAF-BASED PLANT IDENTIFICATION METHOD BASED ON KERNEL DESCRIPTOR 45 2.1 The framework of leaf-based plant identification method .1 Pixel-level features extraction .2 Patch-level features extraction .1 Generate a set of patches from an image with adaptive size .2 Compute patch-level feature .3 Image-level features extraction .1 Results on ImageCLEF 2013 dataset .2 Results on Flavia dataset .3 Results on LifeCLEF 2015 dataset.

66 3 FUSION SCHEMES FOR MULTI-ORGAN BASED PLANT IDEN- TIFICATION 67 3.2 The proposed fusion scheme RHF .3 The choice of classification model for single organ plant identification .2 Single organ plant identification results .3 Evaluation of the proposed fusion scheme in multi-organ plant identification. 87 4 A FRAMEWORK FOR AUTOMATIC PLANT IDENTIFICATION WITHOUT DEDICATED DATASET AND A CASE STUDY FOR BUILDING IMAGE-BASED PLANT RETRIEVAL 88 4.2 Challenges of building automatic plant identification systems .3 The framework for building automatic plant identification system with- out dedicated dataset .4 Plant organ detection .5 Case study: Development of image-based plant retrieval in VnMed ap- plication. 104 CONCLUSIONS AND FUTURE WORKS 105 4. 106 Bibliography 108 PUBLICATIONS 121 APPENDIX 122 v ABBREVIATIONS No.

Abbreviation Meaning 1 AB Ada Boost 2 ANN Artificial Neural Network 3 Br Branch 4 CBF Classification Base Fusion 5 CNN Convolution Neural Network 6 CNNs Convolution Neural Networks 7 CPU Central Processing Unit 8 CMC Cumulative Match Characteristic Curve 9 DT Decision Tree 10 En Entire 11 FC Fully Connected 12 Fl Flower 13 FN False Negative 14 FP False Positive 15 GPU Graphics Processing Unit 16 GUI Graphic-User Interface 17 HOG Histogram of Oriented Gradients 18 ILSVRC ImageNet Large Scale Visual Recognition Competition 19 KDES Kernel DEScriptors 20 KNN K Nearest Neighbors 21 Le Leaf 22 L-SVM Linear Support Vector Machine 23 MCDCNN Multi Column Deep Convolutional Neural Networks 24 NB Naive Bayes 25 NNB Nearest NeighBor 26 OPENCV OPEN source Computer Vision Library 27 PC Persional Computer 28 PCA Principal Component Analysis 29 PNN Probabilistic Neural Network 30 QDA Quadratic Discriminant Analysis vi 31 RAM Random Acess Memory 32 ReLU Rectified Linear Unit 33 RHF Robust Hybrid Fusion 34 RF Random Forest 35 ROI Region Of Interest 36 SIFT Scale-Invariant Feature Transform 37 SM SoftMax 38 SURF Speeded Up Robust Features 39 SVM Support Vector Machine 40 SVM-RBF Support Vector Machine-Radial Basic Function kernel 41 TP True Positive 42 TN True Negative vii MATH SYMBOLS No. Symbol Meaning P 1 Summation - sum of all values in range of series 2 R Set of real number d 3 R Set of real number has d dimensions 4 π π = 3. 5 kwk L2 normalize of vector w 6 xi The i-th element of vector x 7 sign(x) The sign function that determines the sign. Equals 1 if x ≥ 0, −1 if x < 0 8 ∈ Is member of 9 max The function takes the largest number from a list 10 arctan(x) It returns the angle whose tangent is a given number 11 cos(θ) Function of calculating cosine value of angle θ 12 sin(θ) Function of calculating sine value of angle θ 13 m(z) The magnitude of the gradient vector at pixel z 14 θ(z) The orientation of gradient vector at pixel z 15 θ̃(z) The normalized gradient vector 16 exp(x) ex 17 argmax(x) It indicates the element that reaches its maximum value 18 ⊗ The Kronecker product 19 xT Transposition of vector x Q 20 Product of all values in range of series 21 q The query-image set 22 si (Ik ) The confidence score of the plant species i−th when using image Ik as a query from a single organ plant 23 c The predicted class of the species for the query q 24 C The number of species in dataset 25 km̃ The gradient magnitude kernel 26 ko The orientation kernel 27 kp The position kernel 28 m̃(z) The normalized gradient magnitude viii LIST OF TABLES Table 1.1 Example dichotomous key for leaves .2 Methods of plant identification based on hand-designed features .3 A summary of available crowdsourcing systems for plant informa- tion collection.4 The highest results of the contest obtained with the same recog- nition approach using hand-crafted feature.1 Leaf/leafscan dataset of LifeCLEF 2015 .2 Accuracy obtained in six experiments with ImageCLEF 2013 dataset.3 Precision, Recall and F-measure in improved KDES with Interac- tive segmentation for ImageCLEF 2013 dataset .4 Comparison of the proposed method with the state-of-the-art hand-designed features-based methods on Flavia dataset.5 Precision, Recall and F-measure of the proposed method for Flavia dataset .1 An example of test phase results and the retrieved plant list de- termination using the proposed approach.2 The collected dataset of 50 species with four organs .3 Single organ plant identification accuracies with two schemes: (1) An CNN for each organ; (2) An CNN for all organs.

The best result is in bold.4 Obtained accuracy at rank-1, rank-5 when combining each pair of organs with different fusion schemes in case of using AlexNet. The best result for each pair of organ is in bold.5 Obtained accuracy at rank-1, rank-5 when combining each pair of organs with different fusion schemes in case of using ResNet. The best result is in bold.6 Obtained accuracy at rank-1, rank-5 when combining each pair of organs with different fusion schemes in case of using GoogLeNet. The best result is in bold.7 Comparison of the proposed fusion schemes with the state of the art method named MCDCNN [76].

The best result is in bold.8 Rank number (k) where 99% accuracy rate is achieved in case of using AlexNet. The best result is in bold.9 Rank number (k) to achieve a 99% accuracy rate in case of using for ResNet. The best result is in bold.1 Plant images dataset using conventional approach .2 Plant images dataset built by crowdsourcing data collection tools.3 Dataset used for evaluating organ detection method.4 The organ detection performance of the GoogLeNet with different weight initialization.5 Confusion matrix for plant organ detection obtained (%) .6 Precision, Recall and F-measure for organ detection with Life- CLEF2015 dataset .7 Confusion matrix for detection 6 organs of 100 Vietnam species on VnDataset2 (%) .8 Four Vietnamese medicinal species databases .9 Results for Vietnamese medicinal plant identification. 102 x LIST OF FIGURES Figure 1 Automatic plant identification.

2 Figure 2 Examples of these terminologies used in the thesis. 3 Figure 3 One observation of a plant. 4 Figure 4 (a) Example of large inter-class similarity: leaves of two distinct species are very similar; (b) example of large intra-class variation: leaves of the same species vary significantly due to the growth stage. 5 Figure 5 Challenges of plant identification.

(a) Viewpoint variation; (b) Occlusion; (c) Clutter; (d) Lighting variation; (e) color variation of same species. 5 Figure 6 Confusion matrix for two-class classification. 6 Figure 7 A general framework of plant identification.1 Botany students identifying plants using manual approach .2 (a) Main graphical interface of IDAO; (b), (c), (d) Graphical icons for describing characteristics of leaf, fruit and flower respectively.3 Snapshots of Leafsnap (left) and Pl@ntNet (right) applications 13 Figure 1.4 Some types of leaves: a,b) leaves on simple and complex back- ground of the Acer pseudop latanus L, c) a single leaf of the Cercis siliquastrum L, d) a compound leaf of the Sorbus aucuparia L,.5 Illustration of flower inflorescence types (structure of the flower(s) on the plant, how they are connected between them and within the plant) [11].6 The visual diversity of the stem of the Crataegus monogyna Jacq.7 Some examples branch images .8 The entire views for Acer pseudoplatanus L.9 Fundamental steps for image-based plant species identification .10 Accuracy of plant identificaiton based on leaf images on complex background in the ImageCLEF 2012 [18] .11 Feature visualization of convolutional net trained on ImageNet from [58] .