Luận án Tiến sĩ: Định vị và Định danh Người dùng bằng Kết hợp Hình ảnh và Wifi

MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY THUY PHAM THI THANH NGHIÊN CỨU VÀ PHÁT TRIỂN CÁC KỸ THUẬT ĐỊNH VỊ VÀ ĐỊNH DANH KẾT HỢP THÔNG TIN HÌNH ẢNH VÀ WIFI PERSON LOCALIZATION AND IDENTIFICATION BY FUSION OF VISION AND WIFI DOCTORAL THESIS OF COMPUTER SCIENCE Hanoi − 2017 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY THUY PHAM THI THANH NGHIÊN CỨU VÀ PHÁT TRIỂN CÁC KỸ THUẬT ĐỊNH VỊ VÀ ĐỊNH DANH KẾT HỢP THÔNG TIN HÌNH ẢNH VÀ WIFI PERSON LOCALIZATION AND IDENTIFICATION BY FUSION OF VISION AND WIFI Specialization: Computer Science Code No: 62480101 DOCTORAL THESIS OF COMPUTER SCIENCE SUPERVISORS: 1. Thi Lan Le 2. Trung Kien Dao Hanoi − 2017 DECLARATION OF AUTHORSHIP I, Thuy Pham Thi Thanh, declare that this thesis titled, "Person Localization and Identification by Fusion of Vision and WiFi", and the work presented in it are my own. I confirm that:  This work was done wholly or mainly while in candidature for a PhD research degree at Hanoi University of Science and Technology.

 Where any part of this thesis has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated.  Where I have consulted the published work of others, this is always clearly at- tributed.  Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work.

 I have acknowledged all main sources of help.  Where the thesis is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself. Hanoi, January 2017 PhD STUDENT Thuy Pham Thi Thanh SUPERVISORS Assoc. Thi Lan Le Dr.

Trung Kien Dao i ACKNOWLEDGEMENT This thesis is done at International Research Institute MICA, Hanoi University of Science and Technology. First, I would like to express my sincere gratitude to my advisors Assoc. Thi Lan Le and Dr. Trung Kien Dao for the continuous support of my Ph.D study and related research, for their patience, motivation, and immense knowledge.

Their guidance helped me in all the time of research and writing of this thesis. I could not have imagined having a better advisor and mentor for my Ph. Besides my advisors, I would like to thank the scientists, the authors of the pub- lished works which are cited in this thesis. I am provided with valuable information resources from their works for my thesis.

In the process of implementation and comple- tion of my research, I has received many supports from the board of MICA directors. My sincere thanks go to Prof. Yen Ngoc Pham , Prof. Eric Castelli and Dr.

Son Viet Nguyen, who provided me with an opportunity to join researching works in MICA, and who gave access to the laboratory and research facilities. Without their precious support would it have been impossible to conduct this research. I would also like to thank board of directors of the University of Technology and Logistics where I worked. I received financial support and time from my office and leaders for completing my doctoral thesis.

I gratefully acknowledge the financial support for publishing papers and conference fees from research projects B2013. I would like to thank my colleagues at Computer Vision Department and Pervasive Spaces and Interaction Department for their accompaniment during my research. A special thanks to my family. Words can not express how grateful I am to my mother and father for all of the sacrifices that they have made on my behalf.

I would also like to thank to my beloved husband and my sisters. Thank you for supporting me for everything. Hanoi, January 2017 PhD Student Thuy Pham Thi Thanh ii CONTENTS DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT ii CONTENTS v SYMBOLS vi LIST OF TABLES x LIST OF FIGURES xvii 1 LITERATURE REVIEW 13 1.1 WiFi-based localization .2 Vision-based person localization .1 Motion-based detection .2 Classifier-based detection .3 Person localization based on fusion of WiFi and visual properties .4 Vision-based person re-identification. 29 2 WIFI-BASED PERSON LOCALIZATION 30 2.2 Probabilistic propagation model .2 Reduction of Algorithm Complexity .3 Fingerprinting database and KNN matching .1 Testing environment and data collection .2 Experiments for propagation model.

49 iii 3 VISION-BASED PERSON LOCALIZATION 50 3.1 Chromaticity-based feature extraction and shadow-matching score calculation .2 Shadow-matching score utilizing physical properties .3 Density-based score fusion scheme .1 Fusion of background subtraction and HOG-SVM .1 Dataset and evaluation metrics .5 Person tracking and localization .2 Person tracking and data association .3 Person localization and linking trajectories in camera network .2 Linking person’s trajectories in camera network .2 Evaluation metrics for person tracking in one camera FOV. 90 4 PERSON IDENTIFICATION AND RE-IDENTIFICATION IN A CAMERA NETWORK 92 4.1 Face recognition system .3 Testing data and results .2 Appearance-based person re-identification .2 Improved kernel descriptor for human appearance .2 Results and discussion. 115 5 FUSION OF WIFI AND CAMERA FOR PERSON LOCALIZA- TION AND IDENTIFICATION 117 5.1 Fusion framework and algorithm .2 Dataset and Evaluation .1 Experimental results on script 1 data .2 Experimental results on script 2 data. 132 PUBLICATIONS 137 BIBLIOGRAPHY 139 A 154 v ABBREVIATIONS TT Abbreviation Meaning 1 AHPE Asymmetry based Histogram Plus Epitome 2 AmI Ambient Intelligent 3 ANN Artificial Neural Network 4 AoA Angle of Arrival 5 AP Access Point 6 BB Bounding Box 7 BGS Background Subtraction 8 CCD Charge-Coupled Device 9 DSM Direct Stein Method 10 EM Expectation Maximization 11 FAR False Acceptance Rate 12 FN False Negative 13 FOV Field of View 14 FP False Positive 15 fps f rame per second 16 JPDAF Joint Probability Data Association Filtering 17 GA Genetic Algorithm 18 GLOH Gradient Location and Orientation Histogram 19 GLONASS Global Navigation Satellite System 20 GMM Gaussian Mixture Model 21 GMOTA Global Multiple Object Tracking Accuracy 22 GNSS Global Navigation Satellite Systems 23 GPS Global Positioning System 24 HOG Histogram of Oriented Gradient 25 HSV Hue Saturation Value 26 ID Identity 27 IP Internet Protocol 28 KLT Kanade Lucas Tomasi 29 KNN K-Nearest Neighbors 30 LAP Linear Assignment Problem vi 31 LBP Local Binary Pattern 32 LBPH Local Binary Pattern Histogram 33 LDA Linear Discriminant Analysis 34 LMNR Large Margin Nearest Neighbor 35 LoB Line of Bearing 36 LOS Line of Sight 37 LR Large Region 38 MAC Media Access Control 39 MHT Multiple Hypothesis Tracking 40 MOTA Multiple Object Tracking Accuracy 41 MOG Mixture of Gaussian 42 MOTP Multiple Object Tracking Precision 43 MSCR Maximally Stable Colour Regions 44 NLoS None-Line-of-Sight 45 PCA Principal Component Analysis 46 PDF Probability Distribution Function 47 PLS Partial Least Squares 48 PNG Portable Network Graphics 49 PPM Probabilistic Propagation Model 50 RBF Radial Basis Function 51 RDC Relative Distance Comparison 52 Re-ID Re-Identification 53 RFID Radio Frequency Identification 54 RGB Red Green Blue 55 ROI Region of Interest 56 RSS Received Signal Strength 57 RSSI Received Signal Strength Indication 58 SD Shadow 59 SDALF Symmetry Driven Accumulation of Local Features 60 SIFT Scale Invariant Feature Transform 61 SKMGM Spatial Kinetic Mixture of Gaussian Model 62 SLAM Simultaneous Localization and Mapping 63 SMP Stable Marriage Problem 64 SPOT Structure Preserving Object Tracker 65 SR Small Region vii 66 STGMM Spatio Temporal Gaussian Mixture Model 67 STL Standard Template Library 68 SURF Speeded Up Robust Features 69 SVM Support Vector Machine 70 SVR Support Vector Regression 71 TAPPMOG Time Adaptive Per Pixel Mixtures of Gaussians 72 TDoA Time Difference of Arrival 73 TLGMM Two-Layer Gaussian Mixture Model 74 TN True Negative 75 ToA Time of Arrival 76 TP True Positive 77 UWB Ultra-Wideband 78 VOR VHF Omnidirectional Range 79 WLAN Wireless Local Area Network 80 WPS WiFi Positioning System viii LIST OF TABLES Table 2.1 Genetic algorithm configuration .2 Optimized system parameters for the first and the second scenarios of testing environments .3 Evaluations for the first scenario with distance and RSSI features.4 Localization results for the first scenario using different features of distance and RSSI, without using coefficient λ.5 Evaluations for the second scenario with distance and RSSI features.1 Performance of human detectors with HOG-SVM method, com- bination of HOG-SVM and Adaptive GMM with and without shadow removal (SR) on MICA2 dataset.2 Reference points on image and floor plan coordinate systems.3 Evaluations for homography transformation.4 Testing results of person tracking and localization in Cam1’s FOV.5 Testing results of person tracking and localization in Cam2’s FOV.6 Testing results of person tracking and localization in Cam4’s FOV.1 Comparative face recognition results and time consuming for gallery and probe sets.2 Datasets for person re-identification testing.

In the last column, √ the number of sign ( ) shows the ranking for intra-class variation of the datasets.3 The comparative evaluations of Person Re-ID on HDA dataset.4 The testing results on iLIDS VID dataset for the proposed method, original KDES and the method in [157].5 The comparative evaluations on Rank 1 (%) for person Re-ID with different methods and datasets. (The sign ”×” indicates no information available. For iLIDS dataset, there are two data settings as described in [19] and in [10]).1 The comparative results of the proposed fusion algorithm against the evaluations in chapter 4 with testing data of script 1.2 The experimental results for person tracking by identification and person Re-ID with the second dataset.1 Technical information of WiFi-based localization system.2 Technical information of vision-based localization system.3 Technical information of fusion-based localization system. 157 x LIST OF FIGURES Figure 1 Person surveillance context in indoor environment.

4 Figure 2 Multimodal localization system fusing WiFi signals and images. 4 Figure 3 Surveillance region with WiFi range covers disjoint camera FOVs. 6 Figure 4 Framework for person localization and identification by fusion of WiFi and camera.1 Flowchart of WiFi-based person localization.2 The angle-based positioning technique using AoA.3 The position of a mobile client is determined by (a) the inter- section of three circle (circular trilateration) with the radius for each is the distance di and (b) the intersection of two hyperbolas (hyperbolic lateration).4 Framework of person localization system using fixed cameras.5 Human detection in an image [149].6 Human detection results with shadows (the images on the left column) and without shadows (the images on the right column) [134].7 Human tracking and Re-ID results in two different cameras [22].8 Camera-based localization system in [154]: (a) Original frame, (b) Foreground segmentation by MOG, (c) Extraction of foot region from (b), (d) Gaussian kernel of (c) is mapped on floor plan.1 Diagram of the proposed WiFi-based object localization system.2 An example of radio map with a set of pi RPs and the distance values di (L) from each RP to L APs.3 WiFi signal attenuation through walls/floors.4 Optimization of system parameters using GAs.5 Weights of different values of θ based on dissimilarity.6 Weights of different values of λ based on dissimilarity.7 Distribution of APs in the first scenario of testing environment.8 Ground plan of the second floor in the second testing scenario.9 Radio map (a) with (b) 2000 fingerprint locations collected on the 8th floor in the first testing scenario.10 Radio map (a) with (b) 1200 fingerprint locations collected on the 2nd floor in the second testing scenario.11 Deterministic propagation model compared to measurements in the first scenario.12 Deterministic propagation model compared to measurements in the second scenario.13 Probabilistic propagation model for the first scenario.14 Probabilistic propagation model for the second scenario.15 Localization results for the first scenario, with distance and RSSI features.16 Distribution of localization error for distance and RSSI features in the first scenario.17 Localization reliability for distance and RSSI features in the first scenario.18 Localization results for distance and RSSI features, without using coefficient λ.19 Distribution of localization error for distance and RSSI features, without using coefficient λ .20 Localization reliability for distance and RSSI features, without using coefficient λ .21 Localization results for the second scenario, with distance and RSSI features.