Luận án theo dõi và lập chỉ mục khuôn mặt trong chuỗi video 3D - Telecom ParisTECH

Face Tracking and Indexing In Video Sequences Reporters: M. Farid MELGANI Examiners: Mme. Bernadette DORIZZI Student: M. Malik MALLEM Ngoc-Trung Tran M.

Kevin BAILLY Supervisors: M. Maurice CHARBIT A thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy in the LTCI Department Telecom ParisTECH July 2015 Abstract Face tracking in video sequences is an important problem in computer vision because of multiple applications in many domains, such as: video surveillance, human computer in- terface, biometrics. Although there are many recent advances in this field, it still remains a challenging topic, in particular if 6 Degrees of Freedom (DOF) - three dimensional (3D) translation and rotation - or fiducial points (to capture the facial animation) needs to be estimated in the same time. Its challenge comes mainly from following factors: il- lumination variations, wide head rotation, expression, occlusion, cluttered background, etc.

In this thesis, contributions are made to address two of mentioned major difficul- ties: the expression and wide head rotation. We aim to build a 3D tracking framework on monocular cameras, which is able to estimate accurate 6 DOF and facial animation while being robust with wide head rotation, even profile. Our method adopt the 3D face model as a set of 3D vertices needed to compute continuous values of 6 DOF in video sequences. To track wide 3D head poses, there are significant difficulties regarding to the data collection, where the pose and/or a large number of fiducial points is generally required in order to build the statistical shape and appearance models.

The pose ground-truth is expensive to be collected because of requirement of some specific devices or sensors, while the manual annotation of correspondences on large databases can be tedious and error prone. Moreover, the problem of correspondence annotation is the difficulty of locating the hidden points of self-occlusion faces at the profile. The result is a general approach that wants to tackle the out-of-plane rotations have to usually use a view-based or adaptive models. The problem matters since the different numbers of fiducial points on views, for example, profile and frontal, that causes a gap in term of tracking between them using a single model.

The first main contribution of our thesis is to use of a large synthetic data to overcome this problem. Leveraging on the such data covering the full range of head pose and the on-line information correlating between frames, we propose the combination between them to robustify the tracking. The local features are adopted in our thesis to reduce the high variation of facial appearance when learning on a large dataset. ii To be rigorous with simultaneously tracking of the facial animation, out-of-plane ro- tations and towards to in-the-wild, the combination of synthetic and real datasets is proposed to train using cascaded-regression tracking model.

In the perspective of cas- caded regression, the local descriptor is very significant for high performance. Hence, we utilize the method of feature learning in order to allow the representation of local patches more discriminative than hand-crafted descriptors. Furthermore, the proposed method introduces some modifications of the traditional cascaded approach at later stages in order to boost the quality of the found fiducial points or correspondences. Lastly, Unconstrained 3D Pose Tracking (U3PT) dataset, our own recordings, would be nominated for the evaluation of in-the-wild face tracking in the community.

The dataset captured on ten subjects (five videos/subject) in the office environment with the cluttered background. People, who are captured, moving comfortably in front of the camera, rotating their heads in three direction (Yaw, Pitch and Roll), even 90 degree of Yaw, doing some occlusion or expression. In addition, these videos are recorded in different light conditions. With the available ground-truth of 3D pose computed using the infrared camera system, the robustness and accuracy of tracking framework, which aims to work in-the-wild, could be accurately evaluated.

Keywords: head pose tracking, pose estimation, 3D face tracking, in-the-wild face tracking, Bayesian tracking, face alignment, cascaded regression, synthetic data. iii Publications During this study, the following papers were published or under submission: U3PT: A New Dataset for Unconstrained 3D Pose Tracking Evaluation Ngoc Trung Tran, Fakhreddine Ababsa and Maurice Charbit, International Conference on Computer Analysis of Images and Patterns (CAIP), 2015. Cascaded Regression of Learning Feature for Face alignment Ngoc Trung Tran, Fakhreddine Ababsa, Sarra Ben Fredj and Maurice Charbit, Advanced Concepts for Intelligent Vision Systems (ACIVS), 2015. Towards Pose-Free Tracking of Non-Rigid Face using Synthetic Data Ngoc Trung Tran, Fakhreddine Ababsa and Maurice Charbit, International Conference on Pattern Recognition Applications and Methods (ICPRAM), 2015.

A Robust Framework for Tracking Simultaneously Face Pose and Animation using Synthesized Faces Ngoc Trung Tran, Fakhreddine Ababsa and Maurice Charbit, Pattern Recognition Letter, 2014. 3D Face Pose and Animation Tracking via Eigen-Decomposition based Bayesian Approach Ngoc-Trung Tran, Fakhreddine Ababsa, Jacques Feldmar, Maurice Charbit, Dijana Petrovska- Delacretaz and Gerard Chollet International Symposium on Visual Computing (ISVC), 2013. 3D Face Pose Tracking from Monocular Camera via Sparse Representation of Syn- thesized Faces Ngoc-Trung Tran, Jacques Feldmar, Maurice Charbit, Dijana Petrovska-Delacretaz and Gerard Chollet International Conference on Computer Vision Theory and Applications (VISAPP), 2013. Towards In-the-wild 3D Head Tracking Ngoc Trung Tran, Fakhreddine Ababsa and Maurice Charbit, Journal of Machine Vision and Applications (MVA), 2015.

(Submitted) Abbreviations 2D Two Dimensional 3D Three Dimensional AAM Active Appearance Model ASM Active Shape Model CLM Constrained Local Model DOF Degrees-of-Freedom GMM Gaussian Mixture Model IC Inverse Compositional IRLS Iteratively Re-weighted Least Squares LDM Linear Deformable Model KF Kalman Filter KLT Kanade–Lucas–Tomasi ML Maximum Likelihood MAE Mean Average Error PCA Principal Component Analysis RANSAC RANdom SAmple Consensus RGB Red Green and Blue RMS Root Mean Squared POSIT Pose from Orthography and Scaling with ITerations SDM Supervised Descent Method SfM Structure from Motion SSD Sum of Squared Differences SVD Singular Value Decomposition SVM Support Vector Machine 3DMM 3D Morphable Model Contents Contents v List of Figures viii List of Tables xi 1 Introduction 1 1. 8 2 State-of-the-art 10 2.1 Linear Deformable Models (LDMs) .1 Optical Flows Regularization .3 Multi-view Tracking .1 On-line adaptive models .2 View-based models .1 Off-line learning vs. On-line adaptation .2 Real data vs. 23 v Contents vi 3 A Baseline Framework for 3D Face Tracking 25 3.1 The General Bayesian Tracking Formulation .2 Face Tracking using Covariance Matrices of Synthesized Faces .3 Face Tracking using Feature Reconstruction through Sparse Representation 36 3.1 Reconstructed-Error Function .4 Face Tracking using Adaptive Local Models .1 Adaptive Local Models .3 Out-of-plane tracking.

47 4 Pose-Free Tracking of Non-rigid Face in Controlled Environment 49 4.2 Wide Baseline Matching as Initialization .3 Large-rotation tracking via Pose-wise Appearance Models .1 View-based Appearance Model .2 Matching Strategy by Keyframes .3 Rigid and Non-rigid Estimation using View-based Appearance Model .4 Flexible tracking via Pose-wise Classifiers .1 Templates for Matching using Support Vector Machine .2 Large off-line Synthetic Dataset .3 Local Appearance Models .4 Fitting via Pose-Wise Classifiers .3 Out-of-plane tracking. 69 5 Towards In-the-wild 3D Face Tracking 70 5.1 Cascaded Regression of Learning Features for Landmark Detection .3 Local Learning Feature .4 Coarse-to-Fine Correlative Cascaded Regression .1 Learing Feature on Inner vs Boundary points .2 Coarse-to-Fine Correlative Regression (CFCR) .2 In-the-wild 3D Face Tracking .2 Real and synthetic data for in-the-wild tracking .3 Unconstrained 3D Pose Tracking Dataset .2 Datasets for Visualization .3 Our dataset: Unconstrained Pose Tracking dataset. 99 6 Conclusion and Future Works 100 A Algorithms in Use 103 A.1 Nelder-Mead algorithm .4 Local Binary Patterns. 107 Bibliography 109 List of Figures 1.1 Three head orientations: Yaw, Pitch and Roll.2 Some challenging conditions of 3D face tracking: illumination, head rota- tion, clustered background in one sample video sequence.1 Some 3D face models from left to right: AAM, Cynlinder, Mesh and 3DMM.1 The frontal and profile views of Candide-3 model.2 One example of the perspective projection in our method to obtain 2D projected landmarks from the current model given the parameters.3 The diagram of baseline framework for 3D face tracking: data generation, training and tracking stages.4 Landmark initialization, 3D model fitting using POSIT and the rendering of synthesized images.5 The landmarks used in the framework, examples of mean and covariance matrices at the eye and mouth corners.6 The visualization of two and three first components of descriptor projec- tion using PCA learned through synthetic images of some feature points.7 Some video examples of BUFT videos at different views.8 The framework using reconstructed features via sparse representation with two modifications from the baseline.9 The sparse representation to estimate the coefficients of the positive (green) and negative (red) local patches of one corner of eyebrow.10 The construction of posistive and negative patches of eye corners.11 The framework using adaptive local models based on the baseline.12 An sample video of BUFT dataset using local adaptive model.13 Some tracking examples on BUFT dataset using local adaptive model (green) and FaceTracker (yellow).14 The visualization of our (Yaw, Pitch, Roll) estimation (blue) and ground- truth (red) on the video jam7.avi using local adaptive model.15 The visualization of our (Yaw, Pitch, Roll) estimatation (blue) and ground- truth (red) on the video vam2.avi using local adaptive model.16 The RMS error of 12 selected points for tracking in our framework (red) compared to [97] (blue).

The vertical axis is RMS error (in pixel) and the horizontal axis is the frame number.1 The basic idea of wide baseline matching. 50 viii List of Figures ix 4.2 Computing 3D points Lk as the 3D intersection points.3 From left to right: The 2D SIFT keypoints of the keyframe, SIFT match- ing, outliers removed by RANSAC.4 The pipeline of Pose-Wise Appearance Model based Framework.5 The structure of mapping table with the key as three orientation and the content as the local descriptors.6 The cross-validation to select two parameters: the number of nearest neighbors Nq and the threshold kHub. The validation of a) first row: the number of nearest poses, b) second row: kHub for |Y aw| ≤ 30◦ , and c) third row: kHub for |Y aw| > 30◦. The vertical axis is RMS error.7 Tracking examples on VidTimid and Honda/UCSD.8 The way how to pick up positive (blue) and negative samples (red), and the response map computed at the mouth corner after training.9 Some weight matrices or templates of local patches T (x) in our imple- mentation.10 From left to right of training process: 143 frontal images, landmark anno- tation and 3D model alignment, synthesized images rendering, and pose- wise SVMs training.11 a) The Candide-3 model with facial points in our method.

(b) The way to compute the response map at the mouth corner using three descriptors via SVM templates.12 The pipeline of tracking process from the frame t to t + 1.13 The RMS of our framework (red curve) and FaceTracker [97] (blue curve). The vertical axis is RMS error (in pixel) and the horizontal axis is the frame number.14 Our tracking method on some sample videos of VidTimid and Honda/UCSD.1 Some results of face alignment on some challenging images of 300-W dataset.2 The visualization of weight matrices when learning local descriptors of eye and mouth corners using RBM.3 The overview of our approach. In training, we learn sequentially RBM models, coarse-to-find regression (correlative and global regression).4 CED evaluation of local correlative regression on specific landmarks. i → j means using i-th landmark to detect j-th landmark.5 The cross-validation of feature size, the number of hiddens, the number of random samples and the number of regressors.6 The evaluation of the effect of inner and boundary points on cross-validation set of 300-W dataset.7 Some results on 300-W dataset.8 The annotation of 51 landmarks in synthetic images and its automatic rendering in different views.9 Examples of landmark annotation of Multi-PIE at different views.10 Examples of automatic rendering of synthetic images.11 Landmark detection on some large rotations using synthetic training data.

86 List of Figures x 5.12 Examples of landmark detection on real images using the model trained from the synthetic data.