Tách nguồn âm thanh sử dụng mô hình NMF - Luận án tiến sĩ Duong Thi Hien Thanh

MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY DUONG THI HIEN THANH AUDIO SOURCE SEPARATION EXPLOITING NMF-BASED GENERIC SOURCE SPECTRAL MODEL DOCTORAL DISSERTATION OF COMPUTER SCIENCE Hanoi - 2019 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY DUONG THI HIEN THANH AUDIO SOURCE SEPARATION EXPLOITING NMF-BASED GENERIC SOURCE SPECTRAL MODEL Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION OF COMPUTER SCIENCE SUPERVISORS: 1. NGUYEN QUOC CUONG 2. NGUYEN CONG PHUONG Hanoi - 2019 DECLARATION OF AUTHORSHIP I, Duong Thi Hien Thanh, hereby declare that this thesis is my original work and it has been written by me in its entirety. I confirm that: • This work was done wholly during candidature for a Ph.

research degree at Hanoi University of Science and Technology. • Where any part of this thesis has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated. • Where I have consulted the published work of others, this is always clearly at- tributed. • Where I have quoted from the work of others, the source is always given.

With the exception of such quotations, this thesis is entirely my own work. • I have acknowledged all main sources of help. • Where the thesis is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself. Hanoi, February 2019 Ph.

Student Duong Thi Hien Thanh SUPERVISORS Assoc. Nguyen Quoc Cuong Dr. Nguyen Cong Phuong i ACKNOWLEDGEMENT This thesis has been written during my doctoral study at International Research Institute Multimedia, Information, Communication, and Applications (MICA), Hanoi University of Science and Technology (HUST). It is my great pleasure to thank numer- ous people who have contributed towards shaping this thesis.

First and foremost I would like to express my most sincere gratitude to my supervi- sors, Assoc. Nguyen Quoc Cuong and Dr. Nguyen Cong Phuong, for their great guidance and support throughout my Ph. I am grateful to them for devoting their precious time to discussing research ideas, proofreading, and explaining how to write good research papers.

I would like to thank them for encouraging my research and empowering me to grow as a research scientist. I could not have imagined having a better advisor and mentor for my Ph. I would like to express my appreciation to my supervisor in Master cource, Prof. Nguyen Thanh Thuy, School of Information and Communication Technology - HUST, and Dr.

Nguyen Vu Quoc Hung, my supervisor in Bachelors course at Hanoi National University of Education. They had shaped my knowledge for excelling in studies. In the process of implementation and completion of my research, I have received many supports from the board of MICA directors and my colleagues at Speech Com- munication department. Particularly, I am very much thankful to Prof.

Pham Thi Ngoc Yen, Prof. Eric Castelli, Dr. Nguyen Viet Son and Dr. Dao Trung Kien, who pro- vided me with an opportunity to join researching works in MICA institute and have access to the laboratory and research facilities.

Without their precious support would it have been being impossible to conduct this research. My warmly thanks go to my colleagues at Speech Communication department of MICA institute for their useful comments on my study and unconditional support over four years both at work and outside of work. I am very grateful to my internship supervisor Prof. Nobutaka Ono and the mem- bers of Ono’s Lab at the National Institute of Informatics, Japan for warmly welcoming me into their lab and the helpful research collaboration they offered.

I much appreciate his help in funding my conference trip and introducing me to the signal processing research communities. I would also like to thank Dr. Toshiya Ohshima, MSc. Yasu- taka Nakajima, MSc.

Chiho Haruta and other researchers at Rion Co., Japan for ii welcoming me to their company and providing me data for experimental. I would also like to sincerely thank Dr. Nguyen Quang Khanh, dean of Information Technology Faculty, and Assoc. Le Thanh Hue, dean of Economic Informatics Department, at Hanoi University of Mining and Geology (HUMG) where I am work- ing.

I have received the financial and time support from my office and leaders for completing my doctoral thesis. Grateful thanks also go to my wonderful colleagues and friends Nguyen Thu Hang, Pham Thi Nguyet, Vu Thi Kim Lien, Vo Thi Thu Trang, Pham Quang Hien, Nguyen The Binh, Nguyen Thuy Duong, Nong Thi Oanh and Nguyen Thi Hai Yen, who have the unconditional support and help during a long time. A special thank goes to Dr. Le Hong Anh for the encouragement and his precious advice.

Last but not the least, I would like to express my deepest gratitude to my family. I am very grateful to my mother-in-law and father-in-law for their support in the time of need, and always allow me to focus on my work. I dedicate this thesis to my mother and father with special love, they have been being a great mentor in my life and had constantly encouraged me to be a better person. The struggle and sacrifice of my parents always motivate me to work hard in my studies.

I would also like to express my love to my younger sisters and younger brother for their encouraging and helping. This work has become more wonderful because of the love and affection that they have provided. A special love goes to my beloved husband Tran Thanh Huan for his patience and understanding, for always being there for me to share the good and bad times. I also appreciate my sons Tran Tuan Quang and Tran Tuan Linh for always cheering me up with their smiles.

Without love from them, this thesis would not have been completed. Thank you all! Hanoi, February 2019 Ph. Student Duong Thi Hien Thanh iii CONTENTS DECLARATION OF AUTHORSHIP. i DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT.

iv NOTATIONS AND GLOSSARY. viii LIST OF TABLES. xi LIST OF FIGURES. AUDIO SOURCE SEPARATION: FORMULATION AND STATE OF THE ART 10 1.1 Audio source separation: a solution for cock-tail party problem .1 General framework for source separation .2 State of the art .1 Gaussian Mixture Model .2 Nonnegative Matrix Factorization .3 Deep Neural Networks .1 Interchannel Intensity/Time Difference (IID/ITD) .2 Rank-1 covariance matrix .3 Full-rank spatial covariance model .3 Source separation performance evaluation .1 Energy-based criteria .2 Perceptually-based criteria.

NONNEGATIVE MATRIX FACTORIZATION 24 2.1 NMF in a nutshell .2 Cost function for parameter estimation .3 Multiplicative update rules .2 Application of NMF to audio source separation .1 Audio spectra decomposition .2 NMF-based audio source separation .3 Proposed application of NMF to unusual sound detection .2 Proposed methods for non-stationary frame detection .1 Signal energy based method .2 Global NMF-based method .3 Local NMF-based method .2 Algorithm settings and evaluation metrics .3 Results and discussion. SINGLE-CHANNEL AUDIO SOURCE SEPARATION EXPLOITING NMF-BASED GENERIC SOURCE SPECTRAL MODEL WITH MIXED GROUP SPARSITY CONSTRAINT 44 3.1 General workflow of the proposed approach .3 Model fitting with sparsity-inducing penalties .1 Block sparsity-inducing penalty .2 Component sparsity-inducing penalty .3 Proposed mixed sparsity-inducing penalty .4 Derived algorithm in unsupervised case .5 Derived algorithm in semi-supervised case .1 Semi-GSSM formulation .2 Model fitting with mixed sparsity and algorithm .2 SiSEC-MUS dataset .3 SiSEC-BNG dataset .2 Single-channel source separation performance with unsuper- vised setting .3 Results and discussion .3 Single-channel source separation performance with semi-supervised setting .3 Results and discussion. MULTICHANNEL AUDIO SOURCE SEPARATION EXPLOITING NMF-BASED GSSM IN GAUSSIAN MODELING FRAMEWORK 68 4.1 Formulation and modeling .1 Local Gaussian model .2 NMF-based source variance model .3 Estimation of the model parameters .2 Proposed GSSM-based multichannel approach .2 Proposed source variance fitting criteria .1 Source variance denoising .2 Source variance separation .3 Derivation of MU rule for updating the activation matrix .1 Dataset and parameter settings .1 Algorithm convergence: separation results as func- tions of EM and MU iterations .2 Separation results with different choices of λ and γ 81 4.3 Comparison with the state of the art. 91 CONCLUSIONS AND PERSPECTIVES.

96 LIST OF PUBLICATIONS. 113 vii NOTATIONS AND GLOSSARY Standard mathematical symbols C Set of complex numbers R Set of real numbers Z Set of integers E Expectation of a random variable Nc Complex Gaussian distribution Vectors and matrices a Scalar a Vector A Matrix T A Matrix transpose H A Matrix conjugate transposition (Hermitian conjugation) diag(a) Diagonal matrix with a as its diagonal det(A) Determinant of matrix A tr(A) Matrix trace A B The element-wise Hadamard product of two matrices (of the same dimension) with elements [A B]ij = Aij Bij .(n) A The matrix with entries [A]ij kak1 `1 -norm of vector kAk1 `1 -norm of matrix Indices f Frequency index i Channel index j Source index n Time frame index t Time sample index viii Sizes I Number of channels J Number of sources L STFT filter length F Number of frequency bin N Number of time frames K Number of spectral basis Mixing filters A ∈ RI×J×L Matrix of filters aj (τ ) ∈ RI Mixing filter of j th source to all microphones, τ is the time delay aij (t) ∈ R Filter coefficient at tth time index aij ∈ RL Time domain filter vector aij ∈ CL b Frequency domain filter vector aij (f ) ∈ C b Filter coefficient at f th frequency index General parameters x(t) ∈ RI Time-domain mixture signal s(t) ∈ RJ Time-domain source signals I cj (t) ∈ R Time-domain j th source image sj (t) ∈ R Time-domain j th original source signal x(n, f ) ∈ CI Time-frequency domain mixture signal J s(n, f ) ∈ C Time-frequency domain source signals I cj (n, f ) ∈ C Time-frequency domain j th source image vj (n, f ) ∈ R Time-dependent variances of the j th source Rj (f ) ∈ C Time-independent covariance matrix of the j th source Σj (n, f ) ∈ CI×I Covariance matrix of the j th source image b x (n, f ) ∈ CI×I Σ Empirical mixture covariance b x (n, f ) ∈ CI×I Σ Empirical mixture covariance V ∈ RF+×N Power spectrogram matrix W ∈ RF+×K Spectral basis matrix H ∈ RK×N + Time activation matrix F ×K U ∈ R+ Generic source spectral model ix Abbreviations APS Artifacts-related Perceptual Score BSS Blind Source Separation DoA Direction of Arrival DNN Deep Neural Network EM Expectation Maximization ICA Independent Component Analysis IPS Interference-related Perceptual Score IS Itakura-Saito ISR source Image to Spatial distortion Ratio ISTFT Inverse Short-Time Fourier Transform IID (i.d) Interchannel Intensity Difference ITD (i.d) Interchannel Time Difference GCC-PHAT Generalized Cross Correlation Phase Transform GMM Gaussian Mixture Model GSSM Generic Source Spectral Model KL Kullback-Leibler LGM Local Gaussian Model MAP Maximum A Posteriori ML Maximum Likelihood MU Multiplicative Update NMF Non-negative Matrix Factorization OPS Overall Perceptual Score PLCA Probabilistic Latent Component Analysis SAR Signal to Artifacts Ratio SDR Signal to Distortion Ratio SIR Signal to Interference Ratio SiSEC Signal Separation Evaluation Campaign SNMF Spectral Non-negative Matrix Factorization SNR Signal to Noise Ratio STFT Short-Time Fourier Transform TDOA Time Difference of Arrival T-F Time-Frequency TPS Target-related Perceptual Score x LIST OF TABLES 2.1 Total number of different events detected from three recordings in spring 40 2.2 Total number of different events detected from three recordings in sum- mer .3 Total number of different events detected from three recordings in winter 42 3.1 List of snip songs in the SiSEC-MUS dataset.2 Source separation performance obtained on the Synthetic and SiSEC- MUS dataset with unsupervised setting.3 Speech separation performance obtained on the SiSEC-BGN. ∗ indi- cates submissions by the authors and “-” indicates missing information [81, 98, 100].4 Speech separation performance obtained on the Synthetic dataset with semi-supervised setting.1 Speech separation performance obtained on the SiSEC-BGN-devset - Comparison with closed baseline methods.