Doctoral dissertation in computer science enhancing performance of mathematical

MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY BUI HAI PHONG ENHANCING PERFORMANCE OF MATHEMATICAL EXPRESSION DETECTION IN SCIENTIFIC DOCUMENT IMAGES DOCTORAL DISSERTATION IN COMPUTER SCIENCE Hanoi—2021 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY BUI HAI PHONG ENHANCING PERFORMANCE OF MATHEMATICAL EXPRESSION DETECTION IN SCIENTIFIC DOCUMENT IMAGES Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION IN COMPUTER SCIENCE SUPERVISORS: 1. Hoang Manh Thang 2. Le Thi Lan Hanoi—2021 DECLARATION OF AUTHORSHIP 1, Bui Hai Phong, declare that the thesis titled "Enhancing performance of mathe- matical expression detection in scientific document images" has been entirely composed by myself. I assure some points as follows: a This work was done wholly or mainly while in candidature for a Ph.

research degree at Hanoi University of Science and Technology. a The work has not be submitted for any other degree or qualifications at Hanoi University of Science and Technology or any other institutions. a Appropriate acknowledge has been given within this thesis where reference has been made to the published work of others. a The thesis submitted is my own, except where work in the collaboration has been included.

The collaborative contributions have been clearly indicated. Hanoi, September, 2021 PhD Student SUPERVISORS 1. Hoang Manh Thang 2. Le Thi Lan ACKNOWLEDGEMENT I decided to pursue a PhD.

in Computer Science at MICA International Research Institute, Hanoi University of Science and Technology (HUST) in 2017. It has been one of the best decisions I could have made. HUST is a really special place where I have accumulated immense knowledge. I would like to thank Executive Board and all members of MICA Research Institute, HUST for the kind support in the PhD.

I wish to express my deepest gratitude to my supervisors Assoc. Hoang Manh Thang and Assoc. Le Thi Lan for their continuous instruction, advice and support in the PhD course. The thesis cannot be fulfilled without the specific direction of my supervisors.

I wish to thank all members of Computer Vision Department, MICA Research Institute, HUST for the frequent support in the PhD. T wish to thank Executive Board and all members of School of Graduate Education; School of School of Electronics and Telecommunications and School of Information and Communication Technology, HUST for the specific comments and suggestion for the thesis. I wish to thank all members of Faculty of Information Technology, Hanoi Archi- tectural University for the support in the professional work in the completion of the PhD. I wish to thank Professor Akiko Aizawa and members of Aizawa Laboratory, Na- tional Institute of Informatics, Tokyo, Japan where I have obtained many scientific experiences during the internship of the PhD.

I wish to thank anonymous reviewers for valuable comments during the completion of the PhD. I gratefully acknowledge the funding from SAHEP HUST project number T202()- SAHEP-008 and Domestic Master/ PhD Scholarship Programme of Vingroup Innova- tion Foundation 2019-2021. I wish to express my sincere gratitude to my family and friends for the continuous support. and encouragement in the completion of the PhD.

Student ii ABSTRACT Mathematical expressions (MEs) play an important role in scientific documents and a huge number of scientific documents have been produced over years. Therefore, the demand of document digitization for rescarching and studying purposes has contin- uously increased. Detection and recognition of MEs in documents are considered as essential steps for document digitization. The detection of expressions aims to locate the position of expressions within documents.

Mcanwhile, the recognition of MEs aims at converting expressions from image format to string. In the documents, mathematical expressions are classified in two categories: isolated (displayed) and inline (embedded) expressions. An isolated expression displays in a separate line, an inline expression is mixed with other components (texts). Mathematical expressions may consist of math- ematical operators (e.

Large expressions may consist of multiple text lines. Meanwhile, small expressions may consist of one character. The accuracy of the detection of isolated expressions has been gradually improved. However, the detection of inline expressions is considered as a challenging task.

In practice, the detection and recognition of MEs in document images are closely related. The accuracy of the detection allows to obtain accuracy of the recognition. In contrast, the incorrect detection may cause errors in the recognition of MEs This thesis presents three main contributions in the detection and recognition of MEs in scientific document images: (1) First, a hybrid method of two stages has been proposed for the effective detection of MEs. At first stage, the layout analysis of entire document images is introduced to improve the accuracy of text linc and word segmentation.

At second stage, both isolated and inlinc MEs in document images are detected. Both hand-crafted and deep learning features are extensively investigated and combined to improve the detection accuracy. In the handcrafted feature extraction approach, the Fast Fourier Transform (FFT) is applied for text line images for the detection of isolated MEs. The Gaussian parameters of projection profile are applied as the feature extraction for the detection of inline MEs.

After the feature extraction, various machine learning classificrs have been fine tuned for the detection. In the deep learning approach, the CNNs (Alexnet. and ResNet) have been optimized for the detection of MEs. The fusion of handcrafted and deep learning features based on the prediction scores has been applied.

The merit of the method is that it can operate directly on the ME images without the employment of character recognition. (2) Second, an end-to-end framework for mathematical expression detection in sci- iti entific document images is proposed without using any Optical Character Recognition (OCR) or Document Analysis techniques as in conventional methods. The distance transform is firstly applied for input document images in order to take advantages of the distinguished features of spatial layout of MEs. Then, the transformed images are fed into the Faster Region with Convolutional Neural Network (Faster R-CNN) that.

has been optimized to improve the accuracy of the detection. Specifically, the optimiza- tion and gencration strategies of anchor boxes of the Region Proposal Network have been proposed to improve the accuracy of expression detection of various sizes. The proposed methods for the detection of MEs have been tested on two public datascts (Marmot and GTDB). The obtained accuracies of isolated and inline expressions in the Marmot dataset.90% while those in the GTDB datasct are 91.

The performance comparison with conventional methods shows the effectiveness of the proposed method. (3) Finally, the detection and recognition of MEs have been integrated in a system. The MEs in document images have been detected and recognized. The recognition results are represented in Latex.

The application aims to support end users to use the detection and recognition of MEs in document images conveniently. Student iv CONTENTS DECLARATION OF. ii PERS ERA CE ccs 5 wcacsisin tna: main piaieraacanaienemiese wre seein 6/6199 eum nin ene nae srelatavecerecernin iii CŨOINTENTTE:Gcc¿⁄22212022221222c62 6 0146103154 60066E66%4% OTST viii ABBREVIATIONS. Hà viii LƯẾT CE TABS sca aiasac cian aia 5a asa es sara waa was aa aS Oi as RRS wa xi 0u AOE RI "ốc ca ốc.

xviii INTRODUCTION © osiccccccusceewne ees ean eases tae Cee 1 Nợ an 1 0:2: Bypotheaas iccssiescceiciennncissn ieee Re 00160 1xa8 1 0. Objectives of the thesis. 002 cc eee ee eee ee eee eee 2 0. Introduction of the ME detection and recognition.

Introduction of ME dctcction. Introduction of ME rccognition. Contributions of this thesis. cc ce eee eee neces 7 0:6: Structure of this Chests.

0s:cscs:e0ssaes-ees sais eoueansermanseinmaianisientas 8 CHAPTER 1. 10 {:1; ĐocumioHt SDRIWHNNEC c2 a1 644 cáo Cai vanitt tài gi20: 64064 824,423:G3 2:4 11 1. ME detection methods im document images.1- ile based detection sic aa as ima ina mn 13 1. Handcrafted feature extraction methods for the ME detection.

Dcep neural network for ME detection.- Đbep:nietril:HCÊNWDRNL. Deep neural network models for ME detection.Q— Sen 2n einicinieg ssmceicie sd teisee sa peeetayesamasene 22 1. Traditional approaches for ME recognition. Ncural network approaches for ME; recogmition.

Datascts and evaluation metrits 226000 60008. SE a 27 BAD A ahora at RSE RTT ES AIST 27 14:2: Bvaluation moetates s.ccisccuncins aii ae ai eR 3 50704167c4. Pxisting systems for ME rccognition. 33 1000000011710: HN To se c+oecsostoS-t2E0C.

THE DETECTION OF MEs USING THE LATE FUSION OF HANDCRAFTED AND DEEP LEARNING FEATURES. Overview of the proposed method. Page scementation a c:z:z:zz cxs:zs:zã2 3:54 án ti4at6iS4c2diá4140S8 c4 Sið2:48©-aGi 88 x46 38 2. Handerafted feature extraction for ME detection.

Handcrafted feature extraction for isolated ME detection. Handcrafted feature extraction for inline ME detection. Decp lcarning mcthod for ME detcction. Late fusion of handcrafted and deep learning features for ME detection.

Post-processing for ME detectioni. Ecperamieribal RenilENs vi ác: cv 00100012 saan s do wales Na sale as eens en 60 2. Performance evaluation of the detection of MEs using different machine krurriine6Ì19GTIHINN 22 các c2 ng lá GA na CN cial ws wide isla p duy 08 cua sua 61 2. Performance evaluation of the detection of MEs using the fusion of hand- crafted and deep learning featurcs with đifferent operations.

Performance evaluation of the detection of isolated and inline MEs on dif- ferent piiblic viitamets 1 See ee NE NE 63 2. Evaluation of the impact of image resolution on the ME detection. Evaluation of the impact of the post-proccssing. Visualization of extracted features of images using the handcrafted and deep learning feature appronchios 215.

see eu 130664 S6 0AEc4i5/403602%/11221466ic6/46 68 207-7. error: analysis and discussion 226i skeen ee 73 2. Measurement of execution: time visi. ci cede ca ee ei keels eek o's 75 2.85 Sammaty OF the CHARGE eo can cece cena gee coger ceetepeceesepeeees 76 CHAPTER 3.

THE DETECTION OF MEs USING THE COMBINATION OF THE DISTANCE TRANSFORM AND FASTER R-CNN. Overview of the proposed method for ME detection using the DT and the Faster R-ONN HidđiÍỔÍỔÕ. The detection of MES nsing the DT and the Faster R-ƠNN. Distancc transform of document Imagœ.

ME dctcction using a Fastcr R-ƠNN. H€EIỚN DEODOSBÍ TICEWDEÌE=¿. Fully connected detection nctwork. Loss function of the training Faster R-ƠNN.

cc eee ce eee eee eens 89 3. Loss function of the training process of Faster R-CNN. Evaluation of the impact of the DT and anchor box gencration to the per- formancc of the ME detection. Comparison of Faster R-ƠNN models in ME detection.

Comparison of the proposed and statc-of-the-art methods used in ME detection 95 3. Performance comparison of the proposed method on cross datasets. Illustration of feature extraction of the Resnet-50. 98 ached», MaET ONY ORLA yitths Sata) CCR 101210216 052V20446042Á60L40204412ze 102 3.

Measuremecnt of cxccutiơn tìme. Đunniuary 0Ý ho CHADECE c.- co 6c 02c ki 0024 k4n g1 nà k4 g1 4E'G2 814 4. THE DETECTION AND RECOGNITION OF MEs IN DOCUMENT IMAGRS-‹:::::-::::::2///21222c222c/: 1222 ãcvŸs 105 4. Overview of the proposed system for the detection and recognition of MESs.

ME recognition using the WAP network. Watcher modulc of the WAP network. Parser:module-of thơ WẤTP ngEWOIÌ::. Training the WAP ngtwoTÌ:.

Weaker tinea ate CaaS oats scene saree ie eae sie HỆ HE ĐC eae een mepew sien 113 4. Performance evaluation of the detection and recognition of MEs. Error analysis and discussion. Measurement of execution tĩmG.4: Qumamary:oÊ the Chap Ts2¿22 212220225224 50 c3 c6 5 S0 eee eee 120 re EIN ce nce eso rernrarecere ec mrecmpnecene yt aye aces everett moms ereyes es acu- ere ereimynonoate 121 PRB EIC AT EON so s:ccsrore sieves taraiate alatainrare era i aisle alana erage eeeaiais 123 iD UGREARIY: <5 scenes 25555 5c eo eee re ras =) dd0se SS RcseeSeaasaaseaee 125 ABBREVIATIONS No.

Abbreviation Meaning Scent aouk wwe = m mg Bì (HH Ra a won ee C 16 17 18 19 20 21 22 23 24 25 26 CNN DT ExpRate FFT Faster R-CNN GRU HOG HPP IoU kNN LSTM Mask R-CNN ME OCR ResNet RF RNN ROIs RPN SSD SVM t-SNE VPP WAP WER.