A Study on Statistical Machine Translation of Legal Sentences - Nghiên cứu của Bùi Thanh Hưng

A Study on Statistical Machine Translation of Legal Sentences by BUI THANH HUNG submitted to Japan Advanced Institute of Science and Technology in partial fulfillment of the requirements for the degree of Doctor of Philosophy Supervisor: Professor AKIRA SHIMAZU School of Information Science Japan Advanced Institute of Science and Technology June, 2013 i Abstract Machine translation is the task of automatically translating a text from one natural language into another. Statistical machine translation (SMT) is a machine translation paradigm where translations are generated on the basis of statistical models whose parameters are derived from the analysis of bilingual text corpora (Philipp Koehn, 2010). Many translation models of statistical machine translation are proposed such as word-based, phrase-based, syntax-based, a combination of phrase-based and syntax-based translation, and hierarchical phrase-based translation. Phrase-based and hierarchical-phrase-based model (tree-based model) have become the majority of research in recent years, however they are not powerful enough to legal translation.

Legal translation is the task of how to translate texts within the field of law. Translating legal texts automatically is one of the difficult tasks because legal translation requires exact precision, authenticity and a deep understanding of law systems. The problem of translation in the legal domain is that legal texts have some specific characteristics that make them different from other daily-use documents as follows:  Because of the meticulous nature of the composition (by experts), sentences in legal texts are usually long and complicated.  In several language pairs such as Vietnamese-English and Japanese-English the target phrase order differs significantly from the source phrase order, selecting appropriate synchronous context-free grammars translation rule (SCFG) to improve phrase- reordering is especially hard in the hierarchical phrase-based model  The terms (name phrases) for legal texts are difficult to translate as well as to understand.

Therefore, it is necessary to find ways to take advantage to improve legal translation. To deal with three problems mentioned above, we propose a new method for translating a legal sentence by dividing it based on the logical structure of a legal sentence, using rule selection to improve phrase-reordering for the hierarchical phrase-based machine translation, and propose paraphrasing to increase translation. For the first problem mentioned above, we propose dividing and translating legal text basing on the logical structure of a legal sentence. We recognize the logical structure of a legal sentence using statistical learning model with linguistic information.

Then we segment a legal ii sentence into parts of its structure and translate them with statistic machine translation models. In this study, we applied the phrased-based and the tree-based models separately and evaluated them with baseline models. For the second problem, we propose a maximum entropy based rule selection model for the tree-based model, the maximum entropy based rule selection model combines local contextual information around rules and information of sub-trees covered by variables in rules. For the last problem, we propose sentence paraphrasing and noun phrase paraphrasing approach.

We apply a monolingual sentence paraphrasing method for augmenting the training data for statistical machine translation systems by creating it from data that is already available. We generate named-entity recognition (NER) training data automatically from a bilingual parallel corpus, employ an existing high-performance English NER system to recognized name- entities at the English side, and then project the labels to the Japanese side according to the word alignment. We apply splitting the long sentence into several noun phrases that could be translates independently. With this method, our experiments on legal translation show that the method achieves better translations.

Keywords: phrase-based machine translation; tree-based machine translation; logical structure of a legal sentence; CRFs; Maximum Entropy Model, rule selection; linguistic and contextual information; paraphrasing, NER iii Acknowledgments Firstly, I would like to thank my supervisor, Professor Akira Shimazu for his kindly guidance, warn encouragement and helpful support. He has given me much invaluable knowledge not only how to formulate research ideal or to write a good paper but also the vision and much useful experiment in the academic life. I would like to thank Professor Kiyoaki Shirai, who has been discussing and giving me inspirations. I would like to thank Professor Hiroyuki Iida for his help in my sub-theme research.

He has given me as good as possible conditions for my work during this time. I would like to thank Associate Professor Nguyen Le Minh. He is a respectable dedicated person. He always gave me all the time and supported everything I needed from using software tools to listening to my problems, making kind suggestion.

I also appreciate the help and the encouragement from professor Ho Tu Bao, professor Duong Anh Duc, professor Le Hoai Bac, professor Dinh Dien and many other faculty members of Ho Chi Minh University of Science and Ha Noi University of Technology. A special thank to colleagues and friends in Shimazu-Lab, Shirai-Lab and in JAIST from the first day I came to Japan. I have received a lot of help from them. They gave me invaluable advices, comments, and most importantly cheered me up all the time.

I am deeply indebted to the Ministry of Education and Training of Vietnam for granting me a scholarship. Thanks also to the JAIST Foundation for providing me with their travel grants which supported me to attend and present my work at international conferences I would like to thank my friends, all members of my family for sharing my happiness, difficulties all the time and supporting me as always. Finally I have to give a big thank you to my wife, my son and my daughter, without their encouragements I would never have began, and much less completed this thesis. iv Content Abstract ii Acknowledgments iv Introduction 1 1.1 Statistical Machine Translation .2 Machine Translation in Legal Domain .2 Motivation and Problem .1 Word-Based Translation Model .2 Phrase-Based Translation Model .3 Syntax-based Translation Model .4 Tree-Based Translation Model.

30 3 Dividing and Translating Legal Sentence based on Its Logical Structure 31 3.1 Logical Structure and Recognition of Logical Structure of a Legal Sentence .1 Logical Structure of a Legal Sentence .2 Recognition of the Logical Structure of a Legal Sentence .3 Translating Split Sentences with Phrase-Based and Tree-Based Models. 47 4 Rule Selection for Tree-Based Statistical Machine Translation 51 4.1 Maximum Entropy Rule Selection Model (MaxEnt RS model) .2 Lexical and Syntax for Rule Selection .2 Lexical Features of Nonterminal .3 Lexical Features around Nonterminal .3 Integrating MaxEnt RS Model into the Tree-based Translation Model .4 Detail of Experiment .5 The result and Discussion. 73 5 Paraphrasing to Increase Translation 75 5.2 Noun Phrase Paraphrasing .1 Alignment and Automatic English NER .2 Japanese NE Candidates Generation .3 Training Data Selection .4 Integrating Noun Phrase Paraphrasing into SMT. 90 6 Conclusion and Future Works 91 6.1 Summary of the Thesis.

92 Publications 94 Bibliography 95 vii List of Figures Figure 1.1: The machine translation pyramid .2: Structure of typical statistical machine translation system .3: Architecture of the statistical machine translation approach based on Bayes’ decision rule.1: The process of word-based translation .2: Phrase-based machine translation: The input is segmented into phrases, translated one-to-one into phrases in English and possibly reordered. 14 Figure 2-3: Word alignment from English to Vietnamese. 19 Figure 2-4: Word alignment from Vietnamese to English. 20 Figure 2-5: Intersection/Union of word alignment .6: Unigram matches; adapted from (Turian et al.1: Four cases of the logical structure of a legal texts sentence .2: The recognition of the logical structure of a legal sentence .3: Examples of sentence segmentation .1: Rule selection for tree-based Vietnamese-English statistical machine translation diagram .2: Sub-tree covered nonterminal X1 .3: Parent feature of sub-tree covered nonterminal X1: NP .4: Sibling feature of sub-tree covered nonterminal X1: N .5: The model of Moses-chart.

1: Semantic Representation of “For the Government, it must announce it officially without delay” .2: Paraphrase process for sentence “For the Government, it must announce it officially without delay” .3: (a) Word Alignment from English to Japanese. (b) Word Alignment from Japanese to English. (c) The Merged Result of Both Directions .4: (a) An eligible case; (b) An ineligible case. In (b), the word alignment pair ei – jk is against the rule, while l > i+3 or l < i.

84 viii List of Tables Table 3.1: A sentence with IOB notation for the sequence learning model .3: Statistics on logical parts of the corpus .4: Experimental results for recognition of the logical structure of a legal sentence 39 Table 3.5: Experiments with feature sets of Japanese sentences .6: Experiments with feature sets of English sentences .7: Statistics of the corpus .8: Statistics of the test corpus .9: Number of requisition part, effectuation part in the test data .10: Translation results in Japanese-English .11: Translation results in English-Japanese .12: Positive translation examples in Moses-chart .13: Negative translation examples in Moses-chart .1: Lexical features of nonterminals .2: Lexical features of nonterminal of the example .3: Lexical features around nonterminal .4: Lexical features around nonterminal of the example .5: Statistical table of train and test corpus .6: BLEU-4 scores (case-insensitive) on Vietnamese-English corpus .7: Statistical table of rules .8: Number of possible source-sides of SCFG rule for Vietnamese-English corpus and number of source-sides of the best translation .1: Types of paraphrases (Lexical and Syntactic) .2 Statistics of the corpus .4: Statistics of the corpus .5: The statistics of the number of zones in the test data. 89 ix 1 Introduction In this chapter we briefly address the research context, the research motivations, as well as the major contributions of the thesis. First, we introduce the Machine Translation approaches. Second, we state the research motivation which the thesis focuses to solve.

Third, we present the main contribution of the thesis. Finally, we outline the structure of the thesis 1.1 Machine Translation Machine translation (MT) is the task of automatically translating a text from one natural language into another. The ideal of machine translation can be traced back to the seventeenth century, but it became realistically possible only in the middle of the twentieth century (Hutchins, 2005). Soon after the first computers were developed, researchers began on MT algorithms.

The earlier MT systems consisted primarily of large bilingual dictionaries and sets of translation rules. Dictionaries were used for word level translation, while rules controlled higher level aspects such as word order and sentence organization. Starting from a restricted vocabulary or domain, rule based systems proved useful. But as the study progressed, researchers found that it is extremely hard for rules to cover the complexity of natural language, and the output of the MT systems were disappointing when applied to larger domains.

Little breakthrough was made until the late 1980’s, when the increase in computing power made statistical machine translation (SMT) based on bilingual language corpora possible. In the beginning, much scepticism about SMT existed from the traditional MT community because people doubled whether statistical methods based on counting and mathematical equations can be used for the sophisticated linguistic problem. However, the potential of SMT was justified by pioneering experiments carried out at IBM in the early 1990s (Brown et al. Since then the statistical approach has become the dominant method in MT research.

Several criteria can be used to classify machine translation approaches, yet the most popular classification is done attending to the level of linguistic analysis (and generation) required by the system to produce translations. Usually, this can be graphically expressed by the machine translation pyramid in Figure 1.1: The machine translation pyramid Generally speaking, the bottom of the pyramid represents those systems which do not perform any kind of linguistic analysis of the source sentence in order to produce a target sentence. Moving upwards, the systems which carry out some analysis (usually by means of morphosyntax-based rules) are to be found. Finally, on top of the pyramid a semantic analysis of the source sentence turns the translation task into generating a target sentence according to the obtained semantic representation.