Kiểm thử tự động cho các ứng dụng đa phương thức tương tác luận án tiến sĩ

MINISTRY OF EDUCATION & TRAINING THE UNIVERSITY OF DANANG ---------- LE THANH LONG AUTOMATIC TESTING OF INTERACTIVE MULTIMODAL APPLICATIONS ENGINEERING DOCTORAL THESIS Da Nang, 11/2017 MINISTRY OF EDUCATION & TRAINING THE UNIVERSITY OF DANANG ---------- LE THANH LONG AUTOMATIC TESTING OF INTERACTIVE MULTIMODAL APPLICATIONS Major: Computer science Code of Major: 62 48 01 01 ENGINEERING DOCTORAL THESIS Supervisors: 1. Nguyễn Thanh Bình Da Nang, 11/2017 REASSURANCES I hereby certify this thesis done by my work, under the guidance of Prof. Ioannis Parissis and Assoc. Nguyễn Thanh Bình.

I certify that the research results presented in the thesis are true and are not copied from any other documents. All quotations origins are recorded clearly and completely. Student Lê Thanh Long 1 TABLE OF CONTENTS TABLE OF CONTENTS. INTERACTIVE MULTIMODAL APPLICATIONS.

Features of Multimodal Interaction. Design Spaces and Theoretical Frameworks. The TYCOON Theoretical Framework. CASE Design Space.

The CARE Properties. Intoduction to Software Testing. Model-Based Testing. Operational Profile-Based Testing .3 Requirement-Based Testing.

Testing Interactive Multimodal Applications. BACKGROUND OF A NEW TEST MODELING LANGUAGE. The Interactive Multimodal Application Memo. Probabilistic Finite State Machines.

Setting a Level of Abstraction for Testing. Generation of Tests at Dialog Controller Level. Taking into Account Conditional Probabilities. Evaluation of the Results of Extending the CTT with Conditional Probabilities.

TTT: A NEW TEST MODELING LANGUAGE FOR TESTING INTERACTIVE MULTIMODAL APPLICATIONS. The User Actions Traces. Definition of the TTT Language. Basic Structure of a TTT Model.

Supporting Conditional Probability Specifications for all the CTT Operators. Storing the Traces of the User Actions. Transformation Rules from CTT to Test Model by Using the TTT Language. Taking into Account Multimodality.

Generating Tests for Multimodal Events. Checking the Validity of CARE Properties. Redundancy-Equivalence Property. Modeling the Interactive Multimodal Application Memo by the TTT Language.

Advantages and disadvantages of the TTT Language. TTTEST: THE SUPPORT TOOL FOR TESTING INTERACTIVE MULTIMODAL APPLICATIONS. Test Execution Environment. The TTTEST Tool .Translating TESTCTT Model into C Program.

Automatic Translation Solution. Modeling the NotePad Application by the TTT Language. Testing the Memo Application. Testing the Map Navigator Application.

Evaluation of the Resulted Test Cases. 112 CONCLUSIONS AND FUTURE WORKS. Acronyms Meaning 1 API Application Programming Interface 2 AUT Application Under Test 3 CARE Complementarity, Assignment, Redundancy and Equivalence 4 CASE Concurrent, Alternate, Synergistic and Exclusive 5 CTT ConcurTaskTrees 6 DFA Deterministic Finite State Automaton 7 FSMs Finite State Machines 8 HMD Head Mounted Display 9 IMA Interactive Multimodal Application 10 ICO Interactive Cooperative Objects 11 MBT Model-Based Testing 12 NFA Nondeterministic Finite State Automaton 13 ObCS Object Control Structure 14 OPBT Operational Profile-Based Testing 15 PFSM Probabilities Finite State Machine 16 RBT Requirement-Based Testing 17 SQL Structured Query Language 18 TW Temporal Window 19 UML Unified Modeling Language 20 TTT Task Tree – based Test 21 TTTEST Testing IMA by means of the TTT language 5 LIST OF FIGURES Figure 1. The TYCOON Theoretical Framework for studying multimodality.

The CASE design space. A window of Tuple editor. The class Editor. The ObCS of the class Editor.

The interactive multimodal application "Memo". The Memo application struture [4]. The CTT for the Memo application. The CTT with unconditional probabilities for the Memo application.

A multimodal application organized along the PAC-Amodeus model. FSM Example for the Memo application [20]. The behavior of the choice operator. The behavior of the concurrency operator.

The behavior of the deactivation operator. The behavior of the option operator. The behavior of the suspend-resume operator. The extended CTT with conditional probabilities for the Memo application.

The TTTEST Testing Environment. The TTTEST tool interface. Transformation diagrams from TESTCTT model into C program. Translating TESTCTT to C program with Lex/Yacc.

The extended CTT for application NotePad. Multimodal interaction with a map. The extended CTT for the Map navigator application. 106 6 LIST OF TABLES Table 2.

The CTT Operators [27]. Test data generated with unconditional probabilities and conditional probabilities. Extensions of CTT operators with conditional probabilities. The TTT Syntax.

A basic structure of a TESTCTT. A structure of a function. The CTT Syntax. The behavior of choice operator.

The behavior of Concurrency operator. The behavior of Deactivation operator. The behavior of Suspend-resume operator. The behavior of Option operator.

The SQL-like syntax. The conditional constructs syntax. Transformation rules from augmented CTT to Test Model. The semantics of modalities operator.

Events are generated by Modalities operator. The behavior of TestEquivalence operator. The result of TestEquivalence operator. The behavior of TestRedundant_EquivalenceEarly operator.

The result of TestRedundant_EquivalenceEarly operator. The behavior of TestcomplementaryEarly operator. The result of TestComplementaryEarly operator. TESTCTT model for the Memo application.

Transformation of choice operator. Transformations from create table statement in the TTT language into the C language. High-level NotePad model. The result of TestEquivalence operator.

The result of TestRedundantEquivalenceEarly operator. The result of TestComplementaryEarly operator. The TESTCTT model for the Map Navigator application. Multimodal Events are generated for the Map Navigator.

Results of the Experiment 1. Results of the Experiment 2. Results of the Experiment 3. Context and Motivation Interactive Multimodal Applications (IMAs) support communication with the user through different modalities such as voice and gesture.

They can greatly improve human-computer interaction, because they can be more intuitive, natural, efficient, and robust. Multimodality brings an intuitive, natural affinity between the machine and the user, such as in virtual reality mobile application. Efficiency is obtained when the user can use equivalent modalities for the same tasks while robustness can result from the integration of redundant or complementary inputs [35]. The CARE properties (Complementarity, Assignment, Redundancy and Equivalence) can be used as a measure to assess the usability of the multimodal interaction.

Equivalence and assignment represent the availability and, respectively, the absence of choice between multiple modalities for performing a task while complementarity and redundancy express relationships between modalities. The flexibility and robustness of interactive multimodal applications result in an increasing complexity of the design, development and testing. Therefore, ensuring their correctness requires thorough validation [34]. Approaches based on formal specifications automating the development and the validation activities have been proposed to deal with this complexity.

In [21], Laya Madani et al. present a technique of test case generation for testing CARE properties by means of a synchronous approach. According to the proposed approach, CARE properties are translated into an enhanced version of the Lustre synchronous language. An improved method presented in [22] uses task trees and a fusion model to perform test data generation for IMAs.

The above presented approach uses several notations, inspired fom existing modeling languages, to build test models : a model of the application behavior, a model of the interactive tasks, operational profiles (annotations on CTT) and 9 modality specifications. The variety of notations makes the modeling process hard. So, as an additional improvement to this previous research work, in this thesis “Automatic Testing of Interactive Multimodal Applications”, our objective is to define a single specification and modeling language, called TTT (Task Tree based Test) making possible to express, in a single and consistent syntax:  Scenarios and conditional operational profiles for IMAs,  Test oracles,  Expected properties of the IMAs. We built an automatic test generation approach based on this test modeling language.

The approach allows specifying multimodal events of interactive multimodal applications and CARE properties as well as checking the validity of CARE properties. We also develop a tool that automates the test of such interactive multimodal applications. Main Contributions of the Thesis The thesis has the following main contributions: (1) On the basis of analyzing the characteristics of ConcurTaskTrees (CTT), a well-know notation for specifying interactive application, extensions of CTT operators with conditional probabilities are proposed in order to take into account IMA. More precisely, the user behavior on IMA is often influenced by conditions on the application and its environment.

(2) A new test modeling language is defined for interactive multimodal applications based on task trees. This language defines all the CTT operators, supports state definition, multimodality and conditional probabilities for IMA. (3) The transformation rules from CTT into a test model in the TTT language are formally developed. (4) A solution to generate test data for interactive applications is proposed.

10 (5) A specification of multimodal interactions and CARE properties is integrated into the TTT language. The specification consists of two different issues when testing IMA: generating tests for multimodal events and checking the validity of the CARE properties. (6) The TTTEST tool is developed for automating the test of such interactive multimodal applications. Structure of the Thesis Chapter 1 presents the background of interactive multimodal applications which includes the definition of multimodality and key features of multimodal interaction.

Testing methods for interactive multimodal applications are also summarized in this chapter. In chapter 2, we present, with more details, the testing approach that we focus on and the related background consisting of task trees, finite state machines, multimodal interaction, CARE properties, operational profiles, conditional probabilities. In chapter 3, we propose a test modeling language for testing interactive multimodal applications, called TTT, which makes it possible to express scenarios and conditional operational profiles. The transformation rules from CTT specifications into TTT test models are also developed.

Chapter 4 presents the TTTEST tool for testing interactive multimodal applications which includes the test execution environment, and the underlying implementation of this tool. We also introduce two interactive multimodal applications and describe how to test them. INTERACTIVE MULTIMODAL APPLICATIONS Interactive Multimodal Applications (IMA) [35] ensure the access to various commercial services and are increasingly involved in many critical domains such as flight or industrial process control. Designing IMA is a complex and error-prone activity, because of the importance of the human-computer interaction aspect.

This is especially true when these interactions use multiple modalities (voice, gesture…). They have the potential to greatly improve human-computer interaction, because they can be more intuitive, natural, efficient, and robust. Flexibility is obtained when the user can use equivalent modalities for the same tasks while robustness can result from the integration of redundant or complementary inputs. As a result, thoroughly testing such applications is particularly important and requires more effort than for traditional interactive applications.

In this chapter, we present the background of interactive multimodal application which includes the definition of multimodality and key features of multimodal interaction. Testing methods for interactive multimodal applications are also summarized in this chapter. Multimodality A modality is a channel or path of communication between the human and the computer. It is one of different senses through which the human can perceive the output of the computer (audition, vision, touch, smell, and taste); modalities will also cover the input devices and sensors allowing the computer to receive information from the human such as speech, pen, touch, manual gestures, gaze and head and body movements [7].

So, a modality is defined as the representational application, and the physical I/O device used to convey expressions of the representational application [34]. Multimodal interaction processes two or more combined user input modes in a coordinated manner with multimedia application output. Multimodal interaction aims at recognizing naturally occurring forms of human language and 12 behavior [35]. Multimodal interaction can come either from the user action or from the environment.

Multimodal data can be analyzed concurrently, as well offline or online, in order to get richer and more robust information over context and participants. The advantage of multimodal interaction is increased usability. The weaknesses of one modality are offset by the strengths of another. On a mobile device with a small visual interface and keypad, a word may be quite difficult to type but very easy to say.