Nghiên cứu xác trị bài thi đọc hiểu tiếng Anh VSTEP - Luận án tiến sĩ

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES ******** WEI YU DOCTORAL DISSERTATION A STUDY ON THE VALIDITY OF A VIETNAMESE DEVELOPED TEST OF ENGLISH READING PROFICIENCY (Nghiên cứu xác trị bài thi đọc hiểu của một định dạng bài thi tiếng Anh của Việt Nam) MAJOR: ENGLISH LANGUAGE TEACHING METHODOLOGY CODE: 9140231.01 HANOI, 2023 This dissertation was completed at the University of Languages and International Studies, Vietnam National University, Hanoi. This dissertation was defended on October 26th, 2023 This dissertation can be found at: - National Library of Vietnam - Library and Information Center - Vietnam National University, Hanoi i DECLARATION OF AUTHORSHIP I hereby certify that the thesis I am submitting is entirely my own original work except where otherwise indicated. I am aware of the University’s regulations concerning plagiarism, including those regulations concerning disciplinary actions that may result from plagiarism. Any use of the works of any other author, in any form, is properly acknowledged at their point of use.

Date of submission: November 3rd, 2023 Ph. candidate’s signature: ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Nguyen Hoa (Supervisor) I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Duong Thu Mai (Co-supervisor) iii TABLE OF CONTENTS LIST OF FIGURES.

x LIST OF TABLES. xii LIST OF ABBREVIATIONS. xix CHAPTER I: INTRODUCTION .1 Rationale of the study .2 Objective of the study.3 Scope of the study .4 Clarifications of the key terms .5 Context of the study .1 Introduction of VSTEP.1 Development of VSTEP.2 Administration of VSTEP.3 Test takers of VSTEP.4 Test format of VSTEP.5 Cut-off scores of VSTEP.2 Introduction of VSTEP.1 Test purpose of VSTEP.2 Test format of VSTEP.3 Proficiency level description of VSTEP.6 Significance of the study .7 Overall structure of the thesis. 15 iv CHAPTER II: LITERATURE REVIEW .1 Review of L2 reading .1 Nature of L2 reading .2 Constructs of L2 reading.1 The processing perspective.2 The task perspective .3 The reader purpose perspective .2 Review of test validation approaches .1 Test validation approaches from product-based perspective .1 Item Response Theory .2 Test validation approaches from process-based perspective.1 Think aloud protocols.

50 CHAPTER III: THEORETICAL FRAMEWORK .2 Studies on the validity of VSTEP.1 Evidence-based validation frameworks .1 Bachman and Palmer’s test usefulness framework .2 Weir’s socio-cognitive framework .2 Argument-based validation frameworks .1 Kane’s interpretive argument framework .2 Chapelle, Enright, and Jamieson’s interpretive argument framework .3 Bachman and Palmer’s assessment use argument framework .4 Empirical validation studies with interpretive argument framework .5 Interpretive argument for the VSTEP.1 Justification for the use of interpretive argument .2 Articulation of an interpretive argument for the VSTEP. 81 CHAPTER IV: RESEARCH METHODOLOGY.1 Ontology and epistemology for the design of the study .2 The convergent mixed methods design for the study .3 Participants of the study .1 Test takers for the VSTEP.2 Experts for interview and judgement .3 Participants for think aloud protocols .4 Participants for eye tracking .4 Instruments of the study .1 A version of the VSTEP.3-5 reading test paper .3-5 reading test specifications .3 Expert interview questions .4 The modified framework of task characteristics for the VSTEP.5 Expert judgment forms .6 Think aloud protocols operation instructions .7 Eye tracking metrics .8 The reading processing checklist .5 Data collection procedures of the study .1 Data collection procedures for research questions 1, 2 and 3 .2 Data collection procedures for research question 4 .3 Data collection procedures for research question 5 .6 Tools for data analysis .1 Compleat Lexical Tutor.7 Data analysis procedures of the study .1 Data analysis procedures for research question 1 .2 Data analysis procedures for research question 2 .3 Data analysis procedures for research question 3 .4 Data analysis procedures for research question 4 .5 Data analysis procedures for research question 5. 138 CHAPTER V: FINDINGS AND DISCUSSIONS ON VALIDITY OF VSTEP.3-5 READING TEST FROM THE PRODUCT-BASED PERSPECTIVE .1 Findings and discussions for research question 1 .1 Findings from test tasks .1 Comparison of test tasks in the investigated test and test specifications .2 Expert interview on test tasks .2 Findings from Rasch model analysis of test items.2 Item estimates according to Rasch model analysis .3 Item and person calibration according to Rasch model analysis.2 Findings and discussions for research question 2 .1 Differential item functioning by genders .2 Differential item functioning by majors .3 Findings and discussions for research question 3 .1 Separation and reliability measures. 164 CHAPTER VI: FINDINGS AND DISCUSSIONS ON VALIDITY OF VSTEP.3-5 READING TEST FROM THE PROCESS-BASED PERSPECTIVE .1 Findings and discussions for research question 4 .1 Expert judgement on reading subskills in the investigated test .2 Discussions from expert judgement .3 Think aloud protocols .1 Reading strategies in think aloud protocols .2 Reading subskills in think aloud protocols.4 Discussions from think aloud protocols.2 Findings and discussions for research question 5 .1 Eye tracking of successful and unsuccessful test takers across item types 200 6.2 Discussions from eye tracking of successful and unsuccessful test takers across item types .3 Eye tracking of test takers at different proficiency levels and results of reading processing checklists .4 Discussions from eye tracking of test takers at different proficiency levels and results of reading processing checklists.

231 CHAPTER VII: CONCLUSION .1 Interpretive argument for the VSTEP.3-5 reading test revisited .1 A recap of the generalization inference .2 A recap of the explanation inference .2 Implications of the study .3 Limitations of the study .4 Suggestions for future studies. 239 LIST OF PUBLICATIONS RELATED TO DISSERTATION. 240 LIST OF PUBLICATIONS OF OTHER FIELDS. 242 APPENDIX 1: Control file for WINSTEPS.

267 APPENDIX 2: Person measures of the investigated VSTEP. 269 APPENDIX 3: Expert interview. 278 APPENDIX 4: Expert judgement form A. 280 APPENDIX 5: Expert judgement form B .3-5 reading test think aloud protocols experiment operation instructions.

285 APPENDIX 7: Reading processing checklist. 288 APPENDIX 8: Interpretive argument for the VSTEP. 290 ix LIST OF FIGURES Figure 2.1 Khalifa and Weir’s cognitive model of reading (2009) .2 Task Characteristics Framework (Bachman and Palmer, 1996) .1 Test usefulness framework (Bachman & Palmer, 1996) …………….2 A socio-cognitive framework for test validation (Weir, 2005, p.3 Model of Toulmin’s argument structure (1958, 2003).4 Interpretive argument framework (Kane, 2006, 2013, 2016) .5 Interpretive argument framework (Chapelle et al.6 Synthesized Bachman & Palmer’s assessment use argument framework (2010) .1 AOI defining on the screen page .2 The layout of experiment materials of eye tracking .3 Eye tracking mode in the study .4 Successful calibration and validation of the eye tracker camera on the host computer screen.1 The Wright map of the investigated test .1 Heat map from the successful test taker on Item 1 .2 Heat map from the unsuccessful test taker on Item 1 .3 Gaze plot from the unsuccessful test taker on Item 29 .4 Gaze plot from the unsuccessful test taker on Item 29 .5 Heat map from the successful test taker on Item 2 .6 Heat map from the unsuccessful test taker on Item 2 .7 Heat map from the successful test taker on Item 23 .8 Heat map from the unsuccessful test taker on Item 23 .9 Gaze plot from the successful test taker on Item 6 .10 Gaze plot from the unsuccessful test taker on Item 6 .11 Gaze plot from the successful test taker on Item 27 .12 Gaze plot from the unsuccessful test taker on Item 27 .13 Gaze plot from the successful test taker on Item 10 .14 Gaze plot from the unsuccessful test taker on Item 10 .15 Heat map from the successful test taker on Item 40 .16 Heat map from the unsuccessful test taker on Item 40. 223 xi LIST OF TABLES Table 1.1 Format of each subtest of VSTEP.2 Cut-off scores of VSTEP.3 Proficiency level description of VSTEP.1 Summary of the warrant, assumptions, and backing associated with each inference in the TOFEL interpretive argument (Chapelle et al.2 Summary of the generalization inference for the VSTEP.3 Summary of the explanation inference for the VSTEP.3-5 reading test81 Table 4.1 The convergent mixed methods design for the study.2 Sampling of participants in the study .3 Information of the test takers for the investigated test .4 Information of the experts for interview and judgement .5 Information of the participants for think aloud protocols .6 Information of the participants for eye tracking .7 Basic information of the VSTEP.3-5 reading test paper .8 The modified framework of task characteristics for the VSTEP.9 Reading subskills and descriptors of VSTEP.10 Results on reading subskills and descriptors of VSTEP.11 Correspondence between subskills and each item of the investigated reading test .12 Eye tracking metrics to be examined in the study .13 The reading processing checklist .14 Participants and dates of the think aloud protocols data collection .15 Participants and dates of the eye tracking data collection .16 Initial think aloud protocols coding scheme .17 Modified think aloud protocols code scheme .18 Think aloud protocols coding demo .19 Information of raw score, converted score and corresponding level of VSTEP.3-5 reading test from think aloud protocols’ participants .20 Information of the eye tracking experiment participants .1 Comparison of text length in the investigated test and test specifications .2 Comparison of language of input in the investigated test and test specifications.3 Comparison of text domain in the investigated test and test specifications.4 Comparison of text topic in the investigated test and test specifications .5 Comparison of text level in the investigated test and test specifications .6 Comparison of response type in the investigated test and test specifications.7 Frequency on the results of interview questions B .8 Standardized residual variance in Eigenvalue units .9 Item fit of the investigated test .10 Item measure of the investigated test .11 DIF by genders .12 DIF by majors .13 Separation and reliability measures of the investigated test .1 Reading subskills and descriptors of VSTEP.2 Results of the expert judgement of reading subskills in the investigated test .3 Frequency of used strategies from participants’ think aloud protocols .4 Frequency on the subskill of “understanding explicit information” from xiii participants’ think aloud protocols .5 Frequency on the subskill of “understanding word meanings in context” from participants’ think aloud protocols .6 Frequency on the subskill of “understanding cohesive devices” from participants’ think aloud protocols .7 Frequency on the subskill of “making inferences” from participants’ think aloud protocols.8 Frequency on the subskills of “integrating information” from participants’ think aloud protocols .9 Frequency on the subskill of “summarizing main ideas” from participants’ think aloud protocols .10 Frequency on the subskill of “understanding explicit/implicit author’s opinion/attitude” from participants’ think aloud protocols .11 Frequency on the subskill of “understanding text function” from participants’ thinking aloud protocols .12 Test takers’ eye tracking data on Item 1 .13 Test takers’ eye tracking data on Item 29 .14 Test takers’ eye tracking data on Item 2 .15 Test takers’ eye tracking data on Item 23 .16 Test takers’ eye tracking data on Item 6 .17 Test takers’ eye tracking data on Item 27 .18 Test takers’ eye tracking data on Item 10 .19 Test takers’ eye tracking data on Item 40 .20 Eye tracking data from participants at different proficiency levels .21 Proportions on reading subskills and strategies used from participants’ reading processing checklists .