The silent majority an examination of nonresponse in college stu

University of Massachusetts Amherst ScholarWorks@UMass Amherst Open Access Dissertations 9-2012 The Silent Majority: An Examination Of Nonresponse In College Student Surveys Ethan A. Kolek University of Massachusetts Amherst Follow this and additional works at: https://scholarworks.edu/open_access_dissertations Part of the Education Commons Recommended Citation Kolek, Ethan A., "The Silent Majority: An Examination Of Nonresponse In College Student Surveys" (2012). Open Access Dissertations.7275/wn37-0y87 https://scholarworks.edu/open_access_dissertations/613 This Open Access Dissertation is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Open Access Dissertations by an authorized administrator of ScholarWorks@UMass Amherst.

For more information, please contact scholarworks@library. THE SILENT MAJORITY: AN EXAMINATION OF NONRESPONSE IN COLLEGE STUDENT SURVEYS A Dissertation Presented by ETHAN A. KOLEK Submitted to the Graduate School of the University of Massachusetts Amherst in partial fulfillment of the requirements for the degree of DOCTOR OF EDUCATION September 2012 School of Education Educational Policy and Leadership © Copyright by Ethan A. Kolek, 2012 All Rights Reserved THE SILENT MAJORITY: AN EXAMINATION OF NONRESPONSE IN COLLEGE STUDENT SURVEYS A Dissertation Presented by ETHAN A.

KOLEK Approved as to style and content by: _______________________________________ Elizabeth Williams, Chair _______________________________________ Gary Malaney, Member _______________________________________ Gloria DiFulvio, Member ____________________________________ Christine B. McCormick, Dean School of Education ACKNOWLEDGMENTS I have many people to thank for their help, support, encouragement, interest, and love that have made it possible for me to write this dissertation. I have been fortunate to have had many teachers (both inside and outside of the classroom), family members, and friends who have shaped who I am and how I think. Miss Pretat, Mrs.

Pawelczyk, David Brownell, Toby Gillis, Liz Rapoza, David Boyce, Priscilla Bates, and Jim Bean took an interest in me as an adolescent and encouraged my curiosity and creativity. I have thought of them, as well as many of my Star Island friends, as I have reflected on my dissertation journey. My parents, Carol and Drew, and their respective partners, Carlton and Sandy, have always supported me and have encouraged me throughout this process. My friendships with my brother, Adam, and good friends, Aric Russom and Dan Saunders, helped sustain me while I was in the midst of writing.

I would like to thank my Center for Community Engagement colleagues for their encouragement and understanding, particularly my supervisor, Molly Mead, who created my part-time position which enabled me to complete this dissertation. My partner, Tamatha Gaumnitz, has been incredibly supportive throughout my time in graduate school and during the time I spent writing this dissertation. Tamatha was instrumental in the success of my focus group study, helping to recruit many participants and finding a place to hold the focus groups at the university. Tamatha, thank you for being so understanding when I curtailed nearly all social commitments, abandoned my domestic responsibilities, and locked myself in my office to finish this project.

I have been very lucky to have your love and encouragement. iv I am grateful for the small, but vibrant, intellectual community in the higher education program, especially the members of our informal “Doc Reading Group,” which included Aaron Kuntz, Dan Saunders, Tom Fleenor, Raldy Laguilles, April Stroud, Shelly Perdomo, and Karen Howard. The members of my committee, Liz Williams, Gary Malaney, and Gloria Difulvio, have asked great questions and shared important perspectives. Ryan Wells graciously helped me learn logistic regression, which I employed in this project.

A special thanks goes to Dan Saunders, who read and commented on several chapters and prodded me to finish my dissertation. Gary Malaney’s sarcasm and cynicism were reassuring when I first arrived at the University. I have valued his encouragement, advice and friendship. During the nearly six years that I spent in the Student Assessment, Research and Evaluation Office, I worked closely with my mentor, Liz Williams.

I am grateful to her for that incredibly rich experience, through which I learned more than I could have imagined about research and higher education. At work, in the classroom, or while working on independent projects, Liz always challenged me to be a better writer, thinker, and researcher. For over a decade, Liz has been a wonderful supervisor, teacher, advisor, collaborator and friend. v ABSTRACT THE SILENT MAJORITY: AN EXAMINATION OF NONRESPONSE IN COLLEGE STUDENT SURVEYS SEPTEMBER 2012 ETHAN A., UNIVERSITY OF MASSACHUSETTS AMHERST Ed.D, UNIVERSITY OF MASSACHUSETTS AMHERST Directed by: Adjunct Assistant Professor Elizabeth Williams Nonresponse is a growing problem in surveys of college students and the general population.

At present, we have a limited understanding of survey nonresponse in college student populations and therefore the extent to which survey results may be biased. The purpose of this dissertation is to explore three facets of nonresponse in surveys of college students in order to strengthen our empirical and conceptual understanding of this phenomenon. This dissertation seeks to contribute to our understanding of who participates in surveys and who does not, how students experience the process of being asked to complete surveys, and whether or not students’ perspectives about surveys suggest that college student surveys should be conceptualized as organizational surveys. To begin to answer these questions, I conducted three studies – a secondary data analysis that examines student characteristics associated with the odds of completing a survey, a “survey on surveys” study that asks students about their experiences with surveys, and a series of focus groups to understand how students made vi sense of surveys at their institutions.

Taken together, these findings provide a basis for a more developed and nuanced understanding of nonresponse in student surveys. vii TABLE OF CONTENTS Page ACKNOWLEDGMENTS. vi LIST OF TABLES .xv CHAPTER 1: INTRODUCTION .1 Introduction: Survey Research and the Study of College Students .3 Response Rates in Decline .4 Purpose of the Study .9 Significance of the Study .13 People: Populations, Samples, Respondents, and Response Rates .22 Section 1: Declining Response Rates .23 Documenting Declines in Survey Response Rates in the General Population .23 Potential Reasons for Declining Response Rates .26 Declining Response Rates Summary .27 Section 2: Nonresponse Bias .28 viii Causal Nonresponse Models .29 Separate Cause Model.29 Common Cause Model .30 Direct Survey Variable Cause Model .30 Causal Model Summary.31 Approaches to Studying Nonresponse Bias .31 Comparing Response Rates .32 Comparisons to Existing Data .32 Use of Auxiliary Variables .33 Early and Late Respondents Comparisons .33 Follow-Up Surveys .35 Archival or Record-Linking Approach .36 Surveys on Surveys .37 Summary of Approaches to Studying Nonresponse Bias .38 Empirical Studies of Nonresponse Bias .38 Higher Response Rates Do Not Lessen Nonresponse Bias .40 Studies Suggesting Increased Response Rates Lessen Nonresponse Bias.42 Nonresponse Bias Meta-Analyses .43 Nonresponse Bias Summary .44 Section 3: Response Theories .46 Web Survey Response Steps .50 Social Exchange Theory .50 Leverage Salience Theory.51 Active and Passive Nonresponse .54 Other Theories of Survey Response from the Public Opinion Literature .55 Response Theories Summary.56 Section 4: Factors Relating to Survey Response .60 Societal Level Factors Summary .61 Survey Design Characteristics .66 Survey Design Characteristics Summary.68 Characteristics of Individuals .71 General Survey Attitudes and Previous Survey Experience .71 Factors Relating to Survey Response Summary .73 Section 5: Survey Methodology in Surveys of College Students .73 Response Rate Decline in College Student Surveys .74 Potential Reasons for Declining Response Rates in College Student Surveys .76 College Student Survey Response Rate Summary .77 Factors Relating to College Student Nonresponse.77 Salutations, Personalization, and Sponsorship in Web Surveys of Students .80 College and University Characteristics .83 Person-level Factors in College Student Surveys .89 Section 6: Organizational Research .91 How Organizational Surveys Differ from General Population Surveys .95 Critique of Organizational Research Studies on Survey Nonresponse .102 Summary of Organizational Nonresponse Findings .104 College Students as Organizational Members .105 x “National” Studies as Composites of Individual Organizational Studies .107 Summary of Literature Review .111 Design and Research Questions .114 Participants and Data Sources .119 Past Survey Behavior .123 Inter-Item Correlations.124 Logistic Regression Results .126 Logistic Regression Statistics .128 Interpreting Exponentiated Logged Odds .133 4: SURVEYS ON SURVEYS .140 Self-Reported Response Rate .141 xi Data Analysis .146 Self-Reported Response Rates .146 Self-Reported Response Rates and Number of Requests .149 Self-Reported Response Rates and Demographics .149 Reasons for Participating in the Current Survey.150 Differences by Semester .151 Correlations Between Motivations .152 Class Year and Motivation.153 Gender and Motivation .156 5: FOCUS GROUP STUDY .159 Focus Groups: Methods .160 Focus Groups: Sites and Participants .160 Focus Group Sites .160 Focus Group Recruitment .160 Focus Group Participant Information .161 Focus Groups: Administration and Protocol.162 Focus Groups: Analysis .163 Coding and Trustworthiness .164 Focus Groups: Limitations .166 Focus Groups: Results .167 Understanding the Nature of Surveys .168 Surveys as Referenda .168 Sampling and Measurement .171 Students’ Perceptions of Nature of Surveys: Summary .173 xii Survey Experiences .175 Respondent-(Un)friendly Questionnaires .179 Number of Survey Requests .181 Two Important Considerations in the Response Decision Process .184 Surveys Should Have Meaning.186 Students Want Surveys to Have an Effect .187 Black Hole and Trust .189 Students Want to See Survey Results .199 Who Responds and Who Does Not Respond to Student Surveys .199 How Students Experience the Survey Process.202 Should We Treat College Student Surveys as Organizational Surveys? .205 Implications for Higher Education Research .211 Conduct Surveys Sparingly.212 Implications for Future Research .216 APPENDICES A: RESEARCH QUESTIONS AND METHODS MAP .219 B: LOGISTIC REGRESSION STATISTICS FOR EACH REGRESSION MODEL .220 xiii C: SURVEY ITEMS .222 D: SAMPLE FOCUS GROUP FLIER .223 E: FOCUS GROUP INFORMED CONSENT LETTER .224 F: FOCUS GROUP INFORMATION FORM .225 G: FOCUS GROUP PROTOCOL .255 xiv LIST OF TABLES Table Page 2.1: Overall NSSE Response Rates and Number of Institutions .2: College Alcohol Study Historical Response Rates .1: Select University and College Characteristics .2 Disposition of Sample for Replication Study .3 Logistic Regression Independent Variables .4: Database Variables Included in Logistic Regression Models .5: CIRP Variables Included in Logistic Regression Models .7: Holland Personality Types .8: Inter-Item Correlations for Model 1 and Model 2 .9: Inter-Item Correlations for Model 3 and Model 4 .11: Logistic Regression Classification of Cases .12: Logistic Regression Results .13: Example of Gender to Interpret Odds and Odds Ratios .1: Comparison of Respondent Demographics to Population Demographics.2: Excluding Current Survey, Number of Institutional Surveys Respondents Reported Being Asked to Complete during Current Semester .3: Students’ Self-Reported Response Rates .4: Number of Surveys Students Reported Completing by Number of Surveys Students Reported Being Asked to Complete .5: Reasons for Participation in Current Survey .6: Gender Differences in Self-Reported Motivation .7: Spring Inter-Item Correlations: Motivations to Complete Survey .8: Fall Inter-Item Correlations: Motivations to Complete Survey .1: Focus Group Participant Characteristics .2: Majors of Focus Group Participants .254 xvi CHAPTER 1 INTRODUCTION Introduction: Survey Research and the Study of College Students Higher education research attempts to answer numerous questions about the college student experience. What causes some students to persist, and others to drop out? Does attending an institution with a racially diverse student body contribute to students’ success in an increasingly heterogeneous world? How widespread a problem is Internet addiction among students? As befitting an applied field with a wide breadth of research questions, higher education researchers rely on the theoretical underpinnings and methodologies of several social science disciplines (Smart, 2005), in particular sociology, psychology and economics, as well as the methods of other applied fields of study (e.

public health and management). Moreover, the field of higher education does not align itself solely within a particular tradition of inquiry or epistemology. For example, studies conducted from constructivist (e. Jones & Hill, 2003), feminist (e.

Twombly, 1993), and post-positivist perspectives (e. Pike 2008) are all published in higher education journals. Higher education researchers rely on a range of data collection methods to answer their research questions, including interviews, observations, administrative data, tests, content analyses, and experiments (Hutchinson & Lovell, 2004). Although the research orientations and data sources of higher education studies are quite diverse, the single most common method of collecting data on college students is through surveys (Hutchinson & Lovell, 2004, Porter, 2011).