The estimation and evaluation of optimal thresholds for two seque

Virginia Commonwealth University VCU Scholars Compass Theses and Dissertations Graduate School 2013 The Estimation and Evaluation of Optimal Thresholds for Two Sequential Testing Strategies Amber R. Wilk Virginia Commonwealth University Follow this and additional works at: https://scholarscompass.edu/etd Part of the Biostatistics Commons © The Author Downloaded from https://scholarscompass.edu/etd/3163 This Dissertation is brought to you for free and open access by the Graduate School at VCU Scholars Compass. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of VCU Scholars Compass. For more information, please contact libcompass@vcu.

Wilk All rights reserved THE ESTIMATION AND EVALUATION OF OPTIMAL THRESHOLDS FOR TWO SEQUENTIAL TESTING STRATEGIES A Dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at Virginia Commonwealth University. Mathematics and Statistics Composite Secondary Education, Utah State University, 2007 Advisor: DONNA K MCCLISH, PH. PROFESSOR, DEPARTMENT OF BIOSTATISTICS Virginia Commonwealth University Richmond, Virginia July 2013 Acknowledgements This dissertation would not have been possible without the help of so many people in so many ways. First and foremost, my advisor Dr.

Donna McClish was instrumental in the development, progression, and completion of this research. Without your unwavering support, kind words, guidance and persistent help, willingness to let me complain, encouragement to keep going, and meeting at a last minutes’ notice to talk both professionally and personally, this dissertation could not have been completed. Your passion for your work inspires me. Thank you to the rest of my dissertation committee: Dr.

Christine Schubert-Kabban, Dr. Leroy Thacker, Dr. Jessica Ketchum, and Dr. Edmund (Trey) Wickham, whose ideas and comments are all greatly appreciated.

Particular thanks goes to Dr. I have had your constant endorsement and encouragement throughout the past five years. Thank you for the constructive feedback, the opportunity to learn and develop as a researcher, and motivation to keep going when I wanted to quit. Thank you for being a mentor and my friend.

I am thankful to all of my supervisors from various positions I have held as a research assistant over the course of my graduate career. Thank you to Joanne Einsmann and John Matro for giving me time away from work when I needed to study, and for always supporting me during the two years I was part of your group and beyond. Thank you to the team I worked with at Merck & Co., for the unique opportunity to gain experience working in a pharmaceutical environment while realizing I was still a student. ii iii Finally, thank you to my current supervisor, Dr.

Cathy Bradley, for never failing to ask how my work was progressing. There were key people in the department who helped me get through school. I owe a very important debt to Yvonne Hargrove, Gayle Spivey, Russ Boyle, Dr. Roy Sabo, and Dr.

Additionally, I want to thank my cohort of peers for the system of support we created: Adam Sima, Jiayi Hou, and especially Caroline Carrico, who shared an office / cubicle with me the entire five years and was always there to laugh, complain, gossip, and talk with. Thank you all for being excited with me and for me as we all graduate and become Biostatisticians. I am grateful to my friends and family, near and far, for the constant inspiritment, comfort, and encouragement. I will never be able to surely express how appreciative I am.

A very special thank you goes to two of my best friends, Kimberly Parker and Claire Russell. We have shared many laughs and tears, dissertation work parties and breaks from studies, vacations and road trips, and so much more. We started graduate school together, and now we finish it together. As we go down our own separate paths, I know that I have found two lifelong friends, and I could not be more grateful and thankful to the both of you.

Erich Fritz, you have been my friend through the good times and the not so good times, and I could not have finished this dissertation without your invariable encouragement. Thank you for helping me realize and reach my potential. I owe my deepest gratitude to my parents, Annalisa and Ed, for molding me into the person I am today and motivating me with their own examples of hard work. Thank you for your indefinite unconditional love, support, and confidence, and for believing in iv me even when I didn’t believe in myself.

From a young age you taught me to always strive to be my best no matter what. Thank you for cheering me on and reminding me when I needed to take a break. Thank you for your endless words of encouragement through phone calls, text messages, emails, and visits. Thank you for always being there; I definitely could not have done this without you, and I can’t wait to share the next chapter of my life.

v Table of Contents Page Acknowledgements .ii List of Tables. ix List of Figures. Methods of Combining Tests. ROC Curves and Related Measurements .1 Empirical ROC Curves .2 Binormal ROC Curves .3 Measures of Accuracy.

Optimal Operating Point(s). Motivation and Goals .8 2 Review of Relevant Statistical Topics. Receiver Operator Characteristic (ROC) Curves. Useful Measures to Select an OOP.

Area Under the Curve (AUC) and its Influence on the OOP. OOP for the Univariate Case using the YI and Binormal Assumption. Use of More than One Diagnostic Test (Sequential Testing).1 Believe the Positive (BP) .2 Believe the Negative (BN) .3 Visual Representation of BP and BN. Univariate Cases in the Literature.

BP and BN Cases in the Literature. Sequential Testing Formulae .1 BP System of Equations .2 BN System of Equations. Newton-Raphson (NR) with Ridging .1 Non-Linear Programming using SAS PROC NLP. Bias and Root Mean Square Error (RMSE).

Simultaneous Confidence Intervals (SCI) and Confidence Regions (CR). Determining Legitimate BP/BN Strategies. Determining Valid OOPS. Prediction of Total Sample Size and SCI Widths.

Population Parameters Results .1 Identifying Legitimate Strategies .2 Comparing BP and BN .3 Effect of Population Parameters on Sensitivity, Specificity and GYI. Believe the Positive (BP) Simulation Results .1 Formal Regressions to Determine Relationships. Believe the Negative (BN) Simulation Results .1 Formal Regressions to Determine Relationships. Assessing the Normality of the OOPs for a Subset of Simulation Sets.

Varying r for a Subset of Simulation Parameter Sets. Predicting SCI Widths and Total Sample Size. Wieand Cancer Example .76 6 Discussion and Future Work .94 7 List of References. Appendix A: Exact Gradient and Hessian Equations for BP and BN.

Appendix B: Simulation Results. Appendix C: SCI Widths and Total Sample Size Prediction Equations. Appendix D: SAS Programs .1 Determining Legitimate BP and BN Strategies .2 BP Simulations and Empirical Estimates .3 BN Simulations and Empirical Estimates .5 Prediction Models for SCI Widths and TSS .187 ix List of Tables Page Table 4.4: Formal Regression Analysis to Determine Relationships, BN .6: Total Sample Size – OLS Regression, R2 .2: Cancer Sample Descriptive Statistics1, by Group .1: BP, Population Parameters’ Optimal Points, listed as Se / Sp / GYI, AUC = (0.2: BP, Population Parameters’ Optimal Points, listed as Se / Sp / GYI, AUC = (0.3: BP, Population Parameters’ Optimal Points, listed as Se / Sp / GYI, AUC = (0.4: BP, Number of Invalid OOPs, listed as θ1 / θ2 / θ1 & θ2 .5: BP, Median Bias in (θ1, θ2), listed as θ1 / θ2 .6: BP, Median RMSE in (θ1, θ2), listed as θ1 / θ2 .7: BP, Median Bias in Sensitivity and Specificity for (θ1, θ2), listed as Se / Sp .8: BP, Median RMSE in Sensitivity and Specificity for (θ1, θ2), listed as Se / Sp .9: BP, Percent of Overestimation of Sensitivity and Specificity, listed as % Se / % Sp .10: BP, Coverage for SCIs and CRs for (θ1, θ2), listed as SCI / CR .11: BP, Median SCI Widths for (θ1, θ2), listed as θ1 / θ2 .12: BP, Median CR Area for (θ1, θ2) .13: BP, Median Bias in Empirical (θ1, θ2), listed as θ1 / θ2 .14: BP, Median Bias in Sensitivity and Specificity for Empirical (θ1, θ2), listed as Se / Sp .15: BP, Percent of Overestimation of Empirical Sensitivity and Specificity, listed as % Se / % Sp .16: BN, Population Parameters’ Optimal Points, listed as Se / Sp / GYI, AUC = (0.17: BN, Population Parameters’ Optimal Points, listed as Se / Sp / GYI, AUC = (0.18: BN, Population Parameters’ Optimal Points, listed as Se / Sp / GYI, AUC = (0.19: BN, Number of Times Invalid OOPs, listed as θ3 / θ4 / θ3 & θ4.20: BN, Median Bias in (θ3, θ4), listed as θ3 / θ4 .21: BN, Median RMSE in (θ3, θ4), listed as θ3 / θ4 .22: BN, Median Bias in Sensitivity and Specificity for (θ3, θ4), listed as Se / Sp .23: BN, Median RMSE in Sensitivity and Specificity for (θ3, θ4), listed as Se / Sp .24: BN, Percent of Overestimation of Sensitivity and Specificity, listed as % Se / % Sp .25: BN, Coverage for SCIs and CRs for (θ3, θ4), listed as SCI / CR .26: BN, Median SCI Widths for (θ3, θ4), listed as θ3 / θ4 .27: BN, Median CR Area for (θ3, θ4) .28: BN, Median Bias in Empirical (θ3, θ4), listed as θ3 / θ4 .29: BN, Median Bias in Sensitivity and Specificity for Empirical (θ3, θ4), listed as Se / Sp .30: BN, Percent of Overestimation of Empirical Sensitivity and Specificity, listed as % Se / % Sp .31: BP, Assessing Normality .32: BN, Assessing Normality .33: BP, Varying r (TSS = 200, k = 1, ρ = (0.34: BN, Varying r (TSS = 200, k = 1, ρ = (0.141 xii List of Figures Page Figure 2.2: Small, Medium, and Large Population Overlaps (Respectively) .3: BP and BN Diagnoses .1: ROC Curves and b .2: BP, Normally Distributed .3: BN, Not Normally Distributed .4: BN, Normality Assumption Questionable .1: Empirical ROC Curves of Tests .2: Histograms of CA125 and CA19-9 by Cancer Group.3: QQPlots of CA125 and CA19-9 by Cancer Group .4: Histograms of Transformed CA125 and CA19-9 by Cancer Group .5: QQPlots of Tranformed CA125 and CA19-9 by Cancer Group .6: Scatterplot of Transformed CA125 and CA19-9, by Cancer Group .7: BP CR Ellipse, Transformed and Original Units Respectively .8: BN CR Ellipse, Transformed and Original Units Respectively .85 xiii Abbreviations AUC Area Under the Curve BN Believe the Negative BP Believe the Positive CDF Cumulative Distribution Function CR Confidence Region FPR False Positive Rate GYI Generalized Youden Index NR Newton-Raphson OLS Ordinary Least Squares OOP Optimal Operating Point RMSE Root Mean Square Error ROC Receiver Operator Characteristic SCI Simultaneous Confidence Interval Se Sensitivity Sp Specificity TPR True Positive Rate TSS Total Sample Size YI Youden Index Abstract THE ESTIMATION AND EVALUATION OF OPTIMAL THRESHOLDS FOR TWO SEQUENTIAL TESTING STRATEGIES By Amber R. A Dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at Virginia Commonwealth University.

Virginia Commonwealth University, 2013 Advisor: Donna K. McClish Professor, Department of Biostatistics Many continuous medical tests often rely on a threshold for diagnosis. There are two sequential testing strategies of interest: Believe the Positive (BP) and Believe the Negative (BN). BP classifies a patient positive if either the first test is greater than a threshold θ 1 or xiv xv negative on the first test and greater than θ 2 on the second test.

BN classifies a patient positive if the first test is greater than a threshold θ 3 and greater than θ 4 on the second test. Of interest is to determine if these optimal threshold, or optimal operating point (OOP), estimates are “good” when calculated from a sample.