A metaevaluation of energy efficiency evaluations

Western Michigan University ScholarWorks at WMU Dissertations Graduate College 4-2014 A Metaevaluation of Energy Efficiency Evaluations Brandy Brown Western Michigan University, brandypratt@gmail.com Follow this and additional works at: https://scholarworks.edu/dissertations Part of the Energy Systems Commons, Statistics and Probability Commons, and the Sustainability Commons Recommended Citation Brown, Brandy, "A Metaevaluation of Energy Efficiency Evaluations" (2014).edu/dissertations/238 This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact wmu-scholarworks@wmich. A METAEVALUATION OF ENERGY EFFICIENCY EVALUATIONS by Brandy Brown A dissertation submitted to the Graduate College in partial fulfillment of the requirements for the degree of Doctor of Philosophy Interdisciplinary Ph.

in Evaluation Western Michigan University April 2014 Doctoral Committee: Dr. Chris Coryn, Ph. Daniela Schroeter, Ph. David Hartmann, Ph.

A METAEVALUATION OF ENERGY EFFICIENCY EVALUATIONS Brandy Brown, Ph. Western Michigan University, 2014 This study systematically reviews the methodological characteristics of energy efficiency evaluations and uses metaevaluation to assess its quality. Metaevaluation is used to systematically assess the quality of evaluation products, confirm that evaluations deliver sound findings and conclusions, are useful to the client, are credible, are ethically conducted, and are done as cost-effective as possible. The results of this study show that the ability to accurately assess evaluation for methodological quality using evaluations reports as a primary data source depends on the presence of detailed descriptions of evaluation methods.

Furthermore, the study suggests that methodological variations of energy efficiency evaluations coalesce along energy efficiency evaluation types. The study concludes that energy efficiency evaluation practitioners can augment efforts to improve the industries evaluation methods, policies and processes by integrating metaevaluation into its practice. © 2013 Brandy Brown ACKNOWLEDGEMENTS I would like to thank my dissertation committee for guiding me through my learning process; Dr. Chris Coryn, thank you for introducing me to evaluation and providing me with every opportunity to grow as a thought leader.

Daniela Schroeter, thank you for your firm standards and constant encouragement. David Hartmann, thank you for believing in my abilities and giving me an opportunity to exercise them. I would also like to express gratitude to two individuals who shaped this study; Dr. Daniel Stufflebeam, thank you for allowing me to reference your work and apply it to energy efficiency evaluation.

Rafael Friedmann, I greatly appreciate your guidance and encouragement as I became an energy efficiency evaluator. I would like to extend a special thank you to Mary Ramlow for continuously problem solving and encouraging me at every step of this process. I’d also like to thank my family who collectively serves as my foundation. To my parents: You’ve nurtured me, challenged me, and supported me through everything.

I can never repay you for your priceless investment. I promise follow in your foot steps by paying forward what you’ve gifted me, to my children and any others whom I may help along the way. ii Acknowledgments—Continued To my husband: You are my rock. Thank you for your support, love, and endless humor.

To my children, my little blessings: Thank you for being my motivation and the embodiment of all that is possible. Your love sustains me. Brandy Brown iii TABLE OF CONTENTS ACKNOWLEDGEMENTS. ii LIST OF TABLES.

3 Defining Energy Efficiency. 5 Energy Efficiency Policy. 6 The Energy Efficiency Industry. 8 Energy Efficiency Evaluation.

12 Rationale for Study. 13 Structure of the Dissertation. REVIEW OF RELATED LITERATURE. 16 Evaluation: A Brief History.

18 Evaluation as a Discipline. 21 Research on Evaluation. 24 iv Table of Contents—Continued CHAPTER Energy Efficiency Methodology. 27 Standards and Practice.

32 Rationale for the Methodology. 36 California Evaluator's Protocol Compliance. 45 Qualitative Reliability and Validity. 45 Data Processing and Analysis.

49 Conformity to Program Evaluation Standards. 50 Conformity to CPUC Evaluation Standards. 53 Overall Systematic Review Findings. 59 v Table of Contents—Continued CHAPTER Process Evaluation Findings.

61 Process Evaluation Approach. 64 Impact Evaluation Findings. 68 Threats to Validity. 73 Implications of the Results.

74 Contribution to the Discipline of Evaluation. 75 Recommendations for Future Research. Program Metaevaluation Checklist Example. Level of Rigor Instrumentation.

Sampling and Uncertainty Levels of Rigor. Systematic Review Instrumentation. 99 vi LIST OF TABLES 1. Inclusion Criteria for CALMAC Results.

Example of Program Metaevalaution Checklist. Program Metaevaluation Checklist Score Ranges. California Protocol Checklist. Evaluation Design Characteristics.

Checklist for Threats to Validity. Levels of Rigor. Average Metaevaluation Ratings. Metaevaluation Score Ranges.

Rating Frequency among Energy Efficiency Evalautions. Level of Rigor. Distribution of Evaluations by Research Design. Elements of Design.

Methods of data Collection. Types of Analysis. 60 vii List of Tables—Continued 18. Threats to Validity.

Use of Design Elements. Use of Sampling. Data Collection Methods. Use of the California Protocol Approach.

Use of Qualitative Inquiry Methods. Research Designs among Impact Evaluations. Design Elements in Impact Evaluations. Sampling among Impact Evaluations.

Data Collection among Impact Evaluations. Analysis Methods among Impact Evaluations. Threats to Validity in Impact Evaluations. Alignment with California Protocol among Impact Evaluations.

70 viii CHAPTER I INTRODUCTION Energy efficiency evaluation methods have stagnated as the industry has adapted a “business as usual” approach, which ignores contemporary evaluation methods (Friedmann, 2011, p. In this way, energy efficiency evaluation is out of step with the larger evaluation community in assessing the quality of its work. Over the last twenty years, evaluators have emphasized evaluating their work to ensure that they are producing quality, methodologically sound evaluations (Mark, 2007). In doing so, evaluators have strived toward building an empirical basis for the practice of evaluation, which simultaneously should improve existing methods of evaluation.

Energy efficiency evaluation has yet to address this issue. Moreover, energy efficiency evaluation approaches vary so widely that evaluation results cannot be reasonably compared across particular studies or jurisdictions (Kushler, Nowak, & Witte, 2011). This lack of consistency among energy efficiency evaluation approaches has resulted in a wide variation of evaluation methodology and practice (Schiller & Goldman, 2012). In the last few years, scrutiny of energy efficiency evaluation has grown and manifested as debates on the accuracy of energy efficiency evaluation results and concern with funding evaluation activities 1 (Vine, 2012).

Because inconsistent methods open the door to attacks on the validity of evaluation results, it is important to demonstrate sound methods in an effort to bolster the credibility of energy efficiency research (Schiller & Goldman, 2012). Although energy efficiency evaluators understand that practice may differ from theory, they desire to establish a common framework of standard practices in an effort to communicate best practices (Kushler, Nowak, & Witte, 2011; Dougherty, Sutter, & Randazzo, 2012; Williamson & Kasman, 2012; NAPEE, 2006; Schiller & Goldman, 2012). However, empirical evidence that depicts energy efficiency evaluation as practiced does not exist. There is a chasm in the literature in regards to empirical evidence of common energy efficiency evaluation methodologies and practices.

The available literature focuses almost exclusively on calculations of energy savings and the measure types (a measure type is a category of energy efficiency equipment such as lighting or heating ventilation, and cooling) associated with those savings. Consequentially, evidence-based energy efficiency practice has not been compared to general evaluation standards in order to assess its quality. Therefore, prior to the collective development of national energy efficiency evaluation standards, there must be a thorough study on the current common practices in energy efficiency evaluation. Thus, the purpose of this study is to fill the void in the literature by examining the current practice of evaluation within the field of energy efficiency.

It will investigate the quality of energy efficiency evaluation by applying generally accepted standards for evaluation. This chapter provides an introduction to energy efficiency evaluation. It defines 2 energy efficiency, describes the field of energy efficiency, addresses energy efficiency policy, and describes the energy efficiency industry and evaluation within that industry. Furthermore, it elucidates the focus of the study, its purpose, and its importance.

Finally, the chapter ends with an explanation of the research questions. Background After the 1973 oil crisis, energy efficiency became an increased priority in the United States (Laitner, 2012). Since that time, energy use per unit of gross domestic product (GDP) decreased 42% with energy efficiency responsible for three quarters of that change (Geller, 2001). Also during this period, the average fuel economy of new cars increased from 16 mpg to 24 mpg; energy efficiency of appliances increased 56%; and sales of more energy efficient lighting increased five times over (Geller, 2001).

However, there is still much work needed to increase the efficient use of energy in the United States and elsewhere. The Environmental Protection Agency (EPA) asserts that in the commercial sector more than 30% of energy use is squandered (McLean-Conner, 2009). Likewise, it was estimated that in 2009 the United States wasted more than half (58%) its total energy because of inefficiency (Lawernce Livermore National Laboratory, 2010). These weaknesses in the ways in which energy is used leaves great potential for the future of energy efficiency.

Several organizations have identified the current potential for energy efficiency. The American Council for an Energy Efficient Economy (ACEEE) maintains that 3 national comprehensive energy efficiency policies could: (a) cut national energy use by another 33% by 2020, (b) dramatically lower greenhouse gas emissions, and (c) save consumers and business $500 billion from 2000-2020 (Geller, 2001). They are not alone in their assessment of the potential impacts of energy efficiency. Energy Star, an EPA initiative, stresses that replacing one incandescent light bulb with a compact fluorescent light bulb (CFL) in each home in the United States would save over $600 million in annual energy costs and produce enough electricity to light three million homes each year (McLean-Conner, 2009).

Energy efficiency is considered the most economical, effective, and swiftest means of reducing energy consumption in the United States (Gold, Furrey, Nadel, Laitner, & Elliott, 2009). The motivations driving consumers to reduce energy use may vary. However, two resounding themes exist in the move toward more efficient energy use; first, reducing cost, and second, improving the environment (McLean-Conner, 2009). When consumers reduce their energy use, they benefit from a reduction in their energy costs.

Since consumers pay for energy based on the amount of energy they consume, products that use less energy result in a reduced cost of energy. This financial savings serves as a genuine motivation for consumers to use less energy. Coupled with recent economic conditions, the financial effect of energy efficiency is an attractive way for consumers to reduce cost. Environmentally conscious consumers are also motivated to enhance their energy efficiency, as it is a proven path to achieving the second theme, improving the 4 environment (Mills & Rosenfeld, 1996).

The International Energy Agency reported that more efficient uses of energy in transportation, buildings, and industrial processes could reduce the global energy need by one third by the year 2050 (International Energy Agency, 2006). Such a reduction in energy consumption would diminish the amount of pollution that would be present otherwise. As households and businesses in the United States look for the most convenient and inexpensive ways to save money and go green, energy efficiency has grown in popularity (Griskevicius, Tyber, & Van den Bergh, 2010). Within the last decade, “going green” has become socially desirable and marketable.

Defining Energy Efficiency According to the Lawrence Berkeley National Laboratory, energy efficiency is defined as “using less energy to provide the same service” (Lawrence Berkley National Laboratory, 2012, p. For example, a television that uses less energy than an older model television is considered more energy efficient. This concept differs from energy conservation, which involves forgoing use of something, or using it less, in order to reduce one’s energy usage or consumption.