Morell Evaluation in the Face of Uncertainty

Anticipating Surprise and Responding to the Inevitable
Erscheinungsjahr 2011
ISBN: 978-1-60623-859-2
Verlag: Guilford Publications
Format: EPUB
Kopierschutz: Adobe DRM (»Systemvoraussetzungen)

E-Book, Englisch, 303 Seiten, EPUB, Format (B × H): 152 mm x 228 mm

Unexpected events during an evaluation all too often send evaluators into crisis mode. This insightful book provides a systematic framework for diagnosing, anticipating, accommodating, and reining in costs of evaluation surprises. The result is evaluation that is better from a methodological point of view, and more responsive to stakeholders. Jonathan A. Morell identifies the types of surprises that arise at different stages of a program's life cycle and that may affect different aspects of the evaluation, from stakeholder relationships to data quality, methodology, funding, deadlines, information use, and program outcomes. His analysis draws on 18 concise cases from well-known researchers in a variety of evaluation settings. Morell offers guidelines for responding effectively to surprises and for determining the risks and benefits of potential solutions.

Autoren/Hrsg.

Morell, Jonathan A.

Inhaltsverzeichnis

1.  From Firefighting to Systematic Action


 Adding “Surprise” to the Mix


 Historical Roots: Evaluation, Planning, and System Behavior


 From Explaining Surprise to Dealing with It


 Development Path of This Book


 Guiding Principles


 How to Read This Book


 In Sum


2.  Structure of the Unexpected


 Where Does Surprise Come From?


 Beyond Simple Distinctions


 In Sum


3.  Placing Surprise in the Evaluation Landscape


 When Is the Probability of Surprise High?


 When Is Surprise Disruptive to Evaluation?


 In Sum


4.  Minimizing Foreseeable Surprise


 Theory: Using Explanatory Power and Simplified Relationships


 Exploiting Past Experience: Capitalizing on What We Already Know


 Limiting Time Frames to Minimize the Opportunity for Surprise


 In Sum


5.  Shifting from Advance Planning to Early Detection


 Leading Indicators


 System-Based Logic Modeling


 In Sum


6.  Agile Evaluation


 Data


 Agile Methodology


 Retooling Program Theory


 Agility and Stakeholder Needs


 In Sum


7.  How Much Is Too Much?: Appreciating Trade-Offs and Managing the Balance


 A Framework for Appreciating Design Trade-Offs


 Maximizing Choice, Minimizing Risk


 Evaluation Design


 In Sum


8.  Applying the Examples to Categories of Cases: The Life Cycle View


 “Unintended Consequences”: Unity across Programs and Their Evaluations


 Interpreting Cases through a Life Cycle Perspective


 In Sum


9.  Applying the Examples to Categories of Cases: The Social/Organizational View


 Navigating through the Cases


 Placement of Cases on the Social/Organizational Map


 Categorizations Derived from the Data


 In Sum


10.  Lessons from Individual Cases: Tactics for Anticipating Surprise


 In Sum


11.  Lessons from Individual Cases: Responding to Surprise


 The Middle


 Leading Indicators and Agile Evaluation


 In Sum


12.  Unanticipated Program Outcomes


 Case Descriptions


 Applying the Cases to Unintended Program Outcomes


 Comparing the Cases


 Predicting the Need for Agile Evaluation


 In Sum


13.  Concluding Thoughts


Cases


Case 1. Grasping at Straws and Discovering a Different Program Theory: An Exercise in Reengineering Analysis Logic in a Child Care Evaluation Setting,
Dennis P. Affholter


Case 2. Shifting Sands in a Training Evaluation Context,
James W. Altschuld and
Phyllis M. Thomas


Case 3. Evaluating Programs Aimed at Promoting Child Well-Being: The Case of Local Social Welfare Agencies in Jerusalem,
Anat Zeira


Case 4. Assessing the Impact of Providing Laptop Computers to Students,
J. Dan Strahl, Deborah L. Lowther, and
Steven M. Ross


Case 5. Quasi-Experimental Strategies When Randomization Fails: Propensity Score Matching and Sensitivity Analysis in Whole-School Reform,
Gary L. Bowen, Roderick A. Rose, and
Shenyang Guo


Case 6. Unexpected Changes in Program Delivery: The Perils of Overlooking Process Data When Evaluating HIV Prevention,
Bryce D. Smith


Case 7. Evaluating Costs and Benefits of Consumer-Operated Services: Unexpected Resistance, Unanticipated Insights, and Déjà Vu All Over Again,
Brian T. Yates


Case 8. Keep Up with the Program!: Adapting the Evaluation Focus to Align with a College Transition Program’s Changing Goals,
Kristine L. Chadwick and
Jennifer Conner Blatz


Case 9. Assumptions about School Staff’s Competencies and Likely Program Impacts,
Laura Hassler Lang, Christine E. Johnson, and
Shana Goldwyn


Case 10. Mixed Method Evaluation of a Support Project for Nonprofit Organizations,
Riki Savaya and
Mark Waysman


Case 11. Evaluating the Health Impacts of Central Heating,
Jeremy Walker, Richard Mitchell, Stephen Platt, and
Mark Petticrew


Case 12. Recruiting Target Audience: When All Else Fails, Use the Indirect Approach for Evaluating Substance Abuse Prevention,
Molly Engle


Case 13. Unintended Consequences of Changing Funder Requirements Midproject on Outcome Evaluation Design and Results in HIV Outreach Services,
Lena
Lundgren, Therese Fitzgerald, and
Deborah Chassler


Case 14. Generating and Using Evaluation Feedback for Providing Countywide Family Support Services,
Deborah L. Wasserman


Case 15. Trauma and Posttraumatic Stress Disorder among Female Clients in Methadone Maintenance Treatment in Israel: From Simple Assessment to Complex Intervention


Miriam Schiff and
Shabtay Levit


Case 16. From Unintended to Undesirable Effects of Health Intervention: The Case of User Fees Abolition in Niger, West Africa,
Valéry Ridde and
Aissa Diarra


Case 17. Unintended Consequences and Adapting Evaluation: Katrina Aid Today National Case Management Consortium,
Amanda Janis and
Kelly M. Stiefel


Case 18. Evaluation of the Integrated Services Pilot Program from Western Australia,
Peter Hancock, Trudi Cooper, and
Susanne Therese Bahn

Über Autor(innen)

Jonathan A. Morell is Director of Evaluation at The Fulcrum Corporation and Editor of
Evaluation and Program Planning. Formerly he was Senior Policy Analyst at the Vector Research Center, a division of Jacobs Engineering. He is active in the American Evaluation Association (AEA), where he has been instrumental in founding two topical interest groups: Systems, and Business and Industry. He is a recipient of the Marcus Ingle Distinguished Service Award and the Paul F. Lazarsfeld Evaluation Theory Award from the AEA. His professional life has integrated his role as an evaluation practitioner with his theoretical interests. As a practitioner, he evaluates organizational change, R&D, and safety programs. He is also deeply involved in organizational design. His theoretical interests include the nature and use of logic models, the role of Lean Six Sigma methodologies in evaluation, complex system behavior, and the nature of practical action. He maintains a blog on issues related to evaluation and evaluation surprises.