This is an abstract of a presentation I gave at AEA 2014. Click here for the slide deck.

How to measure impact if impact is a moving target? Some examples: 1) a qualitative pre-post design uses semi-structured questions to determine people’s distress during refugee resettlement; 2) a post-test only quantitative study of innovation adoption requires comparisons between design engineers in different parts of a company; 3) a health awareness program requires a validated survey. But what if halfway through, different outcomes are suspected: family dynamics among the refugees; skills of production engineers; mental health in the health case? No problem if the solution is a post-test only design using existing data or flexible interview protocols. But those are just a few arrows in the quiver of evaluation methods. How can design choices be maximized when so many of those choices assume stability of impact? The answer depends on making design choices based on understanding why and when outcomes shift, and the amount of uncertainty about outcomes.