I’m doing a presentation at the U.S. Department of State’s Fourth Annual Conference on Program Evaluation: “Diplomacy, Development, and Defense – Working Together to Achieve Foreign Policy Goals” June 7-8, 2011
I’m looking for comment and critique. Draft of presentation slides
Purpose and Assumptions
The focus of the proposed presentation is methodology to evaluate unanticipated and unintended consequences of program action. It is based on an evaluation theory and a set of case studies developed by the presenter. The specific track for this proposal is “building evaluation capacity”, with an immediate impact on “development”, and a longer term impact on the “building democracy” element of the “diplomacy” theme. The proposed presentation is based on two principles.
§ Good evaluation must often depends on maintaining the integrity of an evaluation design over time. For instance, it may be necessary to interview service recipients within a narrow window of opportunity after they received a service, or to apply a previously validated scale over a protracted period of time, or to maintain good relations with program staff so that observations of their work can be made.
§ In order to assure the integrity of an evaluation’s design, scarce resources are needed to build and maintain an “evaluation infrastructure”. To continue the previous example, it takes time and effort for evaluators to maintain a set of agreements with program managers and policy makers, or to develop and validate scales. Once these resources are used, there is that much less opportunity to adjust the evaluation design in the face of unexpected program behavior.
These principles are challenged by the fact that the consequences of programs are often different from what planners expected, and therefore, different from the outcomes that evaluators had planned to measure. How then, to best maintain the power of an evaluation design and still be responsive to changing needs? This is the question that the panel will address. As the answer unfolds it will have major consequences for how research questions are formulated, strategic planning, and program monitoring. Consequences for question formulation and strategic planning derive from unintended outcomes whose roots are in narrow conceptualizations of program theory and outcome. Consequences for monitoring derive from the fact that “lead time” for detecting incipient program change is a critical element in evaluation response to unanticipated change. Good monitoring can increase lead time.
Unexpected Outcomes in Development and Democracy
The techniques to be advocated are particularly relevant in development settings because programs in those contexts are extremely prone to outcomes that were not anticipated by planners and policy makers. This uncertainty exists because development programs often involve rich, tight linkages that affect many aspects of the systems in which they reside, and also because the environments in which they exist can be unstable.
The role of unexpected program outcome is particularly important when trying to assess the relationship between program outcome and democracy because of a mismatch between what is known about the relationship between democracy and development on the one hand, and the causal path between a development program and the precursors to democracy on the other. We have a good idea about what brings about democracy:
The good news, however, is that the conditions conducive to democracy can and do emerge—and the process of “modernization,” according to abundant empirical evidence, advances them. Modernization is a syndrome of social changes linked to industrialization. Once set in motion, it tends to penetrate all aspects of life, bringing occupational specialization, urbanization, rising educational levels, rising life expectancy, and rapid economic growth. These create a self-reinforcing process that transforms social life and political institutions, bringing rising mass participation in politics and—in the long run—making the establishment of democratic political institutions increasingly likely. Today, we have a clearer idea than ever before of why and how this process of democratization happens.
From the point of view of program theory and evaluation though, any given development program may affect any democracy precursor in many ways, not all of which will be foreseen by planners and evaluators. Moreover, the evaluation challenge is compounded in the typical situation where many different development programs coexist within the same system boundaries. Finally, given the many paths and interactions that may develop, the same result may come about through many different paths. To illustrate:
Example: Difficulty of Relating Programs to the long term goal of promoting democracy
Imagine an overly simple scenario in which two programs are operating, one whose primary goal is job training, and one that provides affordable cell phone service in rural areas. Again to oversimplify for the sake of illustration, we may state a program theory for each program. The job training program leads directly to “occupational specialization”. The cell phone program increases the richness of social contact and the ability of farmers to peg their prices to world commodity prices. These are reasonable immediate outcomes to expect, and they do need to be justified (i.e. evaluated) as such. Understanding their consequences for democracy, however is complicated. One problem is that many possible interactions can take place. Will the richer social networks make it easier for people to find ways to specialize their work? Will better commodity pricing allow more people to take advantage of the job training? Will unequally distributed rising levels of wealth and education support or upset the existing social structure? Might the changed social structure set in motion its own undesirable consequences? Any of these outcomes are possible, as are many more that one could conjure with a little bit more time and imagination. From the point of view of evaluation these uncertain outcomes are problematic for methodological reasons. Again to take two overly simple examples. What if, after the evaluation was established, we suspected an unanticipated interaction between occupational specialization and social relationships? Or, we suspected that agricultural pricing in the location of the cell phone intervention was having larger scale impact because of its contribution to a tipping point change in a region that was experiencing the effects of other development programs? Supposing both were occurring and we cared about their relative impact? The first example is about micro-level short term change. The second is longer term and larger scale. They may or may not interact. What they have in common is that the original evaluation infrastructure would be inadequate to determine the state of affairs. Different people would need to be interviewed. Different statistical data bases would have to be queried. Different data collection timelines would have to be worked out. Different comparison groups would be needed. None of this is cheap or easy.
Methods for Evaluating Unintended Program Impact
The approach to be advocated in the proposed presentation is set out in the writing of the presenter, and can be summarized in the graphic below.1
§ There is a continuum that ranges from events that “could reasonably be foreseen” to those that are impossible to predict because they emanate from the operations of complex systems.
§ Different evaluation tools are differentially useful at different points along this continuum. For instance applying diverse program theories is particularly useful for anticipating outcomes, monitoring and evaluation for early detection of unexpected change, and multiple data sources to make the evaluation more capable of measuring a wider range of program behavior.
§ It is impossible to deal with all eventualities, but there are ways to “chip away” at the problem and by so doing to make evaluation more robust in the face of program change.
§ Any design choice to make an evaluation more robust in the face of change carries its own potentially negative consequences. Thus choices have to be made carefully. For instance the resources needed to maintain multiple data sources may diminish resources for interaction with stakeholders, or effort devoted to analysis.
 Morell, J. A. (2005). “Why are there unintended consequences of program action, and What Are the Implications for Doing Evaluation?” American Journal of Evaluation 26(4): 444 – 463
Morell, J. A. (2010). Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable. New York, Guilford.
 Inglehart, R. and C. Welzel (2009). “How Development Leads to Democracy: What We Know About Modernization.” Foreign Affairs 88(2): 33 – 49.