People have been asking me if I could briefly summarize my work on evaluation in the face of uncertainty. It took me a while, but I finally came up with the following.
NEED FOR CREATIVE THINKING ABOUT UNEXPECTED PROGRAM BEHAVIOR
I believe:
1) In powerful evaluation methodology.
2) Powerful methodology must encompass a broad range of quantitative and qualitative data collection techniques and research designs.
3) In order to have powerful evaluation designs, advance planning and rigorous implementation strategies are needed.
I know that:
1) Programs change in unexpected ways.
2) Program change may render many parts of an evaluation obsolete.
So, how to maintain maximum evaluation power in the face of uncertain program behavior? Answering this question has been the motivation for my work during the past few years.
THEORY:
I developed a theory of unexpected program behavior in which there is a continuum from events that might reasonably be foreseen, to those that are impossible to determine in advance. By “impossible” I mean really impossible because they spring from the dynamics of complex adaptive systems.
The same factors that make for uncertainty in program behavior also make for uncertainty in evaluation behavior. This is because both are similar social constructions — collections of people and resources, organized for a particular purpose, and set within a social / organizational / political / economic context.
TECHNIQUE
I propose a variety of methodologies that are differentially useful at different points along the continuum. None of the specific methods are esoteric or innovative, although some come from fields and may be unfamiliar to evaluators, and others may put familiar methods to innovative uses. What’s important is the relationships among these methods, and the value of setting them within a theory of unexpected behavior in programs and evaluations.
EMPIRICAL DATA
The theory and techniques are informed by a set of eighteen case studies. These have been analyzed and used to support various parts of my arguments. Two a priori frameworks are used to organize the data: 1) relationship between program and evaluation life cycles, and 2) social/organizational factors, (e.g. whether an issue springs from internal program behavior or the program’s environment). Three other categorizations emerged from the data analysis: 1) pilot and feasibility tests, 2) resistance to evaluation, and 3) incorrect assumptions early in the evaluation life cycle.
GOOD REASONS NOT TO FOLLOW MY ADVICE
Evaluation designs and their execution follow the same dynamics that drive any system. This means that perturbing the system, or adding elements to them, can create their own complications. For instance, it may be desirable to add clinical records review to interviews with clients, thereby assuring data availability if the interviews fail to materialize. But adding the review adds time and cost to the evaluation, which may result in missing a window of opportunity to educate stakeholders, or fewer resources for data analysis. A framework is provided to help guide decisions about what changes to make, and what positive and negative consequences they might have.
THE FUTURE
I’m focusing on two initiatives to further what I have done so far.
Agent based modeling and complex adaptive systems
1) Continue the work I and my colleagues have been doing to tightly integrate traditional evaluation with agent based modeling and simulation, complete with a continual feeding of knowledge back and forth between the model and the empirical data collection. I want to do this for two reasons. 1) Provide evaluators with greater lead time in scouting for unanticipated changes that may be affecting the programs they are evaluating. Lead time is critical because the longer the lead time, the greater the possibilities for adjusting evaluation designs. 2) It’s worth exploring whether the principles of complex systems are capable of doing a better job of explaining program behavior than our traditional methods of understanding how social systems work. If anyone has a lead on funding to do this work, please let me know.
Find more examples
2) Collect as many additional cases of unexpected program behavior as possible, and use the data to refine or change the views I have been advocating. I am on a determined hunt for more examples in order to advance my thinking about evaluating in the face of surprise. If you have any, or know people who might, please contact me.
Links to what I have written recently are at: http://www.jamorell.com
Evaluation Uncertainty 
Good evening,
This is just a thought based on my experience with families and
family systems. The model we used had the idea that interventions
with families did not necessarily focus on the specific behaviors of
one individual, but the system as a whole. For example, while it
would be impossible to predict the day to day actions and decisions of
an unpredictable parent in order to design contingencies for every
possible action, we looked at the propensity of the parent towards
unpredictability as the main issue, as well as the reaction by others
close to that parent when they were unpredictable.
For evaluation, then, rather than having the ability to forecast any
one positive or negative action taken by the organization, one could
look at whether the organization is capable of making more positive
decisions, whether the organizational structures can support “healthy
actions”, or whether the political and social contexts that surround
the program have the ability to support positive actions and decision
making. I am not sure that this is necessarily breaking ground, but
hopefully this perspective is helpful.
Jason Burkhardt
Hi Jason
Thanks for your response. The second paragraph in particular got me to thinking. It seems to me that there are really two issues here. 1) Capacity to make decisions , and 2) recognition of what is adaptive, i.e. “positive”.
Capacity for change
As for “capacity for change”, there are certainly many bodies of research and intellectual traditions that speak to this topic. Learning organization, readiness for change, innovation adoption, and adoption of evidence based practice are some that pop immediately to mind.
What is “positive”
I find this second topic much more interesting. One reason I like it is that it is values based. Another is that dealing with it involves some wonderfully knotty methodological issues.
Values
What is positive for one person or group may be negative for others. Or even more complicated is the possibility that the same change can be both positive and negative for the same group of stakeholders. Thus thinking about the consequences of change brings up some juicy values questions.
Another related topic I find interesting is that “positive” can be multidimensional. (So can “negative” for that matter.) For instance imagine a community facing urban decline. The city decides that a change is needed and is able to establish a program that guarantees college tuition to all kids who successfully get through its school system. A pretty good idea, isn’t it? I think so. But consider the downside. By making residency in the community so desirable, people who should move (say for a better job during an economic downturn) stay put. Do I really want to assert that the college tuition program is an unalloyed good?
On an organizational level I wonder what “positive action” means. What about the company that fires workers, cuts the wages of the remaining workforce, shuts down plants (thereby causing terrible disruption to communities), and thereby assures its survival as a profitable entity, an employer of thousands, and the source of a great deal of tax revenue. Did the company take “positive action”? With respect to itself as a whole, yes. With respect so some of its components, yes. With respect to other components, no. Such a company certainly has an admirable ability to change in the face of a changing environment, but how do we want to judge whether it engaged in positive or healthy change?
Methodology
Another reason I like the “positive action” topic is that it brings up some really interesting methodology questions. Think of all the possibilities I spun above with respect to the various outcomes that a program might have depending on one’s reference point. Addressing each one would complicate an evaluation methodology, adding time, cost, and complexity to the evaluation design. So, how to address as many as possible and keep the evaluation viable? (By “viable” I mean methodologically defensible and able to provide information within stakeholders windows of opportunity for using the information.) Chapter 7 of my book deals with this topic. It’s title is: How Much is too Much? Appreciating Trade Offs and Managing the Balance”. I had a lot of fun figuring that one out.