People have been asking me if I could briefly summarize my work on evaluation in the face of uncertainty. It took me a while, but I finally came up with the following.

I believe:

1) In powerful evaluation methodology.

2) Powerful methodology must encompass a broad range of quantitative and qualitative data collection techniques and research designs.

3) In order to have powerful evaluation designs, advance planning and rigorous implementation strategies are needed.

I know that:

1) Programs change in unexpected ways.

2) Program change may render many parts of an evaluation obsolete.

So, how to maintain maximum evaluation power in the face of uncertain program behavior? Answering this question has been the motivation for my work during the past few years.

I developed a theory of unexpected program behavior in which there is a continuum from events that might reasonably be foreseen, to those that are impossible to determine in advance. By “impossible” I mean really impossible because they spring from the dynamics of complex adaptive systems.

The same factors that make for uncertainty in program behavior also make for uncertainty in evaluation behavior. This is because both are similar social constructions — collections of people and resources, organized for a particular purpose, and set within a social / organizational / political / economic context.

I propose a variety of methodologies that are differentially useful at different points along the continuum. None of the specific methods are esoteric or innovative, although some come from fields and may be unfamiliar to evaluators, and others may put familiar methods to innovative uses. What’s important is the relationships among these methods, and the value of setting them within a theory of unexpected behavior in programs and evaluations.

The theory and techniques are informed by a set of eighteen case studies. These have been analyzed and used to support various parts of my arguments. Two a priori frameworks are used to organize the data: 1) relationship between program and evaluation life cycles, and 2) social/organizational factors, (e.g. whether an issue springs from internal program behavior or the program’s environment). Three other categorizations emerged from the data analysis: 1) pilot and feasibility tests, 2) resistance to evaluation, and 3) incorrect assumptions early in the evaluation life cycle.

Evaluation designs and their execution follow the same dynamics that drive any system. This means that perturbing the system, or adding elements to them, can create their own complications. For instance, it may be desirable to add clinical records review to interviews with clients, thereby assuring data availability if the interviews fail to materialize. But adding the review adds time and cost to the evaluation, which may result in missing a window of opportunity to educate stakeholders, or fewer resources for data analysis. A framework is provided to help guide decisions about what changes to make, and what positive and negative consequences they might have.

I’m focusing on two initiatives to further what I have done so far.

Agent based modeling and complex adaptive systems
1) Continue the work I and my colleagues have been doing to tightly integrate traditional evaluation with agent based modeling and simulation, complete with a continual feeding of knowledge back and forth between the model and the empirical data collection. I want to do this for two reasons. 1) Provide evaluators with greater lead time in scouting for unanticipated changes that may be affecting the programs they are evaluating. Lead time is critical because the longer the lead time, the greater the possibilities for adjusting evaluation designs. 2) It’s worth exploring whether the principles of complex systems are capable of doing a better job of explaining program behavior than our traditional methods of understanding how social systems work. If anyone has a lead on funding to do this work, please let me know.

Find more examples
2) Collect as many additional cases of unexpected program behavior as possible, and use the data to refine or change the views I have been advocating. I am on a determined hunt for more examples in order to advance my thinking about evaluating in the face of surprise. If you have any, or know people who might, please contact me.

Links to what I have written recently are at:

2 thoughts on “Jonny Morell’s very brief summary of his work on evaluation uncertainty

  1. Good evening,

    This is just a thought based on my experience with families and
    family systems. The model we used had the idea that interventions
    with families did not necessarily focus on the specific behaviors of
    one individual, but the system as a whole. For example, while it
    would be impossible to predict the day to day actions and decisions of
    an unpredictable parent in order to design contingencies for every
    possible action, we looked at the propensity of the parent towards
    unpredictability as the main issue, as well as the reaction by others
    close to that parent when they were unpredictable.

    For evaluation, then, rather than having the ability to forecast any
    one positive or negative action taken by the organization, one could
    look at whether the organization is capable of making more positive
    decisions, whether the organizational structures can support “healthy
    actions”, or whether the political and social contexts that surround
    the program have the ability to support positive actions and decision
    making. I am not sure that this is necessarily breaking ground, but
    hopefully this perspective is helpful.

    Jason Burkhardt

  2. Hi Jason

    Thanks for your response. The second paragraph in particular got me to thinking. It seems to me that there are really two issues here. 1) Capacity to make decisions , and 2) recognition of what is adaptive, i.e. “positive”.

    Capacity for change
    As for “capacity for change”, there are certainly many bodies of research and intellectual traditions that speak to this topic. Learning organization, readiness for change, innovation adoption, and adoption of evidence based practice are some that pop immediately to mind.

    What is “positive”
    I find this second topic much more interesting. One reason I like it is that it is values based. Another is that dealing with it involves some wonderfully knotty methodological issues.

    What is positive for one person or group may be negative for others. Or even more complicated is the possibility that the same change can be both positive and negative for the same group of stakeholders. Thus thinking about the consequences of change brings up some juicy values questions.

    Another related topic I find interesting is that “positive” can be multidimensional. (So can “negative” for that matter.) For instance imagine a community facing urban decline. The city decides that a change is needed and is able to establish a program that guarantees college tuition to all kids who successfully get through its school system. A pretty good idea, isn’t it? I think so. But consider the downside. By making residency in the community so desirable, people who should move (say for a better job during an economic downturn) stay put. Do I really want to assert that the college tuition program is an unalloyed good?

    On an organizational level I wonder what “positive action” means. What about the company that fires workers, cuts the wages of the remaining workforce, shuts down plants (thereby causing terrible disruption to communities), and thereby assures its survival as a profitable entity, an employer of thousands, and the source of a great deal of tax revenue. Did the company take “positive action”? With respect to itself as a whole, yes. With respect so some of its components, yes. With respect to other components, no. Such a company certainly has an admirable ability to change in the face of a changing environment, but how do we want to judge whether it engaged in positive or healthy change?

    Another reason I like the “positive action” topic is that it brings up some really interesting methodology questions. Think of all the possibilities I spun above with respect to the various outcomes that a program might have depending on one’s reference point. Addressing each one would complicate an evaluation methodology, adding time, cost, and complexity to the evaluation design. So, how to address as many as possible and keep the evaluation viable? (By “viable” I mean methodologically defensible and able to provide information within stakeholders windows of opportunity for using the information.) Chapter 7 of my book deals with this topic. It’s title is: How Much is too Much? Appreciating Trade Offs and Managing the Balance”. I had a lot of fun figuring that one out.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s