Applying the complexity concept of “sensitive dependence” to understanding how a program works

Jonathan (Jonny) A Morell
President, 4.669… Evaluation and Planning
jamorell@jamorell.com
blog
YouTube  
This is the first of what I hope will be many posts that show how specific constructs from complexity science can be useful for doing evaluation. There will only be many posts if others contribute. Please do.

What complex behavior is this post about?
This post is about “sensitive dependence”.

A system’s sensitivity to initial conditions refers to the role that the starting configuration of that system plays in determining the subsequent states of that system. When this sensitivity is high, slight changes to starting conditions will lead to significantly different conditions in the future (Santa Fe Institute).

… refers to the idea that current and future states, actions, or decisions depend on the sequence of states, actions, or decisions that preceded them – namely their (typically temporal) path.  For example, the very first fold of a piece of origami paper will determine which final shapes are possible; origami is therefore a path dependent art (Santa Fe Institute).

What is an evaluation scenario where “sensitive dependence” may be relevant?
Figure 1 is a typical logic model. It’s simple and stylized, but it’s the kind of thing we like to draw. (There is usually a feedback loop or two, but I’m leaving them out to  keep the picture simple.) I realize that one can make distinctions between logic models, theories  of action, and theories of change, and  that those distinctions may affect what I am about to say. But for now, I am content to call these “models”. I don’t think the LM/ToA/ToC differences will make a big difference in my argument.

Maybe I’m wrong but I have a strong sense that whenever we construct models like Figure 1, what we really mean is Figure 2. I suspect that we assume that for the outcome to be achieved, each link needs to be operative. I hope I am wrong because if we really believed in this model, it means we have set the program up for failure. Why? Because each connection must be operative. What are the odds?

We want people to design programs like Figure 3 because a program with this logic has a high probability of succeeding. The reason it could succeed is because there are three separate paths that can lead to the desired outcome.

Let’s say that we are handed a program like Figure #3. Or hopefully, that we have helped our customers realize that they should design programs like number three. What does this mean for evaluation? In one sense, not much. After all, in all three models we still have to measure each element and determine if the indicated relationships are operating. Nor would the methodology be much different. More or less, whatever combination of time series data, comparison group data, and qualitative assessments that we would deploy for one model we would deploy for the others. The differences are in data interpretation.

What would make a difference would be if our customers looked at number three and asked: “Well, now that you have showed us what the correct path is, we can design programs like that  in the future and save ourselves a lot of time and effort. Isn’t that right?” Put another way, our customers would be saying: “You have showed us that our program theory was wrong. The real program theory is much simpler and cheaper and easier than we thought”.

How should we respond? If we had enough confidence in our methodology, we would answer in the affirmative, that yes indeed, we have discovered an operative path, i.e. a correct (and simpler) program theory. If we had a bit less confidence in our methodology, we would hedge our bets and only claim that we may have determined a correct program theory. But both answers would contain three assumptions: 1) The original, elaborate program theory was wrong. 2) There is a simpler program theory. 3) Our evaluation provides a reasonable idea of what that correct program theory is.

What are the consequences for the evaluation scenario if sensitive dependence is present?
If sensitive dependence is operating, the following may be true.

• Each time the model runs, i.e. each time the program is implemented, there are three possible paths to success that may be traveled.
• The path that will be traveled cannot be predicted in advance.
• The model in Figure 3 may correct in the sense that each time the program is implemented, one of those three paths will be traveled.

If this were the case, what would be the implications for evaluation? There are many. But here are a few that I can think of.