Complexity is about stability and predictability

Table of Contents

Complexity is About Stability and Predictability

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Fractals
Example 4: Phase Transitions
Example 5: Logistic Maps
Example 6: Power Laws
Example 7: Cross Linkages
Example 8: Emergence

What Does All This Mean for Evaluators?

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Power Laws
Example 4: Timeframes, Attractors, and Power Laws
Example 5: Emergence
Example 6: Fractals
Example 7: Phase Shifts


Complexity is About Stability and Predictability

Figure 1: Ban the Butterfly

Figure 1: Ban the Butterfly

I have been thinking about how complexity is discussed in evaluation circles. A common theme seems to be that because programs are complex we can’t generalize evaluation findings over space and time because of the inherent uncertainties that reside in complex systems. (Sensitive dependence on initial conditions, evolving environments, etc.) The more I think about the emphasis on instability and unpredictability, the less I like it. See figure 1. Ban the butterfly! Continue reading

Posted in Uncategorized | 2 Comments

Big Data in Evaluation

I have been developing an interest in “big data” as it may relate to the field of evaluation. The interest comes from two sources. 1) As the Editor of the journal Evaluation and Program Planning, I am on the lookout for cutting edge material to present to our readers. As someone who has done research and theoretical work on unintended consequences of program action, I see big data as a methodology that may help to reveal such unintended consequences. As a result of these two interests I’m looking for:

  • Examples where a big data approach has revealed consequences of program action that were not anticipated, and
  • People who may want to write about big data as it applies to the field of evaluation.

If you can help please get in touch with me at

Posted in Looking for examples -- Unintended consequences of program behavior | 1 Comment

How to evaluate complex health interventions?

I just came back from Toronto, where I was visiting Sanjeev Sridharan, who runs a most interesting organization called the Evaluation Center for Complex Health Interventions. I was there to demonstrate the applicability of agent-based modeling as an evaluation tool. During the talk nobody asked me a question I could not easily answer. But then came lunch the next day, when Sanjeev casually asked “How would you evaluate a complex health intervention”? Hmmmm. On the train home a few days later I worked out an outline of an answer. I figured I’d lay it out there for the world to have a whack at. See below for my ruminations. Whack away.

Because I have been thinking so much about agent based systems lately I came up with the notion that concepts drawn from that domain could be useful for constructing evaluation strategies. I’m not arguing (right now) for doing formal agent based modeling. But I do think that the idea of agent behavior could be a useful trick of the mind to help design evaluations.

Some brief background on agents
What is an “agent”? It is an entity that Continue reading

Posted in Complex Systems, Programs, and Evaluation | 10 Comments

Matching management systems to system behavior

Matt Keene and I have been having a back and forth on the topic of how management systems are built with respect to the way in which systems actually behave. Below is a record of our conversation to date.

Jonny’s thoughts
Desirability of Narrow Rigid Planning
First, I don’t place any negative value judgments on the need for rigid planning. There are good reasons for it given the nature of the world. Second, I don’t think that environmental management is different from any other kind of management in this regard. Everyone is in the same boat.  Third, the problem is not management systems. To blame the systems is like shooting the messenger.  Management systems match the world in which management takes place, and that is what is giving us trouble. Continue reading

Posted in Uncategorized | 4 Comments

What is the relationship between path dependence and system stability? With explanation of why I care.

I realized it might help to explain what led me to ask this question in the first place. I submitted a proposal to AEA to talk about how traditional evaluation methods can be used in complex systems. Part of that explanation will have to involve understanding the CAS implications of stability in program impact across time and place. See the end of this post for that proposal.

I’m looking for some sources and opinions to help with a question that has been troubling me lately.  I’m struggling with the question of the relationship between path

  • dependence and
  •  system stability.

Or maybe I mean the relationship between path dependence and the ability to predict a system’s trajectory. I’m not sure about the best way to phrase the question.  In any case read on to see my confusion.

I’m bumping into a lot of people who believe that systems are unstable/unpredictable because of path dependence. This is one of those notions that seems right but smells wrong to me. It seems too simple, and it does not make sense to me because it implies that if systems are predictable there is no path dependence operating.  That can’t be right, can it? Here is a counter example. Continue reading

Posted in System Stability and Sustainability | 12 Comments

Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community

What I tried to do is to look at the link between the intellectual traditions of evaluation and the sociology of evaluation from the point of view of evolutionary biology. For simplicity I’m assuming only two times in what is a long continuous process, and only two intellectual traditions. Also, I’m ignoring the possibility of linkages and networks among multiple traditions, and the multitude of reasons why the market for evaluation may change other than the behavior of evaluators. With those gross simplifications, my notions goes like this. Continue reading

Posted in Logic Models and Program Theory, Uncategorized | 1 Comment

Simple, Complicated, Complex and Chaotic VS. Complexity Science. Jonny finally resolved his confusion

In addition to my hissy fit about the “agreement x certainty” matrix I have also been in a bit of a lather about the typology in the Cynefin model that identifies four states of systems – simple, complicated, complex, and chaotic. I like this model and I think it is exceedingly useful for helping people understand the program and evaluation scenarios they are working in. But at the same time I always bristled at it, and I finally figured out why.

As I see it, the way Cynefin draws on complexity concepts only partially overlaps with the way complexity science deals with complexity. Those four domains only partially overlap with what CAS researchers and theoreticians think of as complex systems. Continue reading

Posted in Complex Systems, Programs, and Evaluation | 20 Comments

A rolling conversation about the “Agreement x Certainty” space.

ECLIPS (Evaluation Communities of Learning, Inquiry, and Practice about Systems) is a project sponsored by the National Science Foundation to improve the evaluation programs that support Science, Technology, Engineering and Math (STEM). The project is housed at InSites. Beverly Parsons is the PI. I am on the advisory board. Recently I  and some of the project staff (Beverly Parsons, Pat Jessup and Marah Moore) had a rambling conversation about the “certainty x agreement” graph that is becoming so common among those of us who are trying to apply systems concepts in evaluation. I am not a big fan of the graph. Others are. Below is a somewhat cleaned up and edited version of our back and forth on this topic.  We used colors to differentiate our responses.

Here is how to read this. Black is my original post. All the discussion breaks up the post. To read my original you have to go through and read only the black first. Blue are my responses to what other people said. All the other colors are comments Bev and Pat and Marah made. I don’t remember who used which color.

Continue reading

Posted in Complex Systems, Programs, and Evaluation, Uncategorized | 3 Comments