Emergence: Part 1 of 6 Posts on Evaluation, Complex Behavior, and Themes in Complexity Science

Common Introduction to all 6 Posts

History and Context
These blog posts are an extension of my efforts to convince evaluators to shift their focus from complex systems to specific behaviors of complex systems. We need to make this switch because there is no practical way to apply the notion of a “complex system” to decisions about program models, metrics, or methodology. But we can make practical decisions about models, metrics, and methodology if we attend to the things that complex systems do. My current favorite list of complex system behavior that evaluators should attend to is:

Complexity behavior	Posting date
·      Emergence	up
·      Power law distributions	Sept. 21
·      Network effects and fractals	Sept. 28
·      Unpredictable outcome chains	Oct. 5
·      Consequence of small changes	Oct. 12
·      Joint optimization of uncorrelated outcomes	Oct. 19

For a history of my activity on this subject see: PowerPoint presentations: 1, 2, and 3; fifteen minute AEA “Coffee Break” videos 4, 5, and 6; long comprehensive video: 7.

Since I began thinking of complexity and evaluation in this way I have been uncomfortable with the idea of just having a list of seemingly unconnected items. I have also been unhappy because presentations and lectures are not good vehicles for developing lines of reasoning. I wrote these posts to address these dissatisfactions. From my reading in complexity I have identified four themes that seem relevant for evaluation.

• Pattern
• Predictability
• How change happens
• Adaptive and evolutionary behavior

Others may pick out different themes, but these are the ones that work for me. Boundaries among these themes are not clean, and connections among them abound. But treating them separately works well enough for me, at least for right now. Figure 1 is a visual depiction of my approach to this subject.

Figure 1: Complex Behaviors and Complexity Themes

The black rectangles on the left depict a scenario that pairs a well-defined program with a well-defined evaluation, resulting in a clear understanding of program outcomes. I respect evaluation like this. It yields good information, and there are compelling reasons working this way. (For reasons why I believe this, see 1 and 2.)

• The blue region is there to indicate that no matter how clear cut the program and the evaluation; it is also true that both the program and the evaluation are embedded in a web of entities (programs, policies, culture, regulation, legislation, etc.) that interact with our program in unknown and often unknowable ways.
  

• The green region depicts what happens over time. The program may be intact, but the contextual web has evolved in unknown and often unknowable ways. Such are the ways of complex systems.

• Recognizing that we have a complex system, however, is not amenable to developing program theory, formulating methodology, or analyzing and interpreting data. For that, we need to focus on the behaviors of complex systems, as depicted in the red text in the table. Note that the complex behaviors form the rows of a table. The columns show the complexity themes. The Xs in the cells show which themes relate to which complexity behaviors..

• he black rectangles on the left depict a scenario that pairs a well-defined program with a well-defined evaluation, resulting in a clear understanding of program outcomes. I respect evaluation like this. It yields good information, and there are compelling reasons working this way. (For reasons why I believe this, see 1 and 2.)

•  The blue region is there to indicate that no matter how clear cut the program and the evaluation; it is also true that both the program and the evaluation are embedded in a web of entities (programs, policies, culture, regulation, legislation, etc.) that interact with our program in unknown and often unknowable ways.

• The green region depicts what happens over time. The program may be intact, but the contextual web has evolved in unknown and often unknowable ways. Such are the ways of complex systems.

• Recognizing that we have a complex system, however, is not amenable to developing program theory, formulating methodology, or analyzing and interpreting data. For that, we need to focus on the behaviors of complex systems, as depicted in the red text in the table. Note that the complex behaviors form the rows of a table. The columns show the complexity themes. The Xs in the cells show which themes relate to which complexity behaviors.

Emergence

	Pattern	Predictability	How change happens	Adaptive evolutionary behavior
Emergence			X
Power law distributions
Network effects and fractals
Unspecifiable outcome chains
Consequence of small changes
Joint optimization of uncorrelated outcomes

“Emergence” refers to a situation in which a phenomenon cannot be explained by the functioning of its parts. As a counter example, think of a car. Of course the car is more than the sum of its parts. But it is also true that the unique function of each part, and its contribution to the car, can be explained. If someone asked me what a “cylinder” is, I could describe it. I could describe what it does. When I got finished, you would know how the part “cylinder” contributes to the system called a “car”.

In contrast, think about trying to explain a traffic jam only in terms of the movement of each individual car. The jam moves in the opposite direction to the cars of which it is composed. The jam grows at the back as cars slow down, and the front recedes as cars peel off. Looked at as a whole, the traffic jam is clearly something qualitatively different from the movement of any car in the jam. (NetLogo has good one and two lane simulations that are worth looking at.)

Lately I am becoming more and more convinced that we get into trouble when we do not recognize when traditional program logic needs to be replaced with emergent logic. Most of the models we use involve clear outcome chains between short, intermediate, and long range outcomes. Usually those long range outcomes are aspirational in two ways. 1) Nobody ever stays around long enough to actually evaluate whether the program in question had any impact. 2) It’s just as well that the effort was not made because at any reasonable time into the future, the methodology would fall apart. It would not be powerful enough to detect change.

I know this second statement is far from universally true, and that many very good long term evaluations have been done. Still, it seems true enough to me when I think of the ratio of the number of evaluations that stop at intermediate outcomes, and the ones that go on to measure long term effects.

With respect to the methodology falling apart over the long run, I’m beginning to think that the reason it falls apart is more than the obvious one of not having the resources and the will to stick it out. Rather, I am of a mind to think that the problem is not the methodology, but the program theory on which the methodology is based.

A look at an outcome chain model conveys the impression that whatever may be needed to evaluate long term outcomes is more of what was done before – more effort to maintain the integrity of control groups, better tactics for staying in touch with interviewees, and so on. I am beginning to question the “more of the same” approach. I am beginning to think that the real issue is that over time the program theory has changed.

My idea starts with an assertion that I know is not true but which may be true enough to be useful. Namely, that any long term impact is based on networking effects. By this I mean that whatever a program accomplishes, over time, those accomplishments form connections with other phenomena. Some of those phenomena may be other programs. Some may be change that was taking place anyway. My notion is that long term change is:

• not the additive consequence of all those change activities, but rather,
• an emergent phenomenon that cannot be explained in terms of the individual contributions of its parts.

That’s what I mean by a shift in program theory. The change is from:

• a model that is based on a theory of outcome chain relationships, to
• a model that abandons the outcome chain  in favor of change based on network activity and emergence.

This is an extension of the point I made elsewhere about how messy it gets because specific, well identified programs operate in a cluttered world (Figure 1). It goes beyond the Figure 1 story, however, because that story conveyed an message: If only we could identify all the other elements in the model,

we could ascertain the relationships and better interpret our evaluation data. Yes, I mentioned that some of it was unknowable, but I left the impression that if we could know it, we could do a complete evaluation of the program. I am now saying that is not true because whatever long term effects are occurring are not attributable to any of the specific parts of the system. The effects cannot be explained by the interaction of those parts. It’s not that the problem is technically difficult. It is that it is theoretically impossible. To invoke the car example, there is no cylinder whose contribution to the overall system can be explained. There is only an emergent outcome that stands on its own, independent of all its constituent parts.

Figure 2: Transition of Program Models

Figure 2 depicts the situation. Early in the programs life cycle, it may well be appropriate to use a traditional deterministic theory of change. As time goes on, deterministic relationships among program elements and outcomes disappear. At some point it becomes impossible to explain outcome in terms of deterministic relationships. The only way to do it is to explain outcome in terms of emergent, networked behavior.

Note the “/?” markings on the axes. I did this to reinforce the idea that Figure 2  is misleading in some profound ways. 1) It implies a smooth transition from traditional program theory to complex program theory. It is likely that there are no intermediate forms. 2) It assumes that complex behavior can be ignored early in the program’s life cycle. That may well not be true. 3) It implies that “degrees of complexity” can be quantified. That’s a murky idea, to say the least. 4) It may leave the impression that for any given time and degree/type of complexity, only one model exists. That is certainly not true. Models can always differ with respect to level of detail, aspects of the program and its outcomes that are emphasized, and many, many other characteristics. I talk about this in some detail in my article on project schedules as logic models. 5) It implies that the transition is inevitable. I don’t believe that, either.

That last point is important. I am not claiming that emergence in long term outcomes is always present, or even if it is, that it’s important enough to worry about. Sometimes the cylinder metaphor may be true. Nor am I claiming that emergence is due to interactions among every part of the system. I can imagine cases where some parts are relevant and some are not. In which case, the ability to identify those parts would be a great contribution of any evaluation. I am only claiming that evaluators would do well to consider the possibility that linear program logic, over time, yields to emergent logic.

Thanks to Mat Walton and Guy Sharrock for pointing out the error of my ways on previous drafts of this post.