Evaluating for complexity when programs are not designed that way Part 10 of a 10-part series on how complexity can produce better insight on what programs do, and why

Common Introduction to all sections

This is part 10 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.

I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.

Evaluating for complexity when programs are not designed that way

There are good reasons to design programs with complex behavior in mind, and good reasons not to. (For the reasons not to, see Part 3 Ignoring complexity can make sense, which makes a case for the rationality of letting the sleeping complexity dog sleep.)

The fact that programs are not designed in ways that recognize complexity does not mean that evaluation should ignore complexity. My reasoning is that even if programs are not designed with complex behavior in mind, knowing about complex behavior can still be useful to stakeholders. Figure 1 illustrates what I have in mind.

Blue region of model
Blue represents the original program model. It has much in it that is oblivious to complex behavior (and to common sense, for that matter.)

• It is specified to an excruciating level of detail. (Part 5 A pitch for sparse models, describes why this is problematic.
• It does not recognize the possibility of unexpected negative consequences. (For an explanation of why I think that unexpected outcomes are likely to be undesirable, see the section “Why are Unintended Consequences Likely to be Undesirable?” in From Firefighting to Systematic Action: Toward A Research Agenda for Better Evaluation of Unintended Consequences. For information on how to evaluate unintended consequences, see Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable.)
• It does not recognize that groups of outcomes, or the program as a whole, can have outcomes that cannot be explained in terms of the sum of its parts (Part 7: Why should evaluators care about emergence?)
• It does not recognize network behavior within or across outcomes (Part 9 A few very successful programs, or many, connected, somewhat successful programs?).
• It does not recognize the different distributions that outcomes may have, and the consequences of those distributions. (I did not right a blog post on this, but log-linear distributions are characteristic of much complex behavior. One consequence of this is that an astoundingly successful program may provide most of its benefits to a few service recipients.)

Green region of model
Green represents a program model that recognizes some of the complex behaviors that may be operating. To make my point, I superimposed it on the original model

• Network effects are included.
• Undesirable consequences are acknowledged.
• Data are collected and analyzed with respect to groups of outcomes, without regard to any unique outcome within the group.
• The social implications of distribution shapes are considered over and above the technical aspects of doing statistical analysis.

If I had my choice, I’d evaluate with respect to the green model exclusively, but I acknowledge that stakeholders may need more fine-grained information. In any case, as you know by now, I am a big supporter of using more than one model in any single evaluation.  All models are wrong, but many different models can be both wrong and useful in different ways.