Common Introduction to all sections

This is part 10 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.

I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.

Part Title Post status
1 Complex systems or complex behavior? up
2 Complexity has awkward implications for program designers and evaluators up
3 Ignoring complexity can make sense up
4 Complex behavior can be evaluated using comfortable, familiar methodologies up
5 A pitch for sparse models up
6 Joint optimization of unrelated outcomes up
7 Why should evaluators care about emergence? up
8 Why might it be useful to think of programs and their outcomes in terms of attractors? up
9 A few very successful programs, or many, connected, somewhat successful programs? up
10 Evaluating for complexity when programs are not designed that way up

Evaluating for complexity when programs are not designed that way

There are good reasons to design programs with complex behavior in mind, and good reasons not to. (For the reasons not to, see Part 3 Ignoring complexity can make sense, which makes a case for the rationality of letting the sleeping complexity dog sleep.)

The fact that programs are not designed in ways that recognize complexity does not mean that evaluation should ignore complexity. My reasoning is that even if programs are not designed with complex behavior in mind, knowing about complex behavior can still be useful to stakeholders. Figure 1 illustrates what I have in mind.

Figure 1: Overlay of Complex Model on a non-Complex Program Design

Blue region of model
Blue represents the original program model. It has much in it that is oblivious to complex behavior (and to common sense, for that matter.)

Green region of model
Green represents a program model that recognizes some of the complex behaviors that may be operating. To make my point, I superimposed it on the original model

  • Network effects are included.
  • Undesirable consequences are acknowledged.
  • Data are collected and analyzed with respect to groups of outcomes, without regard to any unique outcome within the group.
  • The social implications of distribution shapes are considered over and above the technical aspects of doing statistical analysis.

If I had my choice, I’d evaluate with respect to the green model exclusively, but I acknowledge that stakeholders may need more fine-grained information. In any case, as you know by now, I am a big supporter of using more than one model in any single evaluation.  All models are wrong, but many different models can be both wrong and useful in different ways.

One thought on “Evaluating for complexity when programs are not designed that way Part 10 of a 10-part series on how complexity can produce better insight on what programs do, and why

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s