How Might Complexity Science Inform the Design and Conduct of Evaluation?

 I’m considering pitching a special  issue to an evaluation journal titled: How Might Complexity Science Inform the Design and Conduct of Evaluation? I have not committed to this project yet, but I do want to get the idea out there. I’m looking for: 1)  critique of this idea, 2) specific topics to cover, 3) friendly journals, and 4) suggestions for authors.

The Role of Complexity Science in Evaluation
This volume is based on the belief that:

• The programs that we evaluate, and their outcomes frequently exhibit complex behavior.
• Complexity Science (as defined by the many intellectual streams that flow into this overall concept) has an extensive literature comprising both theory and research.
• “Complexity” is all the rage in evaluation circles, but except for a few centers of activity, too many efforts to apply complexity either misrepresent what is known about complex behavior or fail to exploit the power of what is known.
• Unless complexity proves itself in our field, it will be relegated to a passing fad. That would be a shame because appropriate use of complexity would make for better understanding of how programs operate and what they do.

Statistics as an Illustrative Example
Comparison with statistics is instructive. How much statistical knowledge does an evaluator need to do a humble “t” test? The answer falls somewhere on a knowledge continuum.

• A single knowledge nugget: P<.05 probably means the two groups are different.
• Much else. Examples include the nature of true score and error, statistical power, other analysis choices (e.g. non-parametircs), the historical roots of the .05 criterion, post-hoc testing, and the central limit theorem.

So how much statistics does an evaluator need? The only answer is: “Don’t cleave to the simple end of the continuum, and all else being equal, more is better. Why is more better? There are practical and epistemological aspects to the answer.

Practical considerations: Too little knowledge will lead to misinterpretation of data. The patterns we report may be different from the patterns that exist. The causal statements we claim may be false.

Epistemological considerations: Too little knowledge will obscure the understanding that statistical thinking provides. For instance, classical statistics are based  on the assumption that individual data points do not matter. What matters are group means, standard deviations, and distribution characteristics. Statistics also assume the legitimacy of understanding causation in terms of probabilities.

• Is there anything wrong with these beliefs? No.
• Do these beliefs make for worthwhile understanding about how the world works? Yes.
• Is it OK for evaluators to go about their work ignorant of these characteristics of statistical thinking? In the short run, yes as long as the evaluator does not bow before the alter of P<.05. But over the long run, missing the consequences of statistical thinking will distort the collective understandings that evolve from those analyses.
  

Comparing the Example  of Statistics to Complexity
So it is with how complexity is treated in the field of Evaluation. In the short term the way we use complexity now is fine. That use contributes to how we develop models, how we interpret data, and how we talk to our customers.

But over time, our present treatment of complexity will distort our collective understanding of what a “problem” is, what an effective intervention might be, and how to assess the consequences of our actions. To avoid this fate, I want to organize a special issue in which each article will apply complexity science to an evaluation. I have three objectives for this issue.

• Convey a sense that there is a field of Complexity Science, with its own bodies of research and theory.
• Give evaluators entry points to that field.
• Nudge evaluators a small bit toward more rigorous use of complexity in the work they do.

Structure of the Volume
Each article will consist of four parts.

Complexity Concept
The author/s will present at least one concept that has been the subject of research and theorizing in the field of Complexity Science. Text in this section will explain the concept/s, overview their intellectual development, and provide references.

Evaluation Case
This section will present an evaluation scenario that is amenable to the application of the complexity concept/s. These cases to be “synthetic” in the sense that they will have strong elements of actual evaluations but will be modified to be amenable to a rich discussion of the implications of complexity. Use of synthetic examples is important because purely “real” cases frequently do not have all the characteristics needed to fully explore everything the author might want to discuss. The criterion for an acceptable case will be “verisimilitude” to the workaday world of evaluation. The reader must be able to read the case and remark to him or herself: “I can see this as the kind of scenario I might be involved in”; or “I have colleagues who do evaluation like that”.

Application of Complexity to the Case
This section will discuss what the evaluation of the case might look like if the named complexity concepts were applied. Authors will be free to focus on whatever parts of the evaluation lifecycle are useful for making their case. Across all articles, the entire evaluation lifecycle will be represented, from initial discussions with clients, to the completion of a report and final briefings. My hope is that readers will benefit from realizing that complexity can have different implications for various aspects of evaluation work.

Comparison Between Complexity and non-Complexity Scenarios
This section will be based on the assertion that there is no metric to determine whether using, or not using, complexity makes for a better evaluation. The only assertion is that they are different and that each has advantages and disadvantages. The purpose if this section will be to explore those advantages and disadvantages.