Over the years I have written, lectured, and done workshops about how (and when) Evaluation should draw from Complexity Science. Over that time my beliefs have evolved with respect to what aspects of Complexity Science matter, and how evaluation methodologies can be applied to understand program activity and the consequences of program action. This post is a brief summary of my latest thinking. Activate the link for more details. It contains the slides I developed for a workshop at the 2022 meeting of the European Evaluation Society.
EES PDW Complexity Applied to Evaluation
What Constructs Matter the Most?
A glance at the glossary at Complexity Explorer makes it obvious that many possibilities exist for constructs in Complexity Science that might be useful in Evaluation. Of all of these, three stand out – emergence, sensitive dependence, and attractors.
Emergence and Sensitive Dependence
“Emergence” and “sensitive dependence” hold a special place with respect to how complexity should influence evaluation theory and practice because of their implications for two foundational beliefs that influence much evaluation practice: 1) Causal chains/networks can be built to explain how intermediate change leads to desired outcomes. 2) Outcomes can be explained in terms of changes in their constituent parts. Sensitive dependence implies that the first belief may not always be correct. Emergence implies that the second may not always be correct.
Attractors
Evaluation would benefit from a way to characterize regularity in the behavior of programs and outcomes. Leaving aside the value of any particular change, what does the past tell us about what that program or outcome will do in the future? It’s an important question because it speaks to both resistance to change and to sustainability of change.
The concept of a social attractor is useful for understanding how the contours of change in the past can inform knowledge of change in the future. As the Systems Innovation Network puts it: “Social attractors define a specific subset of states that a social system may take, which corresponds to its normal behavior towards which it will naturally gravitate.”
Besides the notion of an attractor, complexity offers many other concepts that contain some elements of attractor-like reasoning. Candidates include self-organization, autopoiesis, fitness landscape, equilibrium, homeostasis, and phase space. In my experience though, none of these captures an understanding of both change and stability that can be obtained by visualizing the shape of an attractor.
Applying Evaluation Methods to Complex Behavior
For the most part, the key to applying complexity is in data interpretation. Experimental and quasi-experimental design, content analysis, big data methods, observational techniques, case study designs – these and other methods that are well known in the evaluation community are all that are needed. Research design does come into play, but in a way that traditionally trained evaluators can easily manage. For instance, because systems evolve over time, special emphasis may be needed on longitudinal observations. Outcome measures are also affected, but again, in a way that traditionally trained evaluators can easily manage. For instance, if an important outcome is emergent, it would be necessary to develop an indicator that is not attached to indicators of the outcome’s constituent parts.
Evaluation Uncertainty 
This is a great synthesis. But it is very abstract. Case studies based on real life applications would help demonstrate that these ideas are actually relevant to evaluation practice.