What does evaluation gain from thinking in alien terms? An argument for taking complexity and evolutionary biology seriously

For a long time, I have been arguing that if “complexity” is to be useful in evaluation, evaluators’ should focus on what complex systems do, rather than on what complex systems are. This is because by focusing on behavior, we can make practical decisions about models, methodologies, and metrics.

I still believe this, but I’m also coming to appreciate that thinking within research traditions also matters. I’m not advocating a return to a “complex system” focus, but I do see value in adopting the perspectives of people who do research and develop theory in the domain of complexity. And by extension, this is also true for evolutionary biology, another field that I have been promoting as being useful for evaluators.

Aren’t familiar frameworks adequate?
Why bother with alien concepts from other fields? After all, technical terms, frameworks, theories, and methodologies in one discipline are often reflected in the terms, frameworks, theories, and methodologies of other disciplines. It’s easy for someone in one discipline to listen to someone in another discipline and think: “The words are different, but I could have done that. Why not stick to what I know I can do well?” This is certainly true of complexity and evolutionary biology. One can’t read much in those fields without encountering topics that echo much that is familiar to us. As examples, evaluators are well versed in addressing questions such as:

• How does the program change over time?
• What unexpected consequences might arise?
• What are the boundaries of the system I am dealing with?
• Is the program I am evaluating robust, or might it fall apart easily?
• What affects whether a program can be effective in other settings?
• What explains whether a program will continue to function over time?
• What have been the fates of other programs that have similar characteristics?
• What are the essential parts of the program, and which can vary without affecting outcome?
• How do parts of the system influence each other, and how does that influence change over time?
• Are there other programs nearby that may compete for resources or pursue overlapping outcomes?
• How does context and setting affect what consequences the program may have, and how those consequences come about?

So, if we can think of these questions and answer them, why take a step into unfamiliar, and perhaps uncomfortable territory? One answer is that we should not, and usually, that would be a correct conclusion. But for two considerations: 1) paradigmatic constructs, and 2) methodology.

Constructs
A research tradition has interacting, networked “parts”, e.g. preferred methods of:

• interpreting data
• developing models
• defining data needs
• generating hypotheses
• choosing methodologies
• assembling research teams
• identifying topics to research
• specifying acceptable answers
• constructing convincing arguments

Each of these has meaning in its own right, but together they result in an emergent way of looking at the world that cannot be explained in terms of any of its parts.

Do a thought experiment. Would applying a complexity or an evolutionary biology lens to an evaluation lead to an evaluation that would significantly differ from one that was embedded in our traditional approaches? I’d bet that often the answer would by “yes, it would be different”. Moreover, it would not be different because of any specific element in the list above. It would be different because of all of them interacting with each other, in a networked fashion.

Methodology
Disciplines favor different methodologies, and there are times when one discipline would do well to import a methodology from another. For instance, a major theme in research in evolutionary biology and ecology deals with rates of change of different parameter values over time. Some examples: 1) predator/prey relationships, 2) species extinction and birth, 3) adaptability to new environmental circumstances, 4) appearance of mutations, 5) measures of diversity, and 6) of change in an environment.

Evolutionary biology and ecology have developed theories, backed by mathematical formulations, to articulate relevant hypotheses about these kinds of changes. These approaches have been borrowed to great effect by social scientists. As an example, theories of species birth and extinction have been applied to types of organizations (Organizational Ecology Hannan and Freeman). It’s not hard to imagine relevant questions in evaluation. Examples that spring to mind include sustainability, the spread of innovative programs, the consequences of changing social or cultural conditions, the importance of specific program outcomes, and the rate at which programs change.