Complexity as a Trend in Evaluation: Similarities and Differences with Classical Statistics

Evaluators have always been concerned about the weakness of their efforts to drive more effective programs, more desirable outcomes, and fewer unintended negative consequences. Broadly speaking, these concerns fall into three categories.

  • evaluation methodology,
  • “evaluation use” e., the dynamics by which evaluation works its way into decision making, and
  • limits on what evaluation can say about why programs operate as they do and what consequences they have.

 Complexity as a Trend in Evaluation and Planning

In recent years, the evaluation community has been looking to “complexity” as a source for addressing these difficulties. Continue reading “Complexity as a Trend in Evaluation: Similarities and Differences with Classical Statistics”

A Complex System View of Technology Acquisition Choice

I am involved in a project that involves helping people make a single choice among multiple technologies. They must commit to one, so there is no waffling. This is one more of many such exercises that I have been involved in over the course of my career, and I have never been fully satisfied with any of them. On an intuitive level, everyone knows they cannot make the best choice, but everyone thinks that they should be able to. I finally figured out why they cannot. I don’t mean that people are not smart enough. I mean that it is impossible. The behavior of complex systems makes it impossible.

A Workable, Effective Solution
If there is a technology choice with a very few criteria, and it is absolutely clear what criterion is truly critical, and there is good data on performance, then yes,  it is possible to make the best choice. But how many situations like that are there? So, what to do in the majority of all the other cases?

Before I get into a longish esoteric discussion, I’ll jump to a simple, practical method for making a technology choice. The answer is that we accept the reality of how human beings make decisions. We satisfice within a context of bounded rationality. As Herbert Simon put it, “decision makers can satisfice either by finding optimum solutions for a simplified world, or by finding satisfactory solutions for a more realistic world”.

With respect to technology choice, satisficing dictates two decision making strategies which can be used alone or combined.

  • Find a few acceptable technology choices and pick the one you are most comfortable with.
  • Aggregate the requirements into broad enough categories and accept the imprecision that such aggregation requires.

And now for my explanation of why this simple solution is more than an efficient convenience, but a necessity. Of course, there is too much going on in the world for our humble intellect to find and understand. But it is more than the volume of information and our limited capacities. It is how that information is structured.

What are the System-based Reasons why an Optimal Choice Cannot be Made?
To begin, I need to define what I mean by “best choice”. I mean it in a technical optimization sense, where there is a true joint optimization of all relevant criteria. “Best” can also have a social psychological meaning, i.e., a situation where most interested parties are as satisfied as they can be with the collective choice that was made. But although I am a social psychologist, I’ll stick to a definition of “best” that is near and dear to the hearts of my engineer friends.

Choice Criteria are Networked
Why can’t a best choice be made? The answer is that choice criteria are networked and that the nodes of the network are subject to environmental influences. The result is sensitive dependence and emergent behavior. To illustrate with an example, see Table 1. It contains a list of choice criteria that I adapted from a project I’m working on.

Table 1: Technology Choice Criteria
High level Detailed
Signal detection capability 1.   Data analysis capability
2.   Number of signal types detected
3.   Signal resolution for each data type
Human Factors

 

4.   Usability for operator
5.   Training requirements
6.   Visual presentation quality of output
Interoperability 7.   Data export formats
8.   Data import formats
Operating environment 9.     Time of day
10.   Temperature
11.   Weather conditions
Market 12.   Competing technologies
13.   Market demand
14.   Initial cost
15.   Life cycle cost
16.   Compatibility with technology trends
17.   Synergy with other technologies in places where implemented.

Pull just four elements from the list (see the picture): 1) number of signal types detectable, 2) weather,  3) cost, and 4) training requirements. A wider range of detection needs, the ability to work in bad weather, and low training requirements will all increase cost. The ability to work in adverse weather conditions may affect the types of signal detection that can be used. The greater the diversity of information, the greater the training requirements. What would happen to all the tangled dependencies if new hiring drove up the burden on training, or if a need for higher resolution imaging asserted itself, or if requirements for operation in adverse weather conditions were relaxed? Scale this up to dependencies among the seventeen choice criteria, and even a casual look at the dependencies makes it obvious why a strict ordering of criteria is impossible.

Network Behavior
Node relationships in networks are prone to sensitive dependence. This means that local differences in any one node, (or in a small number of nodes) might ripple through the system and affect relationships among many of the nodes. And the nature of those large-scale changes may be different as a function of different local changes. Moreover, networks can be adaptive in the sense that as influence is transmitted across edges, node and edge relationships can rearrange. I am not claiming that sensitive dependence or network adaptivity will always be at play in networks, only that they often are. Given what I know about interactions among technology requirements, it’s hard for me to believe that they are not at play in networked technology choice requirements.

There is yet another network phenomenon that I am convinced makes a strict ordering of criteria impossible, but which I won’t push too hard because I can’t make a strong case for it. I suspect that the “best” technology choice is not an additive function of its component requirements. Rather, “best” is an emergent characteristic of network behavior. Or put differently, each requirement loses its unique identity.

Influences on Network Nodes
If local change in a network of choice criteria can have such profound effects, how certain can we be that those kinds of changes will occur? Very certain. Consider just a few of the endless possibilities that may affect one or a few choice criteria.

  • Technology costs may rise or fall with market conditions.
  • A competing technology standard may become ascendant.
  • Funds for technology acquisition may increase or decrease.
  • The importance of the reasons for the technology choice may change.
  • Domains (location, business conditions, etc.) where the technology is desirable may narrow or broaden.
  • Choices are based on the best knowledge one has at the time about each relevant criterion. But the discovery of more extensive, or more accurate, knowledge is always a
  • And many, many more.

Summary
In the first section of this blog post I made the observation that people have an intuitive appreciation for the difficulty of making an optimal choice among competing technology acquisition candidates. In the second section I provided a complex system, network-based justification for this intuitive appreciation. I laid all this out to make the point that it is not difficult to make an optimal choice, it is impossible, and that therefore choices need to be made via a process of satisficing rather than optimizing. With respect to technology choice, satisficing dictates two decision making strategies that can be used alone or combined.

  • Find a few acceptable technology choices and pick the one you are most comfortable with.
  • Aggregate the requirements into broad enough categories and accept the imprecision that such aggregation requires.

 

 

 

 

 

 

A Complexity-based Meta-theory of Action for Transformation to a Green Energy Future

This is the abstract for a book chapter I am writing on the evaluation of transformation. It is a draft and will undoubtedly change quite a bit by the time it is published.

This chapter draws from complexity science to present a metatheory of transformation that can be applied to discrete theories of change that are constructed to guide model building, methodology, and data interpretation for the evaluation of individual change efforts. The focus is on six specific behaviors of complex systems – stigmergy, attractors, emergence, phase transition, self-organization, and path dependence. These can be  invoked singly or in combination to understand pattern, predictability, and how change happens. The importance of both “explanation” and “prediction” is woven into the discussion. A definition of “transformation” is offered in which a qualitatively new reality becomes the default choice that constitute a new normal. Indicators of transformation include measurable ranges (as opposed to specific values) for level of energy  use and the time over which the change endures. Because complex systems behave as they do, the recommended theory of change is sparse – it has few well-defined elements or relationships among those elements. There is already good progress in the application of complexity to the evaluation of transformation. An argument is made that these efforts should be strengthened by deliberately incorporating what is known about complex system behavior, and that by so doing, both prediction and explanation would better serve the purpose of practical decision making.

Here is the entire chapter.

Metatheory of transformation 05_15_2020