Introduction
I’m working on the notion that there are circumstances when evaluators should think of programs as species of organisms adapting in an ecological niche. This document contains some preliminary thoughts on that topic. I’m groping toward an article, a series of blog posts, and some YouTube movies. I’m looking for any suggestions anyone might have to help me along.
Inapplicability to Evaluation
One thing I need to be careful about is that evolution is agnostic as to the outcome, it only cares about species viability. We care about goals.
Evolution does not mean “progress” in the sense that we humans think of making life better for people. It’s not hard to imagine a dystopian, but highly sustainable, evolutionarily successful world. (In general, people talk about “sustainability” as if it is an unalloyed good. It’s not. It is neutral with respect to being “desirable” or “undesirable” concerning desired ends. In my business the problem is that systems are too sustainable. You can beat them over the head with data until the cows come home, and still they do not change.)
When is an evolutionary biological perspective needed?
People get turned on when I give my complexity workshops and think that they have to apply principles of complex behavior in everything they do. Hooey. It’s one thing to say there is complex behavior operating. It’s quite something else to say that one has to go to the trouble of dealing with it. There is a large and legitimate need for evaluation of single programs with respect to first-order outcomes. No overwhelming need to deal with complexity there. Ditto evolutionary biology.
Toward the end of Superforecasting: The Art and Science of Prediction, Tetlock has a nice discussion of when incremental forecasting is useful, given the ubiquity of log-linearly distributed rare occurrences that can change the course of events. (See Rumsfeld memo to Bush, Cheney, and Rice.) There is an analogous argument to make about evaluation. If it’s true for evaluation in general, it’s certainly the case for using knowledge of evolutionary biology to shape an evaluation. (Actually I can make a good case that we should only evaluate short-term, proximate outcomes. But that’s another story.)
Also, evolution does not care about whether an organism lives of dies. It cares about whether a species thrives of goes extinct. So applying evolutionary biology to evaluation is only appropriate when that which is being evaluated is a class of programs. It’s the viability of the class that matters, not the individual programs within the class.
How can an evolutionary biological perspective be used in evaluation?
An evolutionary biology perspective can be used in a few different ways.
- Technical, as the authors do in “Organizational Ecology”. They actually apply Lotka-Volterra equations to the birth and death of types of organizations. Using more familiar methodologies, I can see evaluators doing things like estimating how quickly a program’s environment is changing, or the diversity of similar programs in the same ecological niche.
- As a vocabulary and a set of constructs that can help developing models, devising methodologies, and interpreting data. Some examples: 1) Fitness landscape: If a set of programs begin to evolve in a particular direction, what are the consequences of small changes for the fitness of that set of programs?
2) Co-evolution of species and environment: In the U.S. at least, Uber is a great example. It could only exist because the environment was conducive – IT infrastructure, GPS, weaknesses in current taxi services, availability of venture and human capital, and so on. But once the “species” began to thrive, the environment had to adapt to it, e.g. rules about traffic congestion, public conveyance regulations in various cities, dedicated waiting spaces at airports, and so on. Depending on the nature and direction of the adaptation, the species may or may not thrive. (It’s an open question. Uber is losing heaps and gobs of money.)
Birth and extinction
If evaluation is going to take an evolutionary biology perspective, it has to take the concept of species birth and extinction seriously. We care about spurring innovation and stifling ineffective programs.
One very juicy example is the advent of the mail order business, as invented by Sears. This required the establishment of Rural Free Delivery in the late 19th century, high local prices, and a market pull for a broad range of goods. It had a truly profound effect on bringing a variety of goods to a large percentage of the population at lower prices, allowing African Americans to buy and get credit for purchases that were not available locally, and badly affected the income of local merchants, some of whom sponsored book burnings of the Sears catalogue.
My problem is that I cannot think of as good an example for the type of stuff that evaluators would evaluate. It’s easy enough to think of examples, but not big, interesting ones. For instance, there are STEM programs that did not exist before Sputnik and did not exist for girls until 20 or so years ago. There were always private schools in the US, but not charter schools in their present incarnation. How long ago was it that there were no programs in environmental education, or climate change mitigation efforts? In terms of extinction, think of big state mental hospitals in the US, and specialized hospital wards for AIDS patients.
Links with other types of evaluation
I would do well to make the case that an evolutionary biological perspective has ties to other trends in evaluation. I can think of three: complexity, developmental evaluation, and sustainability. I have the first one pretty well worked out. Not so much the other two.
Examples I’m looking for
I’m looking for examples that make the transition from evaluating a program to evaluating a group of similar programs, which is what an evolutionary perspective would require. My difficulty is finding an example that evaluators would recognize as something they might get paid to do.
As of now I’m pondering two possibilities.
- Sustainability (See NDE Summer 2019.)
- Telemedicine/telehealth. This has lots of elements I can use. Ancestors (back to plain old telephone service), rapid evolution, adaptation to changing environment (costs of health care, docs leaving rural areas, etc.), co-evolution as the innovation affects is environment, “species” nested in “genius” (e.g. maternal health and surgical consulting), competition, and much else besides.
To build on the example of telemedicine, someone might get paid to evaluate a telehealth counseling program for nursing mothers in Australia, i.e. a program that had an identifiable source of funding coming from some small corner of the Ministry of Health. But getting paid to evaluate the overall consequences of having a telehealth infrastructure and set of services in the country? A nice piece of social science research to be sure, but I’m not sure how many of our brethren would see it as an “evaluation”. I have a feeling that Foundations might do this at a program level, but I’m not sure.
Evaluation Uncertainty 