Webinar slides and recording for “Using Ecology to Evaluate Policy”

I recently presented a webinar to SAMEA. Slides and a recording of the webinar can be found here.

Webinar description
Is it worth our attention to look to Ecology and Evolutionary Biology for help in developing program theory, choosing methodology, and interpreting data? It is when we evaluate policy because policy change affects groups of connected programs and program environments, as those programs and environments evolve over time. As with all disciplines, Ecology and Evolutionary Biology offer unique ways to identify topics to research, develop models, generate hypotheses, choose among methodological tools, define data needs, specify acceptable answers, and interpret data. Each of these discrete elements has its own utility. But perhaps more important, each element is a node in a network: decisions about each have implications for the others. Under these conditions, a unique analytical mindset emerges. The potential contribution of ecological and evolutionary biological thinking to Evaluation lies both in that mindset, and in the utility of discrete tools and concepts.

Using the Biotic Hierarchy as a Framework for Moving Policy Evaluation Away from a Program Evaluation-like Activity

PDF version Biotic_Hierarchy_as_a_Framework_for_Policy_Evaluation

In a previous blog post and in one of my You Tube videos I tried to make the case that different disciplines orient problem definition and solution in different ways, and that evolutionary biology and ecology lead researchers in different directions than do the social sciences. I followed this line of reasoning with the argument that evaluators’ traditional grounding in social science normally has been productive and that for the most part, we should continue to do what we have always done. Then, I tried to make the point that there are circumstances where thinking in terms of evolutionary biology and ecology can be a powerful addition to how we normally do our work. In this blog post I will show how policy evaluation is one of those circumstances when evolutionary / ecological thinking can be valuable. Continue reading “Using the Biotic Hierarchy as a Framework for Moving Policy Evaluation Away from a Program Evaluation-like Activity”

Evolutionary Biology and Ecology as a Valuable Framework for Some Evaluation

I put together a slide deck on the advantages of adding constructs from evolutionary biology and ecology to traditional evaluation. Adding those constructs is worth the trouble when the focus is on:

  • populations of a program types, and
  • rates of change over time

Specific topics covered are:

  • Policy as ecosystem change
  • Evolution of program forms
  • Sustainability in terms of ecosystems as attractors
  • Use of ecosystem analysis to interpret outcome evaluation
  • Using population trends to interpret data on program outcome

Go here for the slides. UD_EBT_Evaluation_Meeting

What does evaluation gain from thinking in alien terms? An argument for taking complexity and evolutionary biology seriously

For a long time, I have been arguing that if “complexity” is to be useful in evaluation, evaluators’ should focus on what complex systems do, rather than on what complex systems are. This is because by focusing on behavior, we can make practical decisions about models, methodologies, and metrics.

I still believe this, but I’m also coming to appreciate that thinking within research traditions also matters. I’m not advocating a return to a “complex system” focus, but I do see value in adopting the perspectives of people who do research and develop theory in the domain of complexity. And by extension, this is also true for evolutionary biology, another field that I have been promoting as being useful for evaluators.

Continue reading “What does evaluation gain from thinking in alien terms? An argument for taking complexity and evolutionary biology seriously”

Can Knowledge of Evolutionary Biology and Ecology Inform Evaluation?

I posted a longish piece (~3,500 words) on my website with the same title as this post. Section headings are:

  • Case: Early childhood parent support
    • Program design
    • Evaluation design
  • Some useful concepts from evolutionary biology and ecology
    • Population
    • Coevolution
    • Birth/death rates
    • Selection pressure
    • Species and species variation
  • What would the evaluation look like if its design were informed by knowledge of evolutionary biology and ecology?
    • Populations, and birth/death rates
    • Coevolution and population
    • Selection pressure
    • Species and species variation
  • Do we gain anything from applying an evolutionary lens?
    • Paradigmatic concepts
    • Methodology

I’m always interested in having people point out the error of my ways.

Preliminary Notes on The Application of Concepts from Evolutionary Biology and Ecology to Evaluation

I’m working on the notion that there are circumstances when evaluators should think of programs as species of organisms adapting in an ecological niche. This document contains some preliminary thoughts on that topic. I’m groping toward an article, a series of blog posts, and some YouTube movies. I’m looking for any suggestions anyone might have to help me along.

Inapplicability to Evaluation
One thing I need to be careful about is that evolution is agnostic as to the outcome, it only cares about species viability. We care about goals.

Evolution does not mean “progress” in the sense that we humans think of making life better for people. It’s not hard to imagine a dystopian, but highly sustainable, evolutionarily successful world. (In general, people talk about “sustainability” as if it is an unalloyed good. It’s not. It is neutral with respect to being “desirable” or “undesirable” concerning desired ends. In my business the problem is that systems are too sustainable. You can beat them over the head with data until the cows come home, and still they do not change.)

When is an evolutionary biological perspective needed?
People get turned on when I give my complexity workshops and think that they have to apply principles of complex behavior in everything they do. Hooey. It’s one thing to say there is complex behavior operating. It’s quite something else to say that one has to go to the trouble of dealing with it. There is a large and legitimate need for evaluation of single programs with respect to first-order outcomes. No overwhelming need to deal with complexity there. Ditto evolutionary biology.

Toward the end of Superforecasting: The Art and Science of Prediction, Tetlock  has a nice discussion of when incremental forecasting is useful, given the ubiquity of log-linearly distributed rare occurrences that can change the course of events. (See Rumsfeld memo to Bush, Cheney, and Rice.) There is an analogous argument to make about evaluation. If it’s true for evaluation in general, it’s certainly the case for using knowledge of evolutionary biology to shape an evaluation. (Actually I can make a good case that we should only evaluate short-term, proximate outcomes. But that’s another story.)

Also, evolution does not care about whether an organism lives of dies. It cares about whether a species thrives of goes extinct. So applying evolutionary biology to evaluation is only appropriate when that which is being evaluated is a class of programs. It’s the viability of the class that matters, not the individual programs within the class.

How can an evolutionary biological perspective be used in evaluation?
An evolutionary biology perspective can be used in a few different ways.

  • Technical, as the authors do in “Organizational Ecology”. They actually apply Lotka-Volterra equations to the birth and death of types of organizations. Using more familiar methodologies, I can see evaluators doing things like estimating how quickly a program’s environment is changing, or the diversity of similar programs in the same ecological niche.
  • As a vocabulary and a set of constructs that can help developing models, devising methodologies, and interpreting data. Some examples: 1) Fitness landscape: If a set of programs begin to evolve in a particular direction, what are the consequences of small changes for the fitness of that set of programs?

2) Co-evolution of species and environment: In the U.S. at least, Uber is a great example. It could only exist because the environment was conducive – IT infrastructure, GPS, weaknesses in current taxi services, availability of venture and human capital, and so on. But once the “species” began to thrive, the environment had to adapt to it, e.g. rules about traffic congestion, public conveyance regulations in various cities, dedicated waiting spaces at airports, and so on. Depending on the nature and direction of the adaptation, the species may or may not thrive. (It’s an open question. Uber is losing heaps and gobs of money.)

Birth and extinction
If evaluation is going to take an evolutionary biology perspective, it has to take the concept of species birth and extinction seriously. We care about spurring innovation and stifling ineffective programs.

One very juicy example is the advent of the mail order business, as invented by Sears. This required the establishment of Rural Free Delivery in the late 19th century, high local prices, and a market pull for a broad range of goods. It had a truly profound effect on bringing a variety of goods to a large percentage of the population at lower prices, allowing African Americans to buy and get credit for purchases that were not available locally, and badly affected the income of local merchants, some of whom sponsored book burnings of the Sears catalogue.

My problem is that I cannot think of as good an example for the type of stuff that evaluators would evaluate. It’s easy enough to think of examples, but not big, interesting ones. For instance, there are STEM programs that did not exist before Sputnik and did not exist for girls until 20 or so years ago. There were always private schools in the US, but not charter schools in their present incarnation. How long ago was it that there were no programs in environmental education, or climate change mitigation efforts? In terms of extinction, think of big state mental hospitals in the US, and specialized hospital wards for AIDS patients.

Links with other types of evaluation
I would do well to make the case that an evolutionary biological perspective has ties to other trends in evaluation. I can think of three: complexity, developmental evaluation, and sustainability. I have the first one pretty well worked out. Not so much the other two.

Examples I’m looking for
I’m looking for examples that make the transition from evaluating a program to evaluating a group of similar programs, which is what an evolutionary perspective would require. My difficulty is finding an example that evaluators would recognize as something they might get paid to do.

As of now I’m pondering two possibilities.

  • Sustainability (See NDE Summer 2019.)
  • Telemedicine/telehealth. This has lots of elements I can use. Ancestors (back to plain old telephone service), rapid evolution, adaptation to changing environment (costs of health care, docs leaving rural areas, etc.), co-evolution as the innovation affects is environment, “species” nested in “genius” (e.g. maternal health and surgical consulting), competition, and much else besides.

To build on the example of telemedicine, someone might get paid to evaluate a telehealth counseling program for nursing mothers in Australia, i.e. a program that had an identifiable source of funding coming from some small corner of the Ministry of Health. But getting paid to evaluate the overall consequences of having a telehealth infrastructure and set of services in the country? A nice piece of social science research to be sure, but I’m not sure how many of our brethren would see it as an “evaluation”. I have a feeling that Foundations might do this at a program level, but I’m not sure.