What complexity theory do evaluators need to know?

My last blog post dealt with why evaluators should focus on complex behavior as opposed to complex systems. Bob Williams made a comment about how the post made a lot of sense, but that it conveyed the impression that evaluators do not have to worry about complexity theory. Evaluators do need to be concerned with theory, and Bob’s post got me to begin to crystallize some notions that have been marinating in the back of my brain for some time.

My Starting Point
Recently I have been pounding on the idea that a switch from complex systems to the behavior of complex systems would do a lot to further the abilities of evaluators to make practical, operational decisions about program theory, metrics, and methodology. And after all, that’s what it’s all about. We (I at least) get hired when someone says to me: Continue reading

Advertisements
Posted in Uncategorized | Leave a comment

Applying Complexity to Make Practical Decisions About Evaluation

Lately I have been speaking to as many audiences as I can about the need to focus on complex behavior rather than on complex systems. The reason is that there is no practical way to apply the notion of a “complex system” to practical decisions about program models, metrics, or methodology. But it is possible to make those decisions with respect to the things that complex systems do. I just completed a series of three short “coffee break” sessions on this topic for the American Evaluation Association.

Go here for the slides.

www.jamorell.com/documents/AEA_Coffee_Break_Part_1.pdf

www.jamorell.com/documents/AEA_Coffee_Break_Part_2.pdf

www.jamorell.com/documents/AEA_Coffee_Break_Part_3.pdf

If you are a member of AEA you can also hear the audio presentation. Go here for the audio tapes.

https://vimeo.com/269709240/e1b05b4857

https://vimeo.com/267297243/523c1a8c44

https://vimeo.com/265410410/8edd0dd3b7

Posted in Uncategorized | 3 Comments

A Plan for Making Complexity Useful in Evaluation

Recently a friend of mine asked about my understanding of what role complexity can play in Evaluation, and how I would further that role. Below is an edited version of what I sent her.

My goal for the role of complexity in Evaluation

Complexity as discussed in evaluation circles contains a great deal of information that is either wrong, or ill-chosen as elements of complexity that can be useful in Evaluation. Those discussions do not appreciate the broad and deep knowledge about complexity that has roots in many different scientific disciplines. Much (most really) of that knowledge is not applicable in the field of Evaluation, but some of it is. My goal is to influence the field to appreciate and use the knowledge that is applicable.

As I see it, the critical issue revolves around program theory, i.e. people’s beliefs about what consequences a program will have, and why it will have those consequences. The problem is not methodology because for the most part our standard arsenal of quantitative and qualitative tools is more than adequate.  The problem is that evaluators do not choose appropriate methodologies because their designs are customized to test incorrect program theories.

What is complexity?

Continue reading

Posted in Uncategorized | 1 Comment

Case Study Example: Drawing on Complexity to do Hands-on Evaluation

In 2016 I developed a case for a workshop I did at the Canadian Evaluation Society on the use of complexity in evaluation. I was doing some archeology and unearthed it. It offers pretty rich opportunity to think about complex behavior in evaluation, so I decided to share it.

Construction of the Case
This is the example we will use throughout this workshop to illustrate how knowledge of system behavior can be applied in evaluation. The example is hypothetical. I made it up to resemble a plausible evaluation scenario that we may face, but which is elaborated to make sure it contains all the elements needed to explain the topics in the workshop. I am sure that none of us (me included) have ever been involved in an evaluation that is as far reaching and in-depth as the example here. But I am sure that all of us have been involved in evaluations that are similar to parts of the example, and, if you are like me, I bet you have dreamed of being involved in an evaluation of the size and scope of the example.

There are three initiatives. One aimed at adults. One aimed at mothers and young children. One aimed at teens. Each initiative has several individual programs that share some common outcomes, and which also have some unique outcomes. Continue reading

Posted in Uncategorized | 1 Comment

Part 1 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: What is Diversity?

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The other is to maximize the likelihood that we can thrive as circumstances change. Diversity Continue reading

Posted in Uncategorized | Leave a comment

Part 2 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: AEA as an Evolving Organism

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The Continue reading

Posted in Uncategorized | Leave a comment

Part 3 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: Evolution, Diversity and Change from the Middle

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.)

Part 1: What do I mean by diversity?

Continue reading

Posted in Uncategorized | 2 Comments