A Complex System Perspective on Program Scale-up and Replication

I’m in the process of working up a presentation for the upcoming conference of the American Evaluation Association:. Successful Scale-up Of Promising Pilots: Challenges, Strategies, and Measurement Considerations. (It will be a great panel. You should attend if you can.) This is the abstract for my presentation:

Title: Complex System Behavior as a Lens to Understand Program Change Across Scale, Place, and Time
Abstract: Development programs are bedeviled by the challenge of transferability. Whether from a small scale test to widespread use, or across geography, or over time, programs do not work out as planned. They may have different consequences than we expected. They may have larger or smaller impacts than we hoped for. They may morph into programs we only dimly recognize. They may not be implemented at all. The changes often seem random, and indeed, in some sense they are. But coexisting with the randomness, a complex system perspective shows us the sense, the reason, the rationality in the unexpected changes. By thinking in terms of complex system behavior we can attain a different understanding of what it means to explain, or perhaps, sometimes to predict, the mysteries of transferability. That understanding will help us choose methodologies and interpret data. It will also give us new insight on program theory.

There will only be one slide in this presentation.


Based on this slide I’m developing talking points. I know I’ll have to abbreviate it at the presentation, but I do want a coherent story to work from. A rough draft is below. Comments appreciated. Whack away. Continue reading

Posted in Uncategorized | 3 Comments

Case Study Example for Workshop 18: Systems as Program Theory and as Methodology


Case Study Example for Workshop 18: Systems as Program Theory and as Methodology: A Hands on Approach over the Evaluation Life Cycle

This case was developed for a workshop at the American Evaluation Association’s 2015 Summer Evaluation Institute.

Construction of the Case
This is the example we will use throughout this workshop to illustrate how knowledge of system behavior can be applied in evaluation. The example is hypothetical. I made it up to resemble a plausible evaluation scenario that we may face, but which is elaborated to make sure it contains all the elements needed to explain the topics in the workshop. I am sure that none of us (me included) have ever been involved in an evaluation that is as far reaching and in-depth as the example here. But I am sure that all of us have been involved in evaluations that are similar to parts of the example, and, if you are like me, I bet you have dreamed of being involved in an evaluation of the size and scope of the example.

There are three initiatives. One aimed at adults. One aimed at mothers and young children. One aimed at teens. Each initiative has several individual programs that share some common outcomes, and which also have some unique outcomes.

All three initiatives are deliberately implemented Continue reading

Posted in Uncategorized | Leave a comment

Timelines, Critical Incidents and Systems: A Nice Way to Understand Programs

I have been involved in evaluating distracted driving programs for transportation workers. While working on the evaluations I developed an interesting way to understand how programs are acting and what they are doing. Well, “developed” is a strong word. What really happened is that a piece of the idea popped into my head for no apparent reason. It just seemed like a good idea at the time. Over the next few months, aided and abetted by time in the sauna, other pieces crept in. This was no coherent intellectual effort, that’s for sure.

A word about voice. Continue reading

Posted in Uncategorized | Leave a comment

Complexity is about stability and predictability

Table of Contents

Complexity is About Stability and Predictability

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Fractals
Example 4: Phase Transitions
Example 5: Logistic Maps
Example 6: Power Laws
Example 7: Cross Linkages
Example 8: Emergence

What Does All This Mean for Evaluators?

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Power Laws
Example 4: Timeframes, Attractors, and Power Laws
Example 5: Emergence
Example 6: Fractals
Example 7: Phase Shifts


Complexity is About Stability and Predictability

Figure 1: Ban the Butterfly

Figure 1: Ban the Butterfly

I have been thinking about how complexity is discussed in evaluation circles. A common theme seems to be that because programs are complex we can’t generalize evaluation findings over space and time because of the inherent uncertainties that reside in complex systems. (Sensitive dependence on initial conditions, evolving environments, etc.) The more I think about the emphasis on instability and unpredictability, the less I like it. See figure 1. Ban the butterfly! Continue reading

Posted in Uncategorized | 3 Comments

What is the relationship between path dependence and system stability? With explanation of why I care.

I realized it might help to explain what led me to ask this question in the first place. I submitted a proposal to AEA to talk about how traditional evaluation methods can be used in complex systems. Part of that explanation will have to involve understanding the CAS implications of stability in program impact across time and place. See the end of this post for that proposal.

I’m looking for some sources and opinions to help with a question that has been troubling me lately.  I’m struggling with the question of the relationship between path

  • dependence and
  •  system stability.

Or maybe I mean the relationship between path dependence and the ability to predict a system’s trajectory. I’m not sure about the best way to phrase the question.  In any case read on to see my confusion.

I’m bumping into a lot of people who believe that systems are unstable/unpredictable because of path dependence. This is one of those notions that seems right but smells wrong to me. It seems too simple, and it does not make sense to me because it implies that if systems are predictable there is no path dependence operating.  That can’t be right, can it? Here is a counter example. Continue reading

Posted in System Stability and Sustainability | 12 Comments

Program Logic, Program Theory, and Unintended Consequences: Understanding Relationships. Implementing Action


These are the slides for a workshop I did at the Canadian Evaluation Society

Posted in Uncategorized | 1 Comment

Big Data in Evaluation

I have been developing an interest in “big data” as it may relate to the field of evaluation. The interest comes from two sources. 1) As the Editor of the journal Evaluation and Program Planning, I am on the lookout for cutting edge material to present to our readers. As someone who has done research and theoretical work on unintended consequences of program action, I see big data as a methodology that may help to reveal such unintended consequences. As a result of these two interests I’m looking for:

  • Examples where a big data approach has revealed consequences of program action that were not anticipated, and
  • People who may want to write about big data as it applies to the field of evaluation.

If you can help please get in touch with me at jamorell@jamorell.com

Posted in Looking for examples -- Unintended consequences of program behavior | 1 Comment

How to evaluate complex health interventions?

I just came back from Toronto, where I was visiting Sanjeev Sridharan, who runs a most interesting organization called the Evaluation Center for Complex Health Interventions. I was there to demonstrate the applicability of agent-based modeling as an evaluation tool. During the talk nobody asked me a question I could not easily answer. But then came lunch the next day, when Sanjeev casually asked “How would you evaluate a complex health intervention”? Hmmmm. On the train home a few days later I worked out an outline of an answer. I figured I’d lay it out there for the world to have a whack at. See below for my ruminations. Whack away.

Because I have been thinking so much about agent based systems lately I came up with the notion that concepts drawn from that domain could be useful for constructing evaluation strategies. I’m not arguing (right now) for doing formal agent based modeling. But I do think that the idea of agent behavior could be a useful trick of the mind to help design evaluations.

Some brief background on agents
What is an “agent”? It is an entity that Continue reading

Posted in Complex Systems, Programs, and Evaluation | 8 Comments