Timelines, Critical Incidents and Systems: A Nice Way to Understand Programs

I have been involved in evaluating distracted driving programs for transportation workers. While working on the evaluations I developed an interesting way to understand how programs are acting and what they are doing. Well, “developed” is a strong word. What really happened is that a piece of the idea popped into my head for no apparent reason. It just seemed like a good idea at the time. Over the next few months, aided and abetted by time in the sauna, other pieces crept in. This was no coherent intellectual effort, that’s for sure.

A word about voice. Continue reading

Posted in Uncategorized | Leave a comment

Complexity is about stability and predictability

Table of Contents

Complexity is About Stability and Predictability

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Fractals
Example 4: Phase Transitions
Example 5: Logistic Maps
Example 6: Power Laws
Example 7: Cross Linkages
Example 8: Emergence

What Does All This Mean for Evaluators?

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Power Laws
Example 4: Timeframes, Attractors, and Power Laws
Example 5: Emergence
Example 6: Fractals
Example 7: Phase Shifts


Complexity is About Stability and Predictability

Figure 1: Ban the Butterfly

Figure 1: Ban the Butterfly

I have been thinking about how complexity is discussed in evaluation circles. A common theme seems to be that because programs are complex we can’t generalize evaluation findings over space and time because of the inherent uncertainties that reside in complex systems. (Sensitive dependence on initial conditions, evolving environments, etc.) The more I think about the emphasis on instability and unpredictability, the less I like it. See figure 1. Ban the butterfly! Continue reading

Posted in Uncategorized | 2 Comments

What is the relationship between path dependence and system stability? With explanation of why I care.

I realized it might help to explain what led me to ask this question in the first place. I submitted a proposal to AEA to talk about how traditional evaluation methods can be used in complex systems. Part of that explanation will have to involve understanding the CAS implications of stability in program impact across time and place. See the end of this post for that proposal.

I’m looking for some sources and opinions to help with a question that has been troubling me lately.  I’m struggling with the question of the relationship between path

  • dependence and
  •  system stability.

Or maybe I mean the relationship between path dependence and the ability to predict a system’s trajectory. I’m not sure about the best way to phrase the question.  In any case read on to see my confusion.

I’m bumping into a lot of people who believe that systems are unstable/unpredictable because of path dependence. This is one of those notions that seems right but smells wrong to me. It seems too simple, and it does not make sense to me because it implies that if systems are predictable there is no path dependence operating.  That can’t be right, can it? Here is a counter example. Continue reading

Posted in System Stability and Sustainability | 12 Comments

Program Logic, Program Theory, and Unintended Consequences: Understanding Relationships. Implementing Action


These are the slides for a workshop I did at the Canadian Evaluation Society

Posted in Uncategorized | 1 Comment

Big Data in Evaluation

I have been developing an interest in “big data” as it may relate to the field of evaluation. The interest comes from two sources. 1) As the Editor of the journal Evaluation and Program Planning, I am on the lookout for cutting edge material to present to our readers. As someone who has done research and theoretical work on unintended consequences of program action, I see big data as a methodology that may help to reveal such unintended consequences. As a result of these two interests I’m looking for:

  • Examples where a big data approach has revealed consequences of program action that were not anticipated, and
  • People who may want to write about big data as it applies to the field of evaluation.

If you can help please get in touch with me at jamorell@jamorell.com

Posted in Looking for examples -- Unintended consequences of program behavior | 1 Comment

How to evaluate complex health interventions?

I just came back from Toronto, where I was visiting Sanjeev Sridharan, who runs a most interesting organization called the Evaluation Center for Complex Health Interventions. I was there to demonstrate the applicability of agent-based modeling as an evaluation tool. During the talk nobody asked me a question I could not easily answer. But then came lunch the next day, when Sanjeev casually asked “How would you evaluate a complex health intervention”? Hmmmm. On the train home a few days later I worked out an outline of an answer. I figured I’d lay it out there for the world to have a whack at. See below for my ruminations. Whack away.

Because I have been thinking so much about agent based systems lately I came up with the notion that concepts drawn from that domain could be useful for constructing evaluation strategies. I’m not arguing (right now) for doing formal agent based modeling. But I do think that the idea of agent behavior could be a useful trick of the mind to help design evaluations.

Some brief background on agents
What is an “agent”? It is an entity that Continue reading

Posted in Complex Systems, Programs, and Evaluation | 8 Comments

Matching management systems to system behavior

Matt Keene and I have been having a back and forth on the topic of how management systems are built with respect to the way in which systems actually behave. Below is a record of our conversation to date.

Jonny’s thoughts
Desirability of Narrow Rigid Planning
First, I don’t place any negative value judgments on the need for rigid planning. There are good reasons for it given the nature of the world. Second, I don’t think that environmental management is different from any other kind of management in this regard. Everyone is in the same boat.  Third, the problem is not management systems. To blame the systems is like shooting the messenger.  Management systems match the world in which management takes place, and that is what is giving us trouble. Continue reading

Posted in Uncategorized | 3 Comments

Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community

What I tried to do is to look at the link between the intellectual traditions of evaluation and the sociology of evaluation from the point of view of evolutionary biology. For simplicity I’m assuming only two times in what is a long continuous process, and only two intellectual traditions. Also, I’m ignoring the possibility of linkages and networks among multiple traditions, and the multitude of reasons why the market for evaluation may change other than the behavior of evaluators. With those gross simplifications, my notions goes like this. Continue reading

Posted in Logic Models and Program Theory, Uncategorized | 1 Comment