Common Introduction to all 6 Posts

History and Context
These blog posts are an extension of my efforts to convince evaluators to shift their focus from complex systems to specific behaviors of complex systems. We need to make this switch because there is no practical way to apply the notion of a “complex system” to decisions about program models, metrics, or methodology. But we can make practical decisions about models, metrics, and methodology if we attend to the things that complex systems do. My current favorite list of complex system behavior that evaluators should attend to is:

For a history of my activity on this subject see: PowerPoint presentations: 1, 2, and 3; fifteen minute AEA “Coffee Break” videos 4, 5, and 6; long comprehensive video: 7.

Continue reading →

Common Introduction to all 6 Posts

History and Context
These blog posts are an extension of my efforts to convince evaluators to shift their focus from complex systems to specific behaviors of complex systems. We need to make this switch because there is no practical way to apply the notion of a “complex system” to decisions about program models, metrics, or methodology. But we can make practical decisions about models, metrics, and methodology if we attend to the things that complex systems do. My current favorite list of complex system behavior that evaluators should attend to is:

For a history of my activity on this subject see: PowerPoint presentations: 1, 2, and 3; fifteen minute AEA “Coffee Break” videos 4, 5, and 6; long comprehensive video: 7.

Since I began thinking of complexity and evaluation in this way I have been uncomfortable with the idea of just having a list of seemingly unconnected items. I have also been unhappy because presentations and lectures are not good vehicles for developing lines of reasoning. I wrote these posts to address these dissatisfactions.

From my reading in complexity I have identified four themes that seem relevant for evaluation.

• Pattern
• Predictability
• How change happens
• Adaptive and evolutionary behavior

Continue reading →

Common Introduction to all 6 Posts

History and Context
These blog posts are an extension of my efforts to convince evaluators to shift their focus from complex systems to specific behaviors of complex systems. We need to make this switch because there is no practical way to apply the notion of a “complex system” to decisions about program models, metrics, or methodology. But we can make practical decisions about models, metrics, and methodology if we attend to the things that complex systems do. My current favorite list of complex system behavior that evaluators should attend to is:

For a history of my activity on this subject see: PowerPoint presentations: 1, 2, and 3; fifteen minute AEA “Coffee Break” videos 4, 5, and 6; long comprehensive video: 7.

Since I began thinking of complexity and evaluation in this way I have been uncomfortable with the idea of just having a list of seemingly unconnected items. I have also been unhappy because presentations and lectures are not good vehicles for developing lines of reasoning. I wrote these posts to address these dissatisfactions.

From my reading in complexity I have identified four themes that seem relevant for evaluation.

• Pattern
• Predictability
• How change happens
• Adaptive and evolutionary behavior

Others may pick out different themes, but these are the ones that work for me. Boundaries among these themes are not clean, and connections among them abound. But treating them separately works well enough for me, at least for right now.

Figure 1 is a visual depiction of my approach to this subject.

Figure 1: Complex Behaviors and Complexity Themes.

• The black rectangles on the left depict a scenario that pairs a well-defined program with a well-defined evaluation, resulting in a clear understanding of program outcomes. I respect evaluation like this. It yields good information, and there are compelling reasons working this way. (For reasons why I believe this, see 1 and 2.)
• The blue region is there to indicate that no matter how clear cut the program and the evaluation; it is also true that both the program and the evaluation are embedded in a web of entities (programs, policies, culture, regulation, legislation, etc.) that interact with our program in unknown and often unknowable ways.
  

• The green region depicts what happens over time. The program may be intact, but the contextual web has evolved in unknown and often unknowable ways. Such are the ways of complex systems.

• Recognizing that we have a complex system, however, is not amenable to developing program theory, formulating methodology, or analyzing and interpreting data. For that, we need to focus on the behaviors of complex systems, as depicted in the red text in the table. Note that the complex behaviors form the rows of a table. The columns show the complexity themes. The Xs in the cells show which themes relate to which complexity behaviors.

Unspecifiable Outcome Chains

People are enamored by the “butterfly effect”, but I hate it. It is beyond me why evaluators are so drawn to the idea of instability. In my world you can hit programs over the head with data as hard as you can, and they still do not change. My problem is too must stability, not too little. And yet, the notion of sensitive dependence has its place in evaluation. That place is not in uncertainty about what will happen, but uncertainty about the order in which things will happen. I don’t know how frequent a problem this is in evaluation, but I’m pretty sure it exists. I am very sure that evaluators would do well to consider the possibility when they develop program theory.

In what follows I am going to adopt a common view of butterflies and instability. It’s the one that opens Wikipedia’s entry on the butterfly effect: “In chaos theory, the butterfly effect is the sensitive dependence on initial conditions in which a small change in one state of a deterministic nonlinear system can result in large differences in a later state.” Needless to say this is a very simplistic approach to a very complicated and controversial subject. Read the rest of the Wikipedia entry to get a sense of the issues involved. If you really want to get into it, go to: Chaos.

The reason for the difficulty in understanding outcome order is that we can be too confident in our estimations of what fits where in outcome chains. We think the sequence is invariant, which most often it probably is. I am convinced though, that there are times when small perturbations can affect the order. Stated differently, the sequence of outcomes is subject to small random fluctuations in the environment.

I’ll illustrate with a hypothetical example. A friend of mine who does a lot of educational evaluation assures me that it makes some sense. The program in question is designed to improve teachers’ classroom management skills. Figure 2 compares two versions of the program theory. The top of the figure takes the form of a linear sequence. It’s sophisticated in the way it mixes unambiguous relationships and uncertain relationships. The dashed arrows indicate unambiguous relationships: For instance, classroom management leads to job satisfaction, which in turn leads to less tension between teachers and principles. Solid black arrows show ambiguous relationships. For instance, “student satisfaction” is an input toan unspecified collection of other intermediate outcomes.

The bottom form of the model acknowledges limitations in the program theory. It depicts a situation in which better classroom management makes itself felt in a cloud of outcomes that affect each other in elaborate ways, both directly via 1:1 and 1:many relationships, and also via proximate and distal feedback loops. Also, note the two different network possibilities – red solid, and blue dashed. I did that to emphasize that any number of relationships are possible. It would make the picture too complicated, but it is also the case that the network of relationships will be different in each setting where the classroom management program is implemented.

Figure 2: Traditional and Complex Theories of Change

What will the relationships be in any particular setting? That is an unanswerable question because too many factors will be operating to specify the conditions. All we know is that better classroom management leads to any number of student performance outcomes, which in turn will lead to higher test scores.

If there is so much confusion about intermediate outcomes, why might we be able to reliably expect that the classroom management program will result in higher test scores? Complexity provides two ways to think about this: 1) emergence, and 2) attractor space.

Emergence: A good way to explain emergence is to start with a counter example. Think of a car. Of course the car is more than the sum of its parts. But it is also true that the unique function of each part, and its contribution to the car, can be explained. If someone asked me what a “cylinder” is, I could describe it. I could describe what it does. When I got finished, you would know how the part “cylinder” contributes to the system called a “car”.

In contrast, think about trying to explain a traffic jam only in terms of the movement of each individual car. The jam moves in the opposite direction to the cars of which it is composed. The jam grows at the back as cars slow down, and the front recedes as cars peel off. Looked at as a whole, the traffic jam is clearly something qualitatively different from the movement of any care in the jam. (NetLogo has good one and two lane simulations that are worth looking at.) In the “classroom management” case, we might consider “better test scores” as an emergent outcome – one that cannot be explained in terms of the particulars of any of its parts.

Attractor space: There are two ways to think about “attractors” in complexity. The most formal is a mathematical formulation concerning the evolution of a dynamical system over time. As Wikipedia puts it: “In the mathematical field of dynamical systems, an “attractor” is a set of numerical values toward which a system tends to evolve, for a wide variety of starting conditions of the system. System values that get close enough to the attractor values remain close even if slightly disturbed.”

However, there is a more metaphorical, but still useful, way to think about this. Namely, that an attractor is a “space” that defines how something will move through it. There may be many paths within the attractor, but depending on its “shape”, many paths through it will lead to the same place. Marry this to the open systems notion of “equifinality” and it’s not hard to think in terms of a set of causal relationships among a defined set of variables that will lead to the same outcome. In theory there could be an infinite number of elements and paths that would lead to higher test scores, but that does not matter. What matters is that a particular set of outcomes are meaningful intermediate outcomes for a particular program, that it makes sense to measure those outcomes, and that many different combinations of those intermediate outcomes can be relied upon to produce better test scores.

While I am not sure which way to think about the bottom scenario in Figure 2, I do know that there is an important difference between the “emergence” and “attractor” perspective. With emergence, the specific intermediate outcomes do not matter very much. Which ones are manifest and which ones are not is irrelevant to the emergent result. That may be elegant in its way, but it is not all that satisfying to program funders. After all, they do want to know what intermediate outcomes were produced. The attractor way of looking at it does focus attention on which of those intermediate outcomes were manifest, and in what order. It may not be possible to assure the funders that the same outcomes and order will appear the next time, but it is possible to give them some pretty good understanding of what happened. The logic of generality and external validity notwithstanding, knowing what happened in one case can be awfully useful for planning the future.

Common Introduction to all 6 Posts

History and Context
These blog posts are an extension of my efforts to convince evaluators to shift their focus from complex systems to specific behaviors of complex systems. We need to make this switch because there is no practical way to apply the notion of a “complex system” to decisions about program models, metrics, or methodology. But we can make practical decisions about models, metrics, and methodology if we attend to the things that complex systems do. My current favorite list of complex system behavior that evaluators should attend to is:

For a history of my activity on this subject see: PowerPoint presentations: 1, 2, and 3; fifteen minute AEA “Coffee Break” videos 4, 5, and 6; long comprehensive video: 7.

Since I began thinking of complexity and evaluation in this way I have been uncomfortable with the idea of just having a list of seemingly unconnected items. I have also been unhappy because presentations and lectures are not good vehicles for developing lines of reasoning. I wrote these posts to address these dissatisfactions.

From my reading in complexity I have identified four themes that seem relevant for evaluation.

• Pattern
• Predictability
• How change happens
• Adaptive and evolutionary behavior

Others may pick out different themes, but these are the ones that work for me. Boundaries among these themes are not clean, and connections among them abound. But treating them separately works well enough for me, at least for right now.

Figure 1 is a visual depiction of my approach to this subject.

Figure 1: Complex Behaviors and Complexity Themes

• The black rectangles on the left depict a scenario that pairs a well-defined program with a well-defined evaluation, resulting in a clear understanding of program outcomes. I respect evaluation like this. It yields good information, and there are compelling reasons working this way. (For reasons why I believe this, see 1 and 2.)

• The blue region is there to indicate that no matter how clear cut the program and the evaluation; it is also true that both the program and the evaluation are embedded in a web of entities (programs, policies, culture, regulation, legislation, etc.) that interact with our program in unknown and often unknowable ways.
  

• The green region depicts what happens over time. The program may be intact, but the contextual web has evolved in unknown and often unknowable ways. Such are the ways of complex systems.

• Recognizing that we have a complex system, however, is not amenable to developing program theory, formulating methodology, or analyzing and interpreting data. For that, we need to focus on the behaviors of complex systems, as depicted in the red text in the table. Note that the complex behaviors form the rows of a table. The columns show the complexity themes. The Xs in the cells show which themes relate to which complexity behaviors.

Network Structure and Fractals

Any time a program is concerned with the spread of its services or impact over time, the subject of networks becomes a candidate for attention by evaluators. This is because: 1) It can be of value to know the pattern that describes how a program moves from one adopting location to another. 2) Network structure is a useful concept when the spread of a program constitutes an outcome to be evaluated. 3) Network structure is a useful construct when evaluating the degree to which a set of connections is efficient, effective, and resilient in the face of breakage. Continue reading →

Common Introduction to all 6 Posts

History and Context
These blog posts are an extension of my efforts to convince evaluators to shift their focus from complex systems to specific behaviors of complex systems. We need to make this switch because there is no practical way to apply the notion of a “complex system” to decisions about program models, metrics, or methodology. But we can make practical decisions about models, metrics, and methodology if we attend to the things that complex systems do. My current favorite list of complex system behavior that evaluators should attend to is:

For a history of my activity on this subject see: PowerPoint presentations: 1, 2, and 3; fifteen minute AEA “Coffee Break” videos 4, 5, and 6; long comprehensive video: 7.

Since I began thinking of complexity and evaluation in this way I have been uncomfortable with the idea of just having a list of seemingly unconnected items. I have also been unhappy because presentations and lectures are not good vehicles for developing lines of reasoning. I wrote these posts to address these dissatisfactions.

From my reading in complexity I have identified four themes that seem relevant for evaluation.

• Pattern
• Predictability
• How change happens
• Adaptive and evolutionary behavior

Continue reading →

Common Introduction to all 6 Posts

History and Context
These blog posts are an extension of my efforts to convince evaluators to shift their focus from complex systems to specific behaviors of complex systems. We need to make this switch because there is no practical way to apply the notion of a “complex system” to decisions about program models, metrics, or methodology. But we can make practical decisions about models, metrics, and methodology if we attend to the things that complex systems do. My current favorite list of complex system behavior that evaluators should attend to is:

For a history of my activity on this subject see: PowerPoint presentations: 1, 2, and 3; fifteen minute AEA “Coffee Break” videos 4, 5, and 6; long comprehensive video: 7.

Since I began thinking of complexity and evaluation in this way I have been uncomfortable with the idea of just having a list of seemingly unconnected items. I have also been unhappy because presentations and lectures are not good vehicles for developing lines of reasoning. I wrote these posts to address these dissatisfactions. From my reading in complexity I have identified four themes that seem relevant for evaluation.

• Pattern
• Predictability
• How change happens
• Adaptive and evolutionary behavior

Continue reading →

Lately I have been spending a lot of time thinking about two subjects: 1) models (program, logic, change, etc.) and 2) complex behavior. (Not complex systems. I don’t like that subject.)

It occurred to me that different models are relevant at different time scales. Most of the models one sees in the evaluation world involve clear outcome chains between short, intermediate, and long range outcomes. Usually those long range outcomes are aspirational in two ways. 1) Nobody ever stays around long enough to actually evaluate whether the program in question had any impact. Continue reading →

My last blog post dealt with why evaluators should focus on complex behavior as opposed to complex systems. Bob Williams made a comment about how the post made a lot of sense, but that it conveyed the impression that evaluators do not have to worry about complexity theory. Evaluators do need to be concerned with theory, and Bob’s post got me to begin to crystallize some notions that have been marinating in the back of my brain for some time.

My Starting Point
Recently I have been pounding on the idea that a switch from complex systems to the behavior of complex systems would do a lot to further the abilities of evaluators to make practical, operational decisions about program theory, metrics, and methodology. And after all, that’s what it’s all about. We (I at least) get hired when someone says to me: Continue reading →

Lately I have been speaking to as many audiences as I can about the need to focus on complex behavior rather than on complex systems. The reason is that there is no practical way to apply the notion of a “complex system” to practical decisions about program models, metrics, or methodology. But it is possible to make those decisions with respect to the things that complex systems do. I just completed a series of three short “coffee break” sessions on this topic for the American Evaluation Association.

Go here for the slides.

www.jamorell.com/documents/AEA_Coffee_Break_Part_1.pdf

www.jamorell.com/documents/AEA_Coffee_Break_Part_2.pdf

www.jamorell.com/documents/AEA_Coffee_Break_Part_3.pdf

If you are a member of AEA you can also hear the audio presentation. Go here for the audio tapes.

https://vimeo.com/269709240/e1b05b4857

https://vimeo.com/267297243/523c1a8c44

https://vimeo.com/265410410/8edd0dd3b7

Recently a friend of mine asked about my understanding of what role complexity can play in Evaluation, and how I would further that role. Below is an edited version of what I sent her.

My goal for the role of complexity in Evaluation

Complexity as discussed in evaluation circles contains a great deal of information that is either wrong, or ill-chosen as elements of complexity that can be useful in Evaluation. Those discussions do not appreciate the broad and deep knowledge about complexity that has roots in many different scientific disciplines. Much (most really) of that knowledge is not applicable in the field of Evaluation, but some of it is. My goal is to influence the field to appreciate and use the knowledge that is applicable.

As I see it, the critical issue revolves around program theory, i.e. people’s beliefs about what consequences a program will have, and why it will have those consequences. The problem is not methodology because for the most part our standard arsenal of quantitative and qualitative tools is more than adequate.  The problem is that evaluators do not choose appropriate methodologies because their designs are customized to test incorrect program theories.

What is complexity?

Continue reading →

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The other is to maximize the likelihood that we can thrive as circumstances change. Diversity Continue reading →

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The Continue reading →

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.)

Part 1: What do I mean by diversity?

Continue reading →

We  like to complain about evaluation use.
People in my business (me included) like to lament the lack of attention that people pay to evaluation. If only we did a better job if identifying stakeholders. If only we could do a better job of engaging them. If only we understood their needs better. If only we had a different relationship with them. If only we presented our information in a different way. If only we chose the appropriate type of evaluation for the setting we were working in. If only we fit the multiple definitions of “evaluation use” to our setting. And so on and so forth. I’m in favor of asking these questions. I do it myself and I am convinced that asking them leads to more and better evaluation use.

Lately I have been thinking differently.
I’m using this blog post for two reasons. One is that I want to begin a discussion in the evaluation community that may lead to more and better evaluation use. The second reason is that writing this post is giving me a chance to discern the logic underlying a behavior pattern that I seem to have fallen into. As for the logic, as far as I can tell it has two roots: continuous process improvement, and complexity. Continue reading →

This blog is my effort to consolidate and organize some back and forth I have been having about evaluation use. It was spurred by a piece on NPR about the Administration’s position on an after school program. (Trump’s Budget Proposal Threatens Funding For Major After-School Program.) In large measure the piece dealt with whether the program was effective. Arguments abounded about stated and unstated goals, and the messages contained in a variety of evaluations. Needless to say, the political inclinations of different stakeholders had a lot to do with which evaluations were cited. Below are the notions that popped into my head as a result of hearing the piece and talking to others about it.

Selective Use of Data
Different stakeholders glommed onto different evaluations to make their arguments. Continue reading →

Effective programs and useful evaluations require much more appreciation of complex behavior than is currently the case. This state of affairs must change. Evaluation methodology is not the critical inhibitor of that change. Program design is. Our purpose is to begin a dialog between program funders and evaluators to address this problem.

Current Practice: Common Sense Approach to Program Design and Evaluation
There is sense to successful program design, but that sense is not common sense. And therein lies a problem for program designers, and by extension, for the evaluators that are paid to evaluate the programs envisioned by their customers.

What is common sense?  
“Common sense is a basic ability to perceive, understand, and judge things that are shared by (“common to”) nearly all people and can reasonably be expected of nearly all people without need for debate.”

What is the common sense of program design?
The common sense of program design is usually expressed in one of two forms. One form is a set of columns with familiar labels such as “input”, “throughput”, and “output”. The second is a set of shapes that are connected with 1:1, 1:many, many:1 and many:many relationships. These relationships may be cast in elaborate forms, as for example, a systems dynamics model complete with buffers and feedback loops, or a tangle of participatory impact pathways.

But no matter what the specific form, the elements of these models, and hypothesized relationships among them, are based on our intuitive understandings of “cause and effect”, mechanistic views of how programs work. They also assume that the major operative elements of a program can be identified.

To be sure, program designers are aware that their models are simplifications of reality, that models can never be fully specified, and that uncertainties cannot be fully accounted for. Still, inspection of the program models that are produced makes it clear that almost all the thinking that went into developing those models was predominantly in the cause and effect, mechanistic mode. We think about the situation and say to ourselves: “If this happens, it will make (or has made) that happen.” Because the models are like that, so too are the evaluations.

Our common sense conceptualization of programs is based on deep knowledge about the problems being addressed and the methods available to address those problems. Common sense does not mean ignorance or naiveté. It does, however, mean that common sense logic is at play. There is no shame in approaching problems in this manner. We all do it. We are all human.

Including Complex Behavior in Program Design and Evaluation
When it comes to the very small, the very large, or the very fast, 20^th Century science has succeeded in getting us to accept that the world is not common sensical. But we have trouble accepting a non-common sense view of the world at the scale that is experienced by human beings. Specifically, we do not think in terms of the dynamics of complex behavior. Complex behavior has much to say about why change happens, patterns of change, and program theory. We do not routinely consider these behaviors when we design programs and their evaluations.

There is nothing intuitively obvious about complex behavior. Much of it is not very psychologically satisfying. Some of it has uncomfortable implications for people who must commit resources and bear responsibility for those commitments. Still, program designers must appreciate complex behavior if they are ever going to design effective programs and commission meaningful evaluations of those programs.

Pursuing Change
There is already momentum in the field of evaluation to apply complexity. Our critique of that effort is that current discussions of complexity do not tap the richness of what complexity science has discovered, and also, that some of the conversation is an incorrect understanding of complexity. The purpose of this panel is to bring a more thorough, a more research based, understanding of complexity into the conversation.

By “conversation” we mean dialogue between program designers and evaluators with respect to the role that complexity can play in a program’s operations, outcomes, and impacts. This conversation matters because as we said at the outset, the inhibiting factor is recognition that complex behavior may be at play in the workings of programs. Methodology is not the problem. Except for a few exotic situations, the familiar tools of evaluation will more than suffice. The question is what program behavior evaluators have license to consider.

Our goal is to pursue a long-term effort to facilitate the necessary discourse. Our strategy is to generate a series of conferences, informal conversations, and empirical tests that will lead to a critical mass of program funders and evaluators who can bring about a long term change in the rigor with which complexity is applied to program design and evaluation.

This blog post is a pitch for a different way to identify desired program outcomes.

Program Theories as they are Presently Constructed

Go into your archives and pull out your favorite logic models. Or dip into the evaluation literature and find models you like. You will find lots of variability among them in terms of: Continue reading →

I have been to a lot of conferences that had many sessions on ways to assure program sustainability. There is also a lot of really good research literature on this topic. Also, sustainability is a topic that has been front and center in my own work of late.

Analyses and explanations of sustainability inevitably end up with some fairly elaborate discussions about what factors lead to sustainability, how the program is embedded in its context, and so on. I have no doubt that all these treatments of sustainability have a great deal of merit. I take them seriously in my own work. I think everyone should. That said, I have been toying with another, much simpler approach.

Almost every program I have ever evaluated had only one major outcome that it was after. Sure there are cascading outcomes from proximate to distal. (Outcome to waves of impact, if you like that phrasing better.) And of course many programs have many outcomes at all ranks. But in general the proximate outcomes, even if they are many, tend to be highly correlated. So in essence, there is only one.

What this means is that when a program is dropped into a complex system, that program is designed to move the entire system in the direction of attaining that one outcome. We know how systems work. If enough effort is put in, they can in fact be made to optimize a single objective. But we also know that success like that makes the system as a whole dysfunctional in terms of its ability to adapt to environmental change, meet the needs of multiple stakeholders, maintain effective and efficient internal operations, and so on. As I see it, that means that any effort to optimize one outcome will be inherently unstable. No need to look at the details.

My notion is that in order to increase the probability of sustainability, a program should pursue multiple outcomes that are as uncorrelated as possible. The goal should be joint optimization, at the expense of sub-optimizing any of the desired outcomes.

I understand the problems in following my idea. The greater the number of uncorrelated outcomes, the greater the need to coordinate across boundaries, and as I have argued elsewhere in this blog, that is exceedingly difficult. (Why do Policy and Program Planners Assume-away Complexity?)  Also, I am by no means advocating ignoring all that work that has been done on sustainability. Ignoring it is guaranteed to lead to trouble.

Even so, I think the idea I’m proposing has some merit. Look at the outcomes being pursued, and give some thought to how highly correlated they are. What we know about systems tells us that optimization of one outcome may succeed in the short term, but it will not succeed in the long term. Joint optimization of uncorrelated outcomes? That gives us a better fighting chance.

A friend of mine (Donna Podems) is heading  up a project that involves providing a structure for a group of on-the-ground observers so they can apply a systems perspective to understanding what programs are doing and what they are accomplishing.  She asked me for a brain dump, which I happily provided.  What follows is by no means a systematic approach to looking at programs in terms of systems. It’s just a laundry list of ideas that popped into my head and flowed through my fingers. Below is a somewhat cleaned up version of what  sent her.

Hi Donna,

What follows is not a list of independent items. In fact I guarantee there are lots of connections. For instance, “redundancy” and “multiple paths” are not the same thing, but they are related. But time is tight, and I have a Greek meatball recipe to shop for, so let’s assume they are independent. Continue reading →

There is an interesting discussion going on in the Linked-In discussion group of the European Evaluation Society with respect to a question someone asked: How do linear models address the complexity in which we work? I can’t help but to weigh in. I also placed a link to this blog post on the EES discussion thread. My thoughts on this topic run in two directions.

1) Putting a lot of stuff in a model, and
2) What does it mean to “address complexity”?

Putting a Lot of Stuff in a Model

I am a big fan of information density. The more information that can be juxtaposed, the greater the amount of meaning that can be conveyed. The countervailing force to this inclination is that I’m also a big fan of information being readable. My solution is to think of rendering a model as an exercise in the joint optimization of two goals: Continue reading →

Common Introduction to all Three Posts
What is the Contribution of Complexity to Evaluation?
Drawing from Research and Theory in Complexity Studies

Common Introduction to all Three Posts

This is the third of three blog posts I have been writing to help me understand how “complexity” can be used in evaluation. If it helps other people, great. If not, at least it helped me.

Part 1:  Complexity in Evaluation and in Studies on Complexity
In this section I talked about using complexity ideas as practical guides and inspiration for conducting an evaluation, and how those ideas hold up when looked at in terms of what is known from the study of complexity. It is by no means necessary that there be a perfect fit. It’s not even a good idea to try to make it a perfect fit. But the extent of the fit can’t be ignored, either.

Part 2: Complexity in Program Design
The problems that programs try to solve may be complex. The programs themselves may behave in complex ways when they are deployed. But the people who design programs act as if neither their programs, nor the desired outcomes, involve complex behavior. (I know this is an exaggeration, but not all that much. Details to follow.) It’s not that people don’t know better. They do. But there are very powerful and legitimate reasons to assume away complex behavior. So, if such powerful reasons exist, why would an evaluator want to deal with complexity? What’s the value added in the information the evaluator would produce? How might an evaluation recognize complexity and

Part 3: Turning the Wrench: Applying Complexity in Evaluation
This is where the “turning the wrench” phrase comes from in the title of this blog post1. Considering what I said in the first two blog posts, how can I make good use of complexity in evaluation? In this regard my approach to complexity is no different than my approach to ANOVA or to doing a content analysis of interview data. I want to put my hands on a tool and make something happen. ANOVA, content analysis and complexity are different kinds of wrenches. The question is which one to use when, and how.

Complex Behavior or Complex System?
I’m not sure what the difference is between a “complex system” and “complex behavior”, but I am sure that unless I try to differentiate the two in my own mind, I’m going to get very confused. From what I have read in the evaluation literature, discussions tend to focus on “complex systems”, complete with topics such as parts, boundaries, part/whole relationships, and so on. My reading in the complexity literature, however, makes scarce use of these concepts. I find myself getting into trouble when talking about complexity with evaluators because their focus is on the “systems” stuff, and mine is on the “complexity” stuff. In these three blog posts I am going to concentrate on “complex behavior” as it appears in the research literature on complexity, not on the nature of “complex systems”. I don’t want to belabor this point because the boundaries are fuzzy, and there is overlap. But I will try to draw that distinction as clearly as I can. Continue reading →

Common Introduction to all Three Posts
Why do Policy and Program Planners Assume Away Complexity?
How Can Evaluators Apply Complexity in a way that will Help Program Designers?

Common Introduction to all Three Posts
This is the second of three blog posts I have been writing to help me understand how given the reality of how programs are designed, “complexity” can be used in evaluation . If it helps other people, great. If not, at least it helped me.

Part 1:  Complexity in Evaluation and in Studies on Complexity
In this section I talked about using complexity ideas as practical guides and inspiration for conducting an evaluation, and how those ideas hold up when looked at in terms of what is known from the study of complexity. It is by no means necessary that there be a perfect fit. It’s not even a good idea to try to make it a perfect fit. But the extent of the fit can’t be ignored, either.

Part 2: Complexity in Program Design
The problems that programs try to solve may be complex. The programs themselves may behave in complex ways when they are deployed. But the people who design programs act as if neither their programs, nor the desired outcomes, involve complex behavior. (I know this is an exaggeration, but not all that much. Details to follow.) It’s not that people don’t know better. They do. But there are very powerful and legitimate reasons to assume away complex behavior. So, if such powerful reasons exist, why would an evaluator want to deal with complexity? What’s the value added in the information the evaluator would produce? How might an evaluation recognize complexity and still be useful to program designers?

Part 3: Turning the Wrench: Applying Complexity in Evaluation
This is where the “turning the wrench” phrase comes from in the title of this blog post1. Considering what I said in the first two blog posts, how can I make good use of complexity in evaluation? In this regard my approach to complexity is no different than my approach to ANOVA or to doing a content analysis of interview data. I want to put my hands on a tool and make something happen. ANOVA, content analysis and complexity are different kinds of wrenches. The question is which one to use when, and how.

Complex Behavior or Complex System?
I’m not sure what the difference is between a “complex system” and “complex behavior”, but I am sure that unless I try to differentiate the two in my own mind, I’m going to get very confused. From what I have read in the evaluation literature, discussions tend to focus on “complex systems”, complete with topics such as parts, boundaries, part/whole relationships, and so on. My reading in the complexity literature, however, makes scarce use of these concepts. I find myself getting into trouble when talking about complexity with evaluators because their focus is on the “systems” stuff, and mine is on the “complexity” stuff. In these three blog posts I am going to concentrate on “complex behavior” as it appears in the research literature on complexity, not on the nature of “complex systems”. I don’t want to belabor this point because the boundaries are fuzzy, and there is overlap. But I will try to draw that distinction as clearly as I can. Continue reading →

This is the first of three blog posts I am writing to help me understand how “complexity” can be used in evaluation. If it helps other people, great. If not, at least it helped me.

Common Introduction to all Three Posts
Practicality and Theory
The Value and Dangers of Using Evaluation Program Theory
Complexity as an Aspect of Evaluation Program Theory
Appropriate but Incorrect Application of Scientific Concepts to Achieve Practical Ends

Continue reading →

During each of the first three weeks in January I will be publishing a blog post on how complexity can be applied in evaluation. They are not ready yet, but they are close. Below is the common introduction that I will be using for each of the posts.

Common Introduction to all Three Posts

Part 1:  Complexity in Evaluation and in Studies on Complexity
In this section will I talk about using complexity ideas as practical guides and inspiration for conducting evaluation, and how those ideas hold up when looked at in terms of what is known from the study of complexity. It is by no means necessary that there be a perfect fit. It’s not even a good idea to try to make it a perfect fit. But the extent of the fit can’t be ignored, either.

Part 2: Complexity in Program Design
The problems that programs try to solve may be complex. The programs themselves may behave in complex ways when they are deployed. But the people who design programs act as if neither their programs, nor the desired outcomes, involve complex behavior. (I know this is an exaggeration, but not all that much. Details to follow.) It’s not that people don’t know better. They do. But there are very powerful and legitimate reasons to assume away complex behavior. So, if such powerful reasons exist, why would an evaluator want to deal with complexity? What’s the value added in the information the evaluator would produce? How might an evaluation recognize complexity and still be useful to program designers?

Part 3: Turning the Wrench: Applying Complexity in Evaluation
Considering what I said in the first two blog posts, how can I make good use of complexity in evaluation? In this regard my approach to complexity is no different than my approach to ANOVA or to doing a content analysis of interview data. I want to put my hands on a tool and make something happen. ANOVA, content analysis and complexity are different kinds of wrenches. The question is which one to use when, and how.

Complex Behavior or Complex System?
I’m not sure what the difference is between a “complex system” and “complex behavior”, but I am sure that unless I try to differentiate the two in my own mind, I’m going to get very confused. From what I have read in the evaluation literature, discussions tend to focus on “complex systems”, complete with topics such as parts, boundaries, part/whole relationships, and so on. My reading in the complexity literature, however, makes scarce use of these concepts. I find myself getting into trouble when talking about complexity with evaluators because their focus is on the “systems” stuff, and mine is on the “complexity” stuff. In these three blog posts I am going to concentrate on “complex behavior” as it appears in the research literature on complexity, not on the nature of “complex systems”. I don’t want to belabor this point because the boundaries are fuzzy, and there is overlap. But I will try to draw that distinction as clearly as I can.

I’m in the process of working up a presentation for the upcoming conference of the American Evaluation Association:. Successful Scale-up Of Promising Pilots: Challenges, Strategies, and Measurement Considerations. (It will be a great panel. You should attend if you can.) This is the abstract for my presentation:

Title: Complex System Behavior as a Lens to Understand Program Change Across Scale, Place, and Time
Abstract: Development programs are bedeviled by the challenge of transferability. Whether from a small scale test to widespread use, or across geography, or over time, programs do not work out as planned. They may have different consequences than we expected. They may have larger or smaller impacts than we hoped for. They may morph into programs we only dimly recognize. They may not be implemented at all. The changes often seem random, and indeed, in some sense they are. But coexisting with the randomness, a complex system perspective shows us the sense, the reason, the rationality in the unexpected changes. By thinking in terms of complex system behavior we can attain a different understanding of what it means to explain, or perhaps, sometimes to predict, the mysteries of transferability. That understanding will help us choose methodologies and interpret data. It will also give us new insight on program theory.

There will only be one slide in this presentation.

Based on this slide I’m developing talking points. I know I’ll have to abbreviate it at the presentation, but I do want a coherent story to work from. A rough draft is below. Comments appreciated. Whack away. Continue reading →

Case Study Example for Workshop 18: Systems as Program Theory and as Methodology: A Hands on Approach over the Evaluation Life Cycle

This case was developed for a workshop at the American Evaluation Association’s 2015 Summer Evaluation Institute.

Construction of the Case
This is the example we will use throughout this workshop to illustrate how knowledge of system behavior can be applied in evaluation. The example is hypothetical. I made it up to resemble a plausible evaluation scenario that we may face, but which is elaborated to make sure it contains all the elements needed to explain the topics in the workshop. I am sure that none of us (me included) have ever been involved in an evaluation that is as far reaching and in-depth as the example here. But I am sure that all of us have been involved in evaluations that are similar to parts of the example, and, if you are like me, I bet you have dreamed of being involved in an evaluation of the size and scope of the example.

There are three initiatives. One aimed at adults. One aimed at mothers and young children. One aimed at teens. Each initiative has several individual programs that share some common outcomes, and which also have some unique outcomes.

All three initiatives are deliberately implemented Continue reading →

I have been involved in evaluating distracted driving programs for transportation workers. While working on the evaluations I developed an interesting way to understand how programs are acting and what they are doing. The method is based on a few principles, one set focusing on the nature of timelines and schedules, and the other on data collection.

Timelines and schedules
Timelines and schedules matter. These documents are constructed to meet a few objectives. They have to:

• Provide a reasonable plan whose tasks can be executed.
• Represent a reasonable correspondence between budget and work.
• Satisfy customers desires for when the work will be completed, and for how much.

In a sense there is a conflict between the first two objectives and the third, resulting in overly optimistic assessments of budgets and timelines. That’s the direction of the bias in our estimates. Continue reading →

Table of Contents

Complexity is About Stability and Predictability

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Fractals
Example 4: Phase Transitions
Example 5: Logistic Maps
Example 6: Power Laws
Example 7: Cross Linkages
Example 8: Emergence

What Does All This Mean for Evaluators?

Example 1: Attractors
Example 2: Strange Attractors
Example 3: Power Laws
Example 4: Timeframes, Attractors, and Power Laws
Example 5: Emergence
Example 6: Fractals
Example 7: Phase Shifts

Acknowledgements

Complexity is About Stability and Predictability

Figure 1: Ban the Butterfly

I have been thinking about how complexity is discussed in evaluation circles. A common theme seems to be that because programs are complex we can’t generalize evaluation findings over space and time because of the inherent uncertainties that reside in complex systems. (Sensitive dependence on initial conditions, evolving environments, etc.) The more I think about the emphasis on instability and unpredictability, the less I like it. See figure 1. Ban the butterfly! Continue reading →