This is part 10 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.
I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.
Part
Title
Post status
1
Complex systems or complex behavior?
up
2
Complexity has awkward implications for program designers and evaluators
up
3
Ignoring complexity can make sense
up
4
Complex behavior can be evaluated using comfortable, familiar methodologies
up
5
A pitch for sparse models
up
6
Joint optimization of unrelated outcomes
up
7
Why should evaluators care about emergence?
up
8
Why might it be useful to think of programs and their outcomes in terms of attractors?
up
9
A few very successful programs, or many, connected, somewhat successful programs?
up
10
Evaluating for complexity when programs are not designed that way
up
Evaluating for complexity when programs are not designed that way
There are good reasons to design programs with complex behavior in mind, and good reasons not to. (For the reasons not to, see Part 3 Ignoring complexity can make sense, which makes a case for the rationality of letting the sleeping complexity dog sleep.)
The fact that programs are not designed in ways that recognize complexity does not mean that evaluation should ignore complexity. My reasoning is that even if programs are not designed with complex behavior in mind, knowing about complex behavior can still be useful to stakeholders. Figure 1 illustrates what I have in mind.
Figure 1: Overlay of Complex Model on a non-Complex Program Design
Blue region of model Blue represents the original program model. It has much in it that is oblivious to complex behavior (and to common sense, for that matter.)
It is specified to an excruciating level of detail. (Part 5 A pitch for sparse models, describes why this is problematic.
It does not recognize that groups of outcomes, or the program as a whole, can have outcomes that cannot be explained in terms of the sum of its parts (Part 7: Why should evaluators care about emergence?)
It does not recognize the different distributions that outcomes may have, and the consequences of those distributions. (I did not right a blog post on this, but log-linear distributions are characteristic of much complex behavior. One consequence of this is that an astoundingly successful program may provide most of its benefits to a few service recipients.)
Green region of model Green represents a program model that recognizes some of the complex behaviors that may be operating. To make my point, I superimposed it on the original model
Network effects are included.
Undesirable consequences are acknowledged.
Data are collected and analyzed with respect to groups of outcomes, without regard to any unique outcome within the group.
The social implications of distribution shapes are considered over and above the technical aspects of doing statistical analysis.
If I had my choice, I’d evaluate with respect to the green model exclusively, but I acknowledge that stakeholders may need more fine-grained information. In any case, as you know by now, I am a big supporter of using more than one model in any single evaluation. All models are wrong, but many different models can be both wrong and useful in different ways.
Assumptions make up a significant percentage of every person’s everyday thinking. Most are subconscious, implicit and go without recognition (when I leave work, my car will be where I left it this morning). Others rise to the surface of our awareness and we can use that awareness to our advantage by checking the validity of that assumption; for example, I my ask my partner if my assumption that he purchased milk while at the grocery store is correct.
All assumptions represent some amount of risk, suggesting the following questions:
why do some assumptions emerge from our subconscious to be checked while others remain hidden?
how can we best surface assumptions and risks for the purpose of program planning and evaluation?
In order to answer these questions, we must first acknowledge that assumptions are an adaptive mechanism the brain uses to manage the sheer volume of stimuli and information that it takes on every day. If I had to constantly monitor my car’s position, I wouldn’t be able to write this blog or do much of anything else. Assumptions are the cognitive short-cuts that allow us to be so productive and creative in our thinking.
It makes sense that our brains are always looking for ways to consolidate and use assumptions as much as possible to move us through our environment and lives more efficiently. Following this reasoning, the only reason we would consciously identify and address an assumption is because the risk associated with that assumption may supersede the advantages it offers. This is, in part, an answer to the first question:
Assumptions naturally become explicit when their associated risk is a) known and b) understood to be greater than the advantage of maintaining the assumption
To understand how this assumption-risk dynamic plays out in program work, we need to unpack the idea of risk a bit more.
When we are thinking about community development programs, (programs focused on health, economic development, youth development education, etc.) risks include program conditions or components that might:
inhibit positive outcomes
do harm to beneficiaries
waste resources
These three types of risk (among other more context-specific risks) endanger program success and do supersede any advantage gained by maintaining an associated assumption. Therefore, it is critical that programs uncover assumptions associated with the various types of risk so that they can be addressed (checked or consciously accepted) by program staff and leaders.
It can be hard for program staff to uncover assumptions that underly the programs that they work on day-in and day-out. Even when program staff understand the role and typology of assumptions (paradigmatic, prescriptive, causal), it can be hard to identify them without a more targeted path in to the subconscious. For some program teams, using the idea of risk is an effective way of facilitating this brainstorm. For example, one might ask, “What are the outcomes, outside our formal program plans, that have to hold true in order for us to meet our objectives?” Or, “Why do we think this is the best approach given our available resources?” This brings us to an answer to the second question:
It is critical that program professionals, who have context and program specific expertise, intentionally engage in surfacing implicit program assumptions in preparation for planning, evaluation and learning. This can be accomplished either by brainstorming assumptions directly and identifying associated risks or by brainstorming possible program risks and identifying the associated assumptions.
This is part 9 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.
I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.
Part
Title
Approximate post date
1
Complex systems or complex behavior?
up
2
Complexity has awkward implications for program designers and evaluators
up
3
Ignoring complexity can make sense
up
4
Complex behavior can be evaluated using comfortable, familiar methodologies
up
5
A pitch for sparse models
up
6
Joint optimization of unrelated outcomes
up
7
Why should evaluators care about emergence?
up
8
Why might it be useful to think of programs and their outcomes in terms of attractors?
up
9
A few very successful programs, or many, connected, somewhat successful programs?
up
10
Evaluating for complexity when programs are not designed that way
8/15
Very successful programs, or many, connected, somewhat successful programs?
If these kinds of effects can result from networking, one could argue that resources are better invested in multiple programs, each of which is somewhat successful, than in a few programs, each of which is highly successful. (Of course, there is no guarantee whatsoever that these changes will be desirable, nor is it certain that the networking effects will result in greater total of positive change. But for now, I’m looking on the bright side – both the direction and amount of change will be desirable.)
To illustrate, consider the two scenarios depicted in Figure 1. In both, red arrows depict the short-term success of the program on a ten-point scale. The scenario on the left shows four very successful programs, each of which works in isolation from the others. The scenario on the right shows six somewhat successful programs, each of which has dense connections with the others.
Figure 1: Comparison of Few Non-networked Very Successful Programs and Many Networked Moderately Successful Programs
In the short term, the scenario on the left is the most beneficial. After all, just counting outcome points shows a total of 35, versus 19 on the right. But what might happen over time? I have no idea, but I’ll spin three not too farfetched possibilities.
Changes specified in the model: Note that in Figure 2 there are three direct connections with girls’ education: 1) SME capacity, 2) civic skills, and 3) crop yield. The SME and crop yield relationships are plausible because of two constructs not shown in the model, namely family income and affordability of school fees. The model is: crop yield and/or SME capacity –> family income –> affordability of school fees. The civic skills relationship makes sense because as people learn to participate in civic life, the quality of education is likely to get better.
The direct relationships in the previous paragraph are augmented by an indirect relationship: civic skills –> SME capacity. Why might this relationship make a difference? Because SME capacity leads to family income, so all programs that affect SME capacity will influence girls’ education.
Changes not specified in the model: It is not hard to imagine that SME owners will be among those who participate in civic skills training programs, and that such participation will bring business owners together who did not previously know each other. Between the new connections and the new skills, is it too hard to imagine that novel business opportunities will develop, or that creative ways to solve community problems will be revealed?
Sustainability consequences of collective change across multiple programs All the changes described above can be thought of as generalized improvements in a higher-level construct called “community functioning”. Over time, “community functioning” may circle back to the original programs, and thus improve their outcomes as well.
To take just one small possibility as an example. Imagine that the highly competent administrator of the malaria prevention program left for another job. What is the likelihood of an equally competent administrator being available to take his or her place? Those odds are better if the networking effects shown on the right in figure 1 have been operating. Why? Because increased wealth in the community may result in a salary that would make the job desirable, and because the overall quality of life in the community may make it a desirable place to live.
Figure 2 is a graphical view of a possible set of consequences of implementing the models shown in Figure 1.
Figure 2: Graphical Depiction of Networked and Non-networked Program Outcomes Over Time
Initial outcomes in the non-networked scenario exceed the initial outcomes for the same programs in the networked scenario. (Black stripes for initial non-networked, blue stripes for initial networked.)
In the non-networked scenario, the level of program outcomes remain constant over time. (Black stripes versus black solid.)
In the networked scenario outcomes grow so that over time, in five of the six cases, the networked outcomes exceed their initial values. (Blue stripped versus blue solid.)
In three of the six cases, networked values grow to exceed non-networked values. (Blue solid versus black solid.)
A, B, and C represent desirable changes that resulted from network effects that could not have been envisioned for each of the programs in isolation. These cases do not have accompanying black lines because they were never part of the outcome models for the original programs.
D is there to remind us that we should never assume that all program outcomes (especially when not part of the program model) will be desirable. If I wanted to be pessimistic, but maybe not too farfetched, I could have included E, F, G, and H, all of which I would have depicted as negative.
An implicit assumption in the above is that networking relationships have either been designed into the programs or implemented in such a way that they evolve naturally. (For example, all the programs are in the same geographical area or administrative unit.) It is by no means certain that networking relationships will develop. Whether they do, and what they do, are empirical questions that need to be addressed as part of the evaluation.
The above paragraph implies that program designers have considered the possibility of connections among their programs. Part 3 (Ignoring complexity can make sense) makes the point that designers may not be doing their jobs well if they did try to build those connections. But even if they don’t, evaluation can still provide valuable insight about the complex behavior that the program designers are ignoring. That is the subject of the next (and thankfully final) section of this series of blogs (Part 10 Evaluating for complexity when programs are not designed that way).
This is part 8 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.
I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.
Part
Title
Approximate post date
1
Complex systems or complex behavior?
up
2
Complexity has awkward implications for program designers and evaluators
up
3
Ignoring complexity can make sense
up
4
Complex behavior can be evaluated using comfortable, familiar methodologies
up
5
A pitch for sparse models
up
6
Joint optimization of unrelated outcomes
up
7
Why should evaluators care about emergence?
up
8
Why might it be useful to think of programs and their outcomes in terms of attractors?
up
9
A few very successful programs, or many, connected, somewhat successful programs?
8/9
10
Evaluating for complexity when programs are not designed that way
8/19
Why might it be useful to think of programs and their outcomes in terms of attractors?
Exercises to understand the historical behavior of a program, (or a class of programs), is a worthwhile activity in any evaluation. People should always do it. My hope in this blog post, however, is to make a convincing case that the concept of an “attractor” provides a richer way to think about a program’s history, its likely behavior in the future, and its outcomes.
In the mathematical field of dynamical systems, an attractor is a set of numerical values toward which a system tends to evolve, for a wide variety of starting conditions of the system. System values that get close enough to the attractor values remain close even if slightly disturbed.
In mathematics, a dynamical system is a system in which a function describes the time dependence of a point in a geometrical space. Examples include the mathematical models that describe the swinging of a clock pendulum, the flow of water in a pipe, and the number of fish each springtime in a lake.
Figure 1 shows some more examples of attractors. #1 is a map of watersheds in the United States. #2 shows multiple animal species at a watering hole. #3, well let’s just say that this one is my favorite. #4 shows planetary orbits.
The pictures in Figure 1 have a limitation, namely that the constructs they depict are real objects in physical space. It’s not just that certain elevations describe water flow. It’s that physical water follows through those elevations. It’s not just that playgrounds appeal to kids. It’s that real kids inhabit the playground space. This coincidence of attractor and physical object does not apply to all attractors. A particularly shaped physical space can also be used to describe less tangible constructs. To telegraph an example I’ll use later, an attractor space in the shape of a pendulum can be used to describe fluctuations in government policy.
Figure 1: Examples of Attractors
Why bother to think in terms of attractors? Why might it be worth thinking of programs and their outcomes in terms of attractors? Why not just rely on observing history and be done with it? Because thinking in terms of attractors:
Provides insightful visualization of how outcomes may change over time.
Kicks program theory up in the level of abstraction, thus revealing similarities and differences among seemingly different activities.
Can reveal likely system behavior even without good historical data.
To illustrate, I’ll use the example of a federal regulatory agency. Between a lot of evaluation I have done, a lot of reading relevant literature, and quite a bit of talking to close observers, this is a subject I know something about.
There is some very interesting social science research and theory on the question of maintaining safety by emphasizing rule compliance or engaging in cooperative activity between government and industry. It’s a complicated story, but the bottom line is that neither strategy can be entirely successful by itself, and that getting the mix right at any given time is a tricky problem. In fact, the most effective mix moves back and forth with circumstances. A dated, but good explanation of this phenomenon is Shapiro, S. A., & Rabinowitz, R. S. (1997). Punishment versus cooperation in regulatory enforcement: A case study of Osha. Administrative Law Review, 49(4). For a more general discussion, try: Sparrow, Malcom K. The Regulatory Craft: Controlling Risks, Solving Problems, and Managing Compliance 2000.
The image of a pendulum comes to mind. Looking back, it seems as if the emphasis that regulatory agencies place on enforcing compliance and working cooperatively oscillates, moving in one direction and then swinging back in the other. One way to look at this is to say that the attractor space traces the ark of a pendulum. Figure 2 shows what have in mind. Figure 2 can be thought of as a form of program model that deliberately omits a great deal of information in order to highlight the oscillatory behavior of the program. (For much more on models, see Part 5 – A pitch for sparse models.)
Figure 2: Pendulum as a Descriptor of Cooperation and Enforcement in a Regulatory Agency
Considering the argument I made above for thinking in terms of attractors and not exclusively in terms of program history, what’s the advantage here?
Visualization of how outcomes may change over time: Sometimes pictures help understanding, and I think this is one of those cases, particularly with respect to sustainability. Imagine evaluating a program designed to minimize a safety problem that was based on a cooperative program theory. No matter how successful that program, I’d assess the likelihood of long-term sustainability as lower in the scenario at the left rather than the scenario on the right? Why? Because I know the shape of the attractor. Of course the model leaves a lot out. It does not account for relationships between labor and management in the industry, or the inclinations of agency leadership, or developments in safety technology, or any of the myriad factors that may affect sustainability. All I know is the shape of the attractor and the location of the agency on it. Figure 2 is a sparse model that abstracts the attractor shape from all the other reasons that may affect sustainability. But it’s a sparse model that provides a lot of insight that would be obscured if I tried to build a mode that included labor/management relationships, the inclinations of policy makers, or developments in safety technology.
Level of abstraction: Figure 2 may be useful for understanding sustainability for one particular program in one particular regulatory agency, but it is also useful for revealing that the same attractor can be applied to any setting where the mission inclines toward enforcing compliance, but where the social reality of ensuring safety also calls for cooperation.
Historical data: It is always important to have empirical data on how an agency has behaved in the past. It’s one thing to draw a picture such as Figure 2, it is something else to have empirical knowledge of how much cooperation and enforcement is taking place, how long the trend has been going in one or another direction, what the balance point is, and so on. Unfortunately, in most instances an evaluator will not have such data. As examples of the difficulties involved, consider a few of the issues that spring to mind. 1) How obvious is it that a program falls into the categories of enforcement and cooperative emphasis? 2) How much activity is informal, and thus cannot be labeled as a “program” or a “policy” that can be identified and assessed? 3) What’s the metric for cooperation and enforcement? In the absence of good data, an evaluator may have to rely on expert judgement. That might provide a valid assessment, but it’s not very satisfying, either. It’s precisely because precise historical data may be lacking that attractor shapes provide insight. It may not take too much experience and historical knowledge to identify the general shape of an attractor and a reasonable sense of where an agency stands on it.
Before I move on to the next topic, I feel a need to make a point about the compliance/cooperation issue. The example I gave above springs from my own experience in evaluating cooperative programs. Please do not take this as an endorsement of a cooperation-only approach to improving safety. I do not believe that such a policy can be effective.
In the spirit of all models being wrong
I believe that the simple pendulum attractor model is useful because by ignoring many relevant issues, it succeeds at revealing some important program behaviors. But appreciating how the model is wrong is also revealing.
There are really two bobs: Figure 2 ignores the fact that there are always elements of cooperation and enforcement at play. Figure 3 is a more accurate portrayal of the situation.
Figure 3: Another Attractor Model for a Regulatory Agency
Smooth versus discontinuous movement: The smooth, predictable movement of a pendulum bob does not characterize how regulatory agencies behave. Cooperation with industry can change very quickly in light of a plane crash, a railroad derailment that leaks toxic chemicals, a pipeline explosion, or an interstate bus that ran off the road. Programs and policies can be easily canceled in the light of legislative and public outcry. Implementing programs and policies, however, take time. Movement in a cooperative direction is incremental.
These are the kinds of dynamics that makes me such a strong advocate of employing multiple models when evaluating the same program. Figure 2 has the advantage of clearly portraying the attractor that governs the sustainability of cooperative programs. It provides that simplicity, however, by ignoring the model depicted in Figure 3, which recognizes the co-existence of enforcement and cooperation. If my sole interest was in sustainability, I’d stick with Figure 2. But if I was also interested in whether agency staff would accept the cooperative program, I might also look at Figure 3 because I’d posit that the balance of cooperative and enforcement efforts may be relevant to acceptance. “Also” is the operative word in the previous sentence. Both models would be useful, each for a different part of the evaluation.
I can also imagine doing evaluation in a regulatory agency that needed only the model in Figure 3. For example, imagine a program designed to enhance communication and cooperation among factions within the agency that had different opinions about the value of enforcement and cooperation. If I ever got lucky enough to do an evaluation like that, I’d start by invoking the model in Figure 3. It’s not that one model is correct and the other is incorrect. It’s a matter of each model being useful for different reasons.
This is part 7 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.
I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.
Part
Title
Approximate post date
1
Complex systems or complex behavior?
up
2
Complexity has awkward implications for program designers and evaluators
up
3
Ignoring complexity can make sense
up
4
Complex behavior can be evaluated using comfortable, familiar methodologies
up
5
A pitch for sparse models
up
6
Joint optimization of unrelated outcomes
up
7
Why should evaluators care about emergence?
up
8
Why might it be useful to think of programs and their outcomes in terms of attractors?
7/19
9
A few very successful programs, or many, connected, somewhat successful programs?
7/24
10
Evaluating for complexity when programs are not designed that way
7/31
Why should evaluators care about emergence?
What’s the difference between an automobile engine (Figure 1 and a beehive (Figure 2)? After all, in each the whole is larger than the sum of its parts.
The answer is that for the engine, it is possible to explain what each part is and what role that part plays in the functioning of the engine. I can tell you shape of a cylinder, how it is constructed, why it is needed to contain combustible material, how it moves up and down and is attached to the crankshaft, and so on. When I finished, you would know how an internal combustion engine worked and how a cylinder contributes to the overall functioning of the engine.
I could not give you such an explanation for how any single bee contributes to the construction or functioning of a beehive. The beehive materia
Figure 2: Beehive
lizes when all those bees interact with each other as they do their simple bee things. That is emergence.
Change happens when the parts of an engine are assembled. Change happens when bees do bee things. But the type of change is different. Only the latter is an emergent phenomenon
Bees are bees, but what of emergence at the human scale of people, organizations, social groups, and political entities? It’s easy to find many examples. Some of the ones I like are: 1) The fractal nature of market fluctuations cannot be explained by the behavior of individual buyers and sellers. 2) The number of patents per capita in a city (when plotted logarithmically) increases more than the increase in a city’s population. 3) Traffic jams move in a direction opposite the direction of the flow of traffic. 4) the collective consequences of people, policy, business and infrastructure yields specialized districts in cities. In each of these cases, the behavior of the larger unit cannot be explained by breaking it down into its constituent parts.
Emergence matters in evaluation because it implies program theory that acknowledges that phenomena cannot be understood in terms of constituent parts. To take the example of specialized districts in cities. Any effort to understand the consequences of such districts for the city’s appeal to outsiders needs to be understood in terms of the impact of the district on city life. It would not help to do such an analysis by researching the individual people, policies, businesses and infrastructure that comprise the district. Those do not, and cannot, “add up” to “district appeal”. To extend the example, the appeal of the city to outsiders probably has to do with the entire group of specialized districts, and how those districts affect each other. Districts might be a meaningful unit of analysis, but the constituent parts of districts would not.
Why not try to do the analysis in terms of the constituent parts? The reason has nothing to do with our analytical capabilities or our access to data. The reason is that because these are emergent phenomena, it is no more possible to understand behavior in terms of its parts than it is to understand a beehive in terms of individual bees.
Figure 3: Emergent Versus Discrete Causality
The challenge for evaluation centers on program theory, as illustrated in Figure 3. The difference between the model at the top and the model at the bottom does not seem all that dramatic. In fact, the difference is profound. The model at the top states that it is possible to understand “appeal” in terms of the individual contributions of people, policy, business, and infrastructure. The model at the bottom acknowledges that however one might understand “appeal”, it is not in terms of the unique contributions of people, policy, business, and infrastructure. The difference between the two models has very different consequences for
Assumptions are what we believe to hold true. They may be tacit or explicit. It is okay to assume. In fact, it’s inevitable, because in order to make sense of a complex world, one needs to prioritize the variables and relationships that matter most. The danger is when the variables that aren’t prioritized are thought not to exist entirely. That is to assume that we haven’t assumed.
Examining our assumptions about how a program should work is essential for program success, since it helps to unmask risks. Program assumptions should be understood in relation to program outputs and outcomes.
An old adage goes: “You can lead a horse to water, but can’t make it drink”.
In his book Utilization focused evaluation, Michael Patton dramatizes the above adage in a way that makes it easy to understand program outputs, outcomes and assumptions:
The longer term outcomes are that the horse stays healthy and works effectively. The desired outcome is that the horse drinks the water (Assuming the water is safe, horse is thirsty, or that the horse herder has a theory of what makes horses want to drink water, or the science of horse drinking).
But because program staff know that they can’t make the horse drink the water, they focus on things that they can control:
Leading the horse to the water, making sure the tank is full, monitoring the quality of water, and keeping the horse within drinking distance of the water.
In short, they focus on the processes of water delivery (outputs) rather than the outcome of water drunk (or the horse staying healthy and productive-emphasis added).
Because staff can control processes but cannot guarantee attaining of outcomes, government rules and regulations get written specifying exactly how to lead a horse to water.
Quality awards are made for improving the path to water-and keeping the horse happy along the way.
Most reporting systems focus on how many horses get led to the water, and how difficult it was to get them there, but never get around to finding out whether the horse drank the water and stayed healthy.
The outputs (horse taken to the well) and outcomes (horse drinks water and stays healthy) are clear in this illustration. What then are the assumptions in this illustration? Here are my suggestions:
Given the horse drinks water outcome, perhaps the horse is expected to be thirsty or we know when the horse needs to drink water, and the water is expected to taste good, that the horse herder understands the relationship between horse drinking and horse health and other horses are not competing for the same water source. And just because one horse drinks the water, it doesn’t mean all of them will do so for all sorts of reasons that we might not understand.
Given the horse healthy outcome, perhaps the water is expected to be safe. Etc. etc.
Most monitoring, evaluation and learning systems try to track program outputs and outcomes. But critical assumptions are seldom tracked. If they are, it’s factors beyond stakeholders’ control that are tracked–such as no epidemic breaking out to kill all horses in the community (external assumptions). In the above illustration, the water quality could be checked. Also, the horse’s thirst could be manipulated with a little salty food, and there could be a system of managing the horses so that they all get a drink. These are assumptions within stakeholder influence (internal assumptions).
My point here is that examining (internal and external) assumptions alongside program outputs and outcomes unmasks risks to program success.
Volume overview: Working with assumptions. Existing and emerging approaches for improved program design, monitoring and evaluation Apollo M. Nkwake, Nathan Morrow
Clarifying concepts and categories of assumptions for use in evaluation Apollo M. Nkwake, Nathan Morrow
Assumptions at the philosophical and programmatic levels in evaluation Donna M. Mertens
Interfacing theories of program with theories of evaluation for advancing evaluation practice: Reductionism, systems thinking, and pragmatic synthesis Huey T. Chen
Assumptions, conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development Thomas Archibald, Guy Sharrock, Jane Buckley, Natalie Cook
Causal inferences on the effectiveness of complex social programs: Navigating assumptions, sources of complexity and evaluation design challenges Madhabi Chatterji
Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts Nathan Morrow, Apollo M. Nkwake
Conclusion: Agency in the face of complexity and the future of assumption-aware evaluation practice Nathan Morrow, Apollo M. Nkwake
This is part 6 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.
I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.
Part
Title
Approximate post date
1
Complex systems or complex behavior?
up
2
Complexity has awkward implications for program designers and evaluators
up
3
Ignoring complexity can make sense
up
4
Complex behavior can be evaluated using comfortable, familiar methodologies
up
5
A pitch for sparse models
up
6
Joint optimization of unrelated outcomes
up
7
Why should evaluators care about emergence?
7/16
8
Why might it be useful to think of programs and their outcomes in terms of attractors?
7/19
9
A few very successful programs, or many, connected, somewhat successful programs?
7/24
10
Evaluating for complexity when programs are not designed that way
Figure 1 is a nice traditional model of an AIDS prevention/treatment program. The program is implemented, and services are provided. Because of careful planning, the quality of service is high. The combined amount and quality of service decreases the incidence and prevalence of AIDS. Decreased incidence and prevalence lead to improvements in quality of life and other similar measures. Because incidence and prevalence decrease, the amount of service provided goes down. However, at whatever level, the quality of service remains high. All these changes can be measured quantitatively. Change in the outcomes also affects the activities of the program, but for the most part, understanding those changes requires qualitative analysis.
There is nothing wrong with this model and this evaluation. I would dearly love to have a chance to do a piece of work like that. Note, however, an aspect of this program that characterizes every program I have ever seen. All the outcomes are highly correlated with each other. Because of the ways in which change happens, this can have some unpleasant consequences.
The unpleasant consequences can be seen by casting the AIDS program within a model that recognize that the AIDS program is but one of many organisms in a diverse ecosystem of health activities (Figure 2). (For a really good look at the subject of diversity and change, see Scott Page’s Diversity and Complexity.)
Figure 2: Casting the AIDS Model into an Ecosystem Based Health Care Model
Looked at in those terms, the way change happens can have widespread, and probably not very desirable consequences. Table 1 explains the details in the model shown in Figure 2.
Table 1: Explanation of Figure 2-
Upper right
The ecosystem of health services is arranged in a radar chart to show how much each service contributes to overall health in the community. There is no scale on the chart, but the absolute value of each service quality does not matter. All that matters is that under the circumstances, each service is about as good as it can be.
Upper left
This is the model of our very successful AIDS prevention and treatment program, as shown in Figure 1.
Lower left
This is a chart of what can happen to health system resources when an overriding priority is put on AIDS, to the exclusion of everything else. Resources flow from the rest of the system to AIDS. When I say “resources” I do not mean just money. I mean everything about a health care system that is needed for the system to function well.
Lower right
This radar chart shows the status of the system after the AIDS effort has been in operation for a while. Indeed, the AIDS measures improve. But what of the other services? How do they accommodate to nurses choosing to move to AIDS care, or policy makers time and intellectual efforts pointed in the AIDS direction, and so on? It seems reasonable to posit that whatever happens to those other services, it will not be to their advantage. Their environment has become resource poor.
This is what happens when a single objective is pursued when a system is comprised of diverse entities with diverse goals. You will get what you worked for, but the system as a whole may be the worse off for it. What is the solution? The solution is to work at jointly optimizing multiple somewhat unrelated outcomes. “Somewhat” is an important qualifier because the range of objectives cannot be too diverse. In the AIDS example, all health care objectives certainly have some overlap and relationships to each other. It’s not as if the goals to be jointly optimized were as far apart as AIDS and girls’ schooling. Some coherence of focus is needed.
The above advice can be excruciatingly difficult to follow. One problem is that there is nothing obvious about what “joint optimization” means. AIDS prevention, tertiary care, and women’s health – imagine drawing a logic model for the goals of each of these programs. Then imagine the interesting conversations that would ensue on the topic of how much achievement of each goal was appropriate.
Indeed, one way to look at the simple model depicted by Figure 1 is that it is a program operating within an organizational silo. And as I tried to show in Part 3 (Ignoring complexity can make sense), operating within silos is rational and functional. I am by no means arguing that the model in figure 2 is in any way better than the model in figure 1, or that programs must be designed and evaluated with respect to one or the other. My only point in this blog post is to show that there is complex system behavior in the form of evolutionary adaptation that is likely to cause unintended undesirable consequences when efforts are made to pursue a set of highly correlated outcomes.
