What does evaluation gain from thinking in alien terms? An argument for taking complexity and evolutionary biology seriously
For a long time, I have been arguing that if “complexity” is to be useful in evaluation, evaluators’ should focus on what complex systems do, rather than on what complex systems are. This is because by focusing on behavior, we can make practical decisions about models, methodologies, and metrics.
I still believe this, but I’m also coming to appreciate that thinking within research traditions also matters. I’m not advocating a return to a “complex system” focus, but I do see value in adopting the perspectives of people who do research and develop theory in the domain of complexity. And by extension, this is also true for evolutionary biology, another field that I have been promoting as being useful for evaluators.
Deriving Value from Models: Appreciating Partial Truth, Simplification, and Multiple Uses
I’m giving a talk at AEA 2019 in Minneapolis. The graphic summarizes the content. For the slides go here.
AEA Professional development workshop on using complexity in evaluation
I’m conducting a Professional Development session on complexity at AEA 2019 in Minneapolis. The graphic summarizes the content. For the slides go here.
Complex behavior can be evaluated using comfortable, familiar methodologies – Part 4 of a 10-part series on how complexity can produce better insight on what programs do, and why
Common introduction to all sections
This is part 4 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.
I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.
| Part | Title | Approximate post date |
| 1 | Complex systems or complex behavior? | up |
| 2 | Complexity has awkward implications for program designers and evaluators | up |
| 3 | Ignoring complexity can make sense | up |
| 4 | Complex behavior can be evaluated using comfortable, familiar methodologies | up |
| 5 | A pitch for sparse models | 7/1 |
| 6 | Joint optimization of unrelated outcomes | 7/8 |
| 7 | Why should evaluators care about emergence? | 7/16 |
| 8 | Why might it be useful to think of programs and their outcomes in terms of attractors? | 7/19 |
| 9 | A few very successful programs, or many, connected, somewhat successful programs? | 7/24 |
| 10 | Evaluating for complexity when programs are not designed that way | 7/31 |
This blog post will give away much of what is to come in the other parts, but that’s OK. One reason it’s OK is that it’s never a bad thing to cover the same material twice, each time in a somewhat different way. The other reason it’s OK is that that before getting into the details of complex behavior and its use in evaluation, an important message needs to be internalized. Namely, that the title of this blog post is in fact correct. Complex behavior can be evaluated using comfortable, familiar methodologies.
Figure 1 illustrates why this is so. It depicts a healthy eating program whose function is to reach out to individuals and teach them about dieting and exercise. Secondary effects are posited because attendees interact with friends and family. It is thought that because of that contact, four kinds of outcomes may occur.
- Friends and family pick up some of the information that was transmitted to program attendees, and improve their personal health related behavior.
- Collective change occurs within a family or cohort group, resulting in desirable health improvements, even though the specific changes cannot be identified in advance.
- There may be community level changes. For instance, consider two examples: 1) An aggregate improvement in the health of people in a community may change their energy for engaging in volunteer behavior. The important outcome is not the number of hours each person puts in. The important outcome is what happens in the community because of those hours. 2) Better health may result in people working more hours, and, hence earning more money. Income is an individual level outcome, but the consequences of increased wealth in the community is a community level outcome.
- To cap it all off, there is a feedback loop between the accomplishments of the program and what services the program delivers. So over time, the program’s outcomes may change as the program adapts to the changes it has wrought.
Even without a formal definition of complexity, I think we would all agree that this is a complex system. There are networks embedded in networks. There are community-level changes that cannot be understood by “summing” specific changes in friends and family. There are influences among the people receiving direct services. Program theory can identify health changes that may occur, but it is incapable of specifying any of the other changes that may occur. There is a feedback loop whereby the effects of the program influence the services the program delivers. And what methodologies are needed to deal with all this complexity? They are in the Table 1. Everything there are methods that most evaluators can either do themselves or can easily recruit colleagues who can.
| Table 1: Familiar Methodologies to Address Complex Behaviors | |
| Program Behavior | Methodology |
| Feedback between services and impact |
|
| Community level change |
|
| Direct impact on participants |
|
There are two exceptions to the “comfortable, familiar methodology” principle. The first would be cases where formal network structure mattered. For instance, imagine that it were not enough to show that network behavior was at play in the healthy eating example, but that the structure of the network and its various centrality measures were important for understanding the program outcomes. In that case one would need specialized expertise and software. The second case would be a scenario where it would further the evaluation if the program were modeled in a computer simulation. Those kinds of models are useless for predicting how a program will behave, but they are very useful for getting a sense of the program’s performance envelope, and testing assumptions about relationships between program and outcome. If any of that mattered, one would need specialized expertise in system dynamic or agent-based modeling, depending on one’s view of how the world works and what information one wanted to know.
Embracing Uncertainty: The case for mindful development
Guy Sha
rrock, Catholic Relief Services
There is a growing awareness that many aspects of economic and social development are complex, unpredictable, and ultimately uncontrollable. Governments, non-governmental organizations, and international agencies have realized the need for a change in emphasis; a paradigm shift is taking place away from predominantly linear and reductionist models of change to approaches that signal a recognition of the indeterminate, dynamic and interconnected nature of social behavior.
Over the last few years many international NGOs have been adopting a more adaptive approach to project management often with reference to USAID’s ‘Collaborating, Learning and Adapting’ (CLA) framework and model. In the case of Catholic Relief Services this work builds on earlier and not unrelated capacity strengthening interventions – still ongoing – in which projects are encouraged to embed ‘evaluative thinking’ (ET) (Buckley et al., 2015) into their modus operandi.
Ellen Langer, in her excellent book The Power of Mindful Learning (Langer, 1997) introduces the notion of ‘mindfulness’. This concept, underpinned by many years of research, can be understood as being alert to novelty – intentionally “seeking surprise” (Guijt, 2008) – introducing in a helpful manner a sense of uncertainty to our thinking and thereby establishing a space for ‘psychologically safe’ learning (Edmondson, 2008) and an openness to multiple perspectives. This seems to me very applicable to the various strands of CLA and ET work in which I’ve been recently engaged; Langer’s arguments for mindful learning seem as applicable to international development as they are to her own sector of research interest, education. To coin the language of Lederach (2007), Langer seems to “demystify” the notion of mindfulness whilst at the same time offering us the chance to “remystify” the practice of development work that seeks to change behavior and support shifts in social norms. This is both essential and overdue for development interventions occurring in complex settings.
A mindful approach to development would seek to encourage greater awareness in the present of how different people on the receiving end of aid adapt (or not) their behavior in response to project interventions; in short, a willingness to go beyond our initial assumptions through a mindful acceptance that data bring not certainty but ambiguity. According to Langer, “in a mindful state, we implicitly recognize that no one perspective optimally explains a situation…we do not seek to select the one response that corresponds to the situation, but we recognize that there is more than one perspective on the information given and we choose from among these.” (op. cit..: 108). Mindful development encourages a learning climate in which uncertainty is embraced and stakeholders intentionally surface and value novelty, difference, context, and perspective to generate nuanced understandings of the outcome of project interventions. Uncertainty is the starting point for addressing complex challenges and a willingness to “spend more time not knowing” (Margaret Wheatley, quoted in Kania and Kramer, 2013) before deciding on course corrections if needed. As Kania and Kramer (ibid.: 7) remark, “Collective impact success favors those who embrace the uncertainty of the journey, even as they remain clear-eyed about their destination.”
References
Buckley, J., Archibald, T., Hargraves, M. and W.M. Trochim. (2015). ‘Defining and Teaching Evaluative Thinking: Insights from Research on Critical Thinking’. American Journal of Evaluation, pp. 1-14.
Edmondson, A. (2014). Building a Psychologically Safe Workplace. Retrieved from: https://www.youtube.com/watch?v=LhoLuui9gX8
Guijt, I. (2008). Seeking Surprise: Rethinking Monitoring for Collective Learning in Rural Resource Management. Published PhD thesis, Wageningen University, Wageningen, The Netherlands.
Kania, J. and M. Kramer. (2013) ‘Embracing Emergence: How Collective Impact Addresses Complexity’. Stanford Social Innovation Review. Stanford University, CA.
Langer, Ellen J. (1997). The Power of Mindful Learning. Perseus Books, Cambridge, MA.
Lederach, J.P., Neufeldt, R. and H. Culbertson. (2007). Reflective Peacebuilding. A Planning, Monitoring, and Learning Toolkit. Joan B. Kroc Institute for International Peace Studies, University of Notre Dame, South Bend, IN, and Catholic Relief Services, Baltimore, MD.
Ignoring complexity can make sense – Part 3 of a 10-part series on how complexity can produce better insight on what programs do, and why
Common Introduction to all sections
This is part 3 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.
I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.
| Part | Title | Approximate post date |
| 1 | Complex systems or complex behavior? | up |
| 2 | Complexity has awkward implications for program designers and evaluators | up |
| 3 | Ignoring complexity can make sense | up |
| 4 | Complex behavior can be evaluated using comfortable, familiar methodologies | up |
| 5 | A pitch for sparse models | up |
| 6 | Joint optimization of unrelated outcomes | up |
| 7 | Why should evaluators care about emergence? | up |
| 8 | Why might it be useful to think of programs and their outcomes in terms of attractors? | up |
| 9 | A few very successful programs, or many, connected, somewhat successful programs? | up |
| 10 | Evaluating for complexity when programs are not designed that way | up |
Ignoring complexity can make sense
Complexity is in large measure about connectedness. It is about what happens when processes and entities combine or interact. I believe that understanding complex connectedness will make for better models, and hence for more useful methodologies and data interpretation. Of course I believe this. Why else would I be producing all these blog posts and videos?
Still, I would be remiss if I did not advance a contrary view, i.e. that avoiding the implications of complexity can be functional and rational. In fact, it is usually functional and rational. I don’t think evaluators can do a good job if they fail to appreciate why this is so. It’s all too easy to jump to the conclusion that program designers “should” build complex behavior into their designs. I can make a good argument that they should not.
The difference between Figure 1 and Figure 2 illustrates what I mean. Every evaluation I have been involved with comes out of the organizational structure depicted in Figure 1. A program has internal operations (blue). Those operations produce consequences (pink). There is a feedback loop between what the program does and what it accomplishes. Real world cases may have many more parts, but qualitatively they are all the same picture.Figure 2 illustrate s how programs really operate. The core of the program is still there, color coded in the same pink and blue. However, that program contains embedded detail (dark blue and dark red), and is connected to a great deal of activity and organizational structure outside of its immediate boundaries (green, gray, yellow, and white.)
The people working in the program certainly know about these complications. They also know that those complications affect the program they are managing. So why not act on that knowledge? There are good reasons. Think about what would be involved in taking all those relationships into account.
- Different stakeholders will have different priorities.
- Different organizational cultures would have to work with each other.
- Goals among the programs may conflict and would have to be negotiated.
- Different programs are likely to have different schedules for decision making.
- The cost of coordination in terms of people, money, and time would increase.
- Different time horizons for the different activities would have to be reconciled.
- Interactions among the programs would have to be built into program theory and evaluation.
- Program designers would have to interact with people they don’t know personally, and don’t trust.
- Each program will have different contingencies, which instead of affecting a narrow program, would affect the entire suite of programs.
That’s the reality. I’d say its rational to work within narrow constraints, no matter how acutely aware people are of the limitations of doing so.
