What does evaluation gain from thinking in alien terms? An argument for taking complexity and evolutionary biology seriously

For a long time, I have been arguing that if “complexity” is to be useful in evaluation, evaluators’ should focus on what complex systems do, rather than on what complex systems are. This is because by focusing on behavior, we can make practical decisions about models, methodologies, and metrics.

I still believe this, but I’m also coming to appreciate that thinking within research traditions also matters. I’m not advocating a return to a “complex system” focus, but I do see value in adopting the perspectives of people who do research and develop theory in the domain of complexity. And by extension, this is also true for evolutionary biology, another field that I have been promoting as being useful for evaluators.

Continue reading “What does evaluation gain from thinking in alien terms? An argument for taking complexity and evolutionary biology seriously”

Complex behavior can be evaluated using comfortable, familiar methodologies – Part 4 of a 10-part series on how complexity can produce better insight on what programs do, and why

Common introduction to all sections

This is part 4 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.

I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.

Part Title Approximate post date
1 Complex systems or complex behavior? up
2 Complexity has awkward implications for program designers and evaluators up
3 Ignoring complexity can make sense up
4 Complex behavior can be evaluated using comfortable, familiar methodologies up
5 A pitch for sparse models 7/1
6 Joint optimization of unrelated outcomes 7/8
7 Why should evaluators care about emergence? 7/16
8 Why might it be useful to think of programs and their outcomes in terms of attractors? 7/19
9 A few very successful programs, or many, connected, somewhat successful programs? 7/24
10 Evaluating for complexity when programs are not designed that way 7/31

This blog post will give away much of what is to come in the other parts, but that’s OK. One reason it’s OK is that it’s never a bad thing to cover the same material twice, each time in a somewhat different way. The other reason it’s OK is that that before getting into the details of complex behavior and its use in evaluation, an important message needs to be internalized. Namely, that the title of this blog post is in fact correct. Complex behavior can be evaluated using comfortable, familiar methodologies.  

Figure 1 illustrates why this is so. It depicts a healthy eating program whose function is to reach out to individuals and teach them about dieting and exercise. Secondary effects are posited because attendees interact with friends and family. It is thought that because of that contact, four kinds of outcomes may occur.

  • Friends and family pick up some of the information that was transmitted to program attendees, and improve their personal health related behavior.
  • Collective change occurs within a family or cohort group, resulting in desirable health improvements, even though the specific changes cannot be identified in advance.
  • There may be community level changes. For instance, consider two examples: 1) An aggregate improvement in the health of people in a community may change their energy for engaging in volunteer behavior. The important outcome is not the number of hours each person puts in. The important outcome is what happens in the community because of those hours. 2) Better health may result in people working more hours, and, hence earning more money. Income is an individual level outcome, but the consequences of increased wealth in the community is a community level outcome.
  • To cap it all off, there is a feedback loop between the accomplishments of the program and what services the program delivers. So over time, the program’s outcomes may change as the program adapts to the changes it has wrought.
Evaluating Complex Behavior With Common, Familiar Methodologies

Even without a formal definition of complexity, I think we would all agree that this is a complex system. There are networks embedded in networks. There are community-level changes that cannot be understood by “summing” specific changes in friends and family. There are influences among the people receiving direct services. Program theory can identify health changes that may occur, but it is incapable of specifying any of the other changes that may occur. There is a feedback loop whereby the effects of the program influence the services the program delivers. And what methodologies are needed to deal with all this complexity? They are in the Table 1. Everything there are methods that most evaluators can either do themselves or can easily recruit colleagues who can.

Table 1: Familiar Methodologies to Address Complex Behaviors
Program Behavior Methodology
Feedback between services and impact
  • Service records
  • Budges and plans
  • Interviews with staff
Community level change
  • Monitoring
  • Observation
  • Open ended interviewing
  • Content analysis of community social media
Direct impact on participants
  • Interviews
  • Exercise logs
  • Food consumption logs
  • Blood pressure / weight measures

There are two exceptions to the “comfortable, familiar methodology” principle. The first would be cases where formal network structure mattered. For instance, imagine that it were not enough to show that network behavior was at play in the healthy eating example, but that the structure of the network and its various centrality measures were important for understanding the program outcomes. In that case one would need specialized expertise and software. The second case would be a scenario where it would further the evaluation if the program were modeled in a computer simulation. Those kinds of models are useless for predicting how a program will behave, but they are very useful for getting a sense of the program’s performance envelope, and testing assumptions about relationships between program and outcome. If any of that mattered, one would need specialized expertise in system dynamic or agent-based modeling, depending on one’s view of how the world works and what information one wanted to know.

Embracing Uncertainty: The case for mindful development

 

Guy Sharrock, Catholic Relief Services

There is a growing awareness that many aspects of economic and social development are complex, unpredictable, and ultimately uncontrollable. Governments, non-governmental organizations, and international agencies have realized the need for a change in emphasis; a paradigm shift is taking place away from predominantly linear and reductionist models of change to approaches that signal a recognition of the indeterminate, dynamic and interconnected nature of social behavior.

Over the last few years many international NGOs have been adopting a more adaptive approach to project management often with reference to USAID’s ‘Collaborating, Learning and Adapting’ (CLA) framework and model. In the case of Catholic Relief Services this work builds on earlier and not unrelated capacity strengthening interventions – still ongoing – in which projects are encouraged to embed ‘evaluative thinking’ (ET) (Buckley et al., 2015) into their modus operandi.

Ellen Langer, in her excellent book The Power of Mindful Learning (Langer, 1997) introduces the notion of ‘mindfulness’. This concept, underpinned by many years of research, can be understood as being alert to novelty – intentionally “seeking surprise” (Guijt, 2008) – introducing in a helpful manner a sense of uncertainty to our thinking and thereby establishing a space for ‘psychologically safe’ learning (Edmondson, 2008) and an openness to multiple perspectives. This seems to me very applicable to the various strands of CLA and ET work in which I’ve been recently engaged; Langer’s arguments for mindful learning seem as applicable to international development as they are to her own sector of research interest, education. To coin the language of Lederach (2007), Langer seems to “demystify” the notion of mindfulness whilst at the same time offering us the chance to “remystify” the practice of development work that seeks to change behavior and support shifts in social norms. This is both essential and overdue for development interventions occurring in complex settings.

A mindful approach to development would seek to encourage greater awareness in the present of how different people on the receiving end of aid adapt (or not) their behavior in response to project interventions; in short, a willingness to go beyond our initial assumptions through a mindful acceptance that data bring not certainty but ambiguity. According to Langer, “in a mindful state, we implicitly recognize that no one perspective optimally explains a situation…we do not seek to select the one response that corresponds to the situation, but we recognize that there is more than one perspective on the information given and we choose from among these.” (op. cit..: 108). Mindful development encourages a learning climate in which uncertainty is embraced and stakeholders intentionally surface and value novelty, difference, context, and perspective to generate nuanced understandings of the outcome of project interventions. Uncertainty is the starting point for addressing complex challenges and a willingness to “spend more time not knowing” (Margaret Wheatley, quoted in Kania and Kramer, 2013) before deciding on course corrections if needed. As Kania and Kramer (ibid.: 7) remark, “Collective impact success favors those who embrace the uncertainty of the journey, even as they remain clear-eyed about their destination.”

References

Buckley, J., Archibald, T., Hargraves, M. and W.M. Trochim. (2015). ‘Defining and Teaching Evaluative Thinking: Insights from Research on Critical Thinking’. American Journal of Evaluation, pp. 1-14.

Edmondson, A. (2014). Building a Psychologically Safe Workplace. Retrieved from: https://www.youtube.com/watch?v=LhoLuui9gX8

Guijt, I. (2008). Seeking Surprise: Rethinking Monitoring for Collective Learning in Rural Resource Management. Published PhD thesis, Wageningen University, Wageningen, The Netherlands.

Kania, J. and M. Kramer. (2013) ‘Embracing Emergence: How Collective Impact Addresses Complexity’. Stanford Social Innovation Review. Stanford University, CA.

Langer, Ellen J. (1997). The Power of Mindful Learning. Perseus Books, Cambridge, MA.

Lederach, J.P., Neufeldt, R. and H. Culbertson. (2007). Reflective Peacebuilding. A Planning, Monitoring, and Learning Toolkit. Joan B. Kroc Institute for International Peace Studies, University of Notre Dame, South Bend, IN, and Catholic Relief Services, Baltimore, MD.

 

Ignoring complexity can make sense – Part 3 of a 10-part series on how complexity can produce better insight on what programs do, and why 

Common Introduction to all sections

This is part 3 of 10 blog posts I’m writing to convey the information that I present in various workshops and lectures that I deliver about complexity. I’m an evaluator so I think in terms of evaluation, but I’m convinced that what I’m saying is equally applicable for planning.

I wrote each post to stand on its own, but I designed the collection to provide a wide-ranging view of how research and theory in the domain of “complexity” can contribute to the ability of evaluators to show stakeholders what their programs are producing, and why. I’m going to try to produce a YouTube video on each section. When (if?) I do, I’ll edit the post to include the YT URL.

Part Title Approximate post date
1 Complex systems or complex behavior? up
2 Complexity has awkward implications for program designers and evaluators up
3 Ignoring complexity can make sense up
4 Complex behavior can be evaluated using comfortable, familiar methodologies up
5 A pitch for sparse models up
6 Joint optimization of unrelated outcomes up
7 Why should evaluators care about emergence? up
8 Why might it be useful to think of programs and their outcomes in terms of attractors? up
9 A few very successful programs, or many, connected, somewhat successful programs? up
10 Evaluating for complexity when programs are not designed that way up

Ignoring complexity can make sense

Complexity is in large measure about connectedness. It is about what happens when processes and entities combine or interact. I believe that understanding complex connectedness will make for better models, and hence for more useful methodologies and data interpretation. Of course I believe this. Why else would I be producing all these blog posts and videos?

Still, I would be remiss if I did not advance a contrary view, i.e. that avoiding the implications of complexity can be functional and rational. In fact, it is usually functional and rational. I don’t think evaluators can do a good job if they fail to appreciate why this is so. It’s all too easy to jump to the conclusion that program designers “should” build complex behavior into their designs. I can make a good argument that they should not.

The difference between Figure 1 and Figure 2 illustrates what I mean. Every evaluation I have been involved with comes out of the organizational structure depicted in Figure 1. A program has internal operations (blue). Those operations produce consequences (pink). There is a feedback loop between what the program does and what it accomplishes. Real world cases may have many more parts, but qualitatively they are all the same picture.Figure 2 illustrate s how programs really operate. The core of the program is still there, color coded in the same pink and blue. However, that program contains embedded detail (dark blue and dark red), and is connected to a great deal of activity and organizational structure outside of its immediate boundaries (green, gray, yellow, and white.)

Figure 1
Figure 2

The people working in the program certainly know about  these complications. They also know that those complications affect the program they are managing. So why not act on that knowledge? There are good reasons. Think about what would be involved in taking all those relationships into account.

  • Different stakeholders will have different priorities.
  • Different organizational cultures would have to work with each other.
  • Goals among the programs may conflict and would have to be negotiated.
  • Different programs are likely to have different schedules for decision making.
  • The cost of coordination in terms of people, money, and time would increase.
  • Different time horizons for the different activities would have to be reconciled.
  • Interactions among the programs would have to be built into program theory and evaluation.
  • Program designers would have to interact with people they don’t know personally, and don’t trust.
  • Each program will have different contingencies, which instead of affecting a narrow program, would affect the entire suite of programs.

That’s the reality. I’d say its rational to work within narrow constraints, no matter how acutely aware people are of the limitations of doing so.

Can Knowledge of Evolutionary Biology and Ecology Inform Evaluation?

I posted a longish piece (~3,500 words) on my website with the same title as this post. Section headings are:

  • Case: Early childhood parent support
    • Program design
    • Evaluation design
  • Some useful concepts from evolutionary biology and ecology
    • Population
    • Coevolution
    • Birth/death rates
    • Selection pressure
    • Species and species variation
  • What would the evaluation look like if its design were informed by knowledge of evolutionary biology and ecology?
    • Populations, and birth/death rates
    • Coevolution and population
    • Selection pressure
    • Species and species variation
  • Do we gain anything from applying an evolutionary lens?
    • Paradigmatic concepts
    • Methodology

I’m always interested in having people point out the error of my ways.