jamorell Uncategorized, What do research and theory in complexity and systems tell us about evaluation practice and evaluation theory?

The application of Social-Ecological Systems (SES) thinking to evaluation

 

Aaron Zazueta a@zazuetagroup.com    

Among the many approaches to complex systems thinking, the Social-Ecological Systems (SES) proponents have developed a set of concepts to understand and model the interlinked dynamics of social and environmental change. When addressing the transformation of large complex systems, I find particularly useful the SES  concepts of the Social-Ecological Systems, boundaries, domains, scales, agents, adaptive behavior, and emergence and system development trajectory.  I have used these concepts to construct theories of change (TOCs), to help me understand how and what extent projects, programs, or policies interact with social-ecological systems to steer development processes in the direction of a given trajectory or policy goals. The notion that all systems are composed of subsystems that are interconnected helps focus the attention on the phenomena relevant to the desired policy goals. The concepts of domains help to identify further the critical conditions that can enable or hamper change in the direction of a given development trajectory (such conditions can include the presence of sound governance, the availability of knowledge and technology to solve a problem or the necessary institutional capacities). As domains cut across the whole SES, the concept of a domain is helpful to trace system interactions across different scales of space and time.

The agents and their adaptive behavior underlie the phenomena that encompass the system and its components. SES scholars assume that systems operate through the actions and reactions of the agents (the agents’ adaptive behavior). While agents command different resources and are influenced differently by the conditions in the various domains, they are linked, either directly or through other agents). Even relatively minor agents under the right conditions can generate reverberating behaviors across the system. The aggregated adaptive behavior of the agents responding to other agents and other factors external to the system result in the emergence of system-level shapes that can be quite different from the behaviors of the agents. Based on these concepts, it is possible to develop a model of the conditions that are likely to steer agent behavior direction and the extent to which an intervention is has contributed or is likely enable adaptive behavior consistent with the desired long term policy goals.

SES also assumes that the adaptive behavior of the agents also contributes to various degrees of unpredictability and non-linearity. It is thus important not to expect that in complex systems, outputs or results will correspond to inputs. Therefore, when dealing with complex systems, effective development interventions are those which mimic other agents in the system and adopt adaptive management as the approach to steer the development trajectory of the system. Adaptive management entails clear long-term goals (or trajectory direction), identification of alternative management objectives,  the development of a set of the hypotheses of causation and procedures for data collection to adjust hypotheses (ongoing evaluation).

 

jamorell What do research and theory in complexity and systems tell us about evaluation practice and evaluation theory?

Traditions of ‘Complexity and Systems Science’?

Martin Reynolds (The Open University). Applied Systems Thinking in Practice (ASTiP) Group. School of Engineering and Innovation. The Open University, Walton Hall, Milton Keynes MK7 6AA, United Kingdom +44 (0) 1908 654894 | martin.reynolds@open.ac.uk |  Profile | Publications

From a systems thinking in practice (STiP) tradition I would first like to change the formulation from ‘complexity and systems science’ to complexity science and systems thinking (cf. Reynolds et al., 2016). The revised formulation is important for two reasons in appreciating respective lineages. First, contemporary ideas on complexity including the ‘butterfly effect’ and ‘complex adaptive systems’ are very much rooted in the scientific tradition dating from Warren Weaver’s 1947 paper ‘science and complexity’. Second, contemporary systems thinking should be regarded as a transdisciplinary endeavour inclusive of systems science and complexity science, but far beyond the confines of a scientific discipline (Reynolds and Howell, 2020). Note that systems science and complexity science have many common lineages, including pioneering work around cybernetics in the 1940s.  Appreciating the value of complexity science and systems thinking requires in my view attention to the ontological and epistemological dimensions of appreciating complexity and systems.

Complexity as used in complexity science invokes the scientific ontological (real world) premise that everything connects. Ideas of uncertainty and emergence are tied to appreciating reality as an infinite network of interconnections, the effects of which are impossible to precisely predict.  Systems science might be regarded as an endeavor to systematically bound such interconnections,  recognized by an impartial observer as relevant to a particular situation of interest.  By so doing, the ensuing bounded systems might be subject to scientific analysis. In systems science and complexity science, the key epistemological driver is positivism; there being an assumed direct representation between reality and systems (ontological realism; e.g. ‘the’ health system), subject to inquiry from an impartial ‘objective’ observer (scientist).

In contrast, complexity as used in a STiP tradition is an effect of contrasting human perspectives on the framing of interconnections, rather than an effect of interconnections directly. In the STiP tradition ‘systems’ as ontological representations of reality are legitimate, but the representations are always nominal (named by a human ‘observer’), provisional (with boundaries subject to change from other observers), and secondary.   Nominal systems such as (i) natural systems (individual organisms, ecosystems, solar system etc.); or (ii) engineered (purposive) systems (mechanical devices ranging from computers to heating systems), are secondary to a primary understanding and active use of systems as conceptual constructs which may be referred to as (iii) human (purposeful) systems.  Purposeful systems (where the bounded purposes are subject to ongoing adaptive change) are a powerful tool of contemporary STiP.  As distinct from ‘seeing’ reality only as natural or engineered systems,  purposeful systems enable such viewings to be tamed within a primary framing of a learning system (as an epistemological construct).  Such primary framings enable organisations, and interventions in education, health, etc. to be not only evaluated but (re)designed.  The STiP tradition, founded on epistemological constructivism, recognizes complexity as an effect of contrasting viewpoints on reality. Complexity here is a second-order attribute of interconnections in situations of interest – the indirect human framings of interconnections.  Complexity in complexity science is a first-order attribute of the interconnections themselves.

The difference is significant for all practitioners in all professional fields.   In a STiP tradition, complexity exists in all situations (since no human situation comprises only one perspective).   Each individual or group of individuals frame things differently depending on lifeworld experiences including, amongst other demographics, ethnic backgrounds.  STiP flushes out the framings of situations in terms of transparent purposeful systems in order to help improve the situations through more meaningful conversation amongst practitioners.  With increasingly uncertain times where racism is being called out internationally through the killing of black American George Floyd, it is perhaps worth recalling the founding principle of  STiP which takes its cue from C.West Churchman: “a systems approach begins when you see the world through the eyes of another” (1968 p. 23).

 

 

 

 

jamorell Contributions of Evolutionary Biology and Ecology

Webinar slides and recording for “Using Ecology to Evaluate Policy”

I recently presented a webinar to SAMEA. Slides and a recording of the webinar can be found here.

Webinar description
Is it worth our attention to look to Ecology and Evolutionary Biology for help in developing program theory, choosing methodology, and interpreting data? It is when we evaluate policy because policy change affects groups of connected programs and program environments, as those programs and environments evolve over time. As with all disciplines, Ecology and Evolutionary Biology offer unique ways to identify topics to research, develop models, generate hypotheses, choose among methodological tools, define data needs, specify acceptable answers, and interpret data. Each of these discrete elements has its own utility. But perhaps more important, each element is a node in a network: decisions about each have implications for the others. Under these conditions, a unique analytical mindset emerges. The potential contribution of ecological and evolutionary biological thinking to Evaluation lies both in that mindset, and in the utility of discrete tools and concepts.

jamorell What do research and theory in complexity and systems tell us about evaluation practice and evaluation theory?

Constructing a Deep Complexity and Systems Science Foundation for the Field of Evaluation

Constructing a Deep Complexity and Systems Science Foundation for the Field of Evaluation

Jonny Morell, PhD
Meg Hargreaves PhD

This section of the blog Evaluation Uncertainty, Surprises in Programs and their Evaluations is an effort to continue and expand prior work setting evaluation theory and evaluation practice within a deep understanding of complexity and systems science research and theory. Four beliefs motivate our current efforts.

  • Not all evaluation needs to invoke systems concepts, but much of it does.
  • When system concepts are needed, evaluators must use those concepts to make operational decisions about the theoretical frameworks and models they construct, the methodologies they devise, and the meaning they draw from data.
  • To make those wise decisions, evaluators require deep-seated understanding of the research and theory that has been generated by complexity and system science.
  • At present, too little of evaluation’s reliance on systems is based on such deep understanding.

“This section of the blog”, and “begin” and are important phrases in the opening paragraph. We hope that what we are doing here will spur a wider range of discussion and inquiry.

We envision two kinds of contributions to this blog. To start, posts will say nothing at all about evaluation. Rather, evaluators will provide explanations of the intellectual domains within complexity and system science that affect their work. We see those contributions as providing an imperfect, but reasonably wide view of the domains of complexity and system science that influence how evaluation is done. We want these posts to be short, somewhere between one and four paragraphs.

Once a there is sufficient variety of material, contributions will then broaden to a wider set of evaluators, more discussion of complexity and system roots, and examples of how research and theory in the fields of complexity and systems affected practical decisions about theoretical frameworks, models, metrics, and methodologies.

jamorell Complex Systems, Programs, and Evaluation, Program Theory and Program Logic

A Complexity-based Meta-theory of Action for Transformation to a Green Energy Future

This is the abstract for a book chapter I am writing on the evaluation of transformation. It is a draft and will undoubtedly change quite a bit by the time it is published.

This chapter draws from complexity science to present a metatheory of transformation that can be applied to discrete theories of change that are constructed to guide model building, methodology, and data interpretation for the evaluation of individual change efforts. The focus is on six specific behaviors of complex systems – stigmergy, attractors, emergence, phase transition, self-organization, and path dependence. These can be  invoked singly or in combination to understand pattern, predictability, and how change happens. The importance of both “explanation” and “prediction” is woven into the discussion. A definition of “transformation” is offered in which a qualitatively new reality becomes the default choice that constitute a new normal. Indicators of transformation include measurable ranges (as opposed to specific values) for level of energy  use and the time over which the change endures. Because complex systems behave as they do, the recommended theory of change is sparse – it has few well-defined elements or relationships among those elements. There is already good progress in the application of complexity to the evaluation of transformation. An argument is made that these efforts should be strengthened by deliberately incorporating what is known about complex system behavior, and that by so doing, both prediction and explanation would better serve the purpose of practical decision making.

Here is the entire chapter.

Metatheory of transformation 05_15_2020

jamorell Contributions of Evolutionary Biology and Ecology

Using the Biotic Hierarchy as a Framework for Moving Policy Evaluation Away from a Program Evaluation-like Activity

PDF version Biotic_Hierarchy_as_a_Framework_for_Policy_Evaluation

In a previous blog post and in one of my You Tube videos I tried to make the case that different disciplines orient problem definition and solution in different ways, and that evolutionary biology and ecology lead researchers in different directions than do the social sciences. I followed this line of reasoning with the argument that evaluators’ traditional grounding in social science normally has been productive and that for the most part, we should continue to do what we have always done. Then, I tried to make the point that there are circumstances where thinking in terms of evolutionary biology and ecology can be a powerful addition to how we normally do our work. In this blog post I will show how policy evaluation is one of those circumstances when evolutionary / ecological thinking can be valuable. Continue reading “Using the Biotic Hierarchy as a Framework for Moving Policy Evaluation Away from a Program Evaluation-like Activity” →

jamorell Evaluation as a Discipline and a Profession

AEA’s Potential to Serve the Public Good

What you see here is a brief overview of my recent thoughts about the field of evaluation and the future of AEA. See the PDF for a more fleshed out explanation. Fair warning, it runs to 2,500 words. I am not advocating change, I’m only advocating that we recognize where we are and where we are going, and that we contemplate the consequences. AEA_Evaluation_Evolutionary_Path_Long

The challenge of contending points of view in a representative democracy goes back to the founding of the Republic. James Madison saw the polity as a collection of “factions” and believed that a stable democracy required a diversity of contending factions.

One way to think about the purpose of evaluation is to see it as an honest broker to which those factions can turn. This is not to say that any given evaluation can be, or should be, “objective”. Supporters of one or another point of view will inevitably find the results of any single evaluation wanting.

The question is whether over time, and across evaluations, it matters whether we are seen as taking sides. That perception has consequences for evaluation use, and I can see how outsiders would get the impression that we do take sides. If they do, they will shop elsewhere for evaluation information.

I’m not sure if we can or should change. But I do not think we should continue on our path without awareness of where that path is taking us.

 

 

jamorell Contributions of Evolutionary Biology and Ecology

Evolutionary Biology and Ecology as a Valuable Framework for Some Evaluation

I put together a slide deck on the advantages of adding constructs from evolutionary biology and ecology to traditional evaluation. Adding those constructs is worth the trouble when the focus is on:

  • populations of a program types, and
  • rates of change over time

Specific topics covered are:

  • Policy as ecosystem change
  • Evolution of program forms
  • Sustainability in terms of ecosystems as attractors
  • Use of ecosystem analysis to interpret outcome evaluation
  • Using population trends to interpret data on program outcome

Go here for the slides. UD_EBT_Evaluation_Meeting