Category: What do research and theory in complexity and systems tell us about evaluation practice and evaluation theory?

Beverly Parsons
Executive Director, InSites
www.insites.org
bparsons@insites.org
Autopoiesis is one of my favorite systems concepts because of its importance in helping us understand a crucial difference between mechanistic systems and  living systems. The term was coined by Humberto Maturana, a Chilean biologist. It means “self-making” or “self-producing” (the combination of auto meaning “self” and poiesis meaning “making”). In the 1970s, Maturana and his colleague, Francisco Varela, built their theory about what is life from observing how biological cells function. Maturana and Varela viewed the main characteristic of life as self-maintenance through the “internal networking of a chemical system that continuously reproduces itself within a boundary of its own making”.[1]

There are many transformations continually going on in a biological cell while at the same time “there is cellular self-maintenance—the fact that the cell maintains its individuality”.[2] A person, a tree, a bear, and a flower all differ from a chair, a computer, a television, and a glass cup in that the items in the first group engage in self-maintenance via a mechanism of self-regeneration from within whereas this doesn’t happen in the second group[3].

It might seem that an autopoietic system is a closed system but, no. There’s an important distinction between the organization and the structure of a system. “The former refers to the relationships between components which are necessary to define that system as part of a particular class of systems; the latter to the particular physical form which those components take.”[4]

An autopoietic system (e.g., a cell) can retain its organization while the environment around it is changing; it is organizationally closed. At the same time, it can be structurally open, allowing energy and matter to flow in and out of it. An autopoietic system, i.e., a living system, must retain its basic organization to stay alive. Marturana and Varela developed other related concepts (e.g., structural coupling) which address important matters of how an autopoietic system changes in relation to its environment. They also did important work together around the biology of cognition. Debates continue about the applicability of autopoiesis at the level of social systems.[5]

References
Capra, F., & Luigi Luisi, P. (2014). The systems view of life: A unifying vision. Cambridge: Cambridge University Press.
Ramage, M. & Shipp, K. (2009). System Thinkers. London: Springer.
Scharmer, O. (2019). Social systems as if people mattered: Response to the Kuhl Critique of Theory U. accepted for publication at J. Change Management. https://www.presencing.org/critiques/social-systems-as-if-people-mattered
Article at:
https://d293qag76e45fv.cloudfront.net/e26ba972-a6df-9db6-b27e-ad71a4a377a8.pdf

[1] Capra and Luisi (2014). p. 129.
[2] Ibid, p. 130.
[3] Ibid, p. 132.
[4] Ramage and Shipp (2009). p. 201.
[5] Scharmer. (2019).

Meg Hargreaves Senior Fellow, Economics, Justice, and Society Department, NORC	Jonny Morell President, 4.669… Evaluation and Planning

Development Along Two Directions
We are looking for suggestions about content and authors for posts on:

• research and theory in complexity and systems, and
• application to evaluation.Posts should be short and focused.

This tactic may not fit the spirit of “systems”, but it will educate and not overwhelm.

Please contact us if you have ideas to share.
To convey a sense of what we have in mind, here is an example.
Jonny’s Example based on Accident Investigation
Consider accident investigation and its relationship to path dependence. It is possible to trace causation in retrospect and to use that knowledge to minimize the likelihood that a class of accidents will reoccur. One would be foolish ignore these analyses. But precisely what accidents will be affected by the change? Unknown and largely unknowable. Is there any certainty that other causal paths won’t lead to the same type of accidents? No certainty at all.
I’d write between three and six paragraphs. There would be references but no deep explanation. I’d present the principles in a few sentences and give an example of how an evaluation would differ if I did or did not take path dependence seriously.

Rick Davies (Dr), Monitoring and Evaluation Consultant, Cambridge, United Kingdom | UK. Websites: http://www.mande.co.uk  and http://richardjdavies.wordpress.com/ | Twitter: @MandE_NEWS | rick.davies@gmail.com Skype: rickjdavies

My initial interest in the relevance of evolutionary theory was specifically in a field known as evolutionary epistemology. In its simplest form, this views the evolutionary process as a type of learning process, one involving the selective acquisition and retention of information, happening at multiple levels of scale. In the context of PhD research, evolutionary epistemology was used as a means of understanding organisational learning within organisations, and more specifically, in the operations of a large NGO in Bangladesh (Davies, 1998). It also helped generate two practical proposals  – one being a means of participatory impact monitoring and the other being a participatory approach to the exploration of alternate futures. Both involved a particular social implementation of the evolutionary search algorithm: variation, selection and reproduction. The main intellectual influences here have been Donald Campbell, Gregory Bateson and Daniel Dennet.

The second body of ideas that has taken up my time is network analysis, in its many and varied forms. This seems a practical way of thinking about complexity – a body of thinking that overlaps substantially with evolutionary theory. Most attempts to describe/define what complexity is do so by referring to complex systems as networks of some kind. There is a wide range of methods of describing and measuring network structures that is relatively agnostic in terms of the theories that can be used to interpret that kind of data.  One good feature of a network perspective is that it can help connect more abstract thinking about complexity to actual observations and measurements. Some intellectual influences here have been Borgatti, Benkler, Burt., Krebs.

The third body of thinking, which has been of more recent interest, is about the measurement, origins and consequences of diversity. Both evolutionary theory and network analysis can have something to say about diversity. So can other fields that are of interest to me. One of which is known as “collective intelligence” i.e. the study of the circumstances where the behaviour of a group can be more productive (on some measure) than that of the best individual in the group. Some intellectual influences here have been Page,  Suroweicki, Wagner

Rob D. van den Berg
Visiting Professor, King’s College London
rdwinterberg@gmail.com

My experience with complexity and systems in evaluation has been in the nexus between development and environment. From 2004 to 2014 I worked as evaluator at the Global Environment Facility (GEF), that financially supported many projects and programs throughout the world focusing on how environment and development could become a win-win situation, leading to more sustainable systems and innovations that would lead to transition to greener societies and economies, safe biodiversity and prevent climate change. My overall message from that experience is the importance of scale. Many of the investment programs supported by the GEF were great and could have been the steppingstone to preventing the current environmental crises, but these initiatives did not reach the scale needed to achieve that. That depended on whether societies, countries, the world would take over the examples of how transformation could take shape. And we know the world did not do this… So the problem of scale to reach transformation to a different system is a challenge that we need to face if a transformation is needed to prevent disasters. Gradually we started to introduce this in our evaluations, but noting the challenge of scale is as far as the power of an evaluator to change the world goes…

A second issue that gradually emerged for me is uncertainty. Complexity almost always means that what you study is to a large extent unpredictable. Complexity calls for multi-actor and multi-layered programs, that aim to influence a complex system through a range of activities. A typical environment/development program would have: a governmental/policy/regulations layer, supposed to prevent negative outcomes and promote and regulate positive ones; a civil society component, interacting with people to encourage them to change behaviour; a private sector component focusing on new ways to do business; and capacity development to promote the knowledge and capacities to move these components forward. In almost all cases the interactions between these components and the sovereign decisions of stakeholders brought huge uncertainty to the program, and only flexible and adaptive management would be able to turn surprises from obstacles to enabling factors for change.

The third issue concerns risk. Gradually I became aware that evaluation is not risk oriented. For many of us, this was self-evident. Evaluations look at the past, and the risks are no longer alive; they have come to pass or not, and about the only issue evaluators were involved in is whether risks were identified well during the implementation of a program. But this is a very different perspective than facing risks and uncertainty in the future. Evaluating programs that aim to transform a complex system inevitably means that as evaluators we need to adopt a forward-looking perspective in addition to our usual assessment of “what happened”. The main instruments of forward-looking science are scenario-building and risk assessment. Both have a long history in science and in many areas of work, amongst them insurance, pension schemes, to name a few financial and socio-economic ones. The Earth sciences (geography, climate, biology, etc.) have done a lot of work to integrate the forward-looking perspective in the research they do. The most famous is perhaps climate science, which models and calculates developments in climate and identifies risks for our societies and economies and life on planet Earth. In my time at the GEF we made the first tentative steps in this direction, and for me it is clear that evaluation needs to adopt deeper knowledge of these issues and approaches to move forward to support transformational change that would bring us a sustainable future.

 

Bob Williams bob@BOBWILLIAMS.CO.NZ

Criteria are the engine that drives evaluation.  As evaluators, our core task is to address the ‘so what’ question not the ‘what’ question.  Indeed our focus on judgements of worth (e.g. merit, value, significance) differentiate our craft from other forms of social inquiry.  And you cannot arrive at a judgement of worth without criteria; either explicit or implicit.

Yet evaluators frequently treat criteria as unproblematic.  We commonly take evaluation commissioners’ criteria as the basis for our evaluative inquiry, without any questions asked. We boilerplate criteria such as those developed by the OECD’s Development Assistance Committee (DAC).  If we adopted a more critical stance towards criteria setting, we would do neither of those things.

C. West Churchman in the 1970s developed the discipline of Critical Systems. Many in the system field subsequently refined his ideas, especially Gerald Midgley, Martin Reynolds and Werner Ulrich. Critical Systems based approaches have a particular focus on how boundaries are established around a task.  In Critical Systems terms, boundaries decide who or what is acknowledged or ‘in’, and who or what is marginalised or ‘out’.  Ulrich’s Critical Systems Heuristics (CSH) comprise a series of questions around common boundary decisions that promote  deep consideration of those decisions and their consequences.  These considerations can be based on moral or ethical arguments, as well as practical and political realities.

Evaluation criteria are boundary choices.  They form the boundary between what the evaluation considers as ‘worthwhile’ and ‘worthless’, ‘significant’ and ‘insignificant’, ‘valuable’ and ‘irrelevant’.  Within a Critical Systems frame these decisions need critical review.  Take the common criterion of ‘efficiency’ for instance.  Who or what is ‘marginalised’ by such a criterion?  Why chose this as a criterion?  Is an inefficient intervention worthless?  And what kind of projects are marginalised by such an understanding?  What about interventions that are testing ideas or are in the process of development.  They are almost certainly not efficient but may be highly worthwhile.
I find that using the challenging questions that Critical Systems and Critical Systems Heuristics poses are essential to the development and use of criteria in my evaluations.

Some easily available references about the use of Critical Systems in evaluation.Better Evaluation https://www.betterevaluation.org/en/plan/approach/critical_system_heuristics
Werner Ulrich’s website : https://www.wulrich.com/csh.html
Bob Williams (2019) Systemic Evaluation Design; A Workbook https://gum.co/evaldesign

 

 

Martin Reynolds (The Open University). Applied Systems Thinking in Practice (ASTiP) Group. School of Engineering and Innovation. The Open University, Walton Hall, Milton Keynes MK7 6AA, United Kingdom +44 (0) 1908 654894 | martin.reynolds@open.ac.uk |  Profile | Publications

From a systems thinking in practice (STiP) tradition I would first like to change the formulation from ‘complexity and systems science’ to complexity science and systems thinking (cf. Reynolds et al., 2016). The revised formulation is important for two reasons in appreciating respective lineages. First, contemporary ideas on complexity including the ‘butterfly effect’ and ‘complex adaptive systems’ are very much rooted in the scientific tradition dating from Warren Weaver’s 1947 paper ‘science and complexity’. Second, contemporary systems thinking should be regarded as a transdisciplinary endeavour inclusive of systems science and complexity science, but far beyond the confines of a scientific discipline (Reynolds and Howell, 2020). Note that systems science and complexity science have many common lineages, including pioneering work around cybernetics in the 1940s.  Appreciating the value of complexity science and systems thinking requires in my view attention to the ontological and epistemological dimensions of appreciating complexity and systems.

Complexity as used in complexity science invokes the scientific ontological (real world) premise that everything connects. Ideas of uncertainty and emergence are tied to appreciating reality as an infinite network of interconnections, the effects of which are impossible to precisely predict.  Systems science might be regarded as an endeavor to systematically bound such interconnections,  recognized by an impartial observer as relevant to a particular situation of interest.  By so doing, the ensuing bounded systems might be subject to scientific analysis. In systems science and complexity science, the key epistemological driver is positivism; there being an assumed direct representation between reality and systems (ontological realism; e.g. ‘the’ health system), subject to inquiry from an impartial ‘objective’ observer (scientist).

In contrast, complexity as used in a STiP tradition is an effect of contrasting human perspectives on the framing of interconnections, rather than an effect of interconnections directly. In the STiP tradition ‘systems’ as ontological representations of reality are legitimate, but the representations are always nominal (named by a human ‘observer’), provisional (with boundaries subject to change from other observers), and secondary.   Nominal systems such as (i) natural systems (individual organisms, ecosystems, solar system etc.); or (ii) engineered (purposive) systems (mechanical devices ranging from computers to heating systems), are secondary to a primary understanding and active use of systems as conceptual constructs which may be referred to as (iii) human (purposeful) systems.  Purposeful systems (where the bounded purposes are subject to ongoing adaptive change) are a powerful tool of contemporary STiP.  As distinct from ‘seeing’ reality only as natural or engineered systems,  purposeful systems enable such viewings to be tamed within a primary framing of a learning system (as an epistemological construct).  Such primary framings enable organisations, and interventions in education, health, etc. to be not only evaluated but (re)designed.  The STiP tradition, founded on epistemological constructivism, recognizes complexity as an effect of contrasting viewpoints on reality. Complexity here is a second-order attribute of interconnections in situations of interest – the indirect human framings of interconnections.  Complexity in complexity science is a first-order attribute of the interconnections themselves.

The difference is significant for all practitioners in all professional fields.   In a STiP tradition, complexity exists in all situations (since no human situation comprises only one perspective).   Each individual or group of individuals frame things differently depending on lifeworld experiences including, amongst other demographics, ethnic backgrounds.  STiP flushes out the framings of situations in terms of transparent purposeful systems in order to help improve the situations through more meaningful conversation amongst practitioners.  With increasingly uncertain times where racism is being called out internationally through the killing of black American George Floyd, it is perhaps worth recalling the founding principle of  STiP which takes its cue from C.West Churchman: “a systems approach begins when you see the world through the eyes of another” (1968 p. 23).

 

 

 

 

Constructing a Deep Complexity and Systems Science Foundation for the Field of Evaluation

Jonny Morell, PhD
Meg Hargreaves PhD

This section of the blog Evaluation Uncertainty, Surprises in Programs and their Evaluations is an effort to continue and expand prior work setting evaluation theory and evaluation practice within a deep understanding of complexity and systems science research and theory. Four beliefs motivate our current efforts.

• Not all evaluation needs to invoke systems concepts, but much of it does.
• When system concepts are needed, evaluators must use those concepts to make operational decisions about the theoretical frameworks and models they construct, the methodologies they devise, and the meaning they draw from data.
• To make those wise decisions, evaluators require deep-seated understanding of the research and theory that has been generated by complexity and system science.
• At present, too little of evaluation’s reliance on systems is based on such deep understanding.

“This section of the blog”, and “begin” and are important phrases in the opening paragraph. We hope that what we are doing here will spur a wider range of discussion and inquiry.

We envision two kinds of contributions to this blog. To start, posts will say nothing at all about evaluation. Rather, evaluators will provide explanations of the intellectual domains within complexity and system science that affect their work. We see those contributions as providing an imperfect, but reasonably wide view of the domains of complexity and system science that influence how evaluation is done. We want these posts to be short, somewhere between one and four paragraphs.

Once a there is sufficient variety of material, contributions will then broaden to a wider set of evaluators, more discussion of complexity and system roots, and examples of how research and theory in the fields of complexity and systems affected practical decisions about theoretical frameworks, models, metrics, and methodologies.