Applying Complexity to Evaluation: Case Based on the GEF’s Resilient Food Systems Program: Integrated Landscape Management to Enhance Food Security and Ecosystem Resilience

Rick Davies (Dr), Monitoring and Evaluation Consultant, Cambridge, United Kingdom | UK. Websites: http://www.mande.co.uk  and http://richardjdavies.wordpress.com/ | Twitter: @MandE_NEWS | rick.davies@gmail.com Skype: rickjdavies

The Case

The case is the Global Environment Facility’s Food Security Program (FSP). In particular, a project in Ethiopia, implemented by the UNDP:  Integrated Landscape Management to Enhance Food Security and Ecosystem Resilience.

A Network View of Complexity

Network structure has long been recognised as an important facet of complexity (Mitchel, 209:227-290). Independent of complexity theory an extensive body of methods has been developed for describing and analysing network structures (Borgatti et al, 2018). These involve the use of three related methods: network visualisations, matrices and mathematical measures. Nodes in such networks can represent actors, activities, events, or more abstract constructs such as objectives or outcomes. Links in networks can represent a wide variety of kinds of relationships between those entities. For example casual connections, information flows, preferences or social ties.

The Relevance of a Network Perspective to the FSP Project

The official description of this project has all the ingredients of a complex programme: many different elements, with many different relationships between these. Of all these relationships many may already have been anticipated by those designing and implementing the project. But the overall structure of all these relationships is much less likely to be clear in the mind of these stakeholders. Even less likely to be anticipated are the possible consequences of the particular structure of this network of relationships for how the “state” of this system will change as time progresses from one period to the next.

This is what the project structure looks like in more detail. The project has 11 different “activities” that fall within 5 different “themes”. In addition, the project is described as having an “approach”, consisting of three distinct “components”. Elsewhere these might be labelled as the expected outcomes of the activities. Under a third category of “Impact” specific targets have been articulated for each of the approaches (7 in all).  What is not yet present in the information available online, is any articulation of the expected relationships between the activities and the components, including the more specific impact targets. However, these could easily be identified by one or more means of participatory systems mapping ( Wikinson et al, 2021) or participatory network mapping (Davies, 2018). The result will be a representation of the project’s “Theory of Change” that will sufficiently detailed to enable the project to be evaluated, subject to other conditions (e.g. data availability) also being met (Davies,2013).   

The project website also provides information on the stakeholders involved. “At national level, major stakeholders involved in the RFS Ethiopia project include the following ministries: Ministry of Environment, Forest and Climate Change; Ministry of Agriculture; and Ministry of Finance and Economic Development. Other stakeholders directly engaged throughout the project include community members and resource users and managers at the local level; NGOs; national and international partners and agencies; universities in the 12 targeted areas; local authorities of Oromia, Amhara, Tigray, Afar and Somali Regional States; and the Woreda Agricultural, Water and Energy and Environment Protection and Land Use Offices”.

When more detailed information becomes available about the stakeholders at the local level a second network mapping will provide important additional information. The first exercise, described above, will generate a single decontextualized model, a kind of high-level theory. Participatory network mapping of the expected relationships between the stakeholders, in each of the 12 different locations, will provide more context specific theories of how the project is expected to work.

The presence of the project in 12 different locations is likely to generate, by intention, by chance and by force of circumstances, a diversity of approach to achieving the projects outcomes. Such diversity is both an opportunity and a challenge. An acceptance of a diversity of approaches means there is an opportunity for learning by trial and error. It increases the chances of the implemented activities being locally fit for purpose. However, there will be a corresponding challenge: how to draw more general lessons from across the whole set of locations.  This is where evolutionary theory, which is another facet of complexity theory (Axlerod & Cohen, 2000), could be useful. Data sets could be constructed describing attributes of each location, including (a) project design features e.g., project activities, and context features e.g., stakeholders, and (b) outcome features. Genetic algorithms, already available as Excel add-ins, can be used to search and find combinations of such features which are best predictors of outcomes interest (Solver, n.d.). These predictive models can then be explored in detail using within-case inquiries, identifying if and where any causal mechanisms are involved. Compared to experimental approaches that typically seek to identify “average treatment effects” this type of configurational analysis gives more appropriate recognition to the complexity of the world, notably equifinality – effects can arise from multiple different causes, and multifinality – a cause can have multiple different effects.