Aaron E. Zazueta a@zazuetagroup.com;
Nima Bahramalian N.Bahramalian@unido.org;
Thuy Thu Le t.le@unido.org
This article builds on a previous contribution to this blog identifying a set of complex adaptive systems that are particularly useful in the formulation of theories of change (TOCs), find the link to the blog here. These include the concepts of the Social-Ecological Systems, boundaries, domains, scales, agents, adaptive behavior, and emergence and system development trajectory in the formulation of theories of change. This article briefly explains how to use these concepts and presents some aspects of an the article Development trajectories and complex systems-informed theories of change which was published in September 2020 in the American Journal of evaluation (Zazueta et al., 2020). A non-edited version of the article is available here. The article illustrates the use of the approach in the evaluation of the UNIDO /SECO project SMART- Fish in Indonesia (UNIDO, 2019), available here.
The long-term objective of the SMART-Fish was to support the transformation to more efficient and environmentally sustainable fisheries that increased value across the market chain, especially among the small fishermen and women. The project addressed three value chains that have different ecological, economic and social characteristics: Pangasius, Pole and Line Tuna and Seaweed. The evaluation focused on the identification of the conditions conducive to the intended transformation and the assessment of the extent to which the project contributed to the advancement of those conditions. The evaluation had two main phases. In the first phase, the project management team and the evaluation team jointly developed a proposal of the social ecological fisheries system in Indonesia that was subsequently presented to project stakeholders for discussion and application. The first phase consisted of the following steps:
- Definition of a manageable set of domains that could provide an initial framework to identify key enabling conditions for the intended transformation. Drawing from a review of the technical literature, the teams identified five broad domains: policy and regulatory, institutional, technological, financial, and sociocultural.
- Brainstorming to identify the four or five most important enabling conditions in each of the five domains. The result was 32 enabling conditions.
- Regrouping the 32 enabling into clusters and when appropriate relabelin
g domains. This led to the grouping of the 32 conditions into six domains. - Identification of the instances in which each of the 32 conditions had an enabling function to the rest of the conditions across the system. This step consisted of development of a relation matrix which identified 236 significant links among the 32 conditions. Figure1 plots the links among the different enabling conditions.
- Ranking of the 32 enabling conditions in terms of their influence across the system. Using the program NodeXL, we ran several tests to identify the most influential enabling conditions across the system. These tests identified five of such conditions (figure 2).
In phase two, the evaluation team convened three stakeholder focus groups, one for each value chain. Each group was asked to rate the state of the 32 enabling conditions before the project started and at the time of project completion. Subsequently, the groups were asked to rate the extent to which SMART- Fish contributed to the changes in these conditions. When aggregating the responses on the three value chains, changes in the enabling conditions to the long-term objectives were most pronounced in the domains of trade and markets, governance, and production (Figure 3). These were three domains that the project targeted and in which stakeholders reported SMART-Fish making substantial contribution. While the stakeholders acknowledged project contributions in science and technology, the progress made in many of the enabling conditions in this domain was seen as low. Progress was made in the enabling conditions under the financial domain with no link to the project.
The evaluation team also used the focus group data to assess the extent to which the project had contributed to the five key enabling conditions with the most influence on the system previously identified through the network analysis which are presented in Figure 2. This figure indicates that despite the project making important contributions to enabling condition pertaining innovation capacity in science and technology, not much progress was achieved on other conditions pertaining the technology domain.
Despite the complexity of the system, the approach allowed us to develop a model to understand the factors leading to transformation of the fisheries systems in Indonesia and the extent and forms by which the project had contributed to a development trajectory consistent with the long term goal of the project. The approach also to enabled the engagement of stakeholders in the assessment of the contributions made by the project in the trajectory of the desired policies.
UNIDO. (2019). Independent Terminal Evaluation Indonesia SMART-Fish Increasing Trade capacities of Selected Value Chains within the Fisheries Sector in Indonesia. United Nations Industrial Development Organization. https://www.unido.org/sites/default/files/files/2020-01/120110_Indonesia_SMART%20Fish_Terminal%20evaluation.pdf
Zazueta, A. E., Le, T. T., & Bahramalian, N. (2020). Development Trajectories and Complex Systems–Informed Theories of Change. American Journal of Evaluation, I–20. https://doi.org/10.1177/1098214020947782
Evaluation Uncertainty 
Reblogged this on Systems Community of Inquiry.
Perhaps I am overlooking something important, the report on the fisheries project is over 100 pages so I have looked through rather than read in detail, but how is this project “complex” ? With logframe design, and use of standard causal RBM throughout the design, implementation, and monitoring, the project is located firmly in the ordered domain. Complicated, perhaps, but not complex by the usual definitions. What makes this in any way complex ?
Thank you for your comment it is much appreciated,
To clarify:
• The approach presented applies the notion of complexity to the phenomena that are targeted by the project. Not to the project. The phenomena are assumed to be a system
• Complex system thinking is used to develop a system model of the phenomena and its dynamics.
• The system model is then used to assess the extent to which the project intersects with the natural phenomena in ways that are likely to steer that system’s development trajectory towards the long term desired condition (or policy objectives).
Some of the literature refers to “complex project”, thus these approaches have identified the characteristics of a complex project (for example Michael Banberger et al book). This is one way of using complexity in evaluation, which is fine. Our work does not go in that direction, we focus on the understanding of the natural systems and the conditions that are likely to bring about change in the direction of a given trajectory. To do this we draw on the existing scientific and technical knowledge. We subsequently focus on assessing the extent to which the intervention has contributed to that change.
We have submitted another article for publication that uses this approach in the planning of a project. This time we used complex systems thinking to identify the most influential conditions within the system that need to be intervened to steer the system development trajectory towards the desired long-term goal. To do this we gather information on the system components and their interactions through a series of workshops with technical experts and stakeholders. We used mathematical modeling to assess the influence on the rest of the system for each of the identified components. We expect that article will be published early next year. On this occasion we have also used complexity to understand the system, the idea has been to come up with a set of interventions that takes advantage of the complexity of the system to steer the system.