This article replaces a few other posts that I published over the last few months. This one integrates and adds material to those other writings.
Abstract
Evaluation is well stocked with knowledge about how to evaluate programs in terms of systems. Our stock is much less when it comes to evaluating the logic of the systems themselves. But separating the two and then bringing them back together can advance our understanding of both programs and systems. This assertion is illustrated with four examples: 1) causal chains, 2) stocks and flows, 3) network development and structure, and 4) complex systems with an attractor/equilibrium focus. Doing this kind of research is practical because the systems evaluation work requires additional analysis, but no, or minimal, extra data collection. Choosing which system model to construct is similar to the effort needed to construct any model. There are multiple possibilities, and the decision depends on the systems-related questions that are most salient in the setting where we are working. In addition to models, choices must be made about what aspects of a system to measure. Many possibilities are provided, which group into five categories: 1) rates and magnitudes, 2) boundaries, 3) architecture, 4) change process, and 5) resistance to change and sustainability. The program/system distinction is useful but not clean because in some scenarios parts of a program model are system models. Many examples are provided to show how research on systems can enrich the knowledge gained from program evaluation.
Evaluation Uncertainty 
From Bob Williams on a previous post. I suspect he will feel much the same about this one.
Kia ora Jonny
I’ve given the paper a quick scan, and will give it a more detailed read later. I gave it a quick scan because I wanted to see how you conceptualised a system.
Essentially, you frame things in the way that I would generally fit into the hard systems tradition.
The distinction between hard and soft is much debated and is not totally relevant here. However, it is not irrelevant because it determines the kind of questions you can ask and the kind of answers you receive.
The main distinction for me, between hard and soft systems, can be best highlighted by the following defintions of a system:
A system is:
A collection of entities [That are perceived by someone as] Interacting together [To achieve something]
Hard systems are those described by the first and third lines. Soft systems include the entire phrase. Ontologically and especially epistemologically, the differences are critical. In soft systems, a system has to have an identified (but not necessarily intended) purpose. How the entities interact and what purpose they express is entirely in the eye of the beholder. You and I will look at the same situation and may ’see’ a different set of entities, a different set of interactions, and a different purpose. We will literally see different systems. In contrast a hard system approach will tend to display a single set of entities and a specific set of interactions, that interact according to a specific set of rules. You and I would ‘see’ the same system. This is the basis of classic system dynamics and to a large extent, classic CAS.
I know you’re familiar with all this, but it means that the distinction you’re making between seeing a program as a system and a systems view of a program is not especially important to me, as someone who tends to engage with the systems field from a soft systems perspective. It is just two different ways of understanding, in systems terms, the situation you are observing.
I am definitely not saying that my approach to systems is right and yours is wrong. But your puzzle reflects more the constraints of hard systems approaches than soft systems approaches. For you, the distinction you are making is important (which is why you are writing about it); for me, it is just another distinction that may be useful depending on what I want to know.
Okay, having said that, I can now put on my hard systems glasses and look at what you have written in those terms.
Thanks for the reminder. It saved me a couple of hours of repeating myself. On reflection, I regret diving into the soft/hard systems discourse, because it’s a bit like Japanese paper folding. There will always be a hard aspect of a soft systems approach and a soft aspect of a hard systems approach. What’s important is how much you acknowledge that. This time, I’m going to use another distinction that a bunch of people make in various forms. There is observing systems, systems thinking and being systemic. Or that there are three distinctive approaches to systemic practice: ontological approaches, epistemological approaches and axiological approaches. Or to put it a third way, what you look at, how look at it, and why you look at it. Again, all three are present in any systemic practice and you need to be aware of them. Also, the emphasis will differ depending on the chosen methodology.
Reflecting back on your paper, I think you are traversing the area covered in what you are looking at, and how you are looking at it. What’s not so clear is why this is important to you (and maybe other people). Why are you selecting those four particular approaches to exploring the behaviour of a system, and not others? What are the world-views and values that underpin the four you selected (or indeed any approach to applying ideas about the behaviour of what you call a system). You are in effect identifying and deliberating on a system of systemic practice. What makes it a system and what world views does it represent and what are the implications of that for your own practice, thinking and that of others?
All good points. I’ll ponder them but I’m not sure how much more writing I’ll do in the near future. As for the four examples. Those are the ones that sprung to mind when it came to “examples writing” time. Nothing more, nothing less. They seemed sufficient in a paper that was already getting too long.
As for importance. I still think that the approach I laid out would be good for understanding both systems and programs. But if I’m the only one who thinks that way, so be it. That won’t be the first time this happened to me. The other two I can remember (memory fades with age) are using schedules as logic models, and applying knowledge from ecology and evolutionary biology to evaluation. If I get lucky, these three won’t be the last.