I realized it might help to explain what led me to ask this question in the first place. I submitted a proposal to AEA to talk about how traditional evaluation methods can be used in complex systems. Part of that explanation will have to involve understanding the CAS implications of stability in program impact across time and place. See the end of this post for that proposal.
I’m looking for some sources and opinions to help with a question that has been troubling me lately. I’m struggling with the question of the relationship between path
- dependence and
- system stability.
Or maybe I mean the relationship between path dependence and the ability to predict a system’s trajectory. I’m not sure about the best way to phrase the question. In any case read on to see my confusion.
I’m bumping into a lot of people who believe that systems are unstable/unpredictable because of path dependence. This is one of those notions that seems right but smells wrong to me. It seems too simple, and it does not make sense to me because it implies that if systems are predictable there is no path dependence operating. That can’t be right, can it? Here is a counter example.
- Example of path dependence coexisting with system stability
- Think of the “planning fallacy”, the well known phenomenon where no matter how expert and experienced the judgment, estimates of how long (or how cheaply) a project can be completed are almost always wrong. We know the reasons for this. One is psychological – selective memory, confirmation bias, and the like. Another is system related. We cannot know all the problems that will creep in, we can only know some of them.Now, if someone took a developmental evaluation perspective, or maybe any of the systems views that are current in evaluation circles, what would one do? We would try to discern major aspects of the system in which the project was embedded, and look for ways to factor those linkages into the evaluation. We would scope the environment, quickly factor findings into planning, and the like. Why do all these things? Because executing a project involves working with a complex system in which path dependence cannot be predicted.But there is an entirely different way to do this, which is the way the cutting edge of project planning does. They take a purely empirical statistical view. How to characterize my project, e.g. cost, length, technical complexity, what is being planned, amount of R&D needed to get it done, or whatever. Then compare my project to the historical record and get an estimate of likely time and cost. This works pretty well.What have they actually done? They have taken a stance that says: Yes path dependence matters and we cannot predict how things will unfold. But we do not care about any of that because none of it matters in terms of achieving success. All the meandering about paths and choice points is noise. That is an example where a complex system exhibits path dependence that does not matter. If I were evaluating the project planning process I would look at a lot of stuff, but the path dependent uncertainties would not be on my list.
My sense is that there must be some kind of boundary condition, or maybe attractor type such that inside there may be lots of path dependence that won’t make a difference in the system’s condition. Does this seem like a reasonable way to look at things?
I have a feeling that what matters is whether a perturbation to a system dampens or builds up a resonance, but maybe the analogy to physics is incorrect. Or maybe another way to look at it is to say that no single branch matters, but what matters is the alignment of several choice points, and the greater the alignment necessary, the lower the probability that any single path dependent branch will make a difference. (Analogous to the Swiss cheese model of accident causation.) If this notion of mine is even half way right it brings up a few other questions.
- 1- Systems scale so to say “within the boundary” has to be defined. Probably the best way to do this is in terms of behavior that the environment cares about. Do people get better in a hospital? Does R&D funding lead to economic development? And so on.2- Then there is the minor problem that to say “within a boundary” or “within an attractor” is purely descriptive. What is it about the nature of those things that matter?3- The problem of latency also rears its ugly head. A path dependent branch may be immediately apparent, or may not manifest for a long time. (Whatever “long time” means.) So any discussion of this topic has to define some kind of a time frame.4- Then there is the matter of what I’ll call “latent path dependence”. What I mean is some kind of path dependent change that seems to be inconsequential until the environment changes.
Aside from intellectual curiosity I’m interested in this subject because it touches on a deep religious argument in the field of evaluation. Is there ever enough stability over time and place that a program shown to be successful in one context can be trusted to work in another? Partisans of the instability wing delight in waving the flag of path dependence (and its cousin that accursed butterfly). What I’m grasping for a bit more of an understanding based on what we really know about system behavior.
Contributions to my mental health would be much appreciated.
————–Begin AEA expert lecture proposal———–
Squaring Complexity With the Reality of Planning and the Capacity of our Evaluation Tools
Abstract (150 words)
We are in a bind. As a practical matter our designs assume a simpler world than the one in which we live. This is because it is difficult for program planners to theorize about complex behavior, and also because our qualitative and quantitative research methods assume patterns of regularity and stability that do exist, but also do not. Neither we nor our customers are good at contending with complex behavior. We must act like the proverbial bee who, being ignorant of the laws of physics, flies anyway, thereby making honey every day. How should we behave in the face of this contradiction? We should not abandon methods that work successfully, but we must factor complex behavior into our understanding of program theory, our designs, and our data interpretation. This presentation will focus on various aspects of complex systems behavior and illustrate how this factoring process might work.
Relevance Statement (500 words)
As evaluators are in a situation where the methods we are able to use can never be more than partially appropriate for the programs we evaluate. The reason is that the world we live in is complex (in the technical sense of the term), but we act as if it is not. The planners and policy makers for whom we work (and with whom we work) do not think in terms of path dependence, the edge of chaos, attractors, adaptation, phase shifts, power law distributions, or any of the other concepts that Complexity Science has shown to govern our world. Evaluators are not very good at thinking like that either. Of course it is important to try to bring complexity thinking closer to the core of what we and our customers do. But even if we all knew better, we might not act much differently. The gulf between what we do and how the world works is not there because any of us are ignorant, or because we lack the intellectual capacity to learn new concepts and to deploy new tools. There are good political reasons for the status quo. There are good economic reasons for it. There are good sociological reasons for it. There are good historical reasons for it. Of course we need to work at bringing complexity thinking into policy, planning, and evaluation. If we succeed, policy, planning and evaluation will be the better for it. But we must also recognize that policy, planning, and evaluation will unfold largely as if the dynamics of complex systems were not at play. What evaluators need to do is to learn how to live with this tension by applying complex system concepts as best they can in how they design evaluation, how the analyze data, and how they interpret data. This prescription is not very satisfying. But it is realistic, and following it will make us improve our ability to give our stakeholders better insight into the consequences of their actions.