I realized it might help to explain what led me to ask this question in the first place. I submitted a proposal to AEA to talk about how traditional evaluation methods can be used in complex systems. Part of that explanation will have to involve understanding the CAS implications of stability in program impact across time and place. See the end of this post for that proposal.
I’m looking for some sources and opinions to help with a question that has been troubling me lately. I’m struggling with the question of the relationship between path
- dependence and
- system stability.
Or maybe I mean the relationship between path dependence and the ability to predict a system’s trajectory. I’m not sure about the best way to phrase the question. In any case read on to see my confusion.
I’m bumping into a lot of people who believe that systems are unstable/unpredictable because of path dependence. This is one of those notions that seems right but smells wrong to me. It seems too simple, and it does not make sense to me because it implies that if systems are predictable there is no path dependence operating. That can’t be right, can it? Here is a counter example.
- Example of path dependence coexisting with system stability
- Think of the “planning fallacy”, the well known phenomenon where no matter how expert and experienced the judgment, estimates of how long (or how cheaply) a project can be completed are almost always wrong. We know the reasons for this. One is psychological – selective memory, confirmation bias, and the like. Another is system related. We cannot know all the problems that will creep in, we can only know some of them.Now, if someone took a developmental evaluation perspective, or maybe any of the systems views that are current in evaluation circles, what would one do? We would try to discern major aspects of the system in which the project was embedded, and look for ways to factor those linkages into the evaluation. We would scope the environment, quickly factor findings into planning, and the like. Why do all these things? Because executing a project involves working with a complex system in which path dependence cannot be predicted.But there is an entirely different way to do this, which is the way the cutting edge of project planning does. They take a purely empirical statistical view. How to characterize my project, e.g. cost, length, technical complexity, what is being planned, amount of R&D needed to get it done, or whatever. Then compare my project to the historical record and get an estimate of likely time and cost. This works pretty well.What have they actually done? They have taken a stance that says: Yes path dependence matters and we cannot predict how things will unfold. But we do not care about any of that because none of it matters in terms of achieving success. All the meandering about paths and choice points is noise. That is an example where a complex system exhibits path dependence that does not matter. If I were evaluating the project planning process I would look at a lot of stuff, but the path dependent uncertainties would not be on my list.
My sense is that there must be some kind of boundary condition, or maybe attractor type such that inside there may be lots of path dependence that won’t make a difference in the system’s condition. Does this seem like a reasonable way to look at things?
I have a feeling that what matters is whether a perturbation to a system dampens or builds up a resonance, but maybe the analogy to physics is incorrect. Or maybe another way to look at it is to say that no single branch matters, but what matters is the alignment of several choice points, and the greater the alignment necessary, the lower the probability that any single path dependent branch will make a difference. (Analogous to the Swiss cheese model of accident causation.) If this notion of mine is even half way right it brings up a few other questions.
- 1- Systems scale so to say “within the boundary” has to be defined. Probably the best way to do this is in terms of behavior that the environment cares about. Do people get better in a hospital? Does R&D funding lead to economic development? And so on.2- Then there is the minor problem that to say “within a boundary” or “within an attractor” is purely descriptive. What is it about the nature of those things that matter?3- The problem of latency also rears its ugly head. A path dependent branch may be immediately apparent, or may not manifest for a long time. (Whatever “long time” means.) So any discussion of this topic has to define some kind of a time frame.4- Then there is the matter of what I’ll call “latent path dependence”. What I mean is some kind of path dependent change that seems to be inconsequential until the environment changes.
Aside from intellectual curiosity I’m interested in this subject because it touches on a deep religious argument in the field of evaluation. Is there ever enough stability over time and place that a program shown to be successful in one context can be trusted to work in another? Partisans of the instability wing delight in waving the flag of path dependence (and its cousin that accursed butterfly). What I’m grasping for a bit more of an understanding based on what we really know about system behavior.
Contributions to my mental health would be much appreciated.
————–Begin AEA expert lecture proposal———–
Session Title
Squaring Complexity With the Reality of Planning and the Capacity of our Evaluation Tools
Abstract (150 words)
We are in a bind. As a practical matter our designs assume a simpler world than the one in which we live. This is because it is difficult for program planners to theorize about complex behavior, and also because our qualitative and quantitative research methods assume patterns of regularity and stability that do exist, but also do not. Neither we nor our customers are good at contending with complex behavior. We must act like the proverbial bee who, being ignorant of the laws of physics, flies anyway, thereby making honey every day. How should we behave in the face of this contradiction? We should not abandon methods that work successfully, but we must factor complex behavior into our understanding of program theory, our designs, and our data interpretation. This presentation will focus on various aspects of complex systems behavior and illustrate how this factoring process might work.
Relevance Statement (500 words)
As evaluators are in a situation where the methods we are able to use can never be more than partially appropriate for the programs we evaluate. The reason is that the world we live in is complex (in the technical sense of the term), but we act as if it is not. The planners and policy makers for whom we work (and with whom we work) do not think in terms of path dependence, the edge of chaos, attractors, adaptation, phase shifts, power law distributions, or any of the other concepts that Complexity Science has shown to govern our world. Evaluators are not very good at thinking like that either. Of course it is important to try to bring complexity thinking closer to the core of what we and our customers do. But even if we all knew better, we might not act much differently. The gulf between what we do and how the world works is not there because any of us are ignorant, or because we lack the intellectual capacity to learn new concepts and to deploy new tools. There are good political reasons for the status quo. There are good economic reasons for it. There are good sociological reasons for it. There are good historical reasons for it. Of course we need to work at bringing complexity thinking into policy, planning, and evaluation. If we succeed, policy, planning and evaluation will be the better for it. But we must also recognize that policy, planning, and evaluation will unfold largely as if the dynamics of complex systems were not at play. What evaluators need to do is to learn how to live with this tension by applying complex system concepts as best they can in how they design evaluation, how the analyze data, and how they interpret data. This prescription is not very satisfying. But it is realistic, and following it will make us improve our ability to give our stakeholders better insight into the consequences of their actions.
Evaluation Uncertainty 
Johnny, as always you bring good questions and interesting wrestling matches. If we were across a table over a beer, I would ask, “What, exactly, do you mean by “path dependence?” and “What, exactly, do you mean by “system stability?” Then, of course I would go on to explain that it is about certainty and constraint and a matter of the system’s conditions for self-organizing (CDE) and whether the three conditions are constrained or not or how much. i would also talk about how much you need (or don’t) a description that matches reality. I would probably also contrast static, dynamic, and dynamical change and ask you which world you want to or think you are playing in.
Since we aren’t ftf, I will just start with the questions. What do you mean by those apparently critical phrases–path dependent and system stability?
Hi Glenda,
Thanks for your response.
Self Organization
Your comment on self organization pointed me to one possible answer to my question. Self organized systems are very predictable and stable: As the discussion in Wikipedia explains: “The resulting organization is wholly decentralized or distributed over all the components of the system. As such it is typically very robust and able to survive and self-repair substantial damage or perturbations.” Another way to look at this is to say that when there are conditions conducive to self organization, there is minimal path dependent change. Random changes don’t make a difference, or if they do, they all move the system in the same direction. I am sure that self organization is not the only reason for stability and predictability of change, but it does seem to be one of them.
Definitions
As for “path dependence” there are a lot of good explanations out there that pretty much converge on the one that is most easily stated. “Path dependence is the dependence of economic outcomes on the path of previous outcomes, rather than simply on current conditions”. Obviously this definition comes from one particular field, but it comports with whatever else is out there.
“System stability” is harder to pin down because it’s possible to draw from discussions in lots of different fields, e.g. fluid dynamics, control theory, structural engineering, meteorology, and much else besides. All these look at the term from a somewhat different point of view, but the definition that seems most common and most intuitive (a lot of these discussions get very heavily mathematical very quickly) is: “the property of a body that causes it when disturbed from a condition of equilibrium or steady motion to develop forces or moments that restore the original condition.” I like this one because it is not exclusive to a static condition.
As for “dynamic” and “dynamical change”, I’m not sure I see why the difference makes a difference in this discussion. Definitions for “dynamic” across a wide variety of fields all have the theme of motion, kind of like the opposite of static. I could not find a definition for “dynamical change” but I did find one for “dynamical system” which I assume is in the same ballpark: “A dynamical system is a concept in mathematics where a fixed rule describes the time dependence of a point in a geometrical space.” In any case these both touch on what I had in mind, which is” the thing does not change in an unexpected way with respect to some parameter one is measuring.”
Thanks, Johnny. I think the relationship you may be looking for is this ” . . . the thing does not change in an unexpected way with respect to some parameter one is measuring” and how both system stability and path dependence relate to it, rather than the relationship between path dependence and system stability. Is that right? This certainly seems immediately relevant to evaluation.
For us, the relevant difference between dynamic and dynamical change is really relevant here. Dynamic change is predictable–Newton’s laws of motion. You can know enough about initial conditions to predict path and outcome over time. Like a thrown ball, the parabola is perfect, and if not you know why. Dynamic change is path dependent (in a really boring way) and it may be stable, though it motion. It depends on the initial conditions and how open the system is.
Dynamical change is unpredictable–Power law, self-organized criticality, tipping point. Dynamics at one scale influence change at other scales, and the system is scale-free, so everything is changing at some scale all the time. You can be sure that the system structure at whatever scale you’re looking at will change at some time, but it is impossible to know how and when. Dynamical change is path dependent (in the interesting way) and is sometimes stable at some scales. Some people call it metastable as it will return to initial position if disturbed a bit at some points in its evolution.
This is the way we understand the relationships between stability and path dependence and the predictability of change along a given parameter. Dynamic change happens only under relatively closed, low dimension, linear system conditions. Dynamical change happens under relatively open, high dimension, nonlinear conditions. If that sounds like limiting conditions of the CDE, it should.
From this frame, then, your basic question still persists–what do you do to evaluate a system that changes in unpredictable, even unknowable, ways?
Hope this is helpful.
Hi Glenda—
Right on the first paragraph. That is a more direct way of stating it.
As for dynamical change, most of the examples you cite seem very predictable to me. Why is a power law distribution any less predictable than any other kind of distribution? They all have maximum likelihood estimators and variances. (Although I cannot follow the math on what the MLE is for a power law distribution.) If you look at the log transform of a power law distribution you get a very good estimate of the number of events of a particular magnitude that will occur.
Self organization is exceedingly predictable in many physical and biological contexts, and in many others I am sure. There might be a phase transition involved, but that does not make it any less predictable. It only makes it non-linear when you plot the change. Why is dynamical change path dependent? In fact it looks not path dependent to me because once the system hits whatever the critical point is, the system will predictably change. One can measure the state of the system, track whatever parameter is important, and watch the system change from known observable state A to known observable state B.
In fact it seems to me that sets of self-organized systems and sets of path dependent systems are actually mutually exclusive. Here is what Wikipedia says about path dependence in the social sciences:
————Begin Wikipedia quote——–
Path dependence explains how the set of decisions one faces for any given circumstance is limited by the decisions one has made in the past, even though past circumstances may no longer be relevant.
In economics and the social sciences path dependence can refer to either outcomes at a single moment in time or to long run equilibria of a process. In common usage, the phrase implies either:
• (A) that “history matters”—a broad concept,] or
• (B) that predictable amplifications of small differences are a disproportionate cause of later circumstances. And, in the “strong” form, that this historical hang-over is inefficient.
————End Wikipedia quote——–
That seems to me like the antithesis of self organization, where history most certainly does not matter.
Thanks, Johnny. The plot thickens.
History absolutely does matter in self-organizing, as I understand it. Every dissipative structure (dissipating entropy and establishing order) resolves tensions present in the previous moment and sets conditions for the next. Even thinking about obvious self-organizing processes like team formation, it is easy to see that history makes a significant difference. Consider, for example, the sequence in which new members are added to the team. That will certainly determine the patterns of team performance and culture.
Self-organized criticality (SOC, Power Law) is definitely not predictable. Of course you can use statistics to approximate, but then you’re treating the system as if events were randomly distributed. It might be close enough for certain purposes, but it isn’t true. The straight line that comes from the inverse log of power law distribution is about the numbers and sizes of events, not when or where they occur. We could wish that SOC were predictable because then we could be prepared for volcanic eruptions, avalanches and earthquakes (to say nothing of economic collapses).
Oh, I think I see where the problem might lie. It looks like you may be talking about control parameters in deterministic chaotic systems. They are predictable and repeatable and not path dependent. If I’m increasing temperature (control parameter) in a pot of water, I can be sure that when the temp reaches a critical point bubbles will show up regardless of how long or from what source the heat is applied. You will often see this kind of “chaos” represented in bifurcation charts and strange attractor patterns. These systems are not path dependent, and they are usually stable except immediately before and after a critical transition point (during the phase shift, for example).
While both this and SOC are analytical ways to deal with “chaotic” systems, they are quite different in phenomenon and mathematics. Deterministic chaos is open (sometimes), low dimension (always), and nonlinear. I don’t use these metaphors from deterministic chaos often because they depend on such strict boundary conditions that human systems I work in usually don’t qualify. Deterministic chaos behaves quite differently from SOC systems, which are always open, high dimension, and nonlinear.
The SOC is (as I said earlier) the scale-free shifts that cascade in ways that depend on history and cannot be predicted in time. Of course, as you get close to a critical point, it becomes easier to “predict” over the short horizon for a particular scale, but this isn’t a consistent feature of the system. I’m thinking Per Bak was the best description of these kinds of system dynamics that we call dynamical and that we find quite helpful in thinking about change at all levels of human systems. Examples include the aha of learning, the innovative breakthrough, violent conflict, and divorce.
Sorry for this long response. Is this relating to your question(s)? What might I be missing? Perhaps this needs a conversation. I’ve not written much about this (some in our most recent book adaptiveaction.org). I find most people don’t really care about the distinctions at this level of detail, so it is fun to dig in this deep. I teach about these distinctions often (excepting deterministic chaos) and would really love to talk about it with you and others. I think this touches at the core of the challenges of evaluation in complex environments. Maybe in an upcoming webinar or on a tweetchat we could expand on this inquiry. What do you think? Is it interesting to others, too?
It’s interesting to me! Glenda, we haven’t formally met, but I do eval for the social innovation lab. If you do a webinar or twitchat, I’d love to join you.
Christine, we’ll let you know if the conversation goes real-time. In the meantime it will be good to explore this more at AEA next fall, if you’ll be there.
Jonny, thanks for the background. It is really helpful. I agree, we can’t use complexity as an excuse not to evaluate or to generalize or to consider fidelity across contexts. If we try to, we just give a good excuse to the culture wars of qual/quant that have been raging (and serving no one IMHO) for a long time. Instead, I’d like to see complexity theory and practice help us get past trivial distinctions and attend to differences that really make a difference. I think your question is moving toward that.
My challenge is that I think we hold some assumptions that are either contradictory or mutually irrelevant. Or maybe I just misunderstand you, but I can’t tell which is the sticking point. There’s a good chance that it is a bit of each all rolled up into the question mark that sits on my brow. I know it will help me, and I hope it will help you, if I try to capture some of my assumptions that I think are woven into your question and your efforts at answers.
I should also begin with a disclaimer. I’m going to speak strongly and sound incredibly arrogant and dogmatic in the following comments. I’m just wanting to speak as clearly as I can and not muck things up with begging off and framing statements around the inquiry I feel. Please write the tone off to the medium and believe that I’m open to questioning all assumptions, including my own that I either do or don’t currently own up to.
Assumption 1: Not all path dependence (PD) causes instability (IS), and not all instability is caused by path dependence.
The behavior of a rock is about as stable as it gets, but its current moment is certainly determined by the previous (and its previous) moment.
A random shock to a system can cause system instability, but random is, by definition, not path dependent.
I suppose this is the easiest and most direct response to your initial question, but it isn’t a trivial one. People who automatically connect PD and IS do themselves, complexity, and evaluation practice a disservice.
Assumption 2: Neither PD nor IS is a binary proposition. You can have more or less dependence on history. Even among my sisters, one is constantly engaged with things past, I’m always present/forward, and the other two are somewhere in between. My clients (consulting and evaluation) are all over the map as well, so both these characteristics need to have the words “more” or “less” in front of them. Of course, the perfect worlds of maths and computer simulation models can put a finer edge on it, but as practitioners we certainly can’t.
Assumption 3: Complex systems scale–the same patterns show up at different levels of time and across different periods of time. The larger the larger the scale (space, org level, power, time, etc.), the less path dependent the system will tend to be. The smaller the space or time scales, the more path dependent. When I consider the state of Minnesota, arrival of various ethnic groups may pay a very small part in current public policy debates. In the elections from 1900 to 1910 or in the little town of Wobegon, that history could have been quite significant.
This, too, is a non-trivial issue. What is means is that we have to think differently about accuracy and precision in our evaluations. As I understand it, precision is the level of detail (or scale of focus) that we choose. Accuracy is the match between my measure and reality. The more precise my measures, the smaller the scale of focus, and the more obvious IS and PD will be. If I measure the same system with less precision, focus on a larger scale, it will be easier to measure accurately, but PD and IS will seem to magically disappear. At risk of demonstrating the obvious, when I see a river from the plane, I cannot see the effects of last night’s rain, while standing on the shore it will be obvious.
Assumption 4: What we are looking for (the best we can hope for) in messy reality is a “good enough” fit to purpose. An absolute measure of this or that system and its performance is a holy grail–not worth the effort even if we could find it. What we strive for in evaluation is enough information to support decision making. For some things, precision and PD and IS can be sacrificed for the good of national decision making. For other things, precision, PD, and IS are the point, so measures have to be precise enough to capture them. When we are decision evaluations, we must ask, “What decisions will this work support? What is the “best” scale to provide the information we or our clients need?”
Aside: This, by the way is what I think is wrong with ed policy in the US. Different levels of precision and accuracy apply to national policy on education and to this child learning to read in this classroom. In this case, the policy decisions have no regard at all for the misfit that exists. Teachers who are well aware of the lack of fit have no power or voice to do anything about it.
Of course, knowing the scale (and instability, complexity, and path dependence, etc.) is hardly enough. You also need to have in your toolbox models and methods that are effective at the relevant scale. You also need to be able to remember and explain to others the strengths and limitations of whatever scale of description you choose.
I’m looking forward to your session and to other conversations between now and then. I think you’re focusing on real issues of complexity doing (as opposed to complexity thinking). I’ll be curious to hear what you think about these questions in relation to Royce’s and my new book Adaptive Action: Leveraging Uncertainty in Your Organization (Stanford University Press, 2013). You can read about it at adaptiveaction.org. Thanks for this and all. G
Hi Glenda –
Thanks for the comments. I have just a few thoughts.
——–Glenda says——-
If we try to, we just give a good excuse to the culture wars of qual/quant that have been raging (and serving no one IMHO) for a long time. Instead, I’d like to see complexity theory and practice help us get past trivial distinctions and attend to differences that really make a difference. I think your question is moving toward that.
——–End Glenda says——-
——Begin Jonny responds——–
There is indeed a culture war between the qual/quant camps, not to mention the war’s links to matters of economics (who gets the funding) and politics (who gets the access). But from an epistemological point of view I don’t think this is a qual/quant issue. Rather it has to do with beliefs about whether there is enough stability over time and place to assume that findings in one study can be applied in other settings. Of course this often becomes an argument between advocates of quantitative and qualitative methods, but it does not have to. I can easily see case studies that rely very heavily on quantitative data, but for which people would still say “ah but the findings won’t (can’t) generalize. And I can see a qualitative study for which people would claim extension beyond time and place. For instance one of the cases in my book was a qualitative study of how refugees adjusted to resettlement camps. I can’t speak for the authors and they don’t talk about it in their write-up. But I’d bet they would claim that the findings would generalize, at least for the same ethnic groups being resettled under the same policies in the same country.
——End Jonny responds——–
——–Begin Glenda says——-
Complex systems scale–the same patterns show up at different levels of time and across different periods of time. The larger the larger the scale (space, org level, power, time, etc.), the less path dependent the system will tend to be. The smaller the space or time scales, the more path dependent.
…
What is means is that we have to think differently about accuracy and precision in our evaluations…
——–End Glenda says——-
——Begin Jonny responds——–
Yup. This touches on the rough lines of thought I have been mulling over with respect to making good choices about using traditional evaluation tools when we know the systems are complex. Once I think more on this I’ll post it, but don’t hold your breath. I don’t know how long that will take.
——End Jonny responds——–
——–Begin Glenda says——-
What we are looking for (the best we can hope for) in messy reality is a “good enough” fit to purpose. An absolute measure of this or that system and its performance is a holy grail–not worth the effort even if we could find it. What we strive for in evaluation is enough information to support decision making.
——–End Glenda says——-
——Begin Jonny responds——–
Totally right, and a good reason for people in our business to think of ourselves as engineers and not scientists. There is a lot of interesting writing on the differences with respect to theory, research, and conclusions. For more on this see “Evaluation as Social Technology” https://evaluationuncertainty.com/evaluation-as-social-technology/
——End Jonny responds——–
Thanks, Jonny. The question pops up again, why do you think that stability is the key factor in transferability or generalizability? I can imagine two very stable systems where a solution from one could not be applied to another. Two traditional and stable programs, one in the city and one in the country. I can also imagine unstable systems where solutions could be applied in multiple settings. I once knew school teachers in urban NY and on the edge of civilization in AK who found much to share in engaging students with each other and with literature. I think there’s something I’m not getting when you talk about stability as the key determinant. Can you help me out?
Hello! I’m at work surfing around your blog from my new apple iphone!
Just wanted to say I love reading through your blog and look forward to all your posts!
Carry on the superb work!
Hello! I could have sworn I’ve been to this website before
but after browsing through some of the post I realized it’s
new to me. Nonetheless, I’m definitely delighted
I found it and I’ll be book-marking and checking back often!
Nice post. I learn something new and challenging on sites I stumbleupon every day.
It will always be interesting to read content from other authors and practice something from their sites.