Lately I have been spending a lot of time thinking about two subjects: 1) models (program, logic, change, etc.) and 2) complex behavior. (Not complex systems. I don’t like that subject.)
It occurred to me that different models are relevant at different time scales. Most of the models one sees in the evaluation world involve clear outcome chains between short, intermediate, and long range outcomes. Usually those long range outcomes are aspirational in two ways. 1) Nobody ever stays around long enough to actually evaluate whether the program in question had any impact. 2) It’s just as well that the effort was not made because at any reasonable time into the future, the methodology would fall apart. It would not be powerful enough to detect change.
I know this second statement is far from universally true, and that many very good long term evaluations have been done. Still, it seems true enough to me when I think of the ratio of the number of evaluations that stop at intermediate outcomes, and the ones that go on to measure long term effects.
With respect to the methodology falling apart over the long run, I’m beginning to think that the reason it falls apart is more than the obvious one of not having the resources and the will to stick it out. Rather, I am of a mind to think that the problem is not the methodology, but the program theory on which the methodology is based.
A look at an outcome chain model conveys the impression that whatever may be needed to evaluate long term outcomes is more of what was done before – more effort to maintain the integrity of control groups, better tactics for staying in touch with interviewees, and so on. I am beginning to question the “more of the same” approach. I am beginning to think that the real issue is that over time the program theory has changed.
My idea starts with an assertion that I know is not true but which may be true enough to be useful. Namely, that any long term impact is based on networking effects. By this I mean that whatever a program accomplishes, over time, those accomplishments form connections with other phenomena. Some of those phenomena may be other programs. Some may be change that was taking place anyway. My notion is that long term change is
- not the additive consequence of all those change activities, but rather,
- an emergent phenomenon that cannot be explained in terms of the individual contributions of its parts.
That’s what I mean by a shift in program theory. The change is from:
- a model that is based on a theory of outcome chain relationships, to
- a model that abandons the outcome chain belief in favor of change based on network activity and emergence.
Evaluation Uncertainty 
Not entirely sure how this advances things. The aspirational nature of ToCs has been talked about since, well forever really. The multicausal nature of outcomes/impacts is far from a new topic in the discourse. Rick Davies (of Most Significant Change fame) argued that network theory was a form of theory of change about ten years ago (and counting). I thought he was onto something but at the time I think I was the only one who did. So he abandoned it after some very promising developments and moved on to other things. It also reminds me of my organisational development days, when we would say that you never reach your goals because by the time you’ve reached them the goal has changed.
As you know I’m not a huge fan of ToCs (as they are commonly expressed and used) but if done well can at the very least surface someone’s thinking process, mental models and assumptions.
In terms of models (or rather what I call maps) I agree that you use different ones at different stages of a job for different purposes. The Open University course on system diagraming has an interesting little quiz at the end which allows you to select from a small suite of system mapping/diagramming approaches (spray diagrams, rich pictures, CLDs, influence diagrams ….)
Anyway keep going on you thread because someone needs to guide us into something beyond the idea that ’emergence’ = stuff happens.
I certainly did not mean to imply that evaluators have not thought in terms of networks. To me the issue is the shift from “multi causal”, to “emergent”. My notion is that even if you knew all the specific causes, it would be impossible to understand the change in terms of knowing each of them. One needs to understand it on its own terms.
Jonny, Your perspective aligns with work I’ve been doing on a theory of transformational change (as different from a theory of change), namely, that major transformations (systems tipping points and critical mass) results from intersecting, mutually reinforcing, and synergistic networks of endeavors. Evaluating transformation, then, is substantially different from project ad program evaluation, both in the long-term frame that you address and in the emergent, unpredictable, and uncontrollable nature of major transformations. MQP