Lately I have been spending a lot of time thinking about two subjects: 1) models (program, logic, change, etc.) and 2) complex behavior. (Not complex systems. I don’t like that subject.)
It occurred to me that different models are relevant at different time scales. Most of the models one sees in the evaluation world involve clear outcome chains between short, intermediate, and long range outcomes. Usually those long range outcomes are aspirational in two ways. 1) Nobody ever stays around long enough to actually evaluate whether the program in question had any impact. 2) It’s just as well that the effort was not made because at any reasonable time into the future, the methodology would fall apart. It would not be powerful enough to detect change.
I know this second statement is far from universally true, and that many very good long term evaluations have been done. Still, it seems true enough to me when I think of the ratio of the number of evaluations that stop at intermediate outcomes, and the ones that go on to measure long term effects.
With respect to the methodology falling apart over the long run, I’m beginning to think that the reason it falls apart is more than the obvious one of not having the resources and the will to stick it out. Rather, I am of a mind to think that the problem is not the methodology, but the program theory on which the methodology is based.
A look at an outcome chain model conveys the impression that whatever may be needed to evaluate long term outcomes is more of what was done before – more effort to maintain the integrity of control groups, better tactics for staying in touch with interviewees, and so on. I am beginning to question the “more of the same” approach. I am beginning to think that the real issue is that over time the program theory has changed.
My idea starts with an assertion that I know is not true but which may be true enough to be useful. Namely, that any long term impact is based on networking effects. By this I mean that whatever a program accomplishes, over time, those accomplishments form connections with other phenomena. Some of those phenomena may be other programs. Some may be change that was taking place anyway. My notion is that long term change is
- not the additive consequence of all those change activities, but rather,
- an emergent phenomenon that cannot be explained in terms of the individual contributions of its parts.
That’s what I mean by a shift in program theory. The change is from:
- a model that is based on a theory of outcome chain relationships, to
- a model that abandons the outcome chain belief in favor of change based on network activity and emergence.