Over the past year or so I have been thinking a lot about novel methodologies and approaches to help evaluators understand unexpected program behaviors. This is part of my general view that it’s possible to increase the lead time between when indicators of unexpected change first pop up, and when the need arises to adjust evaluation methodologies. And, I believe that the longer the lead time, the better the adjustments.
One result of my ponderings has been the idea that agent based modeling (ABM) should be tightly integrated into traditional program evaluation methods. ABM approaches are desirable for a host of reasons, not the least of which is that they are based on the principles of complex adaptive systems (CAS). Because of this connection, combining ABM and traditional evaluation has two advantages. First, it will help with the “unexpected behavior” problem. Second, it can help shift the way in which CAS is presently used in our business.
There is a great deal of talk about CAS in evaluation these days, but for the most part the principles of CAS are only applied in a heuristic fashion. As an example, it is fashionable to speak of “path dependence” as an explanation for instability of logic models and program theory. That’s a pretty powerful idea as far as it goes, but it’s only an intuitive application of the concept. It is not a formal application of the concept of path dependence to system behavior. The only way to apply CAS in a rigorous way is from within cyberspace, where simulations can be run. So by using ABM we not only improve our ability to understand (and predict) system change, but we also help put our use of CAS in evaluation on a more rigorous footing.
The reference for the article is: Integrating Evaluation and Agent-Based Modeling: Rationale and an Example for Adopting Evidence-Based Practices Jonathan A. Morell, Rainer Hilscher, Stephen Magura, Jay Ford Journal of MultiDisciplinary Evaluation Vol. 6, No 14 (2010) 32-57
Evaluation Uncertainty 
I will be attending our workshop in San Antonio and am very interested in system approaches and modeling and how logic models can be used in complex initiatives. I use logic models extensively and am hoping this workshop will be more than an intro to logic models but will take us beyond their traditional uses and explore how they can be used in novel and complex ways.