Surprises in Programs and their Evaluations

I might be interested in exploring this with you — I was an evolutionary ecologist once-upon-a-time (biology PhD). But I’ll be upfront in saying it’s been a LONG time since I’ve done any population biology (and the sort that your speaking of, while part of my coursework, was not part of my PhD).

I’d also wonder how change in the “environment” impact the fitness of “organisms”? Ie, a program that was “adapted” to its environment a decade ago might be woefully mal-adapted today. Interesting to think about how far this analogy can be taken — programs do not live or die solely by how “fit” (good?) they are, but also politics, mandates….

This would be an interesting intellectual exercise. Would need to think through whether it would useful in a practical sense (though I’m all for science for science sake)?

At the moment, things are very busy with work, but I might be able to set some time aside.

Meghan Guinnee mguinnee@catalystresearch.net

I am new to population ecology, but think that it definitely has some implications that are highly relevant to program sustainability and thereby evaluation. One challenge you might be running into–traditionally, I have seen population ecology as focused on a population of organizations, not just an individual organization. Given that, sustainability-related questions would tend to examine characteristics at the population level at one time point and then again later, to see how characteristics of the population changed as individual organizations survived or did not survive. Which types of programs are more or less common after time has passed? (It always helps me to go back to biological evolution metaphors— over time, the population of primates had larger brains, because individuals with larger brains were more likely to survive and reproduce). Sometimes I find untangling levels of analysis issues helps me. Hope this helps you.
Megan Greeson greesonm@msu.edu

You may want to look at ecological economics methods. In this field, principles of ecology are considered as a guide to organize economic measures. You may find some useful methods to apply to evaluation.

I used ecological economics to frame an evaluation of community sustainability for my dissertation, and I recall struggling quite a bit with metrics and how to “box up” those things that are difficult to measure with typical data until I applied this framework.

Regards,
Jess Chandler jchandler@emiconsulting.com

I could use some help (well, a lot of help actually) to work through an idea I have been pondering lately. It seems to me that the principles of population ecology could be very useful in understanding program sustainability. If that’s right, then those concepts could be useful in the evaluation of sustainability. Continue reading →

I posted a long version of this question to Evaltalk, and got quite a few replies. I am now posting that long version as well as the replies I received. This could be an interesting inquiry. (Whether is it useful is TBD.) Please weigh in as the spirit moves.
If anyone wants to post rather than just respond to existing posts send me email at jamorell@jamorell.com. I’ll set you up with “author” status. Or just send me the material and I’ll post it in your name.

Discussion in the first session of the Azenet Tucson book club – th Theory; using explanatory power section with the introduction to life cycle behavior (p.49). The most common evaluation activity among our members is evaluation of state or federally funded programs (DOE, SAMHSA, OJJDP, BJA). Common characteristics:

• programs have a few years to implement an ‘evidence-based practice’
• evaluation is closely structured around performance measures, often with online reporting requirements
• some projects require comparison groups or other hard-to-implement designs

While programs are ‘start ups’ and are supposed to mature, the rigidity of the reporting requirements means that evaluation often leaps right over implementation issues to outcomes (substituting ‘fidelity’ measures and specific program monitoring – dichotomous = ‘did they or didn’t they do X?” or ‘sustainability’).  Even the Strategic Prevention Framework stuff (SAMHSA,  which is trying to build coalitions to address local substance abuse issues)  doesn’t use implementation theory or program life cycle ideas, at least here in Arizona. So, (surprise!) when we don’t have any theory, the plan doesn’t include any way to measure/record what happened as the program or coalition weathered maturation changes, staff turnover etc., and there’s no way to report on it.   Do you have specific recommendations for social and educational program implementation-stages theory and/or program maturation?  I think that this discussion will continue throughout our reading.

Our book group is part of the Tucson, Arizona contingent of Azenet, an AEA affiliate. Reading Evaluation in the Face of Uncertainty chapters 1-4 has stimulated the rich discussion and experience sharing that we had hoped for, among new and experienced evaluators.  As JM anticipated in the intro, some read cases as they were cited, while others are waiting to read them later and are inserting our own experiences. None of us has discussed systematically common problems, surprises and solutions before.  Can people learn to handle surprises before becoming evaluators/researchers? We agreed that the book would be a great read for an Evaluation II course. We’ve begun to call it Uncertainty in the Face of Evaluation…. More on our discussion next post.