Surprises in Programs and their Evaluations

The NSF asked for proposals to improve evaluation for programs in Science, Technology, Engineering and Mathematics (STEM). We (me, Chris Coryn and Daniela Schroeter) submitted a proposal to test some of the ideas in my book on evaluation surprise by applying them to ongoing evaluation at six STEM sites. Technical section of the proposal.

I want to spark a discussion of what evaluation might look like if it were practiced by people who were working from different ideological frameworks. It has been difficult for me to frame this post because my own politics are distinctly medium rare, and I don’t have the imagination needed to think deeply from other points of view. Still, I think that there are three reasons why this is an important exercise for the members of AEA.

Continue reading →

The range of material we can cover during the workshop is much broader than time will allow. Please look through the slides (see attachment) and post your suggestions for priorities.

As a result of some back and forth with members of AEA’s Systems TIG, I have been thinking about the idea that over and above specific activities in any one organization, what’s also important is its capacity to deal with variability. And presumably, because this is important in organizations, it is important in evaluations as well.
Continue reading →

People have been asking me if I could briefly summarize my work on evaluation in the face of uncertainty. It took me a while, but I finally came up with the following.

NEED FOR CREATIVE THINKING ABOUT UNEXPECTED PROGRAM BEHAVIOR
I believe:
Continue reading →

The range of material we can cover during the workshop is much broader than time will allow. Please look through the slides and post your suggestions for priorities. Logic Models – Beyond the Traditional View: Metrics, Methods, Expected and Unexpected Change

I just returned from a meeting that provoked some fresh thinking about an evaluation problem I have thought about a lot in the past. The meeting was a training session designed to teach people how to address difficult problems that the organization had hitherto been unable to solve. The session was part instrumental, e.g. “When you have difficulty “A”, use tactic “X”. Part of the session was conceptual, e.g. the presenter asked attendees to discuss what they were doing, and a dialogue ensued that showed the insight that might come from thinking of the problem is a different way. This dialogue was repeated many times, thus (hopefully) sensitizing people to how to look at old problems in new ways.
Continue reading →