Surprises in Programs and their Evaluations

This post is to allow participants in the AEA workshop to state their preferences about workshop content, to let others offer an opinion about content for a workshop like this, and for participants to get to know each other.

Please post some information about your professional background.

Slides from a previous version of this workshop can be found at: Workshop slides. Please post your opinion about which sections are most important to you:  1- models, metrics and methodology, 2- graphic design in support of information density, 3- methods of working with stakeholders. Also, are there any particular topics you want covered?

Help

We (Joanne Farley, Tarek Azzam and I) have been musing about whether evaluation as practiced by the members of AEA is framed within too narrow range of political and social ideologies. We suspect that it may be, and that as a result, evaluators miss important elements of program theory, metrics, and methodology.

We are conducting a Think Tank at AEA 2011 as an experiment to test our conjecture. Our plan is to ask AEA members to design an evaluation of a program that we know is embedded in a rich set of values and beliefs. The idea is to take the same program and ask people to design the evaluation from different points of view. Below is a description of the scenario we cooked up. We are looking for ideas about making this one better, or for something entirely different. Thanks in advance to all who take the trouble to weigh in.

Scenario

A government agency has developed a sex education curriculum. In our mythical program

Continue reading →

I’m doing a presentation at the U.S. Department of State’s Fourth Annual Conference on Program Evaluation: “Diplomacy, Development, and Defense – Working Together to Achieve Foreign Policy Goals” June 7-8, 2011

I’m looking for comment and critique. Draft of presentation slides

Proposal

Purpose and Assumptions

The focus of the proposed presentation is methodology to evaluate unanticipated and unintended consequences of program action. It is based on an evaluation theory and a set of case studies developed by the presenter.[1] The specific track for this proposal is “building evaluation capacity”, with an immediate impact on “development”, and a longer term impact on the “building democracy” element of the “diplomacy” theme. The proposed presentation is based on two principles.
Continue reading →

I have been toying with an idea about thinking of “fidelity” in terms of “attractors” as they are cast in complex adaptive systems (CAS). I have yet to convince myself that this idea makes any sense in either a formal or a metaphorical sense. I am even less convinced that even if it does make sense, that it is at all useful in helping to do better evaluation. I’m open to suggestions, so please whack away.

I got interested in this topic because now that “Evaluation in the Face of Uncertainty” has been published, I have been pondering how to advance some of the ideas I set out in that book. The “fidelity/attractor” idea is one of the trails I have been sniffing.

What is the problem?
I think there are two big ideas floating around in our field that are tugging at each other. The first is the notion of “fidelity”, i.e. the notion that for a program to be successful as it moves into different settings, it must maintain a core set of characteristics. (Let’s assume we know what those are.) The second idea is that things are never the same as they move through space and time. We know from research on innovation adoption that “reinvention” is common. And certainly, all our recent talk about developmental evaluation, systems, complexity, and so on speak to the belief that programs are in a constant state of flux. I’m groping for a way to bring “fidelity” and “change” under a single conceptual umbrella.

What are attractors? Continue reading →

Submitted by
Jonny Morell
Joanne Farley
Tarek Azzam

Abstract:
How does political ideology affect program theories, methodologies, and metrics? Participants will be randomly assigned to groups, and asked to sketch an evaluation based on one of three positions. 1) Government has an obligation to alleviate social inequities and thereby promote the public good. 2) Government’s role is to uphold civil order so people to pursue their own goals, with the consequences of their actions being their own personal responsibility. In general, less government is better. 3) The family is the primary unit of social cohesion, and there resides the locus of decisions about issues such as health and education. Government can be active or passive, as long as it supports the centrality of the family as the locus of moral authority and daily living. During report backs and we will compare how the evaluation designs differ with respect to program theory, metrics, and methodology.
Continue reading →

My friend and colleague and partner in crime Sanjeev Sridharan sent a query to a few people asking for reaction to various ideas he has been pondering. I thought it might be a good idea to post my response here.

Sanjeev says:
Sometimes governments invest large amounts of  money on grand ideas — these ideas are often not based on evidence or detailed plans. A kind version of this story is the  “innovation” lies in taking risks based primarily on somewhat vague promises of the original idea. In all likelihood the set of interventions that connect the activities to the long term outcomes  are going to be highly complex even though the language of accountability is framed as though the intervention is either simple/complicated.

Jonny says:

• I talk about part of this issue in Chapter 3 of my book in a section titled: When is the probability of surprise high”. I won’t go into details here, but I argue that given economic, political, social, and human capital realities, governments have little choice but to implement unidimensional solutions to multidimensional problems.
  Continue reading →

I’m sending this email to people who either expressed interest in participating the session or in helping to design it. Welcome fellow partners in crime and trouble makers! The purpose of this message is to get some discussion going to help set up the session. Here is what I have in mind. We can change it and shape it as the discussion evolves.

Theme of Panel
The basic them is how political ideology affects metrics and methodology. To give a simple example of both.

Continue reading →

The NSF asked for proposals to improve evaluation for programs in Science, Technology, Engineering and Mathematics (STEM). We (me, Chris Coryn and Daniela Schroeter) submitted a proposal to test some of the ideas in my book on evaluation surprise by applying them to ongoing evaluation at six STEM sites. Technical section of the proposal.

I want to spark a discussion of what evaluation might look like if it were practiced by people who were working from different ideological frameworks. It has been difficult for me to frame this post because my own politics are distinctly medium rare, and I don’t have the imagination needed to think deeply from other points of view. Still, I think that there are three reasons why this is an important exercise for the members of AEA.

Continue reading →

The range of material we can cover during the workshop is much broader than time will allow. Please look through the slides (see attachment) and post your suggestions for priorities.