We (Joanne Farley, Tarek Azzam and I) have been musing about whether evaluation as practiced by the members of AEA is framed within too narrow range of political and social ideologies. We suspect that it may be, and that as a result, evaluators miss important elements of program theory, metrics, and methodology.
We are conducting a Think Tank at AEA 2011 as an experiment to test our conjecture. Our plan is to ask AEA members to design an evaluation of a program that we know is embedded in a rich set of values and beliefs. The idea is to take the same program and ask people to design the evaluation from different points of view. Below is a description of the scenario we cooked up. We are looking for ideas about making this one better, or for something entirely different. Thanks in advance to all who take the trouble to weigh in.
A government agency has developed a sex education curriculum. In our mythical program
schools will not be required to implement the curriculum, but the program roll-out and dissemination will actively try to influence all stakeholder groups – State Departments of Education, local school boards, and community groups. Also, schools that implement the curriculum will be eligible for technical assistance in its use, and free access to instructional materials.
Long term desired outcomes as articulated by the program’s developers are:
- STD prevention and
- pregnancy prevention.
Intermediate desired changes as articulated by the program’s developers are:
- Increased self esteem of girls who participate
- Increased knowledge of alternatives to intercourse
- Increased ability to negotiate sexual activity with a partner
- Increased knowledge needed to choose methods of contraception
- Increased knowledge about the risks of sexual activity including intercourse, and
- Increased ability to clarify one’s values with respect to engaging in sexual behavior.
Participants will be randomly assigned to one of three groups, and asked to design an evaluation that emanates from one of three political ideologies.
- Government intervention can be effective, and it is legitimate for government to exercise that ability to improve the social welfare.
- Government intervention is most likely to be either not effective or counterproductive, and in any case the right thing for government to do is to do as little as possible so that individuals can act according to their own sense of personal responsibility.
- The family is the center of authority over people’s lives. The decision about whether government should act or not act depends on whether the action furthers or inhibits family authority.
Participants will be asked to proceed with an open mind, i.e. there has to be some recognition that the curriculum has a range of outcomes worth measuring. No fair to dismiss the program out of hand. From this point of view, each group will be asked to produce a high level evaluation plan encompassing:
- Program theory
Will this exercise work to elicit the discussion we are looking for? What else might we do?