Help
We (Joanne Farley, Tarek Azzam and I) have been musing about whether evaluation as practiced by the members of AEA is framed within too narrow range of political and social ideologies. We suspect that it may be, and that as a result, evaluators miss important elements of program theory, metrics, and methodology.
We are conducting a Think Tank at AEA 2011 as an experiment to test our conjecture. Our plan is to ask AEA members to design an evaluation of a program that we know is embedded in a rich set of values and beliefs. The idea is to take the same program and ask people to design the evaluation from different points of view. Below is a description of the scenario we cooked up. We are looking for ideas about making this one better, or for something entirely different. Thanks in advance to all who take the trouble to weigh in.
Scenario
A government agency has developed a sex education curriculum. In our mythical program
schools will not be required to implement the curriculum, but the program roll-out and dissemination will actively try to influence all stakeholder groups – State Departments of Education, local school boards, and community groups. Also, schools that implement the curriculum will be eligible for technical assistance in its use, and free access to instructional materials.
Long term desired outcomes as articulated by the program’s developers are:
- STD prevention and
- pregnancy prevention.
Intermediate desired changes as articulated by the program’s developers are:
- Increased self esteem of girls who participate
- Increased knowledge of alternatives to intercourse
- Increased ability to negotiate sexual activity with a partner
- Increased knowledge needed to choose methods of contraception
- Increased knowledge about the risks of sexual activity including intercourse, and
- Increased ability to clarify one’s values with respect to engaging in sexual behavior.
Exercise
Participants will be randomly assigned to one of three groups, and asked to design an evaluation that emanates from one of three political ideologies.
- Government intervention can be effective, and it is legitimate for government to exercise that ability to improve the social welfare.
- Government intervention is most likely to be either not effective or counterproductive, and in any case the right thing for government to do is to do as little as possible so that individuals can act according to their own sense of personal responsibility.
- The family is the center of authority over people’s lives. The decision about whether government should act or not act depends on whether the action furthers or inhibits family authority.
Participants will be asked to proceed with an open mind, i.e. there has to be some recognition that the curriculum has a range of outcomes worth measuring. No fair to dismiss the program out of hand. From this point of view, each group will be asked to produce a high level evaluation plan encompassing:
- Program theory
- Outcomes
- Metrics
- Methodologies
Will this exercise work to elicit the discussion we are looking for? What else might we do?
Evaluation Uncertainty 
While I agree with the general premise, I do, however, see a fundamental problem if you are really interested in ideology and examining it in any meaningful way. The scenario is a great example, but my concern is the task that you then assign. The way that you have set up the task is itself a prescriptive ideology in that you require participants to develop an evaluation plan that includes (1) a program theory, (2) outcomes, (3) metrics, and (4) methodologies. In doing so, you have already, in part, imposed your own ideology on the think tank participants. Arguably, many approaches do not require specification of a program theory, for example, such as Scriven’s consumer-oriented approach, Stufflebeam’s CIPP model, Stake’s responsive approach, and many, many others. I would encourage you to leave the task open-ended rather than asking them to address the four specific elements that you defined. Otherwise, in my opinion, you are confounding your question by forcing the participants to plan an evaluation according to the specific tasks (i.e., program theory, outcomes, metrics, and methodologies) that you see as central (which is itself an ideology).
I agree with Chris, but I also think there is an issue with the notion of ideology. If we assume that ideology is an organised and internally coherent set of beliefs then I’m not sure that the three ideological dimensions you identify are the kinds of belief structures that are germane to the evaluation task. They don’t actually help influence the worth of an intervention, just whether people think the intervention has merit.
More subtle but critical ideologies are at play – and anyone who has a passing knowledge of realist evaluation will be familiar with them. These are the beliefs that underpin our own beliefs about how interventions work. There are carrot people, there are stick people, there are people who believe that attitudes are affected by information and those that believe that attitudes affect the kind of information we take notice of. And so on. I appreciate that these are closer to what you’d call TOCs – but what are TOC’s but beliefs in how the world works. But they are the ideological structures that underpin what people think is worthwhile. Frankly I don’t have any time for the ideologies that underpin social marketing – one of the great late 20th century frauds invented by the advertising industry to milk the public purse of billions of dollars. But there are those that do.
I also agree with Chris so I won’t bother restating his argument. However, I also take pause at the terminology being used here. You state that you are examining about political ideologies yet your options are (essentially) government mandated, individual-centric or family centered. This seems very limited in its political scope and perception and feels more socially constructed than politically ideological. Perhaps I’m missing something? Are you examining social constructs and how they influence evaluation (which they certainly do) or political, which to me would infere differe econoic and political systems (which again, completely change the influence and outlook of evaluation)? Also, asking people to pretend they are one type of social construct (or political for that matter) just begs for more stereotyping and assumptions of behavior than actual action or perception in creating evaluation.
While Chris makes a good point, it seems to me perfectly legitimate to constrain the task to a single evaluation approach – it’s a workshop and an experiment – lots of parameters have to be constrained. I’m more concerned with Jonny’s original premise. You’ve already assumed a program theory and it’s based on the first political ideology (i.e., government intervention works). The program makes almost no sense from the point of view of the other ideologies. In effect, you’re asking the participants assigned to the other two ideologies to evaluate a program they fundamentally don’t (and can’t) believe in. Participants will be logically unable to define a program theory and without that they can’t proceed.
What if you take a step back and have participants start by designing programs from the POV of each of your three ideologies and then see what different types of evaluations come from those? Your starting point would be your two long term outcomes. Obviously intermediate outcomes and program theory would be radically different. This probably no longer answers your original research question, but it would sure be interesting!
You all make good points. Mitch has it right about needing to constrain the exercise. As for my starting assumption based on the beliefs of the program designers, I did that partially on purpose and partially because I was not smart enough to pick up on what I did. Fell into my own trap, I guess. BUT in my defense… I don’t think the approach we have taken is so farfetched. It’s easy to imagine a situation where the program designers do in fact have a program theory in mind, and by the very act of designing the program, have put their stake in the ground. Along come other groups and challenge what are in essence, facts on the ground, the facts being that the program was developed and is being disseminated. At that point they would be saying something like: “Whoa, not so fast. We can’t stop you from disseminating the program, but we can insist that you consider all the possible outcomes, which because of your narrow point of view, you have not done.”
You’re suggesting they might attack the program on rational grounds? That’s just crazy! But, given that fantasy world view, I think your idea might work and could provide a fun and enlightening time for all. You do need to make sure to build the fantasy nature of the exercise into the scenario quite explicitly.
@jamorell, Ideology in evaluation: Resolution of amounting clash of ideologies seems decisive for better evaluation and policy making in the future – in particular for strategic and program evaluations of large scale and multi-scope policies. This is at least one of the main conclusions from our study(1) of the aggregation problem which arises when incommensurably different value systems are involved in evaluation, such as ecological, social, human and economic impacts. This problem is studied in Slovenian Evaluation Society (including Mojca Golobič (2), landscape planner, on evaluation of territorial cohesion, Srečo Dragoš, sociologits, on meta-ethics in social work, Mirna Macur, sociologist, on self-evaluation and myself, macroeconomist, in methodology of evaluation). Our proposal is to start with Leopold matrix of impacts, transform it into Leontief’s input-output matrix of 3-4 evaluation scopes (or »ideologies« in your terms) and further synthesise it into the correlation matrix. Results are finally presented separately for the primary, secondary and tertiary layer in the form of Venn diagram.
Would you allow rephrasing »ideologies« into »evaluation scopes«? It would be wise to avoid ideological way of thinking in evaluation so that it does not repress the plurality of the concerns. Evaluation is not ideological undertaking but part of policy cycle – don’t you think that one needs to place evaluation in policy context not in the context of politics? For example, three scopes of evaluation may be defined as government concerns, private concerns, family concerns (as Melissa_Nemon also sees it). It may be that there is a set of approaches to achieving the desired goals, which are autonomous between each other: there are certain some government measures that can be fruitfully employed in public domain to directly contribute to achievement of the goal; of course, there are also several possible private measures in private domain (triggered by offering carrots…) and family measures in family domain (triggered by information campaigns…). In this way stakeholders need not to agree which of three mechanisms is valid. In this way policy makers evaluate their option so as they take various legitimate options into account. Not the cannons of the ideology but an alternative mix of approaches is what is comparatively studied in the evaluation.
If this rephrased problem is evaluated in a complex manner (applying incompatible but legitimate values), several interesting methodological issues arise, such as concerning the synthesis of evaluation results between primary (non-overlapping) and secondy (overlypping) impacts. Our the main finding in our case study is the follownig: »The realization that social matter has to be evaluated in a complex way simply means that it has to be evaluated in the golden ratio of its duality, stretched between the explanation of its primary meanings – which are constitutive for it, but in an incommensurable and deeply dividing way – and the explanation of its secondary meanings that are the only ones that lead to a holistic view, but merely in contents that are not of primary importance to anyone.« (3)
@Chris Coryn: One can not say a word without »imposing ideology«. That is true in a formal binary logic, but leads to relativism and is useless in the context of complexity which is not based on binary logic (Aristotle) but applies multi-valued logic (Łukasiewicz, Zadeh). Yes, jamorell is imposing his own ideology but only to conceptualise the process in a complex manner (if rephrased) so that different »ideologies« can indiscriminately participate. Here we have procedural ideology of evaluator which is open minded and substantive ideologies of contributing stakeholders which are not. And jamorell is inquiring the problems related to the latter, if I get it correctly.
I certainly agree with Bob Williams that the three ideological dimensions are germane to the evaluation task, but I found them nevertheless interesting (after rephrasing) because of a cultural stereotype which is involved in their framing which is quite appropriate simplification for an introductory case study. Framing three deeply oppositional ideologies is needed to study interaction between ideology and evaluation so an open ended case study scenarios would throw a lot of mist on the whole undertaking.
@Mitch Fleischer. Requiring participants in evaluation to define distinctive program theories for three »ideologies« is in my view appropriate and even logically necessary for this kind of case study. If there are different »ideologies« (scopes), then there are different internal coherences (cf. Bob Williams), which are probably already well understood (we are talking here about ideologies of everyday life) – not necessary to evaluators but certainly to the real participants. Coexistence of different elaborations of what is perceived as a social reality is not a fantasy but post-normal condition (Ravetz).
Bojan Radej, Slovenian Evaluation Society (Ljubljana, 4. October 2011)
Sources:
(1) Radej B. Synthesis in policy impact assessment. Sage: Evaluation 17/2(April 2011a):133–50; http://www.sdeval.si/Publikacije-za-komisijo-za-vrednotenje/Meso-Matrical-Synthesis-of-the-Incommensurable.html (preprint)
(2) Golobič M., N. Marot. Territorial impact assessment: Integrating territorial aspects in sectoral policies. Elsevier: Evaluation and Program Planning, 34/3(avgust 2011):163–73, http://www.sdeval.si/Studijska-knjiznica/Clanek-Territorial-impact-assessment…-Golobic-Marot.html
(3) B. Radej. 2011. Primary and secondary perspective in policy evaluation. Ljubljana: Slovenian Evaluation Society, Working papers vol. 4, no. 2(2011), 30 pp., http://www.sdeval.si/Publikacije-za-komisijo-za-vrednotenje/Primary-and-Secondary-in-Policy-Evaluation.html