I’m sending this email to people who either expressed interest in participating the session or in helping to design it. Welcome fellow partners in crime and trouble makers! The purpose of this message is to get some discussion going to help set up the session. Here is what I have in mind. We can change it and shape it as the discussion evolves.
Theme of Panel
The basic them is how political ideology affects metrics and methodology. To give a simple example of both.
Metrics: Imagine a program that taught literacy to immigrants. “First order” impacts are pretty straightforward, e.g. ability to read, quality of life, employability, and so on. But what about the “second order” impacts, i.e. things touching on the overall social condition – impact on community and economy, etc. I’d bet dollars to donuts that people with pro and anti-immigration feelings would come up with different measures. For instance what about a measure of whether wages are driven down, or of local disruption to social services in communities? It is certainly reasonable to posit these changes, but I doubt that the pro-immigration advocates would not do it.
Methodology: Imagine a needle exchange program evaluation being designed by two different groups: 1) public health advocates, and 2) people who had very strong moral objections to doing anything to help drug addicts take drugs. People in the latter group might reason as follows: “I am not closed minded, but I really think this is a bad idea. So I am going to design the most rigorous methodology possible – random assignment, no treatment control, alternate treatment (no needle exchange) group, and a long term follow-up. Why do I insist on this? Because unless the conclusions are bullet proof, I ain’t gonna change my mind that needle exchange programs are a bad idea.” I can easily imagine how the pro-needle exchange folk would be satisfied with a less rigorous methodology.
Use of a common example
My thought is to invent a program as an example we can all work from. It can be based on something real, but I have found that in order to have an example that illustrates all the points one wants to illustrate, some synthetic elements are necessary. The example has to be something on the large and complicated side, but well within the scope of the kinds of programs that AEA members might actually get a contract to evaluate. I want to make sure the example resonates with people’s experience.
Panel development process
If you are up for it, I’d like to design this panel in a two-tier social networking framework.
Tier 1: Us, i.e. the people with the responsibility to design and submit the panel to AEA. Any advice we get from others is just that, advice. We can use it or not as our wisdom dictates.
Tier 2: The rest of the world. We exchange our ideas in the “fishbowl” of my blog site. Content, design, assignments, etc. In other words, I’d put this email message up and instead of our hitting “reply all” we would post to the blog. This would have two advantages. First, other people might have worthwhile things for us to consider. Second, it would generate publicity for the panel. (And frankly, I would not mind increasing hits on my blog, either.)
What I’m proposing is an experiment, and to be honest, I’m not expecting too much in the way of public comment. But I think it’s worth a try if you do. Let’s do the first round of responses via email to get a sense of whether you are willing to work in the fishbowl. If you are, I’ll transition the material.
What do we want out of this?
Aside from the fact that doing this panel would be a lot of fun, I assume that we have our reasons for wanting to be involved. It might help shape our discussions if we knew the reasons. If you don’t have any that’s fine, but if you do, let’s hear them. For me, there are two.
1- Unintended consequences of program behavior: You all know that this is my favorite subject. Once I finished my book and had a chance to think about the intellectual frontiers I had not explored, I realized that ideology can be an important reason for unexpected behavior. Either there are variables people did not think to include, or the methodology has some blind spots. I want help exploring this frontier.
2- Diversity: AEA is big on gender, ethnic, and racial diversity. I am in favor of this, but I think the field would be a lot better off if there were more emphasis on ideological and methodological diversity as well. How much poorer would economics and political science be if there were no research and theory based on both Marxian and free market paradigms? Quite a lot poorer, I think. I have a strong feeling that the politics of AEA members clusters in the medium-rare range.( I can’t prove it, but I bet I’m right.) I’m not arguing that we go out and recruit other members, but I do think that AEA would be a lot better off if we had a collective appreciation of the implications of what we don’t have.
Logistics
There are three questions we need to resolve.
1- What TIG should we submit this to? I’m thinking maybe “Evaluation Theory”, but I really don’t have any strong feelings one way or the other.
2- Deadlines. I think we should set ourselves the goal of being able to submit the panel by the end of February.
3- What format should we use? We could do a panel, but there are other possibilities. For instance maybe a think tank, which gives us less time to present, but the audience a lot more opportunity to participate.
Evaluation Uncertainty 
Re posted from the mail-reply correspondence:
Thanks for the preliminary mail on formation of the T.I.G. That was a detailed mail on the agenda part. I prefer to exchange my views on each individual sections as below under the corresponding headings as in your mail.
Theme of the Panel
Metrics, certainly provide us the right way to approach any problem/context/intervention. Sometimes, not in total certainty, the most crucial ones are often found along the boundaries of interpretation (at least in my case). I believe, pushing these ones into the center core of the program is in itself a task to do. And as you hinted, Advocacy could be probably better linked with the metrics and measurements. Well this explains the fact that the methodology for evaluating takes a slight detour (although not a major deviation, but still is a miniature skid from the track.) while in the process of evaluation. In many cases, these diversions have influenced the outcomes. So good news !!
With Methodology, I prefer to hang-on for some time, until every one is starting to contemplate and observe the plans relevant to this T.I.G. I believe, this aspect is very crucial for R.T.E of any program, thus not limiting the potential of designing a program, just for the sake of discussion. Of late, I have felt a lot more perplexed with methodologies that are adopted for a purpose and the outcomes are from out of this box. (may be from a parameter or element, that was considered inappropriate during the formative stages of evaluation)
Use of a common example:
If you have a set of specific program briefs (which I hope you already have in mind) or we can probably start with an example and as we start the panel, we could get streamlined and organized, to work on a common platform.
Note to this section: I am not thoroughly aware of a typical AEA evaluation contract that are handled by AEA evaluators. But I presume, this isn’t a big fix. I would say programs are all the same through out the world. They all run on contexts, facts and results. So I am not fixated to AEA style (if at all there is one particular schema for AEA setting), but if you feel that It could help us in shaping up this team, you can proffer me advice / clues on the same.
What do we want out of this:
I am inclined more onto the point 1 (lacuna in methodology or unraveled program alternatives) than point 2. (Diversities in methods)
A think-tank sounds promising to me. But I will wait to hear from the rest of this T.I.G.
Lets start !! 😉
Jonathan,
I agree with all that you propose re. the panel.
I think tracking any dialogue about the subject on your blog would be quite useful and I certainly don’t mind working is a fishbowl.
I also agree that the Evaluation Theory TIG at least should be the natural home for this discussion given ideology is often mistaken for “theory.” At the very least, the explicit and inexplicit theories that underlay our choice of evaluation questions, methods, and measurements as well as the outcomes we evaluate are framed by ideology in most cases.
I think a Think Tank format would be a good one. However, I’d also suggest we consider Jennifer Greene’s invitation for Presidential Strand session. She summed up what she is looking for in the following:
You are hereby invited to submit a proposal for a 45-minute session for the Presidential Strand at the annual conference that creatively uses alternative representations, specifically as a means to more fully engage with the values dimensions of our work – the diverse values of stakeholders, values embedded in our evaluation designs, values underlying our evaluative judgments, values related to evaluation’s role in society and to our own commitments and ideals.
I’m not sure what “alternate representations” we might use but it would be great to have this dialogue as a Presidential Strand session. And I most truly believe that ideology and values and valuing are inextricably linked. So just a suggestion.
Just FYI. I’d like to do this session because I think it is crucial that evaluators recognize and bring into the light the fact that their evaluations are ideologically driven. This reason is closely connected to the second reason you have for doing the session – discussing how a multiplicity of ideologies are (may be) operative in the field of evaluation.
Thanks for your already hard work on this panel!
Unexpected program outcomes: Metrics, Methodology, and Political Ideology
Overview of proposed Think Tank Session
My inclination is to submit a Think Tank session with the above title. The plan would be to invent a program, break the participants into groups, and ask them to sketch an evaluation in terms of what methodologies and metrics would be needed. Each group would be given instructions to design their evaluation from a particular point of view.
As for explaining how the session came about, I’d frame it in terms of my work on unintended consequences. My book spends a lot of time dealing with how theoretical orientations affect program theory. This think tank is an exploration of one aspect of that topic.
Points of View for the Evaluation Scenario
I’m not sure which ideological positions should be represented, but I think it would be best to use three that seem to be comprising so much of the political debates these days. One would be a somewhat left of center, traditional liberal view. Here we would assume that government can do good and that the way to run society is to find ways for it to do as much good as possible, while recognizing that central control of complex systems is impossible, and that it is destructive to try.
The second point of view would be what I call an economic/political view of small government. Here the assumption would be that private sector activity usually trumps government efforts at change, but that government does have to provide a structure in which free market activity can thrive. This means for instance that infrastructure investments are important, e.g. the air traffic control system. Another example would be certain kinds of market oversight that support honest competition (e.g. enforcement of anti-insider trading activity), and so on.
Finally I’d throw in a social conservative viewpoint, that thrived on value judgments, e.g. that government should not infringe on how parents raise their children. Here the question is not whether government is fostering good child raising, but the question of whether government should dabble in it at all.
Task Given to Break Out Groups
When people entered the room they would be given a color coded card that assigned them to a group and contained pertinent information on their task – the evaluation scenario and the political position they had to take. Then we would split up the 90 minutes evenly – 30 for explaining the task, 30 for small group break outs, and 30 for reporting back.
The groups would be asked to outline the metrics and methods for the evaluation. Both matter. For instance, take immigration. The data (I think) show that immigration is beneficial in the long run, but it certainly does cause local disruption. So what should the methodological decision be about the timeline for the evaluation? How much data should be collected on the (presumably negative) local disruptions? As another example imagine a program on abstinence only sex education. (I know there is lot of data on this topic, but let’s pretend there is not.) If I had moral qualms about premarital sex of any kind, I might insist on a randomized control group design. Why? Because I don’t like sex education, but would accept its value if the standard of proof were high enough. So I’d want the best design I could get. As for metrics? Program proponents might be happy with measures of pregnancy rates. I might want to include measures of possible unpleasant consequences on self image. Why? Because my program theory says that such negative consequences may occur. The proponent of sex education might not have such a program theory.
No matter what ideology the groups are assigned to, they would be instructed to act in good faith. For instance various views of government lean in the direction that government can do no good and should be allowed to do as little as possible. The “good faith” rule is that evidence may reveal that a program is succeeding, and that if it is succeeding, even if less government is better, this program should continue. We need the good faith provision to keep the focus on empirical data and its implications.
I see a few problems in pulling this off. First, we would have to have a thought provoking evaluation scenario. Second, we would need a pithy but cogent explanation of the key aspects of the political ideology that people were assigned to. Third, I am making the assumption that the attendees will be able to get into their assigned roles. Hope springs eternal. I think we could make it work.
Jonathan,
My thinking about the three positions or ideologies you lay out is somewhat different from yours. First, I would suggest that a “traditionally liberal” ideology falls much more into your second position than the first. Also, I disagree that folks left of center (whatever you mean by “center”) do not try to find ways for government to do good. Rather, they identify the areas in which inequities of power, influence, wealth, etc. result in diminished quality of life for certain groups. They also identify areas where certain groups suffer discrimination and injustice due to alternative ideologies operative in society the assumptions and beliefs of which lead to this discrimination and injustice. Folks on the left tend to see that a primary function of government is to level the playing field so that all persons can take advantage of the opportunities for a quality life afforded by an industrialized democracy.
Also, in order to get evaluators thinking at our think tank, I think we need to expand a bit on the second position, particularly in two ways. First, the economic/political ideology assumes a society in which members act primarily out of self interest. This is a crucial assumption linked to the notion of laissez faire economics and markets. Secondly, while I don’t doubt that folks who hold to the political/economic (the right) ideology thoroughly believe in broad democratic values, the ideology itself leaves no room for moral criteria. It is sort of a “what the market will bear” belief; thus, the fact that baseball players make millions while sanitation engineers make far less (but are far more important to the health of the polity) is not a moral issue of justice but rather an outcome of market economics.
Finally, I think we need to expand on the “family values” (or as you have named it, the social conservatist) ideology. This ideology and the groups that adhere to it have an ideological vision of what family including educational values are and reject values counter to their values.
I’m saying we need to expand a bit on what the assumptions and values are in these three ideologies in order for attendees to work through the questions we ask of them.
Do you have any ideas yet re. the evaluation example we might use? Even with Obama’s proposed reforms, the No Child Left Behind legislation has created a great deal of discourse among and between these three positions. Just one example.
Joanne
Joanne–
Thanks for your observations. I think what we need is to define a few “core beliefs” (3 – 5 or so maybe) that characterize each position and which can be set out in bullet points. You seem to have done a lot more thinking than I on this subject, so I vote that you give it a shot and post to the website.
As for the example – I have no idea what a really juicy one would be. I’m open to suggestions.
Before I take a “shot” at bulletting 3-5 key assumptions of ideological positions, let me get some guidance. Jonathan, in some ways, I like your idea of using ideologies operative these days. However, I’m not so sure those are ideologies operative so much in the evaluation field as in the national political landscape (as a number of authors have pointed out, the differences between left of center and right of center politically have become very short indeed!). We may want to focus on those ideologies that have characterized the evaluation field for some time. In some ways, these are more extreme and may make better and clearer examples with which participants of the session can work in considering methods, metrics, etc. I’m thinking here of what we could call traditional liberal views that are infused with traditional “scientific” (which are themselves ideological) perspectives vs. theories (if not ideologies) such as critical theoretical evaluation, transformative evaluation, participatory evaluation, and so forth. In short, I think it would be more relevant and perhaps more feasible for participants to work with some of the diverse theories/ideologies that underlay current evaluation approaches than interrelating national political ideological stances with evaluation approaches (though I agree that ultimately they are related).
What do folks thing? Send me some feedback (ASAP if possible) and I’ll bullet some diverse and distinct theoretical/ideological approaches to suugest we work with.
Re. a single example of evaluation, I’m not as sure about this. My own expertise is primarily in the education field. Thus, I tend toward using a large-scale evaluation effort of a program that comes out of the No Child Left Behind context. Such evaluations offer opportunities (if we are creative and do not just stick to GPRA performance standards) to consider and work with how diverse ideologies would deal with the national goal of enabling our school children to learn better and “perform” better. One such program comes out of OSEP and is the State Improvement Grant program. The grant’s overall goal is for state Departments of Education to work with the lowest performing schools in the state (as identified by mandated annual statewide assessments) in evaluating the effectiveness of interventions to improve classroom instruction and student achievement. Consideration of this grant provides lots of room to discuss how diverse theories/ideologies frame the effort, interpret interventions, conceive of individual learners, and define and measure student achievement.
But just an idea. I’d like to hear more. In fact, while I don’t know of one particularly, a good example would be an evaluation of a large scale program that focuses on increasing the quality of life of persons in poverty with a major focus on economic outcomes such as job obtainment, career mobility, etc. That kind of evaluation would be excellent to demonstrate how diverse theories/ideologies approach the whole problem, program, methods, measures, and expected outcomes. If anyone know of such an evaluation, you might suggest it.
Thanks, Joanne
As I see it, it is precisely the national ideologies we want to focus on. The issue is not our internecine struggles. The issue is that we operate in a larger framework of ideas about how society should be organized, but we implicitly operate as if that framework is much narrower than it really is.
As for the example, I like yours and I’m sure there must be other candidates as well. I really don’t have a good handle on what a really juicy example would be.
Joanne-
Have you been pondering the critical bullet points? I want to compose the submission to AEA.
Jonathan, here is a first stab at characterizing three ideological positions. See if you think they “hold water.”
…..The think tank will explore the interrelationships between diverse ideologies, the values that underlay or are generated by these ideologies and the practice of evaluation. One ideology comes from the left of the American political spectrum. While believing in individual freedoms, this ideology implicitly accepts the organic nature of the polity so that outcomes of political processes impact (e.g., promotes, limts) social goods and the public good. Consequently, it views government’s role as that of taking action to achieve equal opportunity and equality for all. Because the left believes that there are a range of unfair inequities in the quality of American life, they often advocate for change in political, economic and social conditions. Indeed, from their perspective it is the duty of government to alleviate social inequities and ills and to protect civil liberties and human rights.
Another ideology falls on the right of the political spectrum. Adherents of this ideology generally believe in an atomistic society created by individuals pursuing self-interest. Government, they believe, should interfere with the decisions and actions of citizens when there is a need to uphold civil order and a need to protect the freedoms necessary for individuals to pursue their own goals. So long as individuals have the liberty to pursue their own goals, the consequences of their choices are a matter of personal responsibility. The principles of laissez-faire are still positive principles for believers in this ideology as applied to economics and with the advent of interest group politics, to political processes.
Then there are a group of individuals who referred to as social conservatists. Their position is more recent in origins and is a bit muddled as they often share beliefs with persons on the political right and with the Christian right in the United States. Nevertheless, there are some basic principles around which social conservatists cohere. Whereas a major distinction between persons on the left and those on the right is the role of government – its scope of authority and rightful jurisdiction – social conservatists view this issue in light of the institution they believe to be the focal point of life which is the family. For social conservatists, approval of whether government interference expands or contracts has more to do with how governmental expansion or limitation impacts the family as the proper source of authority on issues affecting family values, morality, and family cohesion. Thus, you may find social conservatists agreeing that government should make abortion illegal while disagreeing that government should promote (from their perspective) certain values in education. Their perspective on social change which often distinguishes the left from the right is for the social conservatist a matter of whether or not change will strengthen the fiber of the family as the basic unit of society or weaken it. In short, social conservatists may drift to the left or to the right on a number of public policy issues except when these policies are seen to impact the authority of family and family values.
Draft submission for AEA 2011
150 word limit
Surprise in Evaluation: Appreciating how Political Ideology Affects Program Theory, Metrics, and Methodology
How does political ideology affect program theories, methodologies, and metrics? Participants will be randomly assigned to one of three groups, and asked to sketch an evaluation based on one of three positions. 1) Government has an obligation to alleviate social inequities and thereby, promote the public good. 2) Government’s role is to uphold civil order to allow people to their own goals, with the consequences of their actions being their own personal responsibility. In general, less government is better. 3) The family is the primary unit of social cohesion, and there resides the locus of decisions about issues such as health and education. Government can be active or passive, as long as it supports the centrality of the family as the locus of moral authority and daily living. Each group will report back and we will compare how the same program would be evaluated from each of these positions.
Looks really good, Jonathan! You went right to the hearts of the three positions without having to get into the confusions they can create! I’ll be eager to discuss the evaluation example(s) to use. Thanks for taking the lead on this – I think it will be very instructive.
Good reviews and points on the planned proposal. I really need an example to ponder and critique on the ideological stand point of this panel.
Now let’s get some discussion going on the example.
Think Tank on Political Ideology in Evaluation
The Think Tank proposal requires a 500 word explanation of its contribution to the field. I tried to be both accurate and inflammatory. How did I do?
Evaluation is at its core a conservative business. We do not develop new programs. We do not plan innovative policies. We spend our time, our intellectual capital, and our resources on assessing programs that other people have already committed to. We are constrained to evaluate within the narrow frameworks given to us by our customers – i.e. by stakeholders who have the power, authority, influence, position and drive to make innovation happen. We should not denigrate what those stakeholders do or who they are. Bringing about change is extraordinarily difficult, and those committed to effecting change are acting in good faith to do good as they see that good. Providing empirical understanding of the consequences of those actions, and guidance for improvement, is a noble pursuit. There are debates about this in our field. How much should we attend to the needs of stakeholders who are not involved in program development or execution? What effort should be put into creating knowledge rather than guiding decision makers? How far should we stray from measuring the outcomes that were planned for the program at hand? We cannot do our jobs well without confronting these questions. But as a practical, (and perhaps as a moral) matter, we cannot, (or perhaps should not), stray too far from the needs of the people who pay us. How then, to best serve those needs? We contend that we can help our customers best if we base our work on an appreciation of the hidden assumptions in program theory that are based on political ideology. Program theory drives metrics and methodologies, and metrics and methodologies frame our findings and our conclusions. If we are trapped within our ideologies, we cannot help our customers to understand the full scope of the changes they are trying to implement. We may or may not succeed in getting our customers to include that scope in their operations, but we cannot even try without self knowledge. We believe that precious little of that self knowledge resides within the intellectual stock of the individual members of AEA, and within our membership collectively. That needs to change. We see this Think Tank as a small step in bringing about that change.
Johny,
This is in regard to the reply for the think-tank proposal. You have done a great job outlining the plans for debriefing the ideological perspective of this T.I.G. In a way, your choice of the word, “conservative” in ensconcing the realm of evaluation activity sounds much appropriate to the field itself. Indeed, we do not develop new programs. I guess this observation is grounded within the foundation blocks of conducting evaluations. I would say, often evaluators are ought to be reminded about the fact that they are looking to improve and enhance, rather than creating or inventing, at least at certain levels of producing an impact in a program or policy. Thus we do not innovate on policies, as not to prove a point, but to actually induce and influence outcomes from the perspective of planning a policy or approving a policy mandate. We definitely assess other people / agency / organization’s worth through assessment and diligent observation. If the constraints are posed by the program personnel, in a way to hinder innovation, I believe there could be a deliberate attempt to deviate the purpose of running a program. Hence forth, I would prefer to expand or limit the framework within which we operate as just evaluators to cause a matter of result.
Your question, “How much should we attend to the needs of stakeholders who are not involved in program development or execution ?” is a fascinating inquiry that weighs in terms of delineating power, authority and coercion through the cycle of evaluating the merit of a program. For e.g, you might want to look at the level of participation from the authority exercising decisions about the program. Alternatively, stakeholder consensus stems from the fact that data-collection has to be unbiased in producing positive outcomes for the whole group, rather than to consume or dismiss individual prejudices.
Your take on the question, “How far should we stray from measuring the outcomes that were planned for the program at hand?”, is a challenging one to include in our think-tank inquiry, because it involves subjective judgement leveraged on achieved outcomes of that particular program. May be here, an example of one such program would suffice the boundaries of measuring outcomes.
Rest in the summary of the TT proposal is self-explanatory.
I would like to add a thought, if at all it interests you, with respect to this T.I.G.
How far are we inclined to avoid political en masse while designing an evaluation? And is the answer to this question categorical or generic? And if at all misleading, is it capable of changing the scheme of things from the perspective of an evaluator? (meaning an effective intervention amidst mediocre procedures)
I appreciate your work, up until now in forming this T.I.G. And with an example and further brain storming, We should be able to serve the true purpose.
–
Sudharshan
Jonathan, you’ve done a great job framing the think tank. However, it isn’t until half way into your explanation of what we’re doing that you raise the issue of political ideology and evaluation. Also, clearly, what you have written represents your interest in undertaking this think tank. On the other hand, my interest in it is the intersections between values and valuing, political ideologies, and the practice of evaluation. Perhaps we could revise the description to read something like (suggested deletions are striked through, suggested additions are underscored):
As the theme of this year’s annual meeting reflects, the role of values is central to the practice of evaluation and much discussion has occurred around it. Far less attention has been paid to the often implicit ways in which political ideologies, so rich in normative content, affect the decisions and actions of evaluators in the practice of their craft. This think tank will provide participants with opportunities to explore the intersections between values, ideologies and evaluative choices and interpretations. For example, it can be argued that evaluation is at its core a conservative business. We do not develop new programs. We do not plan innovative policies. We spend our time, our intellectual capital, and our resources on assessing programs that other people have already committed to. We are constrained to evaluate within the narrow frameworks given to us by our customers – i.e. by stakeholders who have the power, authority, influence, position and drive to make innovation happen. We should not denigrate what those stakeholders do or who they are. Bringing about change is extraordinarily difficult, and those committed to effecting change are acting in good faith to do good as they see that good. Providing empirical understanding of the consequences of those actions, and guidance for improvement, is a noble pursuit. There are debates about this in our field. Further, the conservative nature of evaluation has been the impetus for debate around a number of critical issues. How much should we attend to the needs of stakeholders who are not involved in program development or execution? What effort should be put into creating knowledge rather than guiding decision makers? How far should we stray from measuring the outcomes that were planned for the program at hand? We cannot do our jobs well without confronting these questions. But as a practical, (and perhaps as a moral) matter, we cannot, (or perhaps should not), stray too far from the needs of the people who pay us. How then, to best serve those needs? We contend that we can help our customers best if we base our work on an appreciation of the hidden assumptions in program theory that are based on political ideology. Program theory drives metrics and methodologies, and metrics and methodologies frame our findings and our conclusions. If we are trapped within our ideologies and the normative implications they have evaluative decisions and action , we cannot help our customers to understand the full scope of the changes they are trying to implement. We may or may not succeed in getting our customers to include that scope in their operations, but we cannot even try without self knowledge. Moreover, the normative implications of our evaluations may remain implicit and thus, outside the arena of normative discourse. We believe that precious little of that self knowledge resides within the intellectual stock of the individual members of AEA, and within our membership collectively. That needs to change. We see this Think Tank as a small step in bringing about that change.”
Jonathan, you’ve done a great job framing the think tank. However, it isn’t until half way into your explanation of what we’re doing that you raise the issue of political ideology and evaluation. Also, clearly, what you have written represents your interest in undertaking this think tank. On the other hand, my interest in it is the intersections between values and valuing, political ideologies, and the practice of evaluation. Perhaps we could revise the description to read something like (suggested deletions are striked through, suggested additions are underscored):
“As the theme of this year’s annual meeting reflects, the role of values is central to the practice of evaluation and much discussion has occurred around it. Far less attention has been paid to the often implicit ways in which political ideologies, so rich in normative content, affect the decisions and actions of evaluators in the practice of their craft. This think tank will provide participants with opportunities to explore the intersections between values, ideologies and evaluative choices and interpretations. For example, it can be argued that evaluation is at its core a conservative business. We do not develop new programs. We do not plan innovative policies. We spend our time, our intellectual capital, and our resources on assessing programs that other people have already committed to. We are constrained to evaluate within the narrow frameworks given to us by our customers – i.e. by stakeholders who have the power, authority, influence, position and drive to make innovation happen. We should not denigrate what those stakeholders do or who they are. Bringing about change is extraordinarily difficult, and those committed to effecting change are acting in good faith to do good as they see that good. Providing empirical understanding of the consequences of those actions, and guidance for improvement, is a noble pursuit. There are debates about this in our field. Further, the conservative nature of evaluation has been the impetus for debate around a number of critical issues. How much should we attend to the needs of stakeholders who are not involved in program development or execution? What effort should be put into creating knowledge rather than guiding decision makers? How far should we stray from measuring the outcomes that were planned for the program at hand? We cannot do our jobs well without confronting these questions. But as a practical, (and perhaps as a moral) matter, we cannot, (or perhaps should not), stray too far from the needs of the people who pay us. How then, to best serve those needs? We contend that we can help our customers best if we base our work on an appreciation of the hidden assumptions in program theory that are based on political ideology. Program theory drives metrics and methodologies, and metrics and methodologies frame our findings and our conclusions. If we are trapped within our ideologies and the normative implications they have evaluative decisions and action , we cannot help our customers to understand the full scope of the changes they are trying to implement. We may or may not succeed in getting our customers to include that scope in their operations, but we cannot even try without self knowledge. Moreover, the normative implications of our evaluations may remain implicit and thus, outside the arena of normative discourse. We believe that precious little of that self knowledge resides within the intellectual stock of the individual members of AEA, and within our membership collectively. That needs to change. We see this Think Tank as a small step in bringing about that change.”
Or something like this. I think we need this to immediately make the link to values and valuing and as it already does, to evaluation. But it needs to be clearer that the purpose of the think tank is to help us all think through how political ideologies and the values that they implicitly or explicitly rely on affects our decisions as evaluators.
Sudharshan, your raise the following:
How far are we inclined to avoid political en masse while designing an evaluation? And is the answer to this question categorical or generic? And if at all misleading, is it capable of changing the scheme of things from the perspective of an evaluator? (meaning an effective intervention amidst mediocre procedures)
I’m not sure I understand the issue. Could you expand on this a bit.
Thanks,
Joanne
Hi Joanne,
Sorry. If I had made that a little hazy, I am taking an effort to elucidate on my points below,
Please find below the points as under:
Sudharshan, your raise the following:
How far are we inclined to avoid political en masse while designing an evaluation?
I meant that as evaluators, what are the methods and protocols available and those we put into action for seeing results in reality. “Especially, in the context of political dilemmas or situations, how far are we stretching ourselves to practice using models that really matter.” Is there any thing that we can do to advocate models that promote standard practices?.
And is the answer to this question categorical or generic?
I meant that, are there certain programs that involve political intricacies that drive program evaluations from a different ground ? Probably a state of balance between politics in theory and politics in action. Also could we bifurcate the practice of using ideologies amongst different programs? Or can we conduct methodic research to apply a generic model to all evaluations, irrespective of the level of political influence?.
And if at all misleading, is it capable of changing the scheme of things from the perspective of an evaluator? (meaning an effective intervention amidst mediocre procedures)
I meant about, if we are able to conceive an impact of an evaluation by inclusion of ideologies, it might have a wide-spread influence on the outcomes, at certain levels. So
Is it worthy to consider or include such factors and still prove that the program works in terms of intended outcomes ? Essentially a trade-off in procedure without overlooking the outcomes of the program.
Thanks,
Sudharshan