Evaluation as Social Technology

Evaluation as Social Technology
Chapter 5 from: Program evaluation In social research Full chapter

• Introduction
• Why Is Social Research Not Relevant
• A Framework for Solutions
• Differences between Science and Technology
• Advantages of Evaluation as Social Technology
• Interrelationships between Science and Technology

So far I have argued that the value of outcome evaluation can be increased if type, validity, and usefulness are considered separately as distinct aspects of any given evaluation plan. Each of these three elements contains implications for the other two, but those implications will not be clear unless each component part is analyzed separately. There is another aspect of the problem which must also be considered, namely, the basic philosophical model of knowledge seeking upon which outcome evaluation is based. This is a consideration which cuts across elements of type, validity and usefulness, and deals with basic approaches to social research. How are questions formulated? How are variables chosen? What decision rules are used to weigh evidence, draw conclusions, and make recommendations? The answers to these questions reflect a philosophical model of research, and the model chosen can have far-reaching effects on the ultimate value of any research project. There are three main aspects to the argument about to be developed. First, there are crucial differences between scientific and technological models of knowledge development. Second, these differences have profound implications for the practical value of research. Third, evaluation is far more of a technological than a scientific pursuit.

Part 1 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: What is Diversity?

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The other is to maximize the likelihood that we can thrive as circumstances change. Diversity Continue reading “Part 1 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: What is Diversity?”

Part 2 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: AEA as an Evolving Organism

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The Continue reading “Part 2 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: AEA as an Evolving Organism”

Part 3 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: Evolution, Diversity and Change from the Middle

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.)

Part 1: What do I mean by diversity?

Continue reading “Part 3 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: Evolution, Diversity and Change from the Middle”

Ideological diversity in Evaluation. We don’t have it, and we do need it

I’m about to make a case that the field of Evaluation would benefit from theoreticians and practitioners that were more diverse than they are now with respect to beliefs about what constitutes the social good, and how to get there. Making this argument is not easy for me because it means putting  head over heart. But I’ll do my best because I think it does matter for the future of Evaluation.

Examples from the Social Sciences
Think of the social sciences – Economics, Sociology, Political Science.

One does not have to have left wing inclinations to appreciate Marxian critiques of society and the relationships among classes. That understanding can inform anyone’s view of the world whether or not you think that overall, Capitalism is a good organizing principle for society. On the other end of the spectrum, even a dyed in the wool lefty would (should?) appreciate that self-interest and the profit motive are useful concepts for understanding why society works as it does, and that it does (might?) produce some social good despite its faults. Would the contribution of the field of Economics be as rich as it is if one of those perspectives did not exist?

Or to take an example from Sociology. Functionalists like Talcott Parsons and Robert Merton lean toward the notion that social change can lead to dysfunction. The existence of theory like that can shape (support? further?) go-slow views about the pace of social change. Or think of the conflict theories of people like Max Weber and C. Wright Mills. Those views support the idea that conflict and inequality are inherent in Capitalism. That’s the kind of theory that could support or shape a rather different view about the need for social change.

So what we have is a diversity of theory that is in some combination based on/facilitative of, different views of how society should operate. I think the disciplines of Economics and Sociology are better off because of that diversity. More important, we are all better off for having access to these different perspectives as we try to figure out how to do the right thing, or even, what the right thing is.

Evaluation
I am convinced that over the long run, if Evaluation is going to make a contribution to society, it has to encompass the kind of diversity I’m giving examples of above. Why?

One reason is that stakeholders and interested parties have different beliefs about programs – their very existence, choices of which ones to implement, their makeup, and their desired outcomes. How can Evaluation serve the needs of that diversity if there is too much uniformity in our ranks? Also, what kind of credibility do we have if the world at large comes to see our professional associations and evaluations as supportive of only one perspective on the social good and the role of government?

The argument above deals with the design of evaluations and the collection and interpretation of data. But the importance of diversity extends to Evaluation theory as well.

Explaining the value of diversity in Evaluation theory is harder for me because I don’t have a good idea of how it might play out, but I’ll try. It seems to me that right now, all existing Evaluation theory carries the implicit belief that change is a good thing. Change may not work out as we wish because programs may be weak or have unintended consequences. But fundamentally, change is good and the reason to evaluate is to make the change better. Well, what would Evaluation look like if we had evaluation theory that drew from the Functionalist school of Sociology, which takes such a jaundiced view of social change? I have no idea, and emotionally, I’m not sure I want to know because personally I am in favor of intervention in the service of the social good. But on an intellectual level, I know that evaluation based on a conservative (small “c”) view of change would end up producing some very worthwhile insight that I am sure would not come from our present theory.

Moving from Blather to Action
There are numerous impediments to working toward ideological diversity. Mostly, I am convinced that almost everyone in our field has politics that are not too much different from mine. We go into the evaluation business because we think that government is good and we want to make it better. That self-selection bias makes us a pretty homogeneous group that forms into associations that do not throw out the welcome mat for divergent opinion. Maybe the best we can do is make it known that ideological dimensions of diversity are welcome. That itself is not so easy because what does “dimension of diversity” even mean? Still, I think it’s worth a shot.

 

 

Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community

This post is an update to a post I did some time ago. I am adding to  it based on some conversations I recently had at the annual meeting of the Canadian Evaluation Society. The topic I’m dealing with is the development of “Evaluation” through the lens of evolutionary biology. There are two related issues: 1) intellectual traditions, and 2) people’s affiliation as “evaluators”.  Continue reading “Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community”

Ideology in evaluation. Help designing devising a scenario for AEA 2011

Help

We (Joanne Farley, Tarek Azzam and I) have been musing about whether evaluation as practiced by the members of AEA is framed within too narrow range of political and social ideologies. We suspect that it may be, and that as a result, evaluators miss important elements of program theory, metrics, and methodology. Continue reading “Ideology in evaluation. Help designing devising a scenario for AEA 2011”