I’m about to make a case that the field of Evaluation would benefit from theoreticians and practitioners that were more diverse than they are now with respect to beliefs about what constitutes the social good, and how to get there. Making this argument is not easy for me because it means putting head over heart. But I’ll do my best because I think it does matter for the future of Evaluation.
Examples from the Social Sciences
Think of the social sciences – Economics, Sociology, Political Science.
One does not have to have left wing inclinations to appreciate Marxian critiques of society and the relationships among classes. That understanding can inform anyone’s view of the world whether or not you think that overall, Capitalism is a good organizing principle for society. On the other end of the spectrum, even a dyed in the wool lefty would (should?) appreciate that self-interest and the profit motive are useful concepts for understanding why society works as it does, and that it does (might?) produce some social good despite its faults. Would the contribution of the field of Economics be as rich as it is if one of those perspectives did not exist?
Or to take an example from Sociology. Functionalists like Talcott Parsons and Robert Merton lean toward the notion that social change can lead to dysfunction. The existence of theory like that can shape (support? further?) go-slow views about the pace of social change. Or think of the conflict theories of people like Max Weber and C. Wright Mills. Those views support the idea that conflict and inequality are inherent in Capitalism. That’s the kind of theory that could support or shape a rather different view about the need for social change.
So what we have is a diversity of theory that is in some combination based on/facilitative of, different views of how society should operate. I think the disciplines of Economics and Sociology are better off because of that diversity. More important, we are all better off for having access to these different perspectives as we try to figure out how to do the right thing, or even, what the right thing is.
Evaluation
I am convinced that over the long run, if Evaluation is going to make a contribution to society, it has to encompass the kind of diversity I’m giving examples of above. Why?
One reason is that stakeholders and interested parties have different beliefs about programs – their very existence, choices of which ones to implement, their makeup, and their desired outcomes. How can Evaluation serve the needs of that diversity if there is too much uniformity in our ranks? Also, what kind of credibility do we have if the world at large comes to see our professional associations and evaluations as supportive of only one perspective on the social good and the role of government?
The argument above deals with the design of evaluations and the collection and interpretation of data. But the importance of diversity extends to Evaluation theory as well.
Explaining the value of diversity in Evaluation theory is harder for me because I don’t have a good idea of how it might play out, but I’ll try. It seems to me that right now, all existing Evaluation theory carries the implicit belief that change is a good thing. Change may not work out as we wish because programs may be weak or have unintended consequences. But fundamentally, change is good and the reason to evaluate is to make the change better. Well, what would Evaluation look like if we had evaluation theory that drew from the Functionalist school of Sociology, which takes such a jaundiced view of social change? I have no idea, and emotionally, I’m not sure I want to know because personally I am in favor of intervention in the service of the social good. But on an intellectual level, I know that evaluation based on a conservative (small “c”) view of change would end up producing some very worthwhile insight that I am sure would not come from our present theory.
Moving from Blather to Action
There are numerous impediments to working toward ideological diversity. Mostly, I am convinced that almost everyone in our field has politics that are not too much different from mine. We go into the evaluation business because we think that government is good and we want to make it better. That self-selection bias makes us a pretty homogeneous group that forms into associations that do not throw out the welcome mat for divergent opinion. Maybe the best we can do is make it known that ideological dimensions of diversity are welcome. That itself is not so easy because what does “dimension of diversity” even mean? Still, I think it’s worth a shot.
Evaluation Uncertainty 
You are absolutely right but then evaluation is intrinsically diverse since it draws on all the social sciences. This is in fact its major comparative advantage over economics, sociology, etc. It is also clear that combining evaluation models and mixing evaluation methods are keys to evaluation quality. I would also argue that diversity is built into the very definition of evaluation since assessment of worth calls for ascertaining the perspectives of all stakeholders. And the critical assessment of social change interventions is what development evaluation has specialized in. This said, it is good to be reminded of the importance of diversity in evaluation in the current operating context where single narratives dominate the public discourse.
I’d like to be as optimistic as you are that Evaluation already has an adequate dose of diversity. But I’m not. Here are two examples of why I take this position.
Example #1:
Let’s say I am running a program in the U.S. to help immigrants integrate into the larger culture. Someone like me might think of an outcome chain such as: effective recruiting of participants good training and education people learn English and how life is conducted in the U.S. various good things happen, e.g. people do better in working with their kids around schoolwork, they find better jobs, etc. and so forth.
I absolutely guarantee that if I were running the evaluation, I would never think to split the participants into those who were here legally and illegally, and add an outcome chain for the illegals: 1) fewer jobs for people here legally, 2) decrease in respect for the rule of law, 3) uncompensated strain on local social service resources, and 4) increased likelihood of more people trying to enter the U.S. illegally.
Why would I not do this? One reason is that it was exceedingly difficult for someone like me to even think that these are legitimate outcomes that were worth looking at. Second, my bias would be to not publicize these possibilities because I would not want to facilitate these ideas entering the public debate. Third, I just don’t believe they would occur, or at least, not occur to an extent worth considering.
BUT. People who did believe these things are not crazy or irrational. (Just wrong and misguided.) It’s not unreasonable for some legitimate stakeholders to want to know if these outcomes ensued. I feel quite confident in stating that given the normal community of evaluators and the way we go about doing our business, this set of outcomes would never show up as part of our logic models, program theory, or evaluation design.
Now, I am sure that even if we did think about them, our customers would fire us if we insisted in including these outcomes. But that’s another story.
Example #2
I have a pretty good explanation based on complexity behavior as to why many programs will have unintended consequences, and moreover, why such consequences are more likely to be undesirable than welcome. That may be, but my inclination would run along the lines that deliberate efforts to improve the social good are welcome, and that the reason to understand negative consequences is to improve the efforts at change.
But let’s say I did not know all this science and theory, and I was just a plain old, ordinary, sympathizer with Edmund Burke and his intellectual progeny. That would give me a very jaundiced view of interventions in the service of deliberate social change. If I took that view, my mind would run to all kinds of undesirable consequences for any program I might be called upon to evaluate. Not only might I think about these consequences, but unlike the real me, I’d take the position that these kinds of undesirable consequences are a good reason to not implement the change in the first place. I’d be looking for data to make the case that the program should not be continued. I bet that would make for a different methodology and a different program theory.
Dear Mr. Morell,
I thank you for having written such piece and shared your insights. It has been the opportunity for me to learn something new and reflect on your concept.
You raise an important question: “…stakeholders and interested parties have different beliefs about programs – their very existence, choices of which ones to implement, their makeup, and their desired outcomes. How can Evaluation serve the needs of that diversity if there is too much uniformity in our ranks?”
Diversity is something that we, as human beings, recognize as different from our intellectual, emotional, and spiritual capabilities. Different is what we are not, we cannot think of, we cannot ‘feel’, and we do not have. But it does not mean that is bad. Giving diversity a negative nuance is poor quality assessment. I guess we should go deeper. What is different is something that we do not possess but we can acquire thanks to others who are diverse and, in some cases, complementary. So it should be seen as the opportunity to refine our thinking process and mentality.
You are right when you say that there is too much uniformity in programs and evaluations. And I have tried conveying the message several times to senior management. Experienced evaluators have outstanding, extraordinary capabilities together with the capacity to make accurate assessments and address recommendations. But sometime could benefit from creative, indeed different, inputs and ideas. These might be little things like communication products to share good practices, some colorful visuals, new Apps to learn how to be an evaluator on a daily life while playing, and so on. These might be brought to the table by younger folks who have worked in the development industry less years but have grown up with ICTs as extension of their (our) bodies.
In my humble opinion, sometime some people can bring in insights which can help us build on our own ideas and advance programs and evaluation practice if we are in the listening modality of that diversity. Some people learn from classes and books as well as from observation because they have intuitive mindset so can think of issues from different perspectives. I would call that positive diversity looking at it in 3Ds.
Thank you again for having shared your thoughts.
Warm regards,
Laura Gagliardone
Jonny wrote: “I am convinced that over the long run, if Evaluation is going to make a contribution to society, it has to encompass the kind of diversity I’m giving examples of above.”
I have found simple rules “convincing” over the years but they have outlived their usefulness so many times that I’m at the point of abandoning them altogether.
If evaluators wish to make a long-lasting contribution to society, they will focus on preventing the development of negative double binds in public discourse (e.g., between the forces of power and rationality) through the proactive use of nontraditional, multivalent logic(s).
Does developmental evaluation espouse a multivalent form of evaluation logic, or is the bivalent form simply carried over into the complex domain?
Best,
Chad
This is truly thought-provoking. By the very nature of our profession, we’re fixed in the concept that funded programs can and do create change, that the designed changes that they can create are desirable, and that the outcomes can be measured. That inherently excludes the mindsets of ideologies such as social Darwinism or Objectivism.
Whether or not it’s a direct byproduct of those concepts, there are certain beliefs that are likely givens in the evaluation community. The equality of all humans, the need to support the social and economic development of women, and that poverty should be eliminated.
There are a couple of tools that I use in long-term strategic planning and futures studies to spark different kinds of thinking that may be of use in at least simulating more ideological diversity.
1. “And that’s terrible!” There’s a popular internet meme showing Lex Luthor, the Superman villain, stealing pies. The description of the scene includes the line “And that’s terrible!”
After developing a preferred future scenario. I’ll ask the group to think of perspectives that would cause somebody looking at the scenario to say, seriously, “And that’s terrible.” (I find that the meme’s humor truly helps lower mental barriers to thinking this way.) It might take a bit of encouragement to get people to do this, so I try to have a few suggestions ready.
2. A slightly more serious approach is to keep asking, “Who might lose from this?” I have a few categories of loss and phrasings ready, such as “Who might lose money? Who might lose prestige? Who might lose opportunity? Who might become nostalgic for the time before this?”
All the best,
Ann
I really appreciate the opportunity to think about the issue of diversity in evaluation. Many government evaluators have been exposed to different approaches to enhance efficiency, effectiveness and economy in government programs and actions. Internal audit, for example, or spending reviews. Perhaps evaluators should be seen as one element in a more diverse landscape?
(For more on spending reviews, see http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=GOV/PGC/SBO(2013)6&doclanguage=en)
Thanks for this thought provoking discussion!
Felix Meisels, program evaluator
As for the importance of diversity, this passage is relevant:
“When you understand the power of system self-organization, you begin to understand why biologists worship biodiversity even more than economists worship technology. The wildly varied stock of DNA, evolved and accumulated over billions of years, is the source of evolutionary potential, just as science libraries and labs and universities where scientists are trained are the source of technological potential. Allowing species to go extinct is a systems crime, just as randomly eliminating all copies of particular science journals, or particular kinds of scientists, would be.
The same could be said of human cultures, of course, which are the store of behavioral repertoires, accumulated over not billions, but hundreds of thousands of years. They are a stock out of which social evolution can arise. Unfortunately, people appreciate the precious evolutionary potential of cultures even less than they understand the preciousness of every genetic variation in the world’s ground squirrels. I guess that’s because one aspect of almost every culture is the belief in the utter superiority of that culture.
Insistence on a single culture shuts down learning. Cuts back resilience. Any system, biological, economic, or social, that gets so encrusted that it cannot self-evolve, a system that systematically scorns experimentation and wipes out the raw material of innovation, is doomed over the long term on this highly variable planet.
The intervention point here is obvious, but unpopular. Encouraging variability and experimentation and diversity means ‘losing control.’ Let a thousand flowers bloom and ANYTHING could happen! Who wants that? Let’s play it safe and push this leverage point in the wrong direction by wiping out biological, cultural, social, and market diversity!” – Donella Meadows (Leverage Points)
For the best explanation of what diversity is, and its value, see my latest favorite book: Diversity and Complexity by Scott Page.
New to me. I’ll check it out. Thanks!