Evaluation use by people opposed to the program

Recently I posted a message to Evaltalk about a story in the NY Times that dealt with the Administration’s plan to cut funding for teenage pregnancy prevention programs: Programs That Fight Teenage Pregnancy Are at Risk of Being Cut. (Evaltalk is an American Evaluation Association’s listserv.) That message led to an interesting back and forth, which in turn led me to a response. I’m reproducing it here.

What I took from this has very little to do with teen pregnancy or the transferability of program effectiveness across contexts. For me it was a bit of a values crisis. I strongly oppose the Administration’s policy on these programs. But I also strongly favor evaluation use. I am by no means an expert on the effectiveness of pregnancy prevention programs. In fact, all I know about that subject comes from the NYT article, and we all know how bad science reporting is. But it certainly seems as if the Administration has a point about the programs not having much demonstrated effectiveness. Or at least, that they are not crazy or irrational for believing that that is the message in the data. (I’m giving them the huge benefit of the doubt that they actually looked at the data, but I’m willing to accept that they at least peeked at it.)

So, here we have a legitimate stakeholder who is using evaluation as a justification for a policy that I loathe. Frankly, I’m sure they would cut those programs even if the data supporting their effectiveness were incontrovertible. But still, they are attending to evaluation data, and their interpretation of that data is not too far off the wall.

I have been very lucky I my professional life in that: 1) All the programs I have evaluated were in line with my values, 2) my findings may have been critical, but always in line with improving rather than eliminating the program, and 3) I have never had the problem of working with stakeholders who opposed the program I was evaluating. Or maybe it’s more than luck. Maybe it has to do with the values and interests of the people who buy evaluation services, and the choices I have made in what work to chase.

But what is good for me may not be good for Evaluation. Are we really serving the public good by being in a business that supports policy decisions made by people committed enough to a program to put money, personal reputation, and political capital behind it? On the one hand, that is all we can do because we are in a conservative business in the sense that we can only evaluate programs that someone with control over resources has chosen to implement. On the other hand, maybe we don’t pay enough attention to the interests of stakeholders who oppose the program in question. As I said, I never had to face this uncomfortable reality. But as I also said, what is good for me may not be good for Evaluation.