I want to spark a discussion of what evaluation might look like if it were practiced by people who were working from different ideological frameworks. It has been difficult for me to frame this post because my own politics are distinctly medium rare, and I don’t have the imagination needed to think deeply from other points of view. Still, I think that there are three reasons why this is an important exercise for the members of AEA.
1) I have never taken a poll, but my bet is that the members of AEA cluster on the left end of the ideological spread. (Go ahead, prove me wrong. Someone should find out.) I’m not sure we serve our customers and stakeholders well if we do our work from too homogeneous a point of view. 2) The reason we don’t serve our customers and stakeholders well is because the nature of the data we develop, the findings we produce, and the outcome of our efforts at utilization, all combine to provide overly restricted choices relative to the policy decisions that can be made. 3) As an association that cares about the public good, what good are we if we provide weak guidance?
As I see it the problems with a restricted range touch on all aspects of what we do along the entire evaluation project life cycle. 1) What programs or groups of programs do we choose to evaluate? 2) When we design evaluations, what bodies of research literature, groups of experts, and theories do we query? 3) How do we choose which stakeholder groups to involve, and how do we determine the relative importance of each? 4) What positive and negative outcomes do we invest effort in trying to measure? 5) What time frame do we choose for measuring program effects? 6 How much effort do we put into measuring various opportunity costs? 7) How do we interpret data?
So, I propose a thought experiment that involves a collective effort to fill in a table. I define the rows by various political ideologies. I imagine a group of seven categories. 1) Marxist. 2) Socialist 3) Center left (sort of like European Social Democrats), 4) Center right (sort of like European Christian Democrats), 5) Social conservative, 6) conservative in the Edmund Burke sense, and 7) small government conservatism.
As for the columns, I came up with eleven programs that evaluators might be called upon to evaluate. 1) R&D investment by government for work that is near the R side of the R&D continuum. 2) Tax subsidies to wind energy. 3) International development programs to build up civil society. 4) International development programs for improving agriculture. 5) Web 2.0 programs for government to provide services to citizens. 6) Organizational change efforts to make government agencies more effective. 7) The impact of rule making in federal regulatory agencies. 8- Early childhood education. 9) Initiatives to teach under served populations to advocate for themselves. 10) Family support for military families. 11) Immunization promotion.
I certainly don’t think that we should all start filling out a 7×11 matrix, particularly since I started this post by admitting that I can’t do it myself. But I do think it would be of value to all of us if people who resonated to any of the cells took a crack at filling them in. At least, I know that it would be valuable for me.