Common Introduction to all Three Parts
I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.
What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).
Part 1: What do I mean by diversity?
There are two reasons for AEA to build diversity. One is to pursue the social good. The other is to maximize the likelihood that we can thrive as circumstances change. Diversity can look very different in each case. To illustrate, imagine two teams working on a novel evaluation challenge. Team 1 is diverse with respect to gender, age, and race, but made up of people who are all long-standing members of AEA, steeped in AEA’s criteria for quality evaluation, primarily identify as evaluators, sympathetic to the evaluation theories and evaluation methodologies we like to talk about, and graduates of X evaluation graduate program. Team 2 is made up entirely of white men in their 40s who have been active in different professional association (economics, education, etc.), done a lot of evaluation, but do who do not consider themselves “professional evaluators”. For the social good, Team 1 is superior. For devising a good evaluation, Team 2 wins. We have been systematic about promoting diversity for the social good. We have to also get serious about the second aspect of diversity.
Part 2: AEA as an evolving organism
AEA today does not look much s it did only a short time ago.. This is true in terms of the evaluation theories and methodologies we favor, demands made on us by our customers and policy environment, professional identities of our members, our appeal to some evaluators and social scientists but not to others, our conceptions of evaluators competencies, and standards for what constitutes quality work. How did we get here? Certainly not with a plan. We grew organically through an evolutionary dynamic, complete with mutations in the form of intellectual innovations; mutations in the content of evaluator training; acceptance and rejection of those mutations; changes in AEA’s environment in terms of what customers ask from us, larger social forces; activity by our competition; and unpredictable interactions among all of these factors. The trajectory that emerged based on these factors could not have been predicted. There was too much path dependence operating. And so it will be in the future. While that future cannot be predicted, it is fair to say that diversity will increase the likelihood of a desirable direction.
Part 3: Evolution, diversity and change from the middle as a framework for AEA to thrive
Requisite dimensions of diversity include: 1) intellectual capital among our members, 2) intellectual boundary spanning, organizational boundary spanning, and 3) internal structure. Diversity is not an unalloyed good. There needs to be a critical mass in an organization to maintain stability. AEA has been growing organically, developing an appeal to some people and a level of disinterest or aversion by others. That pattern needs to be respected. What we need are mutations that have a fair chance of being accepted. This requires “diversity from within”, i.e. diversity whose range is compatible with who we are, but sensitive to the environment. Fostering such diversity requires a focus on TIGS because they have: 1) influence that spans the gamut from individual members to the Board; 2) profound influence on AEA’s intellectual capital because they influence what people submit to meetings and what gets accepted; and 3) close contact with intellectual frontiers and stakeholder groups that influence subgroups of our membership. Also, TIG members understand the perspectives of their colleagues who do similar work, but who are not involved in AEA. Important information that TIGS can provide includes: trends in methodology and theory; work demands, and career plans. Affiliates can provide complementary information, and also, serve as testbeds for innovation.
What Do I Mean by Diversity?
My sense is that current discussion about diversity focuses on the social good, on what is right and just. There is mention of the role that diversity plays in creativity and problem solving, but for the most part, the discussion seems to be about what is the right thing with respect to social divisions – race, gender, age, sexual preference, and so on (GEDI, Building Diversity Initiative).
In this blog I am switching the focus to the consequences of diversity for AEA as an organism that must evolve and adapt to a changing environment. To see how this perspective shifts attention from our usual treatment of diversity, here are two (admittedly extreme examples.)
Example 1: An evaluator is faced with a need to invoke some cutting edge statistical analysis to deal with an unfamiliar problem. Creative work needs to be done with respect to organizing the data, picking appropriate methods, and interpreting the results. There is no recipe that can be plucked from textbooks. The evaluator has a chance of forming one of two teams.
Team 1: Diverse with respect to gender, age, and race, and completely made up of people with PhDs in mathematical statistics from the same department in the same university.
Team 2: Made up entirely of white men in their 40s, but who come from different universities, and with statistical expertise drawn from graduate studies in statistics, epidemiology, economics and genetics.
Example 2: An evaluator has to design an evaluation that does not fit easily into familiar categories or approaches. Creative work needs to be done with respect to formulating the problem, identifying and engaging stakeholders, discerning program theory, choosing relevant bodies of social theory to invoke, settling on desired outcomes, agreeing on what other programs are “similar”, clarifying the values and ideologies that are driving the program design, and constructing a methodology.
Team 1: Diverse with respect to gender, age, and race, but made up of people who are all long-standing members of AEA, steeped in AEA’s criteria for quality evaluation, primarily identify as evaluators, sympathetic to the evaluation theories and evaluation methodologies we like to talk about, and graduates of X evaluation graduate program.
Team 2: Made up entirely of white women in their 40s, but who have been active in different professional association (economics, education, etc.), and who have done a lot of evaluation but do not consider themselves “professional evaluators”
As for myself, if I wanted to exploit the power of diversity, I’d pick Team 2 in each example. Why? Because the diversity gives us a wider range of intellectual input from which to devise a solution. It’s a good assumption that any single perspective, no matter how good it is or how much success it has had in the past, may be have weaknesses with respect to novel scenarios. Diversity buffers against those weaknesses.
Of course in the real world it is entirely possible to have teams that are diverse in both senses of the term, but these examples do show that in order to think about adaptation and viability, diversity means a lot more than the way we usually think about it.
Note that in both examples I did not say anything about specifics. What is the nature of the statistical challenge or the program that needs to be evaluated? We don’t know, and the ignorance matters. If we knew diversity with respect to what, we might know what kinds of diversity we needed. But we don’t know. That’s the situation evaluation is, and that is the situation in which AEA is in.
Evaluation Uncertainty 