Common Introduction to all Three Parts
I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.
What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.).
Part 1: What do I mean by diversity?
There are two reasons for AEA to build diversity. One is to pursue the social good. The other is to maximize the likelihood that we can thrive as circumstances change. Diversity can look very different in each case. To illustrate, imagine two teams working on a novel evaluation challenge. Team 1 is diverse with respect to gender, age, and race, but made up of people who are all long-standing members of AEA, steeped in AEA’s criteria for quality evaluation, primarily identify as evaluators, sympathetic to the evaluation theories and evaluation methodologies we like to talk about, and graduates of X evaluation graduate program. Team 2 is made up entirely of white men in their 40s who have been active in different professional association (economics, education, etc.), done a lot of evaluation, but do who do not consider themselves “professional evaluators”. For the social good, Team 1 is superior. For devising a good evaluation, Team 2 wins. We have been systematic about promoting diversity for the social good. We have to also get serious about the second aspect of diversity.
Part 2: AEA as an evolving organism
AEA today does not look much s it did only a short time ago.. This is true in terms of the evaluation theories and methodologies we favor, demands made on us by our customers and policy environment, professional identities of our members, our appeal to some evaluators and social scientists but not to others, our conceptions of evaluators competencies, and standards for what constitutes quality work. How did we get here? Certainly not with a plan. We grew organically through an evolutionary dynamic, complete with mutations in the form of intellectual innovations; mutations in the content of evaluator training; acceptance and rejection of those mutations; changes in AEA’s environment in terms of what customers ask from us, larger social forces; activity by our competition; and unpredictable interactions among all of these factors. The trajectory that emerged based on these factors could not have been predicted. There was too much path dependence operating. And so it will be in the future. While that future cannot be predicted, it is fair to say that diversity will increase the likelihood of a desirable direction.
Part 3: Evolution, diversity and change from the middle as a framework for AEA to thrive
Requisite dimensions of diversity include: 1) intellectual capital among our members, 2) intellectual boundary spanning, 3) organizational boundary spanning, and 4) internal structure. Diversity is not an unalloyed good. There needs to be a critical mass in an organization to maintain stability. AEA has been growing organically, developing an appeal to some people and a level of disinterest or aversion by others. That pattern needs to be respected. What we need are mutations that have a fair chance of being accepted. This requires “diversity from within”, i.e. diversity whose range is compatible with who we are, but sensitive to the environment. Fostering such diversity requires a focus on TIGS because they have: 1) influence that spans the gamut from individual members to the Board; 2) profound influence on AEA’s intellectual capital because they influence what people submit to meetings and what gets accepted; and 3) close contact with intellectual frontiers and stakeholder groups that influence subgroups of our membership. Also, TIG members understand the perspectives of their colleagues who do similar work, but who are not involved in AEA. Important information that TIGS can provide includes: trends in methodology and theory; work demands, and career plans. Affiliates can provide complementary information, and also, serve as testbeds for innovation.
AEA as an Evolving Organism
Now that I have made a point about the nature of diversity, I’ll move on to why I think of AEA as an organism. In part three I’ll put the two ideas together, and add an organizational change strategy that emphasizes working at the TIG level.
For now, please accept the notion that it makes sense to think of AEA in terms of evolutionary biology. If you are curious as to why I think it makes sense, go to: (Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community.)
A little historical perspective shows how much we have changed, and how radical that change has been. Here are just a few examples. 1) There was a time when there was a consensus among our members that experiments and quasi-experiments were the most desirable way to evaluate a program. Now there is a raging debate as to whether they are useful at all, or what the narrow conditions are where it is appropriate to use them. 2) There was a time when nobody knew what developmental evaluation was, and now it is a major intellectual pillar of our evaluation thinking. 3) There was a time when the notion of “professionalization” was discussed occasionally, but not taken seriously. Now competencies and standards are essential to our professional identities. 4) There was a time nobody thought of themselves as primarily an “evaluator”. Now many of us do. The old days were not that long ago.
How did this come to be? There was no central plan. Rather:
- Creative work by our members led to previously unknown ways of thinking about evaluation. .
- A selection process took place in which some of this creative thinking was embraced by our members, and some was not.
- A change in the appeal of AEA brought in new members. It also repelled others.
- There were changes in AEA’s environment in terms of the evaluation business, i.e. what did customers ask of us, and how much did they ask of us?
- Education and training arose to serve our developing needs for knowledge. Of course as people chose to avail themselves of some of these opportunities, and to eschew others, a shared knowledge base developed.
- Training content was deployed to serve developing needs. The availability of that training had the effect of increasing the place of that kind of evaluation among our ranks, to the detriment of ideas for which there was little training demand.
- There was an interplay between us and our environment. This took place in two ways. One way was deliberate and planned, as for instance, the work of our Evaluation Policy Task Force. The second was unplanned. Here, many people who either joined AEA, or were somehow educated in the AEA culture, took jobs that made them customers for our services. We taught them what to pay us for.
- Of course larger societal forces were always at play that affected demands for (and of) evaluation.
- We had to adapt to a competitive environment. After all, AEA is not the only organization that has members who provide evaluation services. (As an aside, pay attention to evaluation work cited in the popular press. How much of it is done by AEA members? I don’t think that is a problem, but I do think it is an indicator.)
What I have just described is an adaptive evolutionary process that was driven by mutations in the form of novel ways of thinking about evaluation, and selection conditions that combined to describe a particular direction of change.
Evaluation Uncertainty 