Part 3 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: Evolution, Diversity and Change from the Middle

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.)

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The other is to maximize the likelihood that we can thrive as circumstances change. Diversity can look very different in each case. To illustrate, imagine two teams working on a novel evaluation challenge. Team 1 is diverse with respect to gender, age, and race, but made up of people who are all long-standing members of AEA, steeped in AEA’s criteria for quality evaluation, primarily identify as evaluators, sympathetic to the evaluation theories and evaluation methodologies we like to talk about, and graduates of X evaluation graduate program. Team 2 is made up entirely of white men in their 40s who have been active in different professional association (economics, education, etc.), done a lot of evaluation, but do who do not consider themselves “professional evaluators”. For the social good, Team 1 is superior. For devising a good evaluation, Team 2 wins. We have been systematic about promoting diversity for the social good. We have to also get serious about the second aspect of diversity.

Part 2:  AEA as an evolving organism

AEA today does not look much s it did only a short time ago.. This is true in terms of the evaluation theories and methodologies we favor, demands made on us by our customers and policy environment, professional identities of our members, our appeal to some evaluators and social scientists but not to others, our conceptions of evaluators competencies, and standards for what constitutes quality work. How did we get here? Certainly not with a plan. We grew organically through an evolutionary dynamic, complete with mutations in the form of intellectual innovations; mutations in the content of evaluator training; acceptance and rejection of those mutations; changes in AEA’s environment in terms of what customers ask from us, larger social forces; activity by our competition; and unpredictable interactions among all of these factors. The trajectory that emerged based on these factors could not have been predicted. There was too much path dependence operating. And so it will be in the future. While that future cannot be predicted, it is fair to say  that diversity will increase the likelihood of a desirable direction.

Part 3: Evolution, Diversity and Change from the Middle as a Framework for AEA to Thrive

Requisite dimensions of diversity include: 1) intellectual capital among our members, 2) intellectual boundary spanning, 3) organizational boundary spanning, and 4) internal structure. Diversity is not an unalloyed good. There needs to be a critical mass in an organization to maintain stability. AEA has been growing organically, developing an appeal to some people and a level of disinterest or aversion by others. That pattern needs to be respected. What we need are mutations that have a fair chance of being accepted. This requires “diversity from within”, i.e. diversity whose range is compatible with who we are, but sensitive to the environment. Fostering such diversity requires a focus on TIGS because they have: 1) influence that spans the gamut from individual members to the Board; 2) profound influence on AEA’s intellectual capital because they influence what people submit to meetings and what gets accepted; and 3) close contact with intellectual frontiers and stakeholder groups that influence subgroups of our membership. Also, TIG members understand the perspectives of their colleagues who do similar work, but who are not involved in AEA. Important information that TIGS can provide includes: trends in methodology and theory; work demands, and career plans. Affiliates can provide complementary information, and also, serve as testbeds for innovation.

Evolution, Diversity and Change from the Middle as a Framework for AEA to Thrive

How can AEA organize itself to increase the likelihood that whatever the future, something recognizable as AEA, and something recognizable as evaluation, will thrive? My answer is to make deliberate decisions that will increase diversity in a way that will make us adaptable.

The first step is to consider what needs to be diverse in order to maximize our adaptive potential. I offer the following list of possibilities.

• Intellectual capital, i.e. the range of backgrounds, theoretical perspectives, knowledge, and experience within our membership.
• Connections to the environment, i.e. the range of contacts with people and organizations who buy evaluation, who set policies about evaluation, who might use evaluation, and who do evaluation.
• Internal structure. For instance, AEA has TIGS, regional affiliates, and clear rules about how to become a TIG or an affiliate. We do not have caucuses. I’m not advocating that we should change our structure, only that we should think about it as an aspect of diversity.
• Intellectual boundary spanning. The Systems TIG is a good example. Evaluators felt a good deal of unease over the lack of effectiveness of programs and the impotence of evaluation to help. A few of our members shared this unease, and also, saw the potential value of research and theory in systems to address our unease. Those people organized the Systems TIG, which has in turn has been playing a large role in evaluators’ ability to use systems ideas in their work.

It’s all very well to talk about diversity as if it were an unalloyed good, but it is not. There needs to be a critical mass in an organization that can maintain a common sense of direction. Stability in the face of change is needed or things will fall apart. But for now, let’s assume that more is better. I do believe that AEA needs to increase its diversity. Once we know how to do this, we can worry about how much is too much.

As a practical matter, AEA needs a workable, rational way of increasing its diversity. Whatever that way, it cannot involve simply identifying desirable members and bringing them into the organization. We have been growing organically, developing an appeal to some people and a level of disinterest or aversion by others. That pattern needs to be respected. Why should those others join us? In fact, they won’t and its not reasonable to expect them to. And if they tried, they and we would probably be uncomfortable.

What we need are mutations that have potential to increase our adaptability, and which also have a fair chance of being accepted. I see this as fostering “diversity from within”, i.e. diversity whose range is compatible with who we are. How to make this happen? The answer touches on my faith in the power of  “change from the middle”.

Change From the Middle

Change directed from the top does not work out because it is insensitive to the local realities of what happens in an organization. It assumes a belief in the effectiveness of central direction that does not work very well in a complex system. It’s one thing for the top to express a strategic direction and to implement policies that can facilitate that direction. But that is very different from strategic planning. I’m also not a big fan of change from the very bottom because issues that are too local and constrained are too idiosyncratic with respect to the needs of the organization as a whole.

That leaves change from the middle  — change that can be sensitive both to overarching organizational issues, and to local needs. Change generated there has a chance of being sensitive to both the local and the global. I want to engineer diversity from the middle. Organizationally, that means TIG leadership. Why the TIGS? Because TIGS:

Within AEA

• Have close contact and influence that spans the gamut from individual members, to the Board.
• Drive content at meetings by influencing what their members submit, and what gets presented.
• Know what meaningful diversity looks like with respect to particular aspects of evaluation.

Outside and at the boundaries

• Are in touch with developing trends.
• Have members that are in touch with the organizational and intellectual boundaries that have a direct impact on what evaluation is doing and for whom.

Another way to look at the “within AEA” and “outside AEA” distinction is to think in terms AEA as an entity that serves individuals (its members), and “evaluation” as a construct.

Individual Focus

To say that TIGS are in touch with their members is missing the point because TIGS are not only in touch with their members. This is because TIG members know people who do similar work, but who are not in the AEA orbit. These are people who do evaluation and evaluation-like work. They are subject to many of the same forces as AEA TIG members. They have  knowledge of theories and methods that are not well known within AEA, but which could be useful in evaluation. I’m in favor of capturing the knowledge of our members, but I am not in favor of restricting our capture efforts to our members because if AEA is to make wise decisions about its future, it needs as wide a view as possible.

This blog is a good example. What I have done is to draw on research and theory dealing with the relationship between complexity and diversity to make a point about how AEA should plan for the future. But the knowledge is more generally useful. Any evaluation that deals with a program’s sustainability or robustness in the face of change would do well to draw on that literature. It will affect program theory, methodology, and data interpretation. If someone asked me what’s out there that evaluators should pay attention to but are not, I’d point them to that research.

What information are TIGS in a position to obtain from their members and  their members’ colleagues? If I did the asking, I’d like to know:

• What methodologies, evaluation theories, and social science theories are gaining attention that may be useful in evaluation? Just as important, what is current that is fading in popularity?
• What kind of questions, and what kind of programs, are people being asked to address? Where is demand increasing? Where is demand decreasing?
• What are people doing to build their careers – training and education they are seeking, jobs they are looking for, how they package themselves.

I am agnostic with respect to how this information is obtained – surveys, interviews, focus groups, literature reviews, discussions with a few select experts. It matters not to me as long as the TIG has made an informed choice.

AEA Focus

The above deals with TIG-down in organizational terms. Here the focus shifts to the relationship between TIGS and AEA as an organizational entity. As I see it, TIGS represent “evaluation” in a more differentiated way than does AEA leadership. Whatever “evaluation” means, I can’t believe it is the same thing as “evaluation” from the point of view of “Advocacy and Policy Change”, “Feminist Issues in Evaluation”, “Needs Assessment”, “Program Design”, “Social Impact Measurement”, or “Youth Focused Evaluation”. AEA does need a coherent over arching concept of evaluation. (Of course getting there is impossible, but the attempt to do so is legitimate and important.) But if AEA is to successfully adapt to change, it must also be sensitive to more differentiated notions of evaluation. That cannot happen without knowledge held  by the TIGS.

I have been focusing on TIGS because  they are entities that have both relationships with evaluators who share similar interests and needs; and also, an important role to play in AEA. TIGS are important because they are so tightly integrated into our structure, process, and culture. But affiliates also have a role to play.

Affiliates

Affiliates are important for two reasons. One reason is that affiliates are a useful source of data because all the issues discussed above with respect to TIGS also apply to affiliates. The other reason is way less practical. It’s speculative. I’d forgive anyone who would classify it as a leap of faith. Affiliates are important in evolutionary terms. AEA has a national scope. TIGS have domain –focused scope. In contrast, affiliates have regional boundaries, but are inclusive of all evaluation domains. So one way to look at affiliates is as close relatives that are organized differently and adapting to different environments. Also, because they are small and have simple internal structures, they can change quickly. Finally, because they are close to AEA, change in affiliates stands some reasonable chance of being observed, and perhaps adapted, by AEA as a whole. From a survival point of view, that’s a pretty robust feature.