Part 3 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: Evolution, Diversity and Change from the Middle

Common Introduction to all Three Parts

I have been thinking about what will happen to AEA, and to evaluation, in the future. I can conjure scenarios where AEA and evaluation thrive, and I can imagine scenarios where they whither. What I cannot envision is a future in which AEA and evaluation, as we know them now, stay the same. What I want to do is to start a conversation about preparing for the future. AEA is already active in efforts to envision its future: What will AEA be in 2020? My intent is to inject another perspective into that discussion.

What I’m about to say draws on some thinking I have been doing on two subjects – 1) AEA’s development in terms of evolutionary biology (Ideological Diversity in Evaluation. We Don’t Have it, and We Do Need It, and 2) Using an evolutionary biology view to connect the intellectual development of evaluation and the development of the evaluation community); and the nature of diversity in complex systems. (If you have not read Scott Page’s Diversity and Complexity, I recommend it.)

Part 1: What do I mean by diversity?

There are two reasons for AEA to build diversity. One is to pursue the social good. The other is to maximize the likelihood that we can thrive as circumstances change. Diversity can look very different in each case. To illustrate, imagine two teams working on a novel evaluation challenge. Team 1 is diverse with respect to gender, age, and race, but made up of people who are all long-standing members of AEA, steeped in AEA’s criteria for quality evaluation, primarily identify as evaluators, sympathetic to the evaluation theories and evaluation methodologies we like to talk about, and graduates of X evaluation graduate program. Team 2 is made up entirely of white men in their 40s who have been active in different professional association (economics, education, etc.), done a lot of evaluation, but do who do not consider themselves “professional evaluators”. For the social good, Team 1 is superior. For devising a good evaluation, Team 2 wins. We have been systematic about promoting diversity for the social good. We have to also get serious about the second aspect of diversity.

Part 2:  AEA as an evolving organism

AEA today does not look much s it did only a short time ago.. This is true in terms of the evaluation theories and methodologies we favor, demands made on us by our customers and policy environment, professional identities of our members, our appeal to some evaluators and social scientists but not to others, our conceptions of evaluators competencies, and standards for what constitutes quality work. How did we get here? Certainly not with a plan. We grew organically through an evolutionary dynamic, complete with mutations in the form of intellectual innovations; mutations in the content of evaluator training; acceptance and rejection of those mutations; changes in AEA’s environment in terms of what customers ask from us, larger social forces; activity by our competition; and unpredictable interactions among all of these factors. The trajectory that emerged based on these factors could not have been predicted. There was too much path dependence operating. And so it will be in the future. While that future cannot be predicted, it is fair to say  that diversity will increase the likelihood of a desirable direction.

Part 3: Evolution, Diversity and Change from the Middle as a Framework for AEA to Thrive

Requisite dimensions of diversity include: 1) intellectual capital among our members, 2) intellectual boundary spanning, 3) organizational boundary spanning, and 4) internal structure. Diversity is not an unalloyed good. There needs to be a critical mass in an organization to maintain stability. AEA has been growing organically, developing an appeal to some people and a level of disinterest or aversion by others. That pattern needs to be respected. What we need are mutations that have a fair chance of being accepted. This requires “diversity from within”, i.e. diversity whose range is compatible with who we are, but sensitive to the environment. Fostering such diversity requires a focus on TIGS because they have: 1) influence that spans the gamut from individual members to the Board; 2) profound influence on AEA’s intellectual capital because they influence what people submit to meetings and what gets accepted; and 3) close contact with intellectual frontiers and stakeholder groups that influence subgroups of our membership. Also, TIG members understand the perspectives of their colleagues who do similar work, but who are not involved in AEA. Important information that TIGS can provide includes: trends in methodology and theory; work demands, and career plans. Affiliates can provide complementary information, and also, serve as testbeds for innovation.

Evolution, Diversity and Change from the Middle as a Framework for AEA to Thrive

How can AEA organize itself to increase the likelihood that whatever the future, something recognizable as AEA, and something recognizable as evaluation, will thrive? My answer is to make deliberate decisions that will increase diversity in a way that will make us adaptable.

The first step is to consider what needs to be diverse in order to maximize our adaptive potential. I offer the following list of possibilities.

  • Intellectual capital, i.e. the range of backgrounds, theoretical perspectives, knowledge, and experience within our membership.
  • Connections to the environment, i.e. the range of contacts with people and organizations who buy evaluation, who set policies about evaluation, who might use evaluation, and who do evaluation.
  • Internal structure. For instance, AEA has TIGS, regional affiliates, and clear rules about how to become a TIG or an affiliate. We do not have caucuses. I’m not advocating that we should change our structure, only that we should think about it as an aspect of diversity.
  • Intellectual boundary spanning. The Systems TIG is a good example. Evaluators felt a good deal of unease over the lack of effectiveness of programs and the impotence of evaluation to help. A few of our members shared this unease, and also, saw the potential value of research and theory in systems to address our unease. Those people organized the Systems TIG, which has in turn has been playing a large role in evaluators’ ability to use systems ideas in their work.

It’s all very well to talk about diversity as if it were an unalloyed good, but it is not. There needs to be a critical mass in an organization that can maintain a common sense of direction. Stability in the face of change is needed or things will fall apart. But for now, let’s assume that more is better. I do believe that AEA needs to increase its diversity. Once we know how to do this, we can worry about how much is too much.

As a practical matter, AEA needs a workable, rational way of increasing its diversity. Whatever that way, it cannot involve simply identifying desirable members and bringing them into the organization. We have been growing organically, developing an appeal to some people and a level of disinterest or aversion by others. That pattern needs to be respected. Why should those others join us? In fact, they won’t and its not reasonable to expect them to. And if they tried, they and we would probably be uncomfortable.

What we need are mutations that have potential to increase our adaptability, and which also have a fair chance of being accepted. I see this as fostering “diversity from within”, i.e. diversity whose range is compatible with who we are. How to make this happen? The answer touches on my faith in the power of  “change from the middle”.

Change From the Middle

Change directed from the top does not work out because it is insensitive to the local realities of what happens in an organization. It assumes a belief in the effectiveness of central direction that does not work very well in a complex system. It’s one thing for the top to express a strategic direction and to implement policies that can facilitate that direction. But that is very different from strategic planning. I’m also not a big fan of change from the very bottom because issues that are too local and constrained are too idiosyncratic with respect to the needs of the organization as a whole.

That leaves change from the middle  — change that can be sensitive both to overarching organizational issues, and to local needs. Change generated there has a chance of being sensitive to both the local and the global. I want to engineer diversity from the middle. Organizationally, that means TIG leadership. Why the TIGS? Because TIGS:

Within AEA

  • Have close contact and influence that spans the gamut from individual members, to the Board.
  • Drive content at meetings by influencing what their members submit, and what gets presented.
  • Know what meaningful diversity looks like with respect to particular aspects of evaluation.

Outside and at the boundaries

  • Are in touch with developing trends.
  • Have members that are in touch with the organizational and intellectual boundaries that have a direct impact on what evaluation is doing and for whom.

Another way to look at the “within AEA” and “outside AEA” distinction is to think in terms AEA as an entity that serves individuals (its members), and “evaluation” as a construct.

Individual Focus

To say that TIGS are in touch with their members is missing the point because TIGS are not only in touch with their members. This is because TIG members know people who do similar work, but who are not in the AEA orbit. These are people who do evaluation and evaluation-like work. They are subject to many of the same forces as AEA TIG members. They have  knowledge of theories and methods that are not well known within AEA, but which could be useful in evaluation. I’m in favor of capturing the knowledge of our members, but I am not in favor of restricting our capture efforts to our members because if AEA is to make wise decisions about its future, it needs as wide a view as possible.

This blog is a good example. What I have done is to draw on research and theory dealing with the relationship between complexity and diversity to make a point about how AEA should plan for the future. But the knowledge is more generally useful. Any evaluation that deals with a program’s sustainability or robustness in the face of change would do well to draw on that literature. It will affect program theory, methodology, and data interpretation. If someone asked me what’s out there that evaluators should pay attention to but are not, I’d point them to that research.

What information are TIGS in a position to obtain from their members and  their members’ colleagues? If I did the asking, I’d like to know:

  • What methodologies, evaluation theories, and social science theories are gaining attention that may be useful in evaluation? Just as important, what is current that is fading in popularity?
  • What kind of questions, and what kind of programs, are people being asked to address? Where is demand increasing? Where is demand decreasing?
  • What are people doing to build their careers – training and education they are seeking, jobs they are looking for, how they package themselves.

I am agnostic with respect to how this information is obtained – surveys, interviews, focus groups, literature reviews, discussions with a few select experts. It matters not to me as long as the TIG has made an informed choice.

AEA Focus

The above deals with TIG-down in organizational terms. Here the focus shifts to the relationship between TIGS and AEA as an organizational entity. As I see it, TIGS represent “evaluation” in a more differentiated way than does AEA leadership. Whatever “evaluation” means, I can’t believe it is the same thing as “evaluation” from the point of view of “Advocacy and Policy Change”, “Feminist Issues in Evaluation”, “Needs Assessment”, “Program Design”, “Social Impact Measurement”, or “Youth Focused Evaluation”. AEA does need a coherent over arching concept of evaluation. (Of course getting there is impossible, but the attempt to do so is legitimate and important.) But if AEA is to successfully adapt to change, it must also be sensitive to more differentiated notions of evaluation. That cannot happen without knowledge held  by the TIGS.

I have been focusing on TIGS because  they are entities that have both relationships with evaluators who share similar interests and needs; and also, an important role to play in AEA. TIGS are important because they are so tightly integrated into our structure, process, and culture. But affiliates also have a role to play.

Affiliates

Affiliates are important for two reasons. One reason is that affiliates are a useful source of data because all the issues discussed above with respect to TIGS also apply to affiliates. The other reason is way less practical. It’s speculative. I’d forgive anyone who would classify it as a leap of faith. Affiliates are important in evolutionary terms. AEA has a national scope. TIGS have domain –focused scope. In contrast, affiliates have regional boundaries, but are inclusive of all evaluation domains. So one way to look at affiliates is as close relatives that are organized differently and adapting to different environments. Also, because they are small and have simple internal structures, they can change quickly. Finally, because they are close to AEA, change in affiliates stands some reasonable chance of being observed, and perhaps adapted, by AEA as a whole. From a survival point of view, that’s a pretty robust feature.

Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Part 3 of a 3 Part Series on how to Make AEA, and Evaluation, Relevant in the Future: Evolution, Diversity and Change from the Middle

  1. Bob Williams says:

    Jonny it seems a bit muted to me. I cannot see many people perceiving it particularly controversial. To be honest I see the TIGs as part of the problem rather than part of the solution because they have helped create silos of specific interests and particular dissatisfactions rather than any collective notion that Something Is Wrong. And the majority of evaluators frankly don’t care – that includes the majority of evaluators who not only don’t come to AEA conference, they don’t even belong to the AEA. A friend once estimated that about 80% of evaluators active in the USA do not belong to the AEA. I don’t know what the current membership is but contrast that with the 40,000 or so subscribers to the Linked In discussion group “Monitoring and Evaluation Professionals” – although admittedly many are not US basd.

    In terms of your analysis, while overall it makes sense in terms of changes ‘within’ the field, I think you fail to highlight the nature of the evaluation economy that is more than just a break on adaptation. There is a massive sunk cost in the current state of affairs and substantial risks involved in moving away from it; especially for the individual evaluator but also for the craft as a whole. I’m not sure how ‘complexity’ notions handle such phenomena (‘attractor’ doesn’t quite work for me); soft systems and critical systems would consider it part of a value base – requiring evaluators to identify and develop strategies around the beneficiaries and victims (both personal and ideological) of that value base. I know that this seems a huge nut to crack, but in all the workshops I run, without a single exception, the strongest and most common argument against changing core aspects of evaluation relates to the nature of the evaluation economy. And yes of course everyone tends to blame their problems on those whom they cannot influence. But ironically I’d argue that our customers are actually changing faster than the evaluation field is (although that may not be quite so true in the USA). And yes, we helped create that economy and theoretically at least we can help create the new one …. but we need a strategy to do it. I know from my own experience that evaluation as a field moves too slowly for some of our clients – who are and have headed elsewhere to seek clients. The fiasco over the field’s response to the rise and rise of 3IE and RCT is a classic example of the evaluation field think that what was essentially a political/market issue was merely a technical one that could be solved by arguments over method. Nonsense, the rise of 3IE was because it promised to solve a key political problem experienced by our paymasters …. uncertainty.

    Anyway some hurried random thoughts.

    • jamorell says:

      Hi Bob—
      Needless to say, you have sparked some thoughts. Here goes.

      BOB SAYS
      I see the TIGs as part of the problem rather than part of the solution because they have helped create silos of specific interests and particular dissatisfactions rather than any collective notion that Something Is Wrong.

      JONNY REPLIES
      TIGS may have their problems, but I am a big fan of silos because they are an efficient way of concentrating “resources” (in the broadest sense of the term) to further an objective. In the AEA TIG context, “resources” means intellectual capital and shared interest in developing some aspect of evaluation. Sometimes that aspect is intellectual, e.g. systems or quantitative methods. In other cases it is more domain oriented, e.g. mental health or R&D evaluation. Developing evaluation with those foci seem like a pretty reasonable goal for an interest group in a professional association to have.

      Obviously for a silo to be effective, it will inevitably be counterproductive in other ways because anything that requires coordination outside of the boundaries will be problematic. But the existence of silos is a natural development in any organization, and those silos will always have advantages and disadvantages.

      Also, good, bad, or indifferent, TIGS are what we have. That is not going to change, and they do represent the best (only) chance of effecting change from the middle.

      BOB SAYS
      A friend once estimated that about 80% of evaluators active in the USA do not belong to the AEA. I don’t know what the current membership is but contrast that with the 40,000 or so subscribers to the Linked In discussion group “Monitoring and Evaluation Professionals”.

      JONNY REPLIES
      Cut that estimate in half, and it’s still a big deal. And, I have no problem believing that 80% could be about right. I think that’s just fine. Sure I’d like to see AEA grow, but it will never include anything close to most of the people who do evaluation or evaluation-like work. That’s just fine with me. AEA occupies a niche on the evaluation landscape. I’d like to think we affect the rest of that landscape for the better. If we do, we serve our purpose. (One of them, anyway.)

      More important, this has implications for “evaluation”. Whatever “evaluation” is, it’s fair to say that if it is going to thrive, it has to have a range of manifestations. Diversity makes for robustness and adaptability.

      BOB SAYS
      There is a massive sunk cost in the current state of affairs and substantial risks involved in moving away from it; especially for the individual evaluator but also for the craft as a whole.

      JONNY REPLIES
      Yup. There is absolutely no guarantee that AEA, or “evaluation” will be able to adapt as things change. When environments change systems can break, disappear, species can go extinct. My only point is that the right kind of diversity increases the probability of successful change.

      BOB SAYS
      I’d argue that our customers are actually changing faster than the evaluation field is (although that may not be quite so true in the USA).

      JONNY REPLIES
      That, as I am so fond of saying, is an empirical question. Someone should find out. Not being an academic, I vote for pressing a graduate student in one of the graduate programs in evaluation into service. It would make for a nice piece of work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s