Please post some information about your professional background.

Slides from a previous version of this workshop can be found at: Workshop slides. Please post your opinion about which sections are most important to you:  1- models, metrics and methodology, 2- graphic design in support of information density, 3- methods of working with stakeholders. Also, are there any particular topics you want covered?

We (Joanne Farley, Tarek Azzam and I) have been musing about whether evaluation as practiced by the members of AEA is framed within too narrow range of political and social ideologies. We suspect that it may be, and that as a result, evaluators miss important elements of program theory, metrics, and methodology.

We are conducting a Think Tank at AEA 2011 as an experiment to test our conjecture. Our plan is to ask AEA members to design an evaluation of a program that we know is embedded in a rich set of values and beliefs. The idea is to take the same program and ask people to design the evaluation from different points of view. Below is a description of the scenario we cooked up. We are looking for ideas about making this one better, or for something entirely different. Thanks in advance to all who take the trouble to weigh in.

Scenario

A government agency has developed a sex education curriculum. In our mythical program

schools will not be required to implement the curriculum, but the program roll-out and dissemination will actively try to influence all stakeholder groups – State Departments of Education, local school boards, and community groups. Also, schools that implement the curriculum will be eligible for technical assistance in its use, and free access to instructional materials.

Long term desired outcomes as articulated by the program’s developers are:

• STD prevention and
• pregnancy prevention.

Intermediate desired changes as articulated by the program’s developers are:

• Increased self esteem of girls who participate
• Increased knowledge of alternatives to intercourse
• Increased ability to negotiate sexual activity with a partner
• Increased knowledge needed to choose methods of contraception
• Increased knowledge about the risks of sexual activity including intercourse, and
• Increased ability to clarify one’s values with respect to engaging in sexual behavior.

Exercise

Participants will be randomly assigned to one of three groups, and asked to design an evaluation that emanates from one of three political ideologies.

•  Government intervention can be effective, and it is legitimate for government to exercise that ability to improve the social welfare.
• Government intervention is most likely to be either not effective or counterproductive, and in any case the right thing for government to do is to do as little as possible so that individuals can act according to their own sense of personal responsibility.
• The family is the center of authority over people’s lives. The decision about whether government should act or not act depends on whether the action furthers or inhibits family authority.

Participants will be asked to proceed with an open mind, i.e. there has to be some recognition that the curriculum has a range of outcomes worth measuring. No fair to dismiss the program out of hand. From this point of view, each group will be asked to produce a high level evaluation plan encompassing:

• Program theory
• Outcomes
• Metrics
• Methodologies

Will this exercise work to elicit the discussion we are looking for? What else might we do?

I’m looking for comment and critique. Draft of presentation slides

Proposal

Purpose and Assumptions

The focus of the proposed presentation is methodology to evaluate unanticipated and unintended consequences of program action. It is based on an evaluation theory and a set of case studies developed by the presenter.[1] The specific track for this proposal is “building evaluation capacity”, with an immediate impact on “development”, and a longer term impact on the “building democracy” element of the “diplomacy” theme. The proposed presentation is based on two principles.

§         Good evaluation must often depends on maintaining the integrity of an evaluation design over time. For instance, it may be necessary to interview service recipients within a narrow window of opportunity after they received a service, or to apply a previously validated scale over a protracted period of time, or to maintain good relations with program staff so that observations of their work can be made.

§         In order to assure the integrity of an evaluation’s design, scarce resources are needed to build and maintain an “evaluation infrastructure”. To continue the previous example, it takes time and effort for evaluators to maintain a set of agreements with program managers and policy makers, or to develop and validate scales. Once these resources are used, there is that much less opportunity to adjust the evaluation design in the face of unexpected program behavior.

These principles are challenged by the fact that the consequences of programs are often different from what planners expected, and therefore, different from the outcomes that evaluators had planned to measure. How then, to best maintain the power of an evaluation design and still be responsive to changing needs? This is the question that the panel will address. As the answer unfolds it will have major consequences for how research questions are formulated, strategic planning, and program monitoring. Consequences for question formulation and strategic planning derive from unintended outcomes whose roots are in narrow conceptualizations of program theory and outcome. Consequences for monitoring derive from the fact that “lead time” for detecting incipient program change is a critical element in evaluation response to unanticipated change. Good monitoring can increase lead time.

Unexpected Outcomes in Development and Democracy

The techniques to be advocated are particularly relevant in development settings because programs in those contexts are extremely prone to outcomes that were not anticipated by planners and policy makers. This uncertainty exists because development programs often involve rich, tight linkages that affect many aspects of  the systems in which they reside, and also because the environments in which they exist can be unstable.

The role of unexpected program outcome is particularly important when trying to assess the relationship between program outcome and democracy because of a mismatch between what is known about the relationship between democracy and development on the one hand, and the causal path between a development program  and the precursors to democracy on the other. We have a good idea about what brings about democracy:

The good news, however, is that the conditions conducive to democracy can and do emerge—and the process of “modernization,” according to abundant empirical evidence, advances them. Modernization is a syndrome of social changes linked to industrialization. Once set in motion, it tends to penetrate all aspects of life, bringing occupational specialization, urbanization, rising educational levels, rising life expectancy, and rapid economic growth. These create a self-reinforcing process that transforms social life and political institutions, bringing rising mass participation in politics and—in the long run—making the establishment of democratic political institutions increasingly likely. Today, we have a clearer idea than ever before of why and how this process of democratization happens.[2]

From the point of view of program theory and evaluation though, any given development program may affect any democracy precursor in many ways, not all of which will be foreseen by planners and evaluators. Moreover, the evaluation challenge is compounded in the typical situation where many different development programs coexist within the same system boundaries. Finally, given the many paths and interactions that may develop, the same result may come about through many different paths. To illustrate:

Example: Difficulty of Relating Programs to the long term goal of promoting democracy

Imagine an overly simple scenario in which two programs are operating, one whose primary goal is job training, and one that provides affordable cell phone service in rural areas. Again to oversimplify for the sake of illustration, we may state a program theory for each program. The job training program leads directly to “occupational specialization”. The cell phone program increases the richness of social contact and the ability of farmers to peg their prices to world commodity prices. These are reasonable immediate outcomes to expect, and they do need to be justified (i.e. evaluated) as such. Understanding their consequences for democracy, however is complicated. One problem is that many possible interactions can take place. Will the richer social networks make it easier for people to find ways to specialize their work? Will better commodity pricing allow more people to take advantage of the job training? Will unequally distributed rising levels of wealth and education support or upset the existing social structure? Might the changed social structure set in motion its own undesirable consequences? Any of these outcomes are possible, as are many more that one could conjure with a little bit more time and imagination. From the point of view of evaluation these uncertain outcomes are problematic for methodological reasons. Again to take two overly simple examples. What if, after the evaluation was established, we suspected an unanticipated interaction between occupational specialization and social relationships? Or, we suspected that agricultural pricing in the location of the cell phone intervention was having larger scale impact because of its contribution to a tipping point change in a region that was experiencing the effects of other development programs? Supposing both were occurring and we cared about their relative impact? The first example is about micro-level short term change. The second is longer term and larger scale. They may or may not interact. What they have in common is that the original evaluation infrastructure would be inadequate to determine the state of affairs. Different people would need to be interviewed. Different statistical data bases would have to be queried. Different data collection timelines would have to be worked out. Different comparison groups would be needed. None of this is cheap or easy.

Methods for Evaluating Unintended Program Impact

The approach to be advocated in the proposed presentation is set out in the writing of the presenter, and can be summarized in the graphic below.¹

§         There is a continuum that ranges from events that “could reasonably be foreseen” to those that  are impossible to predict because they emanate from the operations of complex systems.

§         Different evaluation tools are differentially useful at different points along this continuum. For instance applying diverse program theories is particularly useful for anticipating outcomes, monitoring and evaluation for early detection of unexpected change, and  multiple data sources to make the evaluation more capable of measuring a wider range of program behavior.

§         It is impossible to deal with all eventualities, but there are ways to “chip away” at the problem and by so doing to make evaluation more robust in the face of program change.

§         Any design choice to make an evaluation more robust in the face of change carries its own potentially negative consequences. Thus choices have to be made carefully. For instance the resources needed to maintain multiple data sources may diminish resources for interaction with stakeholders, or effort devoted to analysis.

I got interested in this topic because now that “Evaluation in the Face of Uncertainty” has been published, I have been pondering how to advance some of the ideas I set out in that book. The “fidelity/attractor” idea is one of the trails I have been sniffing.

What is the problem?
I think there are two big ideas floating around in our field that are tugging at each other. The first is the notion of “fidelity”, i.e. the notion that for a program to be successful as it moves into different settings, it must maintain a core set of characteristics. (Let’s assume we know what those are.) The second idea is that things are never the same as they move through space and time. We know from research on innovation adoption that “reinvention” is common. And certainly, all our recent talk about developmental evaluation, systems, complexity, and so on speak to the belief that programs are in a constant state of flux. I’m groping for a way to bring “fidelity” and “change” under a single conceptual umbrella.

What are attractors?
“An attractor is a set towards which a dynamical system evolves over time. That is, points that get close enough to the attractor remain close even if slightly disturbed.” (http://en.wikipedia.org/wiki/Attractor#Limit_cycle).

Closely related to the notion of an “attractor” is the idea of phase space: “In mathematics and physics, a phase space, introduced by Willard Gibbs in 1901, is a space in which all possible states of a system are represented, with each possible state of the system corresponding to one unique point in the phase space.” (http://en.wikipedia.org/wiki/Phase_space). Attractors come in three general categories.

A “point attractor” is an attractor where once all energy dissipates from a system, it settles down to a single point. A good example is a pendulum.

A “periodic/limit cycle attractor” is an attractor in which there is continual movement, but all points can be identified. A planet in orbit would be a good example. Another example would be the cyclic behavior defined by predator/prey relationships. “An example of a limit cycle is a predator-prey system. Imagine a lake with trout and a smaller number of pike. Those pike eat the young trout. Because there is so much food the number of pike increases. This increase in pike and all the young trout they eat means that the number of trout decreases. The drop in trout numbers means that the pike have less to eat and the pike numbers then decrease. This allows the trout numbers to increase again and the cycle begins over. The populations of trout and pike rise and fall in a cyclic fashion.” (http://en.wikipedia.org/wiki/Attractor#Limit_cycle.) (By the way, this relationship can very easily transform into formally chaotic behavior if the parameters change a bit, but that’s another story.)

“Strange attractors” are unique from other phase-space attractors in that one does not know exactly where on the attractor the system will be. Two points on the attractor that are near each other at one time will be arbitrarily far apart at later times. The only restriction is that the state of system remain on the attractor. Strange attractors are also unique in that they never close on themselves — the motion of the system never repeats (non-periodic). The motion we are describing on these strange attractors is what we mean by chaotic behavior.” (http://www.stsci.edu/~lbradley/seminar/attractors.html).

What do attractors have to do with fidelity?
I wonder if it helps in evaluation to think of program fidelity as conforming to one or another type of attractor. Doing so would help with program theory because it would help us understand the states of the system that we could consider as being able to have the expected effect. It would help with methodology because different types of programs may require different methodologies.

Point attractor fidelity would be a situation in which in order for a program to be effective, it had to operate within very narrow limits of variation. (I suppose this could be more like a limit or strange attractor operating in a very small phase space, but for me thinking of such narrow variation in terms of a “point” is nice.) If data or theory supported the idea that fidelity had to be this precise, then the evaluation would have to contain a few elements.

First, the process evaluation would have to contain a very rigorous effort to see if the program’s supporting systems (e.g. regulations, funding levels, collaborations with related services), and internal operations (e.g. cross linkages, management control, expertise of service providers, client tracking) allowed the organization to monitor key elements of fidelity within very narrow parameters.

Second, program theory would say that this treatment may not be very useful in messy real world situations. In this case we may want to test the applicability of the program in a variety of settings. In other words, even if the treatment were already proven in test settings, we may need to put considerable effort into testing outcome in a variety of diverse contexts. This would add a lot to the cost and difficulty of doing the evaluation, but it may be necessary because we do not know much about the characteristics of programs and systems that nudge a program away from a narrow range of critical values.

Periodic attractor fidelity reflects a program theory that says that the way a treatment is administered can “wander around” over time, but in fairly predictable ways.

For instance I can imagine a treatment program that is somewhat analogous to the fish example above. Imagine an industrial setting in which historically poor labor – management relations and a blame-based culture prevented the kind of collaborative problem solving that is needed to get to the root causes of accidents, thus preventing substantive improvements in safety. After all, why would labor report on their own bad behavior to management if doing so would result in punishment? (This hypothetical example is loosely based on some real programs I am evaluating.) An innovative program is established to give labor relief from certain kinds of disciplinary actions in order to foster collaborative problem solving. What might happen? The greater the relief from discipline, the greater the participation and the better the generation of corrective actions in the furtherance of safety. But too much relief from discipline might bring the company to a point where labor really did engage in too much dangerous behavior, thus prompting management to tighten up on discipline. This tightening of course, would lower participation rates, thus reducing the amount and/or quality of safety improvements that could take place. After a while the decrease in effectiveness is perceived, and also, memory of all the problems caused to too little discipline fades. At that point relief from discipline goes up, as does the problem solving. Over time (maybe even a period of years), the cycle repeats.

In essence, “program fidelity” in the example consists of a narrow range of two parameters — manager and worker behavior. We can expect continual cycles of program impact based on inevitable tensions between those two parameters. That’s the program theory. From a methodological point of view, we have a need to carefully observe those two parameters and to understand the psychological, social and organizational variables that affect the interplay between them. Why? Because if we understood that interplay, we could give stakeholders good advice on keeping those levels at optimal levels.

Strange attractor fidelity is a situation in which program theory says that small changes in program operation can result in very large changes in the extent to which a treatment conforms to a predetermined set of operating characteristics.

As an example, take another scenario that is loosely based on some real evaluation I am doing. An organization is seeking ways to address serious problems that have proved impervious to many efforts at solution. The organization comes up with the idea that a diverse group of people, pulled from across its membership, and well trained in problem solving, might be able to come up with creative effective solutions. As the organization develops the program, it decides that a large number of elements need to be attended to in order to make the program work: 1) good group leadership, 2) good group followership, (i.e. members who understand how to participate in groups), 3) analytical expertise distributed in group members, 4) management’s ability to choose important but solvable problems for the groups to work on, 5) ability of the group to redefine or partition the problem, 6) available data, 7) motivation / reward system for participation, 8) selection process that brings people into the groups who have certain levels of skill, interest, and motivation, 9) a similar selection process for group leaders.

These are a lot of characteristics, and is easy to imagine lots of interactions among them. What this means is that a great deal of variation in group effectiveness can come from small changes in any of the characteristics. Thus we can’t identify a single sets of values for any one of these characteristics that would make for successful problem solving. An implication is that our program theory would not be very precise. The best we could do is identify some very broad ranges of variation for each parameter and some boundaries for a global measure of “problem solving ability”, which in essence, would define the phase space in which the program had to operate. From a methodological point of view, this means we need a good global measure of “group problem solving ability” because it is impossible (and meaningless) to try to construct one from its constituent parts.

What now?
The above is as far as I have taken my musings on thinking about fidelity in terms of attractors. As I said in the beginning, I have not convinced myself that it is worth thinking of fidelity in this way. One big problem I see is that the typology I am using implies knowing a lot about treatments and their administration. The state of knowledge for almost all programs may well be that we can never know enough to apply the attractor idea.

So that’s why I need help. Comments are very much appreciated. Thanks to all who are willing to help.

Abstract:
How does political ideology affect program theories, methodologies, and metrics? Participants will be randomly assigned to groups, and asked to sketch an evaluation based on one of three positions. 1) Government has an obligation to alleviate social inequities and thereby promote the public good. 2) Government’s role is to uphold civil order so people to pursue their own goals, with the consequences of their actions being their own personal responsibility. In general, less government is better. 3) The family is the primary unit of social cohesion, and there resides the locus of decisions about issues such as health and education. Government can be active or passive, as long as it supports the centrality of the family as the locus of moral authority and daily living. During report backs and we will compare how the evaluation designs differ with respect to program theory, metrics, and methodology.

Relevance Statement
Evaluation is at its core a conservative business. We do not develop new programs. We do not plan innovative policies. We spend our time, our intellectual capital, and our resources on assessing programs that other people have already committed to. We are constrained to evaluate within the narrow frameworks given to us by our customers – i.e. by stakeholders who have the power, authority, influence, position and drive to make innovation happen. We should not denigrate what those stakeholders do or who they are. Bringing about change is extraordinarily difficult, and those committed to effecting change are acting in good faith to do good as they see that good. Providing empirical understanding of the consequences of those actions, and guidance for improvement, is a noble pursuit. Still, although evaluation may be conservative in its scope, the choices we make within that scope are sensitive a wide range of ideologies as we guide stakeholders through exercises to articulate program theory, make choices about what to measure, and decide upon a logic in which to embed our observations. These ideologies have led to well known debates in our field. How much should we attend to the needs of stakeholders who are not involved in program development or execution? What effort should be put into creating knowledge rather than guiding decision makers? How far should we stray from measuring the outcomes that were planned for the program at hand? How tied should we be the values of our customers? We cannot do our jobs well without confronting these questions. But as a practical,(and perhaps as a moral), matter, we cannot,(or perhaps should not), stray too far from the needs of the people who pay us. How then, to best serve those needs? We contend that to provide the greatest value to our customers, we should base our work on an appreciation of the hidden assumptions in program theory that emanate from political ideology. Program theory drives metrics and methodologies, and metrics and methodologies frame our findings and our conclusions. If we are trapped within our ideologies, we cannot help our customers understand the full scope of the changes they are trying to implement. We may or may not succeed in getting our customers to include that scope in their operations, but we cannot even try without self knowledge. We believe that precious little of that self knowledge resides within the intellectual stock of the individual members of AEA (After all, are we any different from all other individuals?), and within our membership collectively. That needs to change. We see this Think Tank as a small step in bringing about that change.

Sanjeev says:
Sometimes governments invest large amounts of  money on grand ideas — these ideas are often not based on evidence or detailed plans. A kind version of this story is the  “innovation” lies in taking risks based primarily on somewhat vague promises of the original idea. In all likelihood the set of interventions that connect the activities to the long term outcomes  are going to be highly complex even though the language of accountability is framed as though the intervention is either simple/complicated.

Jonny says:

• I talk about part of this issue in Chapter 3 of my book in a section titled: When is the probability of surprise high”. I won’t go into details here, but I argue that given economic, political, social, and human capital realities, governments have little choice but to implement unidimensional solutions to multidimensional problems.
  
  It’s the rational choice because it is that, or nothing. The people establishing these programs know better, but they cannot act on the knowledge.

• As for taking risks based on vague promises. I’m in favor of it. The truth is that most important decisions get made that way. Of course I’m in favor of drawing on lots of experience and knowledge to make the most informed choice possible, but the idea of some kind of direct relationship between data and action is illusory for most important decisions. Should the U.S. pass Obama’s health care reform? Should NATO implement a no-fly zone over Libya? Should a university hire professor X or Y? Should Apple market the I-Pod, or Ford the Edsel? Should Head Start be implemented? (Remember the data on Head Start that was drawn upon to justify it’s establishment was a pretty small study.) And so on and so forth. There may be lots of good research and theory to help inform these decisions, but given the uncertainties of the world, there is more “taking risks based on vague promises” that my narrow overly rational mind is comfortable with.

• If I’m right, the value of evaluation is downgraded to helping with incremental improvement, sort of like a CPI situation. The feedback latency can be long and short. Short: I am implementing a new reading program, is the teacher training component part working or do I have to revise it before we scale up to the whole school? Long: NIH has spent years funding multiple studies on helping to get evidence based practice implemented. Over time each study provides feedback for the next round of efforts. But in both cases we are talking about incremental improvements. The big changes are a leap of faith, with hopefully, as much science as possible mixed with the faith.

Sanjeev says:
How does the M & E system differ between interventions systems that are complicated vs. complex?

Jonny says:

• This is yet another question I don’t have a very good answer for, but I do deal with something related in Chapter 5 my book, titled: Shifting from Advance Planning to Early Detection. In there is a discussion of extending M&E to scope for unanticipated change with respect to: 1) using the data to revise logic models, and 2) scope for changes in the program’s environment. I don’t make any distinctions between complicated and complex, but the ideas might be useful for what you are cooking up.

Sanjeev says:
Equally, i have a concern that the language of uncertainty and emergence, can distract the implementers of this program to think more deeply about the mechanisms (non-linear or otherwise) that are necessary to impact long term outcomes.

Jonny says:

• How right you are! I think people are becoming besotted with “complexity”. They are seduced by the sexiness of the exotic ideas to the detriment of using what is known to do some good.

• I do think there are some ideas in complexity that may be duplicated by other frameworks but where casting them in terms of complexity would be useful.. One of my favorites is the notion of fitness landscapes of different shapes. I like to think of programs as organisms evolving on fitness landscapes, complete with competition and cooperation among the various organisms. Sharply peaked landscapes mean small change can result in high payoff or catastrophe, in which case incremental improvement can be risky. With the landscape idea I can also think in terms of local optima, co-evolution, shifting typographies, and the like. But this is a personal decision for me. Could I articulate any one of these notions in other terms, e.g. error variance in environmental change? Of course I could. But I find the fitness landscape framework useful because it’s a neat way of encompassing a lot of different aspects of how a program or an organization functions.

• There are also some constructs in CAS that are useful and not duplicated in other frameworks. My favorite candidate is the notion of “edge of chaos”, the region were an entity is adaptive to its environment at the same time vulnerable. (Of course I have no idea how to use this idea in evaluation except in the most general heuristic fashion, but that’s another story.)

• Another concept is the power law distribution. (I know this is not the exclusive province of CAS, but it does play a big part in discussions of complex behavior.) I think the idea of equal numbers of occurrences over time with a power law distribution of the size of those occurrences is a very useful way to think about how frequently programs will have a noticeable effect, how efficiently they will function, and so on. I see it as an aspect of program theory when there are multiple efforts going on in a system to solve the same problem.

Sanjeev says:
Any thoughts and examples of differences in accountability systems between complicated and complex  would be most welcome.

Jonny says:

• Accountability: I have been thinking about an analogue to this problem with respect to condition based maintenance in big machines – jet engines, locomotives, etc.  Accident prevention is also relevant – normal accidents (Perrow stuff), multiple paths to the same accident, known historical causal path but no future prediction, etc. In all of these domains there is a tendency to look at the condition of parts of the system (which is necessary of course), as a way of understanding the overall system. That’s what accountability is usually all about, at least in it’s M&E sense. As I see it the problem gets worse the more granular the observations. To continue the previous example, do I want to determine whether the teachers who went through the training can teach the new curriculum, or do I want to look at the quality of the workshop instructors, the instructional design of the materials, attendance records, and the like? I’d be comfortable  with the less granular level, but not with a program theory that articulates a specific path by which the teachers are brought to a level of competence. (Let’s leave aside the question of what “granular” means with respect to system boundaries, and take it all intuitively. Otherwise we open up a huge can of worms.) I talk about this stuff in my book and my logic model workshop. There are implications for program theory (pegging the theory to what we actually know), and methodology (what can we measure, when, etc.).

• Complicated and Complex: I don’t know of any way to look at a system and know if it is complex until it displays complex system behavior. Suppose you were looking at a complex system that was wallowing deep in a steep attractor? The system would be incredibly stable and yet be complex. Of course we know characteristics of CAS, e.g. non-linear interactions, multiple feedback loops, path dependence, etc. But all those could be operating to form a system that was so stable that you could whack it over the head with data forever, and it would never change. That’s why I don’t find the “complicated / complex” distinction so useful. And, I can easily imagine an unstable system that was not complex. All you need is one critical path element that could break easily and there would be lots of instability with no complexity.



Theme of Panel
The basic them is how political ideology affects metrics and methodology. To give a simple example of both.

Metrics: Imagine a program that taught literacy to immigrants. “First order” impacts are pretty straightforward, e.g. ability to read, quality of life, employability, and so on. But what about the “second order” impacts, i.e. things touching on the overall social condition – impact on community and economy, etc. I’d bet dollars to donuts that people with pro and anti-immigration feelings would come up with different measures. For instance what about a measure of whether wages are driven down, or of local disruption to social services in communities? It is certainly reasonable to posit these changes, but I doubt that the pro-immigration advocates would not do it.
Methodology: Imagine a needle exchange program evaluation being designed by two different groups: 1)  public health advocates, and 2) people who had very strong moral objections to doing anything to help drug addicts take drugs. People in the latter group might reason as follows:  “I am not closed minded, but I really think this is a bad idea. So I am going to design the most rigorous methodology possible – random assignment,  no treatment control, alternate treatment (no needle exchange) group, and a long term follow-up. Why do I insist on this? Because unless the conclusions are bullet proof, I ain’t gonna change my mind that needle exchange programs are a bad idea.” I can easily imagine how the pro-needle exchange folk would be satisfied with a less rigorous methodology.

Use of a common example
My thought is to invent a program as an example we can all work from. It can be based on something real, but I have found that in order to have an example that illustrates all the points one wants to illustrate, some synthetic elements are necessary.  The example has to be something on the large and complicated side, but well within the scope of the kinds of programs that AEA members might actually get a contract to evaluate. I want to make sure the example resonates with people’s experience.

Panel development process
If you are up for it, I’d like to design this panel in a two-tier social networking framework.
Tier 1: Us, i.e. the people with the responsibility to design and submit the panel to AEA. Any advice we get from others is just that, advice. We can use it or not as our wisdom dictates.

Tier 2: The rest of the world. We exchange our ideas in the “fishbowl” of my blog site. Content, design, assignments, etc. In other words, I’d put this email message up and instead of our hitting “reply all” we would post to the blog. This would have two advantages. First, other people might have worthwhile things for us to consider. Second, it would generate publicity for the panel. (And frankly, I would not mind increasing hits on my blog, either.)

What I’m proposing is an experiment, and to be honest, I’m not expecting too much in the way of public comment. But I think it’s worth a try if you do. Let’s do the first round of responses via email to get a sense of whether you are willing to work in the fishbowl. If you are, I’ll transition the material.

What do we want out of this?
Aside from the fact that doing this panel would be a lot of fun, I assume that we have our reasons for wanting to be involved. It might help shape our discussions if we knew the reasons. If you don’t have any that’s fine, but if you do, let’s hear them. For me, there are two.

1-            Unintended consequences of program behavior: You all know that this is my favorite subject. Once I finished my book and had a chance to think about the intellectual frontiers I had not explored, I realized that ideology can be an important reason for unexpected behavior. Either there are variables people did not think to include, or the methodology has some blind spots. I want help exploring this frontier.
2-            Diversity: AEA is big on gender, ethnic, and racial diversity. I am in favor of this, but I think the field would be a lot better off if there were more emphasis on ideological and methodological diversity as well. How much poorer would economics and political science be if there were no research and theory based on both Marxian and free market paradigms? Quite a lot poorer, I think. I have a strong feeling that the politics of AEA members clusters in the medium-rare range.( I can’t prove it, but I bet I’m right.) I’m not arguing that we go out and recruit other members, but I do think that AEA would be a lot better off if we had a collective appreciation of the implications of what we don’t have.

Logistics
There are three questions we need to resolve.
1-            What TIG should we submit this to? I’m thinking maybe “Evaluation Theory”, but I really don’t have any strong feelings one way or the other.
2-            Deadlines. I think we should set ourselves the goal of being able to submit the panel by the end of February.
3-            What format should we use? We could do a panel, but there are other possibilities. For instance maybe a think tank, which gives us less time to present, but the audience a lot more opportunity to participate.

1) I have never taken a poll, but my bet is that the members of AEA cluster on the left end of the ideological spread. (Go ahead, prove me wrong. Someone should find out.) I’m not sure we serve our customers and stakeholders well if we do our work from too homogeneous a point of view. 2) The reason we don’t serve our customers and stakeholders well is because the nature of the data we develop, the findings we produce, and the outcome of our efforts at utilization, all combine to provide overly restricted choices relative to the policy decisions that can be made. 3) As an association that cares about the public good, what good are we if we provide weak guidance?

As I see it the problems with a restricted range touch on all aspects of what we do along the entire evaluation project life cycle. 1) What programs or groups of programs do we choose to evaluate? 2) When we design evaluations, what bodies of research literature, groups of experts, and theories do we query? 3) How do we choose which stakeholder groups to involve, and how do we determine the relative importance of each? 4) What positive and negative outcomes do we invest effort in trying to measure? 5) What time frame do we choose for measuring program effects? 6 How much effort do we put into measuring various opportunity costs? 7) How do we interpret data?

So, I propose a thought experiment that involves a collective effort to fill in a table. I define the rows by various political ideologies. I imagine a group of seven categories. 1) Marxist. 2) Socialist 3) Center left (sort of like European Social Democrats), 4) Center right (sort of like European Christian Democrats), 5) Social conservative, 6) conservative in the Edmund Burke sense, and 7) small government conservatism.

As for the columns, I came up with eleven programs that evaluators might be called upon to evaluate. 1) R&D investment by government for work that is near the R side of the R&D continuum. 2) Tax subsidies to wind energy. 3) International development programs to build up civil society. 4) International development programs for improving agriculture. 5) Web 2.0 programs for government to provide services to citizens. 6) Organizational change efforts to make government agencies more effective. 7) The impact of rule making in federal regulatory agencies. 8- Early childhood education. 9) Initiatives to teach under served populations to advocate for themselves. 10) Family support for military families. 11) Immunization promotion.

I certainly don’t think that we should all start filling out a 7×11 matrix, particularly since I started this post by admitting that I can’t do it myself. But I do think it would be of value to all of us if people who resonated to any of the cells took a crack at filling them in. At least, I know that it would be valuable for me.