A friend of mind and I were discussing the nature of AEA. I have come to quite a few conclusions about this, but my current thinking is more in the way of questions than answers. As I see it, what’s needed is an exploration of four questions.
- Where does the evaluation that we do fit within all the evaluation that is done?
- How does what we call “evaluation theory” fit within the greater domain of social science?
- What is the nature of AEA’s professional support network?
- What are the implications of AEA’s social values for the field of evaluation?
Where does the evaluation that we do fit within all the evaluation that is done?
Every once in a while, in the popular press, I read about the evaluation of a social program. As far as I can tell, the people doing these evaluations have no involvement in AEA. I don’t see this as problematic. It just means that AEA attracts a breed of evaluator that does not work on the kinds of programs that the popular press likes to write about. But it does mean that a lot of evaluation takes place that is outside of the ambit of AEA’s member’s activity. More important, it is outside the ambit of anything AEA claims for evaluation. So, considering the population of evaluations that are done, what subset is covered by AEA members? Why that particular subset?
How does what we call “evaluation theory” fit within the greater domain of social science?
Over the decades there has been plenty of writing about how evaluation differs from other types of social science with respect to how evaluation should be designed and executed, and how the products of evaluation should be used. It is largely this body of work that sets evaluation apart from other social science. Still, evaluation theory does not exist in isolation. Some of it is a novel extension of what went before. Some of it is a new expression of what already exists, that new expression needed to orient evaluation in unique ways.
What is the nature of AEA’s professional support network?
We know that one function of AEA (as with all professional organizations) is to embed its members in a professional support network. But what is the structure of that network? How does it function and influence the work lives of its members? How does it further their professional interests? How does it develop and evolve? What sub-networks does it contain?
What are the implications of AEA’s social values for the field of evaluation?
AEA as an organization pursues a particular set of values with respect to the dissemination of data, the use of data, the value of data, and the social/organizational processes that should drive the development and execution of evaluation. There are consequences to pursuing these values for the kinds of programs that are evaluated, the nature of the information that is produced, and the range of stakeholders who might look to evaluation as a source of knowledge.
Evaluation Uncertainty 
Jonny, I’m having my usual problem with trying to leave a comment. I’m asked to log into WordPress before the comment can be accepted. And then starts a cycle that just carries on.
Anyway here is my comment
Two thoughts on this Jonny. One is that AEA membership has always been a minority of active evaluators. It has only 5,000 members – about 4,000 of them in the USA. Years ago, working with Iraj Imam, I discovered that there was a whole network of small evaluation shops that mostly ticked boxes for the Feds, and tried to do as much good as they could on the side. None of these organisations saw the AEA speaking for them in any way.
My second point is my concern about Evaluation being conceived primarily as a branch of social science. It this framing that I feel is killing Evaluation (as distinct from evaluation). In my framing, Evaluation is not a science although it uses some aspects that knowledge system in the craft. But it draws from so many other knowledge systems that are not ‘scientific’; community development, policy analysis, journalism, organizational change. And of course, increasingly it is drawing on knowledge systems used by indigenous cultures. Drawing the boundary of the endeavour around the tight definitions and understandings of ‘science’ is both ontologically incorrect and epistemologically barren.
>