Evaluation Criteria and Boundary Critique

Bob Williams bob@BOBWILLIAMS.CO.NZ

Criteria are the engine that drives evaluation.  As evaluators, our core task is to address the ‘so what’ question not the ‘what’ question.  Indeed our focus on judgements of worth (e.g. merit, value, significance) differentiate our craft from other forms of social inquiry.  And you cannot arrive at a judgement of worth without criteria; either explicit or implicit.

Yet evaluators frequently treat criteria as unproblematic.  We commonly take evaluation commissioners’ criteria as the basis for our evaluative inquiry, without any questions asked. We boilerplate criteria such as those developed by the OECD’s Development Assistance Committee (DAC).  If we adopted a more critical stance towards criteria setting, we would do neither of those things.

C. West Churchman in the 1970s developed the discipline of Critical Systems. Many in the system field subsequently refined his ideas, especially Gerald Midgley, Martin Reynolds and Werner Ulrich. Critical Systems based approaches have a particular focus on how boundaries are established around a task.  In Critical Systems terms, boundaries decide who or what is acknowledged or ‘in’, and who or what is marginalised or ‘out’.  Ulrich’s Critical Systems Heuristics (CSH) comprise a series of questions around common boundary decisions that promote  deep consideration of those decisions and their consequences.  These considerations can be based on moral or ethical arguments, as well as practical and political realities.

Evaluation criteria are boundary choices.  They form the boundary between what the evaluation considers as ‘worthwhile’ and ‘worthless’, ‘significant’ and ‘insignificant’, ‘valuable’ and ‘irrelevant’.  Within a Critical Systems frame these decisions need critical review.  Take the common criterion of ‘efficiency’ for instance.  Who or what is ‘marginalised’ by such a criterion?  Why chose this as a criterion?  Is an inefficient intervention worthless?  And what kind of projects are marginalised by such an understanding?  What about interventions that are testing ideas or are in the process of development.  They are almost certainly not efficient but may be highly worthwhile.
I find that using the challenging questions that Critical Systems and Critical Systems Heuristics poses are essential to the development and use of criteria in my evaluations.

Some easily available references about the use of Critical Systems in evaluation.Better Evaluation https://www.betterevaluation.org/en/plan/approach/critical_system_heuristics
Werner Ulrich’s website : https://www.wulrich.com/csh.html
Bob Williams (2019) Systemic Evaluation Design; A Workbook https://gum.co/evaldesign