I am an evaluator who also teaches evaluation courses and conducts research on evaluation. In my work, I draw on the ideas of boundaries and boundary critique from critical systems thinking. These ideas have deep philosophical roots and the potential to alter or affirm the way you see the world, and act and interact within it. They are fundamental to what ‘system’ and a ‘systems approach’ mean. In the limited space here, I share five points about boundaries.
Point #1: Boundaries influence what evidence and values are considered relevant or not. In evaluation, boundaries include: the time frame and scale for evaluating an intervention; the distinction between an intervention and its context; the questions and quality criteria that frame the inquiry; and who or what group(s) are considered stakeholders. Each of these (and other) boundary choices influence the empirical (i.e. data, evidence, facts) and normative (i.e. perspectives, values, principles) bases considered relevant or irrelevant in an evaluation.
Point #2: Boundaries compel us to look critically at who or what may be excluded or marginalized. Using a systems approach is often thought to mean being holistic and pluralistic, including all interrelationships within and influencing the intervention and including all stakeholder perspectives. Clearly, these are aspirations. When we shift our focus to boundaries, these (practically unattainable) aspirations become careful, critical examinations of who or what can defensibly be excluded and potential consequences of such exclusion.
Point #3: Not examining boundaries poses risks for ‘unintended’ consequences and ‘invalid’ claims. It is unavoidable that someone(s) and something(s) will be left out, and that who/what is left out bears directly and indirectly on the evaluation. These can be labeled ‘unintended’ consequences meaning they weren’t what the evaluation and evaluators set out to have happen. For example, if an evaluator takes for granted the stated problem an intervention is designed to address, and frames the questions around how well the intervention addresses this problem, the evaluation will exclude and perhaps further marginalize alternative or contrary/conflicting perspectives on the problem. Whether made apparent or kept implicit, this will have consequences and will condition the validity of the evaluative claims.
Point #4: Critiquing boundaries is a way to enact ethical responsibility and warrant claims. Despite the risks I mentioned in Point #3, boundary judgements are often made implicitly or unknowingly, as is the case when an evaluation uncritically adopts the boundaries set by those commissioning the evaluation and/or defining the terms of reference. This is unacceptable in critical systems thinking. Because boundaries have consequences, which can be profound and counterproductive to an intervention’s and/or evaluation’s stated goals, evaluators have an ethical and professional responsibility to critically examine which boundaries are (and should be) used and how they are (and should be) set.
Point #5: There are no ‘right’ boundaries or ‘experts’ in boundary setting. Critical reflection and deliberation are processes for making ethically defensible boundary choices. Making boundaries explicit and choosing between options within bounded evaluations is not and should not be limited to those on the evaluation team. Additionally, using pre-existing boundaries in theoretical and empirical research, particularly those inherent in methodologies, instruments, and measures, are not sufficient grounds for use. Critical systems thinking challenges the basis of expertise in evaluation and argues that boundaries should be ethically justified through processes of explicit, critical examination of boundary choices and their consequences. Boundary judgements cannot be made solely in one’s head or within an evaluation team; it is important to examine alternative boundaries from different perspectives and value stances. Preferably, evaluators use participatory and deliberative processes.
In Sum: When we cannot include everything and everyone, where does this lead us? Critical systems thinking obligates us to make explicit boundary choices, consider their consequences and alternatives, and make boundary judgements/decisions that we can ethically stand behind, while also being open to critique, debate, and ongoing reevaluation and renegotiation.
Acknowledgments: These are the points that have stuck with me and to which I regularly turn as I consider and make choices in my evaluation practice. They are not my points, but those centrally made in the works of C. West Churchman (US), Werner Ulrich (Switzerland), Gerald Midgley (UK), Martin Reynolds (UK), and Bob Williams (New Zealand). Mostly white, European men. I recognize the risk of this critical systems lineage in perpetuating power imbalances around who and where knowledge is generated from – a risk I find ethically problematic. Therefore, I call attention to the continual evolution of critical systems thinking as advanced in the Inclusive Systemic Evaluation for Gender Equality, Environments, and Marginalized Voices (ISE4GEMs) framework (info here) developed by Anne Stephens (Australia), Ellen Lewis (UK), and Shraventi Reddy (US); Nan Wehipeihana (Māori, NZ) and Kate McKegg (NZ) who practice and write about developmental evaluation drawing on critical systems ideas; and my work (US) on critical systems heuristics. I welcome feedback on who or what I have excluded in this list and will revise accordingly. I included this note to both model and invite further boundary critique. And a shout out to Joseph Madres, for his editorial review of this post!