A Plan for Making Complexity Useful in Evaluation

Recently a friend of mine asked about my understanding of what role complexity can play in Evaluation, and how I would further that role. Below is an edited version of what I sent her.

My goal for the role of complexity in Evaluation

Complexity as discussed in evaluation circles contains a great deal of information that is either wrong, or ill-chosen as elements of complexity that can be useful in Evaluation. Those discussions do not appreciate the broad and deep knowledge about complexity that has roots in many different scientific disciplines. Much (most really) of that knowledge is not applicable in the field of Evaluation, but some of it is. My goal is to influence the field to appreciate and use the knowledge that is applicable.

As I see it, the critical issue revolves around program theory, i.e. people’s beliefs about what consequences a program will have, and why it will have those consequences. The problem is not methodology because for the most part our standard arsenal of quantitative and qualitative tools is more than adequate.  The problem is that evaluators do not choose appropriate methodologies because their designs are customized to test incorrect program theories.

What is complexity?

Continue reading “A Plan for Making Complexity Useful in Evaluation”

Evaluation use by people opposed to the program

Recently I posted a message to Evaltalk about a story in the NY Times that dealt with the Administration’s plan to cut funding for teenage pregnancy prevention programs: Programs That Fight Teenage Pregnancy Are at Risk of Being Cut. (Evaltalk is an American Evaluation Association’s listserv.) That message led to an interesting back and forth, which in turn led me to a response. I’m reproducing it here.

What I took from this has very little to do with teen pregnancy or the transferability of program effectiveness across contexts. For me it was a bit of a values crisis. I strongly oppose the Administration’s policy on these programs. But I also strongly favor Continue reading “Evaluation use by people opposed to the program”

Are Policy Makers, Program Designers and Managers Doing a Good Job if they Rely Too Much on Evaluation?

We  like to complain about evaluation use.
People in my business (me included) like to lament the lack of attention that people pay to evaluation. If only we did a better job if identifying stakeholders. If only we could do a better job of engaging them. If only we understood their needs better. If only we had a different relationship with them. If only we presented our information in a different way. If only we chose the appropriate type of evaluation for the setting we were working in. If only we fit the multiple definitions of “evaluation use” to our setting. And so on and so forth. I’m in favor of asking these questions. I do it myself and I am convinced that asking them leads to more and better evaluation use.

Lately I have been thinking differently.
I’m using this blog post for two reasons. One is that I want to begin a discussion in the evaluation community that may lead to more and better evaluation use. The second reason is that writing this post is giving me a chance to discern the logic underlying a behavior pattern that I seem to have fallen into. As for the logic, as far as I can tell it has two roots: continuous process improvement, and complexity. Continue reading “Are Policy Makers, Program Designers and Managers Doing a Good Job if they Rely Too Much on Evaluation?”

Some Musings on Evaluation Use in the Current Political Context

This blog is my effort to consolidate and organize some back and forth I have been having about evaluation use. It was spurred by a piece on NPR about the Administration’s position on an after school program. (Trump’s Budget Proposal Threatens Funding For Major After-School Program.) In large measure the piece dealt with whether the program was effective. Arguments abounded about stated and unstated goals, and the messages contained in a variety of evaluations. Needless to say, the political inclinations of different stakeholders had a lot to do with which evaluations were cited. Below are the notions that popped into my head as a result of hearing the piece and talking to others about it.

Selective Use of Data
Different stakeholders glommed onto different evaluations to make their arguments. Continue reading “Some Musings on Evaluation Use in the Current Political Context”

Invitation to a Conversation Between Program Funders and Program Evaluators: Complex Behavior in Program Design and Evaluation

Effective programs and useful evaluations require much more appreciation of complex behavior than is currently the case. This state of affairs must change. Evaluation methodology is not the critical inhibitor of that change. Program design is. Our purpose is to begin a dialog between program funders and evaluators to address this problem.

Current Practice: Common Sense Approach to Program Design and Evaluation
There is sense to successful program design, but that sense is not common sense. And therein lies a problem for program designers, and by extension, for the evaluators that are paid to evaluate the programs envisioned by their customers.

What is common sense?  
“Common sense is a basic ability to perceive, understand, and judge things that are shared by (“common to”) nearly all people and can reasonably be expected of nearly all people without need for debate.”

What is the common sense of program design?
The common sense of program design is usually expressed in one of two forms. One form is a set of columns with familiar labels such as “input”, “throughput”, and “output”. The second is a set of shapes that are connected with 1:1, 1:many, many:1 and many:many relationships. These relationships may be cast in elaborate forms, as for example, a systems dynamics model complete with buffers and feedback loops, or a tangle of participatory impact pathways.

But no matter what the specific form, the elements of these models, and hypothesized relationships among them, are based on our intuitive understandings of “cause and effect”, mechanistic views of how programs work. They also assume that the major operative elements of a program can be identified.

To be sure, program designers are aware that their models are simplifications of reality, that models can never be fully specified, and that uncertainties cannot be fully accounted for. Still, inspection of the program models that are produced makes it clear that almost all the thinking that went into developing those models was predominantly in the cause and effect, mechanistic mode. We think about the situation and say to ourselves: “If this happens, it will make (or has made) that happen.” Because the models are like that, so too are the evaluations.

Our common sense conceptualization of programs is based on deep knowledge about the problems being addressed and the methods available to address those problems. Common sense does not mean ignorance or naiveté. It does, however, mean that common sense logic is at play. There is no shame in approaching problems in this manner. We all do it. We are all human.

Including Complex Behavior in Program Design and Evaluation
When it comes to the very small, the very large, or the very fast, 20th Century science has succeeded in getting us to accept that the world is not common sensical. But we have trouble accepting a non-common sense view of the world at the scale that is experienced by human beings. Specifically, we do not think in terms of the dynamics of complex behavior. Complex behavior has much to say about why change happens, patterns of change, and program theory. We do not routinely consider these behaviors when we design programs and their evaluations.

There is nothing intuitively obvious about complex behavior. Much of it is not very psychologically satisfying. Some of it has uncomfortable implications for people who must commit resources and bear responsibility for those commitments. Still, program designers must appreciate complex behavior if they are ever going to design effective programs and commission meaningful evaluations of those programs.

Pursuing Change
There is already momentum in the field of evaluation to apply complexity. Our critique of that effort is that current discussions of complexity do not tap the richness of what complexity science has discovered, and also, that some of the conversation is an incorrect understanding of complexity. The purpose of this panel is to bring a more thorough, a more research based, understanding of complexity into the conversation.

By “conversation” we mean dialogue between program designers and evaluators with respect to the role that complexity can play in a program’s operations, outcomes, and impacts. This conversation matters because as we said at the outset, the inhibiting factor is recognition that complex behavior may be at play in the workings of programs. Methodology is not the problem. Except for a few exotic situations, the familiar tools of evaluation will more than suffice. The question is what program behavior evaluators have license to consider.

Our goal is to pursue a long-term effort to facilitate the necessary discourse. Our strategy is to generate a series of conferences, informal conversations, and empirical tests that will lead to a critical mass of program funders and evaluators who can bring about a long term change in the rigor with which complexity is applied to program design and evaluation.