Eat dessert first. Life is uncertain.
Good advice. True observation.
Any yet, we must soldier on, trying to predict what will happen and to explain why things work out as they do. My professional life has grappled with prediction and explanation through the lens of program evaluation. I have come to appreciate how much, and how little, evaluation can tell us. Sometimes it is “much”. Sometime is “little”. Sometimes it is “much” and “little” at the same time. This blog expresses my engagement with what evaluation can do, what it cannot do, what it should do, and what it should not do. About the graphic.
Contributions to this blog are most welcome. If you are interested, please contact me at jamorell@jamorell.com.
Join 1,623 other subscribers
Evaluation Uncertainty 
I signed up for this seminar because I work full time as an “embedded” program evaluator on a very large NIH grant to the UWisconsin School of Medicine and Public Health. We use logic models to summarize the implementation intentions of the Institute and its 6 cores/30 components because they help to clarify objectives, prod people to think about metrics [which we are hounded about by NIH], and because they tend to spark intense discussions among various sets of key stakeholders. At the same time, I’m reading MQP’s “Developmental Evaluation” book, and “Utilization-Focused Evaluation” which is what we consider ourselves to be doing. What we’re evaluating is enormously complex with fuzzy definitions, lack of agreement among stakeholders about priority outcomes, intense politics, etc etc. What we are tracking & evaluating is not necessarily linear at all, but logic models are my only tool for summarizing confusing complexity. Putting things in matrices exposes huge gaps which is good. I’m looking at your ppt slides which seem very text-dense, but I will continue to study them and provide some more feedback.
“COMPLEXITY”
1) I hate the term “complex” because for me it has both a colloquial meaning (lots of parts, hard to understand, etc.) and a technical meaning having to do with showing particular characteristics — emergence, sensitive dependence, phase shifts, and so on. Many systems that are complex in the colloquial sense are not complex in the technical sense. For that matter it works the other way around as well. Many systems that seem quite simple and easy to understand will exhibit the behavior of a complex system. That said, I use the terms in both their meanings all the time, and so does everybody else I know. If I had my way people would pick one and agree to call it “poodle” to avoid confusion, but I know that will never happen. So, we just have to be careful about knowing what we mean.
STAKEHOLDER ISSUES
2) As for the fuzziness, lack of definitions and the like. You are right that logic models are good for helping to resolve ambiguity, reach agreement, and the like. It does not matter that the system being evaluated may change a lot. Do a thought experiment. Take the programs you are dealing with. Pick it at any single point in time and do the exercise of agreeing on definitions, basic relationships, and the like. Then do it again, and a few times more. I’d bet that there would be al lot of overlap in the issues dealt with. So, even if the logic morphs, the exercise of developing the model can be very useful.
3) Assuming the model does change over time, be sure to keep all the old models. The shift itself is good data for understanding the program.
4) As for the changes over time, take a look at slides 29 – 31. There are different patterns of logic model shifting. Sometimes it helps to know which pattern one is dealing with.
5) With respect to all the stakeholders and lack of agreement, take a look at Part 5 of the slide 84. There is lots of information on picking tactics to deal with various degrees of disagrement among stakeholders.
TEXT DENSE SLIDES
6) There is an important lesson here, which I’m not sure I got right, but which all of us should attend to. I have a personal belief that one is better off putting more, rather than less information on a slide, consistent with maintaining readability, good visual layout, etc. It’s a bias of mine because I always hated lots of slides with just a few little bits of information. Also, more information on a single slide allows people to see relationships. On the other hand, it’s easy to put too much information on a slide and confuse people. It’s not at all unlikely that I got the balance wrong. All of you should weigh in on this. I’d like opinions.
BTW, we use the logic model format from UW Extension’s website: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
I know there are many formats and styles. We added a “metrics page” to the basic format, so that there can be specific metrics attached to each of the elements in the face page of the logic model. Other CTSAs (recipients of Clinical and Translational Science Awards — there are 55 academic research centers with these grants) use different formats with different degrees of wordiness.
I like the idea of many formats. In fact one of the main messages in Section 3 is that multiple forms and styles are desirable.
I also like the idea of attaching metrics. It touches on the notion that there are relationships between models, metrics, and methodology. See slides 32, 33, and 34.
One thing I like to be careful about is levels of detail. Sometimes I populate my models not with specific metrics, but with the constructs that metrics will estimate. (This works for both quantitative and qualitative data.) Then, I index the model to a list of metrics, or develop another version of the model with the additional detail.
The “different formats and different degrees of wordiness” can be a problem for other than aesthetic reasons. Models are useful for reaching common agreement, and differences in models across sites can obscure both agreements and disagreements. If it’s possible to get agreement on a single model at a single level of detail, so much the better. This does not have to mean that individual sites must give up on their cherished models. It just means that it’s useful to have one version of a model that everybody can agree on. Usually, a model like this has to be at an intermediate level of detail. If it’s too high, its useless because it becomes a apple pie and motherhood view. If it’s too detailed, one can never get agreement because in some ways different sites will be different.
Finally, I don’t know your specific situation, but there is always the possibility that there should not be a common model because in fact, there are profound differences in theories of change, important metrics, relationships, program goals, and so on.
On tactics for working with stakeholders, what you have in the slide is PRECISELY how we’ve tried to do it here. We say “This is what we think/observe you want to do and why, and how we think you’re going to know you did it. Did we get it right?”
I’m especially interested in [slide 69] using logic models in other ways, such as to “organize multiple sources of information.”
Many of the other evaluators in other CTSAs are using logic models — there are as many different styles as there are CTSAs and evaluators. It seems as if many people either don’t really know how to use LMs or they mix up outputs/outcomes and/or can’t formulate metrics in ways that really make sense. Sometimes.
As for “formulating metrics”, I have sometimes run into a problem where the surface argument obscures the real argument. A “low level” discussion of whether “A” is a good indicator of “X” may be a stand in for a deeper disagreement about whether “X” should be included in the evaluation. In general arguments about detail are really arguments about principle. I have been led down the garden path more than once.
I am an administrator in higher education. We are about to launch a new campus-wide program and I see the logic model as a way to communicate the program structure and intended outcomes and impacts to various stakeholders. I hope the workshop will help me wrap my head around use of a model for planning, formative evaluation, and summative evaluation. Additionally, I will need to explain the model to the rest of a development team with no evaluation experience (other than me, the committee consists of only faculty–with only one social scientist, I should add). Looking at the presentation, slides #32 and 33 should speak to the team, concrete examples showing feedback loops. I want to learn more, but the more concrete examples, the better.
WHY HAVE A LOGIC MODEL?
Your post says: “I hope the workshop will help me wrap my head around use of a model for planning, formative evaluation, and summative evaluation.” This statement gets at a very important issue. Take a look at slide 78, which identifies three different uses for logic models: evaluation, planning, and advocacy. I think that all of these uses are legitimate. On the other hand, evaluators can get into trouble (and not serve their customer well) if they don’t realize which mix of uses logic models are being put to. I know this because I have been zapped by the problem many times.
I am the Associate Dean, Academic, the Faculty of Pharmaceutical Sciences at the University of British Columbia. As such, I am responsible for ongoing program evaluation and continuous quality improvement related to our curriculum. I am hoping to get a better understanding of how Logic Modelling can be applied to plan, implement and evaluate curriculum.
Remind me during the workshop to talk more about using logic models for planning and implementing, as these are (or at least can be) quite different from using logic models for evaluation. Right now I touch on this on slides 26 and 78, but that’s not a lot. Bring up the topic and we will discuss it.
I will be attending our workshop in San Antonio and am very interested in system approaches and modeling and how logic models can be used in complex initiatives. I use logic models extensively and am hoping this workshop will be more than an intro to logic models but will take us beyond their traditional uses and explore how they can be used in novel and complex ways.
We will certainly go beyond garden variety use, that I promise you. As for “complex”, see my reply to Jan’s first post.
One observation I have made is that people often think that if only logic models could be used to portray a complex system, then all will be well. I have a sense that while this may be true, it is also a way to avoid confronting the real issue. Forget about logic models. The real question is this: IF a system is truly complex, then how should evaluators deal with doing the evaluation? This gets to hairy questions of using models (logic, conceptual, mathematical, or otherwise) to explain and predict. I could talk about this forever and won’t hold back in the workshop unless people muzzle me and make me attend to the other material.
I don’t have a lot of slides on this topic because it would bring the discussion way to far afield, into the topic of my book on unintended consequences, my article on evaluation and agent based modeling, and a lot of writing by other people. (For instance, see Michael Patton’s latest book Developmental Evaluation). But this is a big issue because of the increasing interest in systems approaches to evaluation. If people are interested, I’ll bag some of the prepared material and talk about this topic. It fits best with slides 29 – 31, but those slides do not come close to doing the subject justice.
I’m reading MQP’s Developmental Evaluation now, and continue to think about how complex this Institute is, that I work for. I suspect that some aspects of it are complex, some are just complicated and maybe some are even simple — in terms of the best ways to evaluate what the entity is doing. And obviously it’s critical to think about the purpose of the evaluation! I don’t think we are trying to eventually take anything to scale. We’re trying to reinvent how clinical and translational research is conducted in this institution, improve the processes, transform things… all very lofty. And I’m participating in MQP’s Mon & Tues workshop, so will arrive at your Wed event full of thoughts on developmental evaluation.
Section 3 will be most relevant for you. Check it out before the workshop if you can and see what parts of it appeal to you most.
Hi Jonny,
I am looking forward to your workshop at the AEA Conference in San Antonio. Spending a day on logic models beyond the basics suits me just fine. I am interested in garnering different approaches to explain their utility to others who don’t typically use them (e.g., evaluation students or program managers). For example, it’ll be helpful to hear from you and other workshop participants how you explain: 1) logic models compared to predictive models; 2) limitations of logic models; 3) processes for determining what, of all that may be possible to depict in a logic model, actually gets measured (the model to evaluation plan connection); and 4) selecting a particular depiction/layout of a logic model so that it maximally piques readers’ (program stakeholder) interest to further examine a program.
Your slide set is quite interesting and focused time on it beyond simply reading from the slides will be helpful to me. Hence, my interest in a workshop format where participants will attempt to apply content presented to them by the workshop facilitator. Thanks!
PARTICIPATION BY ATTENDEES
I hope everyone will agree to talk about application from their unique points of view. We will all be bored if I just flap my gums all day. Worse, if I do most of the talking, all of you will get the material from my narrow point of view. As much as I agree with myself, and think everyone else should too, I know full well that narrow points of view do not make for deep understanding.
Thanks for your response. I see logic models as one of the tools we can use when evaluating in complexity. I don’t expect any one tool will provide all that is required and I tend to use them more for their heuristic value, rather than as prediction and control. I appreciate your thoughtful responses to all the posts and I am looking forward to continuing these discussions in person. I do hope others are also interested in taking the discussion in this direction.
There are a few issues that are relevant here.
1- There is the whole issue of evaluating a complex system. That is not a logic model issue but a methodological one. And of course, I see logic models as relating to both methodology and measurement.
2- Still, logic models do play a role, but a somewhat different one from the way we usually use them. We can talk about this as well during the workshop. But remind me to do so when we get to the right parts.
3- With respect to methodology, the big question is how to have powerful methodology when programs are subject to change. That’s what the “agile evaluation” sections of my book are all about.
4- Also with respect to methodology, there is the question of what it even means when a system is truly complex. The problem is that we have to evaluate both the program in terms of its usual components and outcomes, and the program as a complex system. Doing the latter can involve methods quite different from doing the former.
5- The issue I raised in #4 is not that big a deal if we only want to apply the dynamics of complex systems in heuristic terms. But if we want to get formal about it, well that is a horse of an entirely different feather.
6- Don’t forget that we can think of evaluation as having four different roles — description, explanation, causation, and prediction. (Causation and prediction are not the same. Don’t forget that Ptolemy used a totally incorrect theory to make very accurate predictions. And — so I have been told — the guy who invented the hot air balloon thought that fire gave off hydrogen, which being lighter than air, would cause the balloon to float.) Applying CAS notions is different (and a lot easier) for some of these functions of evaluation than for others.
7- If a system is truly complex, then the whole nature of program theory changes, which obviously has an affect on logic models.
8- And now I have to go off on two rants.
Rant 1: People think of complex systems as being unstable, the butterfly effect and all that. NOT always true. Complex systems can also be very, very stable.
Rant 2: People invoke the notion of “complexity” when there is no need to do so. Take a look at slide # 30. I have been in more than one situation where this scenario was explained in terms of “emergence”, which is a core concept in CAS. But there is no reason whatsoever to draw on CAS concepts to explain this situation. Garden variety program theory will work just fine.
Hi Jonny,
First, I do apologize for my late response. I am a researcher in social sciences applied to educational systems (since 1980’s) and I had worked on program evaluation since 2001.
I am mainly interested in how to solve problems related to applying logic models to programs which implies complex (although I already know you don’t like the word) intergovernmental relations, particularly at the partial decentralization process of the Mexican educational system.
I am also responsible for the teaching programs in evaluation of public policies at a MA and PhD levels, and I want to learn more on different perspectives in approaching the communication of logic models both to students, future evaluators, and to the people responsible for public programs.
I took a look to the slides you propose and I look forward to look what you decide for the workshop. For sure, you should select what works better for you.
(English is not my first language, so please excuse my mistakes in the language).
Thanks for the materials you sent-
It’s not that I don’t like the word “complexity”. I love it. Nobody could love it more than I. It is because I love it that I get bent out of shape by its dual use. It’s not that I care so much about the precision of language, which I do. It’s that by using the term loosely, we are drawn into applying the formal notions of complexity in incorrect ways.
As for decentralization, take a look at slide 30. I’ll explain it more during the workshop. I designed it specifically to portray a decentralized scenario. In the meantime, think of it this way. To decentralize a system is to invoke a program theory about topics such as centralized control, response time to changing environments, the role of contextual factors, and a host of other notions. Logic models can be developed to capture all of those concepts. Remind me when we get to this slide, and I’ll talk a lot more about it.
As for your teaching, we should have a side conversation about it. It’s not within the scope of the workshop, but I have some strong feelings about how much evaluation (and what aspects of evaluation) managers and policy makers should have. Also, without too much effort, I could be pushed into the position that such people do not need to know any evaluation at all. What they need is respect for data. But as I said, this is a side conversation.
As for your fluency in English. I would give anything to be half as fluent in any other language as you are in English.