How to evaluate complex health interventions?

I just came back from Toronto, where I was visiting Sanjeev Sridharan, who runs a most interesting organization called the Evaluation Center for Complex Health Interventions. I was there to demonstrate the applicability of agent-based modeling as an evaluation tool. During the talk nobody asked me a question I could not easily answer. But then came lunch the next day, when Sanjeev casually asked “How would you evaluate a complex health intervention”? Hmmmm. On the train home a few days later I worked out an outline of an answer. I figured I’d lay it out there for the world to have a whack at. See below for my ruminations. Whack away.

Because I have been thinking so much about agent based systems lately I came up with the notion that concepts drawn from that domain could be useful for constructing evaluation strategies. I’m not arguing (right now) for doing formal agent based modeling. But I do think that the idea of agent behavior could be a useful trick of the mind to help design evaluations.

Some brief background on agents
What is an “agent”? It is an entity that can 1) sense its environment, and 2) act on what it senses based on a set of rules. Define the agents. Specify the rules. Let the agents interact. Complex behavior emerges. (Or at least it can emerge. Whether it does or not depends on the nature of the interactions.) Anything can be an “agent” – people, wetlands, hospitals, school rooms, county governments, whatever. All they have to be are things that act as if they perceive their environment and respond based on specific rules. The rules do not have be absolute. For instance a rule can be: Look at what the agent next to you does. If that agent does X, you do Y with a probability of 90% the next time you act.” Agents can be nested, e.g. wetlands within fields, or platoons within companies, or kids within classrooms. Wetlands, fields, platoons, companies, kids and classrooms can all be considered agents, each with their own sensing mechanism and decision rules.

Agents live within environments. For instance in a mental health setting “agents” may be therapists who are being asked to adapt a new best practice. Elements of the environment may be the amount of improvement the new therapy offers over the old one, the strength of organizational support for use, and the number of colleagues using the new therapy.

Thinking about agents to design evaluation of complex health interventions
So, how does all this help developing an evaluation of a complex health intervention? My notion is to look at the intervention and to ask: Who are the agents? What is the environment? Of course there is no formulaic answer to this question, and in a sense it is a hopeless question because on some level everything is connected to everything else. Still, not everything is equally connected to everything else, and some things do matter more than others. Based on experience and research, we can probably do a reasonably good job of making practical decisions.

What are some of the decision criteria for choosing? I see four. 1) Length of feedback loops relative to the time-frame of observation, 2) degree of stability, 3) directness of the relationship, and 4) reason to believe the relationship makes an important difference. To continue with my “therapist in a clinic” example, should the evaluation include the therapists personal friends who are not themselves therapists, but who may have an opinion about the new treatment? Should those friends be treated as agents who themselves act within the system? Or maybe their collective opinion might be an environmental condition. Is any of this worth the trouble? If the new treatment touches on sensitive cultural matters that friends may have an opinion on, they sure, including them might make sense. But otherwise, best to leave well enough alone. Or try another scenario. Should the evaluation include reimbursement policies concerning how much of the new treatment will be covered by insurance? If there is a strong relationship, and a short latency period between changing success rates and reimbursement policies, then sure, it would make sense to add them to the evaluation mix. Otherwise, maybe best to leave well enough alone and invest the evaluation resources in more interviewing of therapists and clients, and more rigorous efforts at doing long term follow-up.

Once these decisions are made there is no need to do an evaluation that is a formal complex adaptive system exercise. The methods of data gathering and analysis can be traditional, common, and familiar. There is no obligation to worry about emergent behavior, phase state changes, extent of dependence on initial conditions, or any of the other intellectual tools of the CAS world. (Although I personally would like to see people think this way.)

So why invoke the concept of agents? Because thinking in terms of agents and their environments is useful ploy. 1) Thinking this way offers a nice framework for making decisions about what is important. It’s a simple framework to organize a lot of knowledge. 2) It is an intellectual lens that can offer perspectives on data interpretation that might not otherwise reveal themselves.