In 2016 I developed a case for a workshop I did at the Canadian Evaluation Society on the use of complexity in evaluation. I was doing some archeology and unearthed it. It offers pretty rich opportunity to think about complex behavior in evaluation, so I decided to share it.
Construction of the Case
This is the example we will use throughout this workshop to illustrate how knowledge of system behavior can be applied in evaluation. The example is hypothetical. I made it up to resemble a plausible evaluation scenario that we may face, but which is elaborated to make sure it contains all the elements needed to explain the topics in the workshop. I am sure that none of us (me included) have ever been involved in an evaluation that is as far reaching and in-depth as the example here. But I am sure that all of us have been involved in evaluations that are similar to parts of the example, and, if you are like me, I bet you have dreamed of being involved in an evaluation of the size and scope of the example.
There are three initiatives. One aimed at adults. One aimed at mothers and young children. One aimed at teens. Each initiative has several individual programs that share some common outcomes, and which also have some unique outcomes.
All three initiatives are deliberately implemented at about the same time in the same community because there is an interest in community level change that may not be explainable only in terms of the success of any of the specific initiative or program.
Each initiative, and each program nested within the initiative, receives its own evaluation. In addition to assessing outcomes, each program needs a process evaluation to understand how it was implemented, how well (or poorly) it operated, and how (or whether) it changed over time.
Each evaluation begins at, or very near too, the start of each program’s implementation. The programs all start at about the same time. Each program is funded for three years, with additional funding added for another two years of evaluation. Each program is funded by a different government agencies that have no formal connections until several rungs up the organizational ladder. On the other hand, the funders in each agency know each other and are interested in cross-program and community level change.
Each initiative has its own evaluation team. Each evaluation is funded through a separate RFP process, each of which was won be a different organization. There is no common membership or formal communication among the teams. Some of the evaluators know members of the other teams by reputation and by sight, but none of them have worked together before. Each project does have some budget to allow for cross-program and community level analysis.
Initiative 1: Healthy Eating
This initiative is designed for adults. It has two components. 1) Work with civic organizations, and 2) minimizing food desert coverage.
Civic Organizations – the Program
The program works with civic organizations (churches, service clubs, etc.) in two ways. First, it provides a proven “best practice” curriculum for education and training about healthy eating, along with training on the content of the material and effective ways to deliver the classes.
Second, the program provides technical assistance to help each service organization set up any program it wants to promote healthy eating in its community. There are no restrictions as to what the program can be as long as it is “on topic”, and can be produced and delivered with whatever resources (internal or externally supplied), that the organization can muster.
Civic Organizations – the Evaluation
Participants in the classes are interviewed periodically throughout the evaluation period. Topics include eating and exercise behavior. They are also asked to fill out questionnaires evaluating the classes.
Leaders of the civic organizations are interviewed to determine whether their organizations set up any new programs related to healthy eating. If they have, details are collected with respect to the process through which those programs came into being, and the nature of those programs. Performance data is also collected – what the programs did, and any available data the civic organizations collected on their effectiveness, e.g. how many people were reached, and whether their eating behavior changed. The organization leaders are also queried about their ratings of the technical assistance, and what role it played.
Reducing Food Desert Coverage – the Program
This is an effort aimed at reducing food desert area and intensity in the community. The program consists of a small group of people who have good connections to the formal government in the city and state, and who are equally well known to civic society and informal networks of influential people. This small group of people is pursuing several tactics simultaneously. 1) They are trying to establish farmers markets. 2) They are working with the owners of small stores (e.g. convenience stores) on how to market fresh food. 3) They are attempting to coordinate purchasing of healthy foods among many stores in the community in order to rationalize the supply chain. 4) They implement a social marketing campaign to increase the demand-side pull for stores to stock healthy food.
Reducing Food Desert Coverage – the Evaluation
All the activities of the program staff are recorded, e.g. discussions with commercial produce suppliers, marketing materials and workshops, discussions with farmers organizations and city officials about setting up farmers markets, work with civic groups to build a coalition for devoting city property to a farmers market, etc. Interviews are conducted with the stakeholders that the program staff interacts with. For instance, if the staff try to get small stores and their commercial suppliers to do bulk wholesale purchasing, then interviews will take place with the store owners and the commercial suppliers. Questions will deal with how well the program staff did their facilitation work, whether the discussions were channeled to the substantive technical and business issues involved, whether the staff made a contribution to dealing with these issues, and so on. Data will also be collected on how far each of these initiatives got, and why they developed to the stage that they did. Outcome data is collected on how many stores sell healthy foods and the prices charged for those foods. Finally any data that is available on similar activities in other communities will be identified and used. It is expected that some relevant data will be available, but not at a level of granularity that will allow activity-to-activity comparisons among the communities.
Initiative 2: Early Childhood
This initiative is designed to assist new parents and to affect their children from birth to entry into first grade. It has three components: 1) classes for mothers, 2) classes for fathers, and 3) 1:1 coaching for new mothers.
Classes – the Program
Separate classes are held for mothers and fathers. The classes deal with infant and child behavior, and child rearing techniques. They are provided at local health centers and many of the same civic organizations who are providing the “health eating” services. The classes are offered to anyone who wants to sign up. The curriculum is adapted from well proven best practices for educational activities of this type. The classes meet once per week for a three month period. They are delivered on a rolling basis so that at any one time during the three year funding period, any participating organization may be running several series of classes.. People who wish to attend more than one series are allowed to do so. Thus someone may take a three month series of classes, and come back eight months later to do it again.
Classes – the Evaluation
Questionnaires are used in the middle and end of the class series to evaluate the material, instructors, etc. The usual stuff. Also, participants are interviewed about “critical incidents”, i.e. circumstances with their children where they felt that participating in the classes made a real difference in how they behaved. In cases where both mothers and fathers are taking classes, some questions are added about whether the joint attendance made a difference. These interviews begin at about the middle of the three months, and extend for a year after the last class was taken. To check on the effect of the data collection on the findings, a special interview schedule is implemented. Half the women are interviewed every three months for one year. The other half are only interviewed twice, at six months and twelve months. Important outcomes include the number of people reached by the classes and whether the classes affected how the parents interacted with their children.
1:1 Coaching – the Program
This program consists of in-home visits by a social worker who is an expert in parent-child interaction. Telephone support is also provided for parents who need help or information between visits. While the social worker’s primary job is to help with parent-child interaction issues, the social worker is also encouraged to provide referrals as may be needed for other types of services.
1:1 Coaching – the Evaluation
Participation is randomly assigned to half the mothers who apply. The control group is given only the telephone support. All participants (experimental and control groups) are free to join any of the parenting classes if they so desire. The telephone and coaching group fill out surveys upon entry into the program, and then at 6 12, 18, 24, 48, and 60 months post enrollment.
Minimizing Teen Consumption of Alcohol
The focus here is on reducing alcohol consumption. Two programs are involved. The first involves developing peer-to-peer support programs in schools, along the lines of what is done by SADD. The second program is focused on enforcement of laws against supplying alcohol to underage drinkers.
Peer to Peer – the Program
This program provides materials and examples of ways in which programs like this have been successfully established in other schools. In addition to the materials, it provides an expert who can work with whatever nascent collection of teachers, administrators, and students can be found within the school, who might come together to implement the program.
Peer to Peer – the Evaluation
Evaluation of this component involves records of participation in the groups with respect to frequency of meetings, programming content, numbers of participants, and observation and content analysis of any public messages the group puts out. A social networking analysis is conducted to determine the relationships between the active members of the group and the rest of the students in the school. Surveys are conducted of the students in the school about drinking behavior. These surveys are done annually during the full five year period of the evaluation.
Enforcement – the Program
This program involves a public service campaign reminding people of the danger and illegality of providing alcohol to underage drinkers. The campaign appeals to people’s good judgement, and also to their fear of getting caught due to increased enforcement.
The second part of the program is aimed at stores that sell alcohol. One part of the program is education, as for instance, helping clerks to recognize false ID cards. A second part is high-publicity increased enforcement actions. Finally there is a social sanction component that involves publicizing the names of stores who sell alcohol illegally.
Enforcement – the Evaluation
Members of the community are randomly surveyed to assess the social marketing efforts. 911 and police records are used to determine any police involvement in underage drinking incidents. Records are scrutinized with respect to enforcement efforts related to the illegal sale of alcohol. As a control, data are compiled on other similar communities around the country.
Common Evaluation Elements Across all Initiatives
All the evaluation discussed above also has a monitoring function to detect any related programs that may arise, independent of the programs being evaluated. If such programs are detected, interviews are conducted to ascertain whether the program being evaluated helped motivate the formation of the other programs, and whether there is any formal or informal coordination or competition among them.
Community level change
There is suspicion that because of all the intense activity described above, that there may also be change at the community level that could not be explained only in terms of the specific outcomes of each of the programs. As a result, each evaluation has a community monitoring function to observe changes such as;
- School attendance and school performance
- Use of social and health services for any reason, even not related to the programs
- Number of civic improvement programs started by the population on any topic for any reason
- Other similar activities.
None of the evaluations has the resources to study these changes in depth, but they do have the resources to track the events over the life span of the evaluation, and to do some interviews that may reveal connections between each of the programs themselves, and between those programs and other relevant activity in the community. There are no resources to directly collect data from other communities, but there are enough resources to identify trends from published data.
Evaluation Uncertainty 
I remember this.
>