It seems to me that the kinds of problems you are describing fit into my discussion of “incorrect assumptions early in the evaluation life cycle” (pages 149 – 154.) The most relevant case is #1, where the sponsor insisted on an inappropriate outcome measure.

Stages and Life Cycles
Personally I’m not knowledgeable about implementation stage theories for the kinds of programs you are dealing with. None of the references I have on page 59 are about social or educational programs, although I’m sure there is a big social science literature on this topic. Actually, I see a weakness in my book that I don’t delve into what the various life cycle stages are, and discuss how they relate to evaluation surprise. I may deal with this lack if Guilford ever sees the wisdom of asking me for a second edition.

I talk about concepts such as “start up phases”, and “stability” using the “I’ll know it when I see it” measurement criterion. Not the best measurement system for a guy in my business. Still, I can think of intuitive criteria that may be useful. For instance: Did they just receive funding and are still getting organized? Are they hiring a lot of staff or have the folks that work there been around for a while? Do they have working relationships with closely related systems? (For instance, if schools are supposed to refer to health care clinics, is the referral mechanism in place.) Have the services provided stayed more or less consistent for some time? Has the content of the services been changing a lot even though there is no obvious change in the operating environment?

All that said, I don’t think your problem is lack of an implementation theory. I’m sure that if you went shopping you would find one. The real problem is whether you have any opportunity to collect the necessary data.

Data
Assuming that you are comfortable with a measure of maturity, how would the various programs stack up? Are they all in the start-up phase, or are there some that are legitimately ready for some kind of outcome measure? It might be worth looking at this because the kinds of problems they have may be different. For instance, a mature program may have a problem because you are boxed into collecting inappropriate outcome measures, while a start-up may mean that the whole idea of measuring outcome makes no sense.

I’d bet that in addition to the funder, there are other stakeholders who have an interest in the evaluation, and that their interests align more with what you know needs to be done than do the funder’s requirements. Can you design the evaluation in such a way as to try to meet those needs? That would legitimize other kinds of data collection. I don’t know what position your project officers would take on this, but it’s hard for me to believe that they would be unsympathetic. I realize that it’s easy to suggest more data collection and hard to pay for it. But it may be possible, particularly if you could justify it as a needed precursor to getting the data the funder is asking you for. Or, maybe you could get some of those other stakeholders to kick a bit of money into the pot. You may not need a lot because you already have the evaluation mechanism up and running.

Maybe another possibility is to get your funders to support you in the collection of other data. After all, your project officers must know that they are putting you in a tough position. They are probably trapped themselves in a web or regulations and policy that is making them pass those evaluation requirements on to you. Especially if you could bring other stakeholder to the table asking for other types of data collection, you might get support for getting that data.

Comparison Groups and Other Hard to Implement Designs
I don’t know what you mean by “hard to implement” but I am sympathetic to the difficulty of implementing comparison group studies. (I am also a big fan of comparison groups. Having some in an evaluation I am doing is saving us because there are some strong confounds in the experimental group.) Still, there are designs you may be able to use. What about a no-control group interrupted time series? Or even if you can’t go back far in time, a simple pre-post comparison with valid and reliable data can go a long way to making some reasonable causal inference.

Is there any chance that you are measuring outcomes for which there is national data to compare against? (For instance, I’m thinking of the standardized tests that are given in all the states across the globe.) I realize that you funders may not like these designs, but they can be worthwhile. How much wiggle room do you have in negotiating the design?

Is the Evidence Based Practice Any Good?
I hate to bring this up, but it may be worth pondering. How much faith do people have that the “evidence based practice” is based on enough evidence to believe the program should have the desired outcomes? If nothing else, looking at this question may help set expectations. In the best of all worlds though, the programs you are evaluating would add to the knowledge of evidence based practice, and thus have a longer term impact on the kind of programming your funders are pushing.

[...] Every once in a while we choose blogs that we read. Listed below are the latest sites that we choose [...]…