How can public leaders and program managers track the performance of different sites within a program in a way that reflects impact — i.e., the value added of each site? The most rigorous approach is to run a rigorous program evaluation, such as a randomized controlled trial, by site, but that type of evaluation is not always feasible. Another approach (the most common one) is to use performance measures, since they are low-cost and easy to implement, but there’s a downside: They aren’t necessarily good indicators of impact. That’s because the performance of sites are effected by local conditions and demographics, not just program quality.
Our interview focuses on public leaders can design performance measures to better reflect impact, including through the use of regression-adjusted performance measures — that is, measures that statistically control for factors that aren’t related to program quality. We get insights from Peter Schochet. He’s a Senior Fellow at Mathematica Policy Research, a leading authority on evaluation methodology, and the co-author of an article in the Journal of Policy Analysis and Management on this topic.
In the interview, he describes how regression-adjusted performance measures can be useful, especially if there is good baseline data, measures are not overly complex and align well with the most relevant outcomes of interest, and (when relevant) if there is longer-term follow-up data.