Four fundamental principles of evidence-based policy and practice, drawing from U.S. and European experience: An interview with Howard White, Executive Director, Campbell Collaboration – Episode #150

What principles can help guide public leaders—whether policymakers or public managers—in their use of evidence-based policy to improve results? Howard White (@HowardNWhite) of the Campbell Collaboration joins us to share four fundamental principles:

  • 1. Use the right evidence to answer the right question. Different types of evidence — e.g., monitoring, process evaluation, impact evaluation and systematic reviews — all can produce useful information for decision makers. But each type of evidence should not be used to answer questions that are beyond its usefulness.
  • 2. Don’t rely on single studies. When possible, leaders should avoid making important funding decisions based on single studies, especially those done in one site. That’s because the findings from one study are often different from those of further studies. The best approach is to use systematic reviews (where they exist), meaning syntheses of multiple high-quality studies.
  • 3. Context matters for transferring evidence. Why do findings from one study often not replicate in another? A key reason is that context matters. For example, when a home visiting program found to be effective in the U.S. was tested in Britain, it produced no impact. Why? Likely it was the different context: Britain already provides services to low-income parents that are quite similar to the home visiting program in the U.S. It is why leaders should test out, with rigorous evaluation, programs and initiatives in their own setting, particularly if previous research was conducted in a different context.
  • 4. Evidence-based policy is not a blueprint (aka cookie cutter) approach. This is a way of summarizing the previous two principles. While architects can take the blueprints for one building and build the same building elsewhere, and chefs can take a recipe from one restaurant and cook it in another, public leaders need to be careful when applying research from one place or setting to another.

Dr. White is the Chief Executive Officer of the Campbell Collaboration, a nonprofit best known for its use of systematic reviews to help policymakers and others make well-informed decisions. Previously he was the founding Executive Director of the International Initiative for Impact Evaluation (3ie) and led the impact evaluation program at the World Bank’s Independent Evaluation Group.

Photo credit: European Union

The opportunities and pitfalls of government reorganization: An interview with Bob Behn, Professor, Harvard Kennedy School – Episode #149

Last month, President Trump issued an executive order calling for agencies to analyze their structures and programs in preparation for a major reorganization. The Trump administration’s focus on reorganization raises the question of when is reorganization useful and what pitfalls need be avoided.

To get insights, we’re joined by Bob Behn. Although it is too early to comment on the Trump administration’s reorganization in particular, since it is still being formulated, he draws on lessons from other reorg efforts to provide advice to public leaders.

Bob Behn is one of the nation’s leading experts on leadership and performance management in government. He is a lecturer at the Harvard Kennedy School and the faculty chair of its executive program called Driving Government Performance. He also publishes a free monthly newsletter called Bob Behn’s Public Leadership Report.

An overview of Performance Partnership Pilots for Disconnected Youth, with lessons for other areas of social policy: An interview with Thaddeus Ferber, Vice President, Forum for Youth Investment – Episode #148

One of the most innovative new approaches in Federal policy is known as Performance Partnership Pilots, also called P3. In 2014, Congress authorized several Federal agencies, including Education, Labor and HHS, to enter into up to ten Performance Partnership agreements per year with states, regions, localities, or tribal communities. These agreements allow these jurisdictions to have additional flexibility in using discretionary funds across multiple Federal programs, in exchange for a commitment to achieve significant improvements for disconnected youth, meaning youth that are not in school and not working. The P3 concept, however, could in theory be applied to many other social policy areas as well.

To learn more, we are joined by Thaddeus Ferber, a Vice President at the Forum for Youth Investment, an organization that advocated for and helped bring about the P3 authorization.

How the Rhode Island Innovative Policy Lab (RIIPL) works: An interview with Justine Hastings, Director, RIIPL – Episode #147

In 2015, a unique collaboration was launched call the Rhode Island Innovative Policy Lab (RIIPL). It is a partnership between researchers at Brown University and the Office of the Governor of Rhode Island, with the goal of helping state agencies design evidence-based policies to better serve Rhode Island families.

RIIPL’s goal is to use data and science to improve policy, alleviate poverty and increase equity of opportunity. To do that work, it has created a new linked database of public programs, connecting more than 100 previously independent data sets.

To learn more and hear about some example projects, we are joined my Justine Hastings (@JHastings_Econ). She is an economist at Brown University and the founding director of RIIPL.

The importance of replication and validation in evidence-based policy: An interview with Tammy Chang, U.S. Treasury Department, and Nathaniel Higgins, formerly U.S. Social and Behavioral Sciences Team – Episode #146

In this podcast episode, we explore two real-life stories from the front lines of government performance improvement efforts that highlight the importance of replication and validation in evidence-based policy.

In the U.K., the department of revenue and customs, in conjunction with the U.K. Behavioral Insights Team (BIT), tested new versions of the letter sent to people who were late in paying their taxes. The new versions simply added ones additional sentence, a behavioral “nudge” that drew on the power of social norms. As the New York Times explained:

One nudge was a sentence telling recipients that a majority of people in their community had already paid their taxes. Another said that most people who owe a similar amount of tax had paid. Both messages bolstered tax collection, and combining them had an even stronger effect. Over the last financial year, the letters brought forward £210 million of revenue, Britain’s revenue and customs department says — money that otherwise would have had to be chased in costly court procedures and failed to earn interest for the government.

A few years later, the U.S. Treasury Department, in conjunction with the U.S. Social and Behavioral Sciences Team (SBST) — the Obama Administration’s version of the BIT — decided to test a similar approach. They updated the letters sent to people who owed non-tax debt to the Federal government, simplifying and personalizing the letters and adding similar behavioral nudges as in the U.K.. The results showed no effect on payment rates, underscoring how context matters.

To learn more, we are joined by Tammy Chang, a Senior Economist at U.S. Department of the Treasury within the Bureau of the Fiscal Service, and Nathaniel Higgins, a co-founder and Fellow on the former SBST.

How Massachusetts provides education policymakers with research insights: An interview with Carrie Conaway, Chief Strategy and Research Officer, Massachusetts Department of Elementary and Secondary Education – Episode #145

Massachusetts is known as a leader in providing education policymakers with research findings that they can use to improve policy and practices. The state’s Office of Planning and Research, within the Department of Elementary and Secondary Education (ESE), takes a multifaceted approach:

  • Design a proactive research agenda, also known as a learning agenda
  • Increase internal capacity for research
  • Create partnerships with external researchers

To learn more, we are joined by Carrie Conaway (@clconaway), the Chief Strategy and Research Officer of the ESE. She leads the 15 person Office of Planning and Research.

How Seattle used results-driven contracting to improve homeless services: An interview with Jason Johnson, Deputy Director, Human Services Department, City of Seattle – Episode #144

Because many of the most important functions of state and local governments involve contracting for goods and services supplied by the private sector, improving procurement processes is an important way to strengthen outcomes for citizens. That is the motivation behind our series on results-driven contracting.

Our focus today is Seattle, Washington, whose Human Services Department worked with the Government Performance Lab at the Harvard Kennedy School to improve outcomes for its homeless population. The changes included:

  • Consolidating contracts and allowing for more flexibility
  • Establishing goals for homeless service providers and tracking progress
  • Using active contract management

To learn more, we are joined by is Jason Johnson, the Deputy Director of Human Services Department in Seattle.

How states can optimize their pre-K programs: An interview with Greg Duncan, Professor, UC Irvine, and Member, Pre-Kindergarten Task Force – Episode #143

Forty-two states and the District of Columbia spent $6.2 billion in state funds on pre-kindergarten programs in 2015, highlighting the emphasis that policymakers are placing on pre-k to help students prepare for elementary school. Research has shown both the success of pre-K as well as inconclusive evidence about the sustainability of those gains as children become older. Those findings raise the question: How can states optimize their pre-K programs to provide both the strongest early learning boost and a solid foundation for future learning?

Recently, a group of leading pre-K researchers set out to find consensus about what we know about pre-K education. In April 2017 the Pre-Kindergarten Task Force presented their findings. To learn more, we are are joined by a member of the task force, Greg Duncan. He is a Distinguished Professor in the School of Education at the University of California, Irvine.

In the interview, Duncan summarizes the research evidence around four key decisions (policy levers) facing state policymakers related to pre-K: (1) Whether to fund more or fewer pre-K slots; (2) Whether and how to regulate classroom quality; (3) Whether and how to prescribe curriculum; and (4) How to support the gains of pre-K after children leave pre-K, i.e., in the elementary school years.

How states can use “efficacy networks” to test strategies for school improvement: An interview with Tom Kane, Professor, Harvard Graduate School of Education – Episode #142

The new federal education law, the Every Student Succeeds Act (ESSA), emphasizes the importance of evidence, including defining four levels of evidence-based practices. The law, however, leaves it to states to decide how much they want to build an evidence base and how much to nudge districts toward choosing more effective strategies. So what should state education leaders do who want to leverage the new law and encourage districts to learn and do what works for students?

Tom Kane joins us for a two-part series to provide suggestions. In this podcast episode, he discusses how states could use the authority and resources provided by ESSA to launch a system of “efficacy networks,” meaning collections of local agencies committed to measuring the impact of the interventions they’re using. As he notes, “An overlapping system of efficacy networks working with local [education] agencies would create a mechanism for continuous testing and improvement in U.S. education. More than any single policy initiative or program, such a system would be a worthwhile legacy for any state leader.”

He also describes how the Proving Ground initiative run by the Center for Education Policy Research (CEPR) at Harvard is demonstrating the value of having an efficacy network. CEPR is working with 13 school agencies to develop a model to easily conduct low-cost, local pilots.

Tom Kane is a professor of education and economics at the Harvard Graduate School of Education and faculty director of CEPR. His recent article in the journal Education Next is called, “Making Evidence Locally: Rethinking education research under the Every Student Succeeds Act.”

Transforming the culture of procurement in state and local government: An interview with Jeffrey Liebman, Director, Harvard Kennedy School Government Performance Lab – Episode #141

“Many of the most important functions of state and local governments – from building and maintaining roads to housing the homeless – involve contracting for goods and services supplied by the private sector,” notes the Harvard Kennedy School’s Government Performance Lab in their primer, Results-Driven Contracting: An Overview. “Unfortunately, governments often treat procurement as a back office administrative function, rather than as a core part of their strategy for delivering better performance.”

As a result, increasing the effectiveness of procurements offers an enormous opportunity today for state and local governments to improve their overall performance. As part of Bloomberg Philanthropies’ What Works Cities initiative, the Government Performance Lab has been working with a set of city governments to adopt results-driven contracting strategies and to transform the culture of procurement.

To learn more, we are joined by the Lab’s director, Jeffrey Liebman. He is a professor at the Kennedy School and is also the Director of the Taubman Center for State and Local Government. His recent article published on the website Route Fifty is, “Business as Usual Can Be the Riskiest Procurement Approach.”