Strengthening evaluation capacity within agencies: An interview with Naomi Goldstein, Office of Planning, Research and Evaluation at the Administration for Children and Families, HHS – Episode #20

For public leaders at the federal, state and local levels who want to strengthen their agencies’ abilities to learn what works and to continually improve performance, building program evaluation capacity within their agencies is essential. But what are the building blocks of that capacity? And why is the relationship between an evaluation office and a program office within an agency so important?

To explore these and other related issues, we speak with Naomi Goldstein, the Director of the Office of Planning, Research and Evaluation within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services. In her role, she advises the Assistant Secretary for Children and Families on improving the effectiveness and efficiency of ACF programs. She is one of the leading experts in program evaluation within the federal government and was awarded the Presidential Rank of Distinguished Executive in 2012.

You may also be interested in reading ACF’s evaluation policy, launched in 2012, which is designed to confirm ACF’s “commitment to conducting evaluations and to using evidence from evaluations to inform policy and practice.”

Web extra: Naomi Goldstein discusses the similarities and differences between program evaluation and performance management. [click here] As a postscript, she commented after our interview about the value of combining typical performance management and evaluation approaches, including how experimental evaluations that use administrative data can produce relatively quick and inexpensive results.

A city’s effort to drive innovation and learning on a priority issue: An interview with Kristin Morse, New York City Center for Economic Opportunity – Episode #19

Kristin MorseThe Center for Economic Opportunity (CEO) is a unit within the Mayor’s Office in New York City. It was launched in 2006 by Mayor Michael Bloomberg to develop new and innovative anti-poverty initiatives and to rigorously test them to see what works. It provides about $100 million annually to primarily city agencies to fund pilot programs. The majority of funds come from the city, with additional support from state, federal and philanthropic sources. Since its launch, CEO has worked with 28 city agencies and over 200 community-based providers to pilot 50 programs. In recognition of its work, it won the 2012 Innovations in American Government Award.

The CEO provides insights into how public leaders can focus attention within government, and within their communities, on particular priority issues (in this case, reducing poverty); test new approaches; and rigorously evaluate the results in order to learn what works, scale up effective programs and stop doing what isn’t working. On the latter point, CEO has terminated about 20% of its programs for inadequate results, while at the same time scaling up several programs that have shown strong results.

To learn more, we are joined by Kristin Morse, CEO’s Executive Director.

Web extra: For brevity, the interview does not cover CEO’s Social Innovation Fund work, but more information is available here. This effort is supporting the replication of CEO’s most promising initiatives, including in eight urban areas in the U.S.

Performance budgeting in Austria: An interview with Gerhard Steger, Austrian Ministry of Finance – Episode #18

Gerhard StegerWith a population about the size of Virginia, Austria may be a relatively small nation, but it provides a prominent example of implementing performance budgeting. In particular, a series of budget reforms in recent years has significantly shifted the federal budget process in Austria from one focused on the question, “How much do we spend?” to one with a much stronger focus on the question, “What results are we producing?”

Specific reforms include multiyear budgeting, the ability of ministries (that is, federal agencies) to keep any savings from cost-cutting or efficiencies, and a performance measurement system including the requirement that each ministry set at least five key goals that are approved by parliament.

To tell us about performance budgeting in Austria, we are joined by Gerhard Steger who is the Budget Director for the Austrian Ministry of Finance.

Learning from innovative businesses about creating a culture of experimentation in government: An interview with Jim Manzi, Author of “Uncontrolled” – Episode #17

Jim ManziJim Manzi is the founder and chairman of Applied Predictive Technologies, a business analytics firm. His 2012 book Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics and Society argues for the usefulness of experimental methods—in other words, randomized controlled trials (RCTs)—for addressing important policy issues, from improving education outcomes to increasing economic growth to reducing crime.

In a review of Uncontrolled in the New York Times, columnist David Brooks writes, “Manzi wants to infuse government with a culture of experimentation.” Brooks also notes: “What you really need to achieve sustained learning, Manzi argues, is controlled experiments. Try something out. Compare the results against a control group. Build up an information feedback loop. This is how businesses learn. By 2000, the credit card company Capital One was running 60,000 randomized tests a year — trying out different innovations and strategies. Google ran about 12,000 randomized experiments in 2009 alone.”

Washington State, a leader in the use of cost-benefit analysis: An interview with Steve Aos, Director, Washington State Institute for Public Policy – Episode #17

Steve AosSteve Aos is the Director of the Washington State Institute for Public Policy (WSIPP). The Institute’s mission is to carry out practical, non-partisan research—at legislative direction—on issues of importance to Washington State. Areas of focus have included education, criminal justice, welfare, children and adult services, health and more. The outcome of WSIPP’s work includes cost-benefit analyses of various policy options so that legislatures can make more informed decisions about cost-effective policies.

Although not discussed in the interview, a related initiative to note that builds on WSIPP’s work in Washington State: The Pew Charitable Trusts and the MacArthur Foundation are currently working with 14 states to develop and strengthen their cost-benefit analysis capabilities, as part of the Results First Initiative.

Web extra: Steve Aos explains how trust between WSIPP and the legislature, build over time, is an important aspect of the Institute’s ability to produce analysis that is valued and used. [click here]

The Governor’s Delivery Unit in Maryland: An interview with Mike Powell, State of Maryland – Episode #16

powell profileMike Powell is the Chief Innovation Officer for the State of Maryland under Governor Martin O’Malley. In that role, he helps oversee the Governor’s Delivery Unit.

For public leaders in the United States, the Governor’s Delivery Unit is a useful example of setting specific, ambitious strategic goals, of being transparent with citizens about progress, and of creating a process internally — with close leadership involvement — to “own” those goals and drive progress towards them.

In particular, the Delivery Unit is designed to help achieve 15 strategic goals that are important to citizens and that cut across traditional agency silos. Its website tracks progress, including graphs, background information, and flags that show whether each goal has been delivered, is on track or progressing, or has had insufficient progress so far.

Mike previously worked for IBM as a consultant and was a CityStat analyst in Baltimore earlier in his career.

As background, it’s useful to note another O’Malley Administration initiative mentioned in the interview: StateStat. Every few weeks or months, depending on the agency, each agency leadership team comes to a StateStat meeting in the Governor’s Office to review the agency’s performance data and to discuss, with the Governor’s staff, ways to improve outcomes. The Governor’s Delivery Unit, in contrast, is closely related (and managed by StateStat staff) but is focused on cross-cutting goals that span agencies and that have been identified by the Governor as key benchmarks for state progress.

Web extra: Mike Powell discusses his role as Chief Innovator Officer for Maryland, answering a question that, as he has noted, he gets frequently: “What does a Chief Innovation Officer do?” [click here]

Learning from the Obama campaign about creating a culture of experimentation in government: An interview with Amelia Showalter, former Director of Digital Analytics at the Obama campaign – Episode #15

Amelia ShowalterAmelia Showalter served as Director of Digital Analytics on Barack Obama’s 2012 presidential campaign, leading a team that designed and implemented hundreds of experiments to improve the performance of all types of digital outreach.  Today, she is a consultant who helps organizations achieve better results through testing.

In the interview, we draw on her innovative experience in politics to consider how government — in nonpartisan ways — can learn from the same techniques of testing and experimentation in order to improve program and agency performance. In other words, how can public managers build a culture of rapid organizational testing and improvement?

Web extra: Amelia Showalter talks about the “email derby” in which she and her staff tried to pick the most and least effective versions of fundraising emails and how the derby results underscored the usefulness of randomized testing. [click here]


Using online tools to engage citizens: An interview with Matt Leighninger, Deliberative Democracy Consortium – Episode #14

Matt LeighningerMatt Leighninger is the Executive Director of the Deliberative Democracy Consortium and the author of the recent report for public managers, “Using Online Tools to Engage – and be Engaged by –The Public” published by the IBM Center for the Business of Government. The motivation behind the report: While many federal and state agencies are striving to engage with citizens using online tools and e-government, many public managers find themselves unfamiliar with what tactics and tools work best under different scenarios.


Making telework work: An interview with Scott Overmyer – Episode #13

Scott Overmyer

What does it take to make telework a successful tool for public agencies and workers? To explore that question, we’re joined by Scott Overmyer. He’s the author of the IBM Center for The Business of Government report Implementing Telework: Lessons Learned from Four Federal Agencies. The report offers practical implementation advice to agency leaders and front-line managers.

As background, the Telework Enhancement Act of 2010 expands telework opportunities to over one million federal workers. Today federal agencies are working to implement that law, while state and local officials are also findings ways to promote telework. Telework — working outside of the office — has the potential to save billions of taxpayer dollars along with other benefits.

Policy strategies to prevent unintended pregnancy: An interview with Adam Thomas, Georgetown University – Episode #12

Adam ThomasAlmost half of all pregnancies in the United States are unintended and the women and children involved in these pregnancies are disproportionately likely to experience a range of negative outcomes. For insights into cost-effective policy strategies to reduce unintended pregnancy, my guest is Adam Thomas, a professor at Georgetown University and the former Research Director for the Brookings Institution’s Center on Children and Families. His recent article in the Journal of Policy Analysis and Management — as well as a Brookings policy brief— focus on three evidenced-based strategies to prevent unintended pregnancy and the results from fiscal impact simulations of those strategies. His findings have implications for both federal and state policymakers.

Stay in the loop! Sign up for monthly Gov Innovator podcast updates.
Never display this again