By Sara Feldman, Emily Rhodes, & Lily Alpert of Chapin Hall   

In this article, researchers from the Center for State Child Welfare Data (Data Center) at Chapin Hall at the University of Chicago describe ways in which their work has supported social sector agencies in evidence-based decision making. To learn more, join us for a webinar Sept. 7 from noon-1 p.m. CT.

Today, social sector agencies are increasingly responsible for the outcomes of children and families moving through the child welfare system. At the same time, those organizations are recognizing the critical importance of evidence-based decision making. Typically, most salient for organization leaders and managers is the call to implement evidence-based interventions with children and families. However, the selection of an intervention is not the only juncture at which evidence must inform an adjustment to policy or practice.

Developing a sound improvement plan requires one to make a core set of claims: 

• I observe that there is some problem that needs improvement …
• I think it’s because of the following reason …
• So I plan to implement this new intervention …
• Which I think will result in an improvement to the outcome. 

To ensure that these statements combine to reflect a plan that has the potential to bear fruit, each must be backed up with evidence:

• Problems must be observed using an analysis that is representative and free of bias 
• There should be evidence that hypothesized causes are associated with the outcome in question 
• Selected interventions should show evidence of prior effectiveness, or at least contain testable elements designed to interrupt the source of the problem 

Doing all of this well requires an organization to have certain fundamental capabilities. Doing all of this well requires an organization to have certain fundamental capabilities.   

Performance Measurement: I Observe That…

Core among an organization’s evidence use assets is its ability to measure the outcomes it achieves for the children and families it serves. The capacity to measure change over time and examine how outcomes vary across different groups of children enables an agency to pinpoint opportunities for improvement.

The Data Center worked with Necco, a private child welfare organization and member of the Alliance for Strong Families and Communities, to develop such a resource. Using best practices in performance measurement, we developed metrics to track specific program goals related to foster care, independent living, congregate care, and in-home services. Further, each metric could be displayed to show variation over time, across offices, or by child attributes. The resulting system enabled Necco to answer mission-critical questions about its performance, including, among many others: 

• How effective is Necco at meeting the treatment plan goals for children placed in foster care? 
• What is the likelihood that children served in the foster care program will exit to permanency within one year? 
• Do youth served in the independent living program exit with a stable living arrangement?  

The answers to questions like these can spark conversations about performance trends and future investments. With the ability to see where and with whom outcomes are strong or need improvement, Necco has the evidence it needs to support promising practices and identify opportunities for program improvements.

Implementing Theoretically Sound Interventions: I think It’s Because … So I Plan to … Which I Think Will Result in …

Having observed opportunities for improvement, an organization must decide what to do next. For its interventions to be effective, it needs to be able to articulate a theory of change: Why do we believe that when we do a collection of things it will result in the outcome we want to see?

Recently, the Data Center worked with the Harlem Children’s Zone (HCZ), a nonprofit organization whose mission is to break the cycle of intergenerational poverty in Central Harlem, to help it codify the essential elements of its “pipeline” of programs. Specifically, the Data Center was asked to examine the various theories of change that underlie their core programs and determine the extent to which they: 

• Reflected the scientific literature and established best practices in the field (i.e., the evidence base) 
• Were implemented with fidelity to the theoretical design 

Using a range of methods, including content analysis, interviews, focus groups, observations, and a wide-ranging review of the literature, we found that HCZ champions many program elements that are supported by research and implements them according to established best practices. At the same time, we discovered an occasional disconnect between intended services and the organization’s actual capacity to implement those services in the field. We also saw evidence of a “whatever it takes” ethos that undergirds the organization’s work, where staff at all levels are expected to be nimble in addressing whatever barriers exist for young people and their families to maximize their well-being and their opportunities for future success. With an eye toward codifying the essential elements of HCZ’s core pipeline programs, we had to then ask: How does the Harlem Children’s Zone put into practice an intangible “whatever it takes” ethos in a way that aligns with a set of strengthened, more tangible, essential practices and programs? 

In the end, we were able to identify a set of concrete themes for HCZ to consider when entering into the next phase of its strategic planning effort. For instance, we learned the organization could benefit from enhancing parent involvement, filling gaps in its programs addressing non-cognitive skill development, and clarifying the purpose and scope of its community-building arm. Further, HCZ came to realize that it needed to position its work for future empirical assessment of the “essential elements” that they believe drive their programs’ success. As HCZ’s strategic planning continues, the organization will apply these lessons to help bring on-the-ground activity into better alignment with evidence-based theories of change.

Developing Staff’s Evidence Use Skills

While building capacity for evidence use often involves creating new system structures or engaging with experts, sustaining evidence-based decision making inside social sector organizations requires leaders and managers to be educated and responsible evidence users. In response to that demand, Data Center researchers developed EDGE: Evidence Driven Goals & Excellence. In this long-term, cohort-based immersive program, leaders and managers learn best practices for performance measurement and apply those skills to real world problem solving scenarios they face in the field.

EDGE begins by mapping out the cyclical improvement process and providing tools that hold staff accountable for evidence use at each phase. Students learn best practices in performance measurement and problem solving rooted in a set of priority concepts. These concepts are reinforced throughout the course via lecture, interactive practice exercises using real agency data, and homework assignments. Following the classroom learning period, students form small groups and apply their new skills to a self-identified performance improvement effort. With support from coaches, group members use evidence to identify an outcome that needs improvement and conduct their own grassroots research to uncover the potential drivers of that outcome.

Learn More

To learn more about the projects described above and how the Center for State Child Welfare Data can support social sector agencies in their efforts to build capacity for evidence use, join us for a webinar  Sept. 7 from noon-1 p.m. CT or contact the Data Center.