External Review of Year Two of APS Turnaround Strategy: Finds Promising Early Signs
Before we began our Journey of Transformation, Atlanta Public Schools (APS) was a school district afflicted with chronic under-performance in our schools. The district was in desperate need of a comprehensive, long-term plan designed to provide remedies that addressed past issues while at the same time creating new opportunities to remove barriers for the future.
Nearly three years ago, the Atlanta Board of Education approved the APS Turnaround Strategy that built upon our mission to graduate more children ready for college and career. This strategy implemented such interventions as tutoring, math and reading specialists, school model changes and recruitment of turnaround leaders and teachers.
As part of that strategy, we put our deepest investments into some of our lowest performing schools and collaborated with educational partners
The 2017-2018 school year marked our second full year of turnaround, and we have been evaluating the work and receiving feedback from our principals and independent researchers to make real-time adjustments to the strategy. Having passed the second year of turnaround, I wanted to provide an update of some of the results.
In summary, the results have been mixed, especially those from a recent evaluation from the independent research group Mathematica Policy Research, but there are promising early signs that our investments are making a difference.
As we review, let’s consider the most positive recent news of the APS Turnaround Strategy. First, all 17 APS turnaround schools receiving targeted or partnership support have improved over the past two years. As shown in our most recent Georgia Milestones report from the state, all of these schools decreased the percent of beginning learners, with six showing double-digit decreases.
In another important indicator – the College and Career Ready Performance Index of CCRPI – 13 of the 17 targeted and partnership schools have increased their CCRPI percentile rank after two full years of implementation. (A newly redesigned CCRPI makes comparisons difficult, but the percentile rank among Georgia schools allows us to continue making direct comparisons.)
Beyond state accountability systems and the Georgia Milestones, we wanted additional and independent evaluations of our turnaround so that we could optimize our investments in this work. Thanks to the support of a philanthropic partner, the strategy is being evaluated over a three-year period.
The external evaluation conducted by Mathematica assesses implementation, impact on participating student and adult behaviors, and the overall effectiveness of the strategy on the district.
Mathematica’s evaluations have been informative and insightful. We are using their findings to inform our ongoing efforts to improve school and student outcomes and allow us to make changes in real time.
After completing our second full year of the transformation strategy, Mathematica recently delivered its second impact study, which is available on our School Turnaround page along with the Year One report.
We will present the study during the next work session of the Atlanta Board of Education at 2:30 p.m. on Monday, Feb. 4, at the Center for Learning and Leadership. I encourage the APS community and the public to attend the meeting or view it at https://livestream.com/k12aps.
In the meantime, we continue to unpack the recent findings from Mathematica. From the report, their key highlights are:
- While our Turnaround Strategy efforts are producing improvements in math performance in targeted schools, Mathematica did not find evidence of statistically significant impacts of school-wide targeted support on other student outcomes.
- Evaluators found little evidence that support from math and reading specialists had an impact on academic outcomes.
- They also found no evidence that support from Communities in School improved student suspensions, attendance or academic achievement.
- Finally, our partnership schools are also producing improvements in math performance, but other effects were mixed, varying by outcome and by partner organization.
We are pleased that the Turnaround Strategy has led to marked improvements in math performance. Mathematica noted that the limited impacts with other core subjects is a consistent result among other schools engaged in turnaround. In comparing our findings with national research and turnaround trends, they wrote:
“Despite the limited impacts of targeted supports overall, the second-year impact on math achievement is a promising sign, as many turnaround efforts fail to produce any measurable positive impacts in a comparable or even longer period. When impacts are detected, positive impacts tend to be larger in math than reading, so those measured for targeted school supports follow the trends of turnaround efforts elsewhere.”
So that offers encouragement and support for our work!
In regards to Mathematica finding little evidence of how our specialists impacted academic performance, we found from the implementation phase of their study that specialists worked with students not on their rosters and also provided curricular resources used in their small groups to teachers for classroom use. Additionally, schools reported the specialists supported teachers’ professional learning by modeling lessons and leading professional development sessions.
So it is possible that these activities could have improved student performances in both the targeted group of students and the match comparison group, causing the impacts of the specialists to be underestimated.
As part of the report, Mathematica suggested that we do a better job in capturing data to better understand which supports are most effective.
In regards to our partnership with Community in Schools (CIS), the evaluators again said the results are consistent with other recent research studies and may be partly explained by implementation challenges.
Feedback leading up to the implementation phase of the study released in June 2018 indicated the CIS site coordinators half-time presence as the main challenge to their effectiveness. In response to this feedback targeted tier schools were each assigned a full-time CIS site coordinator for the 2018-19 school year.
CIS is just one wraparound support provided to targeted tier schools. Each targeted tier school is also afforded an additional wraparound position that supports students’ non-academic needs.
Among its recommendations, they suggested that as turnaround schools show improvement, the district should plan how to extend or redirect turnaround supports so the school system can affect lasting change. Looking ahead to the 2019-2020 school year, we want to work in a smart, intentional fashion to how we can phase schools out of support when they show positive progress and how we add supports should other schools be designated as turnaround school eligible by the state.
As we move forward through the third year of School Turnaround in APS, I will continue to provide updates on our ongoing efforts to give every student in Atlanta Public Schools the educational opportunities that lead to college, careers and choice-filled lives.