Are test score outcomes sufficient for understanding charter school impact?

Charter schools are a controversial topic in America. Since their start in 1992, they now serve about 2 million pupils with the number of schools totalling more than 6,000 in 40 states. But do they work? Research thus far has focused heavily on test scores, although some have also evaluated the effects on graduation rates and college attendance. The most rigorous studies are mixed in regards of test scores, with some finding positive effects, some finding null effects, and some finding negative effects, depending on which pupil population is analysed. On average, however it seems clear that charter schools are not better than traditional public schools (at least so far). But in terms of longer-term educational attainment, most studies find positive effects, suggesting that charter schools may improve ‘soft’ skills that are not captured by test scores, which help pupils reach higher levels of academic attainment. But this is not necessarily the case. If charter schools do not raise test scores, the positive impact on graduation and university attendance may instead be due to schools setting lower graduation standards instead of raising quality. Until now, there was little way of knowing which one of these scenarios was true. In a new, so far unreleased paper, Kevin Booker and colleagues go farther than previous studies, and analyse the impact of attending a charter high school in Florida and Chicago on the likelihood to persist in university for at least two years, as well as earnings. The authors analyse pupils who were enrolled in charter schools in eighth grade, some of which later left and went back to the public school sector. These constitute the ‘control’ group, whereas the ones who continue in charter schools constitute the ‘treatment’ group. The authors control for observable characteristics, such as demographic background and previous test scores, and also present robustness tests based on an ‘instrumental-variable model’, which attempts to deal with reverse causality further. Since they have data for the pupils up until they are about 23-25, the authors can analyse both university completion rates and earnings in adulthood (but the latter only in Florida). The results are intriguing. In both Chicago and Florida, the effect on high school graduation and university enrolment is positive. In terms of university persistence, the impact is positive, but it is only statistically significant in Florida. The estimates suggest that pupils attending charter high schools are 13 percentage point more likely to persist at least two years compared to pupils in traditional public high schools. The point estimate in Chicago is 7 percentage point, but, again, it is not statistically significant. Nevertheless, overall, this suggests that charter schools can have significant effects on university persistence. As noted, the authors can only evaluate the effects on earnings in Florida. The estimates suggest that pupils who attended a charter high school have 12.7 per cent higher earnings 10-12 years after they were enrolled in eighth grade. These are strong effects, suggesting that charter schools in Florida do indeed produce better ‘soft’ skills, which in turn produce better labour market performance. The findings are especially striking since previous studies in the same areas have found no impact on pupil test scores. Overall, therefore, this study is a very important contribution – it clearly suggests that analysing test score outcomes is not sufficient for understanding the true impact of charter schools. This also has implications for studies analysing the impact of other market-based reforms, which normally focus on short- or medium-term effects. Market reforms in education may have long-run positive effects, even if they do not improve test scores, and this is crucial to take into account. Gabriel H. Sahlgren This comment piece first appeared as the Editor's Pick in the second trial issue of the CMRE Research Digest (May 2014). The piece reviews a working paper by economists Kevin Booker, Tim Sass, Brian Gill, and Ron Zimmer, 'Charter High Schools’ Effects on Long-Term of Attainment and Earnings', which may be downloaded here. You can download free copies of back issues of the CMRE Monthly Research Digest here. to subscribe.