The importance of a broader approach to education reform evaluation – the evidence from academic/vocational tracking in Sweden

When analysing the effects of education reforms, researchers tend to restrict themselves to traditional outcomes, such as pupil achievement and attainment. There are exceptions, but they are still relatively few. While few question whether it’s worthwhile to study the impact on traditional outcomes, too narrow a focus on them may still be problematic since reforms can also have broader social effects, positive or negative. Even if a reform improves achievement, for example, it may have a negative impact on other long-term outcomes, such as health or crime. Broader effects are clearly important to take into account since it’s otherwise impossible to calculate an unbiased cost-benefit analysis of a reform. In this post, I illustrate this by discussing one relevant and fascinating example: a reform that reduced academic/vocational tracking in Sweden. In the early 1990s, Swedish education went through a hurricane of reforms, one of which significantly altered the upper-secondary education tracking system. Before the reform, pupils either chose an academic or vocational track. Whereas the three-year academic programmes enabled pupils to continue to higher education, the two-year vocational programmes didn’t. In spite of this, many chose the latter; right before the reform, about 45% of pupils attended a vocational track. However, the reform changed all this. The vocational programmes were prolonged to three years and injected with a considerably more general content. All tracks now enabled pupils to continue to higher education; the purely vocational tracks were thus abolished. A key motivation behind the reform was the idea that a general education is necessary for pupils to be able to adapt in a rapidly changing labour market. Another was the desire to ensure that everyone could proceed to university, so that the consequences of choosing the ‘wrong’ track would not be as severe. By abolishing purely vocational tracks, it was argued, all pupils would be better equipped for the future. Were these hopes realised? Caroline Hall, at the Institute for Labor Market Evaluation and Education Policy, and her colleagues have done an excellent job evaluating the reform – which was preceded by a pilot scheme – in a rigorous way and from different perspectives. The overall picture that emerges from this research is fascinatingly complicated. First, while the reform increased the likelihood that pupils in vocational programmes obtained three years of upper-secondary education, there was no impact on university enrolment or earnings. This suggests that the costs of pursuing higher education after attending a vocational track before the reform didn’t discourage pupils to do so, and that the returns to one extra year of work experience among pupils attending the vocational tracks appear to have been as large as the returns to one extra year of education. Also important, the reform significantly increased the likelihood that low-achieving pupils and those with low-educated parents dropped out or graduated without grades in all subjects. Second, however, among girls with high achievement and highly educated parents, the likelihood of childbearing by the age of 20 decreased significantly. This is driven solely by an ‘incapacitation effect’, meaning that those girls simply didn’t have time to have children because of their being enrolled in full-time education – there was no impact on childbearing after the age of 20. Meanwhile, there were no effects among girls with low academic achievement and with low-educated parents – who have a lower opportunity cost to become teenage parents – or among boys. Third, the reform had no average impact on the likelihood of becoming unemployed during the Great Recession 2008-2010. But it actually increased the likelihood of short- and long-term unemployment for low-achieving pupils, an effect that is concentrated among boys. Fourth, in thus far unreleased research, researchers have also found that the reform decreased property crime, while having no impact on violent crime. The fact that these findings are still unreleased, however, means that we should interpret them with caution. So the Swedish reform reducing the vocational and academic divisions in upper-secondary school appears to have had mixed effects, depending on outcome and pupil group analysed. The increase in the drop out rate and likelihood of unemployment among disadvantaged pupils, and the fact that there was no change in university enrolment and earnings either on average or among any sub-groups, imply that the reform wasn’t particularly helpful to say the least. Yet the broader social effects seem to be benevolent. Of course, it’s not necessarily the case that we can interpret the impact on teenage pregnancies among high-achieving girls and those from higher socio-economic backgrounds as a clear-cut positive outcome for the mothers, simply because it’s not certain that teenage pregnancy is causally linked to their having worse outcomes later in life (see new results and review in this paper). Indeed, in the case of the reform discussed, there was no effect at all on university enrolment and future earnings. At the same time, however, the impact on the children born should also be taken into account. Whereas the literature analysing this issue is scant, not of high methodological quality, and mixed to boot, there is some quasi-experimental research suggesting negative effects on children of teenage mothers. Furthermore, the reform’s impact on property crime is a clear-cut social benefit. In general, therefore, this research clearly shows the importance of studying the broader effects of education reforms, not merely their impact on traditional gauges of success. It also gives empirical evidence to the intuitive idea that reforms can have strongly heterogeneous effects, both positive and negative – which all have to be considered when evaluating whether they were worthwhile or should be rolled back.