Version of an article published in Research Fortnight, June 2016:
Just over a year ago I wrote in Research Fortnight about the proposed Teaching Excellence Framework. I contended that a TEF based on metrics was unlikely to make sense unless it was part of a broader system that included a substantial proportion of expert review. But such reviews would impose a material opportunity cost on universities. Given the generally high standards of teaching in UK higher education, the existence of a sound measure of quality that was broadly supported across the sector (the NSS) and other evidence of largely positive student experiences and outcomes, a TEF would seem to be a pointless additional burden.
There would also be the small matter that it might not work. When a very similar scheme, linked to marginal funding, was tried in Australia, it did not last for long. In this case, despite the existence of an expert panel to moderate the results, chaired by probably the best person to do it (the late Sir David Watson) the variations between universities were usually minor and rarely statistically significant. The scores obtained and the resultant funding variations remained highly controversial. Ultimately the system lacked credibility and was quietly dropped.
In the face of the evidence, why are we going ahead with a TEF? Governments are less interested in evidence than in selecting soundbites to support their agendas. The fact that we already provide an outstandingly good higher education experience for most students does not fit the narrative that universities must be continually pressured to fix imagined weaknesses. On this account, the problems of a metrics-based system moderated by an expert panel, so well revealed by the Australian experience, can be dismissed and the negative comments made in the consultation can subjected to the usual excuse that ‘maybe more research is needed – so we will proceed a bit more slowly’.
The White Paper itself provides an extraordinarily weak justification for the TEF. Its authors set the scene by covering familiar yet questionable ground – assertions that students need better information about teaching quality (disputable at best, particularly if it is not at discipline level), that robust measures of teaching quality in HE are not available (untrue), that student satisfaction is important (hardly – it is the quality of learning experiences and outcomes that matter, ‘satisfaction’ being a mere by-product), that contact hours are key to a better experience/greater satisfaction (definitely wrong – did they not look at the Open University?), and that there is a problem with social mobility that the TEF will help address (even if there were such a problem, why should a teaching assessment exercise address it?).
After this poor start, things get better with the proposed metrics. The TEF will initially use three indicators: a composite (calculation unspecified) from NSS data from teaching on course, assessment and feedback, and academic support questions; non-continuation rates; and employment or further study rates. These are standard measures and if the data are properly benchmarked they promise to provide reasonable validity. However, they are hardly without problems. The NSS was never designed to operate at institution level. Even at discipline level, courses have differing profiles of scores on the NSS dimensions; they can be strong on ‘teaching’ and weak on ‘assessment and feedback’ for example — or vice-versa. Combining the measures will simply reduce the size of any differences.
Variations between institutions on a single summative criterion are likely to be minimal. Expect some major differences of opinion and probable challenges to the nice judgements made at the margins between ‘meets expectations’, ‘excellent’ and ‘outstanding’. A quantity that led to labelling a few institutions at the extremes as good or bad might be a better bet; as an Office for National Statistics report on the data sources for the TEF tentatively puts it, it ‘may be possible to identify a small number of institutions which are significantly different, that is significantly better or worse’. The same report argues that the NSS and the retention and destinations data all suffer from the need to define the target population more clearly, to determine the extent of under and over-coverage, and to adjust for non-reponse.
An altogether better effort than the White Paper’s attempt to vindicate the TEF is the technical consultation on the assessment criteria for it. Still in draft form, of course, it has a properly thought out framework (Teaching Quality; Learning Environment; Outcomes and Learning Gain) with criteria and evidence clearly delineated for each. I am tempted to suggest, though, that the principles have been derived from the existence of suitable indicators rather than the other way round. For some inexplicable reason, the NSS questions on personal development, such as improved communication skills, are not included as one of the indicators of learning outcomes and gain. Curiously, the proportion of HEIs falling into the three performance categories is pre-defined (‘norm-referenced’), with 50 to 60 percent of institutions in the middle or average category (although it is labelled ‘excellent’).
The role of assessors and the expert panel, who will be very busy people, is admirably described and detailed. Indeed the proposed assessment process shows just how heavy the additional work and expense will be. As well as the time and payment of many assessors, each HEI will be invited to submit additional evidence (the ‘provider submission’) on matters such as leadership to support teaching excellence, local surveys, teaching and research links, curriculum to involve professional bodies … and many others.
It is inevitable that the labour involved in preparing provider statements and deciding what will go into them will reduce an institution’s capacity in teaching, research and adminstration. If the TEF survives the many obstacles in its way, we may hope for a rigorous evaluation of its impact on teaching quality and learning outcomes. This should incorporate a balance sheet that weighs its considerable costs against any improvements. But I for one will not be holding my breath for it.