I recently wrote about why the proposed publication of the performance tables for student sitting exams this summer makes a mockery of any sense of fairness, comparable accountability and the ability to inform parental choice:
As I explained in that blog, it’s not about about the decisions made, although they could be better thought out – as I will explain below – but rather it is simply about the decision to publish performance tables in full with all the usual measures, in hugely unusual times.
In seems incredibly logical to me that if you are having something called “Compare School Performance” that any comparison needs to be as fair as possible.
The Updated DfE guidance sets out some examples of what the changes mean for the headline performance measures, including attainment 8 which of course feeds progress 8. There are a couple of handy examples of what this all means:
So in a usual year, Poppy would score 67 attainment 8 points. Lovely. Let’s see what’s happening this summer:
As the DfE text explains the English Literature is not counted at all, apart from to double weight the English Literature. The upshot of this is that Poppy now has 64 attainment 8 points.
67 or 64 attainment 8 points doesn’t sound like a great difference does it? Believe me, it is.
It’s fair to assume that any decision to enter early for as in the example, English Literature is often a whole school decision. Not always. But typically this sort of decision affects the whole cohort. Therefore across a whole cohort achieving the same as Poppy, and these rules being applied, and the students attaining 3 fewer attainment 8 points:
This would mean that the school Progress 8 figure would be -0.30 compared to a year where the usual rules are applied. This means if you are achieving P8 of +0.10, you are now coming out with -0.20. This is possibly moving your confidence interval below 0 and therefore give you a “below average” label instead of “average”.
On the flip side, for all the schools that don’t have any early entry, because Progress 8 is a zero sum game, i.e. reductions in some schools scores must be balanced out with gains for others – then all these schools will do a little bit better.
It’s a bit like a football league table showing positions and points scores but not telling you that some teams have played 46 games and some have only played 40. And guess what, some teams are getting relegated on this basis. Doesn’t sound all that fair. Plus there’s an influx of new fans who are looking to support a team for next season and they also have got no idea who played what amount of games.
So, just to be absolutely clear, I’m also not saying that the usual way of calculation should be used, because that would mix TAGs and Exam Results and that just wouldn’t be fair either, in the other direction.
What could happen?
Option A – The affect of not including English Literature (in the DfE example) could be tempered by allowing English Language to count in the English basket but also in the Open basket.
Option B – If any grades are excluded from the attainment 8 calculations then they are replaced in the calculations by the average of the other counting grades. In the DfE example this is also 6 and would have the same affect – making the attainment 8 score 65.
What should happen?
The above options don’t really cut it. The only true solution is not to publish measures that are affected by early entry. It’s that simple. You can’t have zero-sum measures when the field is not even pretending to be level. You can’t really have threshold measures either although they are less problematic. Why can’t the performance tables simply be a list of subjects and the grades, or grade percentages with suppression for low entries. No comparisions, no faux measures masquerading as real information. Just some raw information and a link to the schools website, where they can make their own case.
We need less data-science and more conscience!
dataeducator 
One thought on “Despair-school-performance”