Progress 8 as a performance measure was announced to some fanfare back in Spring 2013.
This would be the new improved measure that would clean up the secondary school performance measures providing a significant improvement on the current system .
In reality the measure is really nothing more than the current value added measure, with restrictions on the combinations of the subjects that can be included in it. The claims of an ex-ante forecasting seem unlikely to come to fruition in the near future mainly due to shifting baselines and GCSE reform.
As discussed here, currently progress 8 is not reliable in either 2014 or 2015 due to the issue of curriculum choice in 2014 and 2015. (Remember these students would have been offered curriculum options prior to the Progress 8 announcements). I believe that even 2016 causes issues for schools with a 3 year KS4.
As demonstrated here, sections of the guidance are not wholly transparent leading to difficulties to those trying to understand progress 8 scores for their own students.
A further initial claim of progress 8 was that it would be a ‘fair’ measure… that a student improving from a grade G to grade F, would have the same impact on the measure as a student improving from a D to a C or an A to an A*.
This was initiated from the DfE and now appears in some blogs such as this very thorough one from Stuart Lock.
Curiously, all written online DfE documentation that made this claim has now disappeared from the gov.uk website although reference to this was made by both Nicky Morgan and Michael Gove in various speeches “our new accountability system will value and reward the progress of every child – low attainers and high performers alike.”
Here is a paper copy of the DfE March 2014 guidance, where this claim is made:
Unfortunately… all of this claim is not true in 2017, part of the reason for this is somewhat unavoidable but the DfE solution to the problem is
So in 2017, we have a mixture of grades happening… English and maths will be graded on a 9-1 grade scale whilst all other GCSEs will retain the A*-G grades for this year. I can’t remember why all subjects couldn’t possibly reform at once but there must be an important reason.
Clearly this causes problems, not only for students, schools and employers. But also for the progress 8 measure. Two grading systems, one with 8 grades – with points scores assigned in a 1-8 range and one with 9 grades, scored 1-9. Two systems, one measure.
This in itself is not the major problem, what is problematic is the unexplained DfE solution to this problem:
Page 22 of the DfE Progress 8 guide:
OK… so let’s see that mapping
So let’s get this straight… “The progress 8 score improves equally, regardless of the grades they are moving between?”
In 2017, a student moving up a grade at the top of the grading scale, receives a point score increment three times of someone lower down the scale. Is that fair?
Well actually… let’s think about it – is this fair, or more appropriately… is this important?
Absolutely yes! It’s very important:
There could be a school of thought that says it doesn’t matter, any individual student is going to be compared against other students with similar KS2 starting points. Therefore the 2017 points imbalance is irrelevant as all students are relative to their peers.
However, it is upsetting to learn that in a measure where all points score increase were created equal, that after one year some grades increases are more equal than others.
What this could mean is that a school teaching all of it’s students equally would be better served focusing on those more likely to attain the higher grades. i.e. by it’s design the progress 8 measure in 2017 could be encouraging prioritisation of the students with higher prior attainment over those with lower “ability”.
Let’s work an example using the attainment 8 measure:
In 2017 School A and School B both have 10 students:
School A has 10 students attaining a grade C (worth 4 points in 2017) in say GCSE Geography
So this equates to 10*4 points = 40 points.
School B has 5 students attaining a grade G (1 point) and 5 attaining a grade A (7 points)
So 5 grade Gs *1 = 5 + 5 grade As *7 = 35 = 40 points
The important thing here is that at this point both schools have the same raw attainment 8 score.
Now… 5 students in both schools – due to excellent teaching are going to improve their grade by one grade.
In school A 5 students improve from a C (worth 4) to a B (worth 5.5) so….
(5*4 )+ (5*5.50) = a new attainment 8 score of 47.5
In school B 5 students also improve, in this case from a G(1pt) to an F (1.5 points)
(5*1.5 )+ (5*7) = a new attainment 8 score of 42.5
Both schools originally had the same attainment 8 score, both improved the grades of 5 students by one grade, but now school A has the better attainment 8 score.
So cynically, would school B be better served putting in effort to improved the grades of their higher ability students all the lower ability students?
This is unfair to those students and this is why this is important.
This is not only unfair within schools but also across schools and I don’t need to spell out the types of schools this favours!
And why? Because the DfE have decided not to distribute the differences between grades equally.
So what would be fairer?
The accusation was that schools/students gamed the systems to suit certain measures, but now I believe the measures are being gamed to suit certain schools/students.