The Regression of Progress 8

Progress 8 as a performance measure was announced to some fanfare back in Spring 2013.

This would be the new improved measure that would clean up the secondary school performance measures providing a significant improvement on the current system .

In reality the measure is really nothing more than the current value added measure, with restrictions on the combinations of the subjects that can be included in it. The claims of an ex-ante forecasting seem unlikely to come to fruition in the near future mainly due to shifting baselines and GCSE reform.

As discussed here, currently progress 8 is not reliable in either 2014 or 2015 due to the issue of curriculum choice in 2014 and 2015. (Remember these students would have been offered curriculum options prior to the Progress 8 announcements). I believe that even 2016 causes issues for schools with a 3 year KS4.

As demonstrated here, sections of the guidance are not wholly transparent leading to difficulties to those trying to understand progress 8 scores for their own students.

A further initial claim of progress 8 was that it would be a ‘fair’ measure… that a student improving from a grade G to grade F, would have the same impact on the measure as a student improving from a D to a C or an A to an A*.

This was initiated from the DfE and now appears in some blogs such as this very thorough one from Stuart Lock.

Curiously, all written online DfE documentation that made this claim has now disappeared from the gov.uk website although reference to this was made by both Nicky Morgan and Michael Gove in various speeches “our new accountability system will value and reward the progress of every child – low attainers and high performers alike.”

Here is a paper copy of the DfE March 2014 guidance, where this claim is made:

P8lie

Unfortunately… all of this claim is not true in 2017, part of the reason for this is somewhat unavoidable but the DfE solution to the problem is flawed unfair.

So in 2017, we have a mixture of grades happening… English and maths will be graded on a 9-1 grade scale whilst all other GCSEs will retain the A*-G grades for this year. I can’t remember why all subjects couldn’t possibly reform at once but there must be an important reason.

Clearly this causes problems, not only for students, schools and employers. But also for the progress 8 measure. Two grading systems, one with 8 grades – with points scores assigned in a 1-8 range and one with 9 grades, scored 1-9. Two systems, one measure.

This in itself is not the major problem, what is problematic is the unexplained DfE solution to this problem:

Page 22 of the DfE Progress 8 guide:

e3gg

OK… so let’s see that mapping

pspoints

PARDON!

So let’s get this straight… “The progress 8 score improves equally, regardless of the grades they are moving between?”

pin

In 2017, a student moving up a grade at the top of the grading scale, receives a point score increment three times of someone lower down the scale. Is that fair?

Well actually… let’s think about it – is this fair, or more appropriately… is this important?

Absolutely yes! It’s very important:

Here’s why:

There could be a school of thought that says it doesn’t matter, any individual student is going to be compared against other students with similar KS2 starting points. Therefore the 2017 points imbalance is irrelevant as all students are relative to their peers.

However, it is upsetting to learn that in a measure where all points score increase were created equal, that after one year some grades increases are more equal than others.

What this could mean is that a school teaching all of it’s students equally would be better served focusing on those more likely to attain the higher grades. i.e. by it’s design the progress 8 measure in 2017 could be encouraging prioritisation of the students with higher prior attainment over those with lower “ability”.

 

Let’s work an example using the attainment 8 measure:

In 2017 School A and School B both have 10 students:

School A has 10 students attaining a grade C (worth 4 points in 2017) in say GCSE Geography

So this equates to 10*4 points = 40 points.

 

School B has 5 students attaining a grade G (1 point) and 5 attaining a grade A (7 points)

So 5 grade Gs *1 = 5 + 5 grade As *7 = 35 = 40 points

The important thing here is that at this point both schools have the same raw attainment 8 score.

Now… 5 students in both schools – due to excellent teaching are going to improve their grade by one grade.

In school A 5 students improve from a C (worth 4) to a B (worth 5.5) so….

(5*4 )+ (5*5.50) = a new attainment 8 score of 47.5 

 

In school B 5 students also improve, in this case from a G(1pt) to an F (1.5 points)

(5*1.5 )+ (5*7) = a new attainment 8 score of 42.5 

Both schools originally had the same attainment 8 score, both improved the grades of 5 students by one grade, but now school A has the better attainment 8 score.

So cynically, would school B be better served putting in effort to improved the grades of their higher ability students all the lower ability students?

This is unfair to those students and this is why this is important.

 

This is not only unfair within schools but also across schools and I don’t need to spell out the types of schools this favours!

And why? Because the DfE have decided not to distribute the differences between grades equally.

So what would be fairer?

This?

this1

Or this?

this2

 

 

The accusation was that schools/students gamed the systems to suit certain measures, but now I believe the measures are being gamed to suit certain schools/students.

Thoughts?

Advertisements

11 thoughts on “The Regression of Progress 8

  1. Andy March 6, 2015 / 3:57 pm

    I get the feeling that someone at the DfE has pulled these numbers (2017 points) out of thin air. Although, once 1-9 grading comes in for all subjects, there will be no more A*-G and we won’t notice it surely?

  2. Philip Bishop March 7, 2015 / 4:27 pm

    I agree that the 2017 points are unfair to A*–G GCSEs.
    However, there is the issue of being fair to the new 9–1 GCSEs (in English Language, English Literature and Maths) that will be in the league tables from 2017. The new 9–1 grades are benchmarked against the old A*–G grades:
    Grade 1 = Grade G
    Grade 4 = Grade C
    Grade 7 = Grade A
    Grade 9 = A bit higher than the current Grade A*
    This puts quite a lot of constraints on the points conversions. A Grade 4, it has been decided, will equal 4 points (which is sensible and, vitally, future-proof). However, to ALSO keep the parity between Grade 4 and Grade C, a Grade C must be also worth 4 points. The same applies for Grade G (must be 1 point) and Grade A (must be 7 points).
    If this isn’t the case, then the link between the old and new GCSE grading system is lost.
    In both the alternatives given in the blog post, a Grade C is awarded more than 5 points. To do this would make a Grade C in, say, Geography more valuable than a Grade 4 (basically, a Grade C) in English Literature (assuming it’s not double-weighted). Clearly, this cannot be allowed to happen.
    I’m not saying that the 2017 points system is perfect and there more than likely IS a better way of doing it (for someone cleverer than me to work out), but I think it is important to highlight that the issue is not as clear-cut as it first seems.

    • dataeducator March 7, 2015 / 5:04 pm

      Thank you for your comments Phil.

      This fairness towards the new grades has been raised with me and my opinion is – that if there is an insistence to make two systems fit one measure then there probably has to be a pinch point. The DfE 2017 points score makes the pinch point be lower grades (and by association lower ability students and therefore schools with larger proportions of lower ability students) It’s also worth noting that the legacy GCSEs will still make up the majority of grades awarded in 2017.

      Can I add, that the new 9-1 GCSEs also have their points scores doubled as part of the calculations. So in the calculations for example a grade 4 (4pts) on the new scale will be made into an 8 in the measures for one of the english components and the maths components. So due to this, they are automatically more valued grade for grade than the legacy GCSEs.

      A solution could have been to simply introduce all the new GCSEs at the same time.

      A second solution could be to report an attainment and progress 2 measure for English and maths, and an attainment and progress 6 measure for Ebacc and Other in 2017 only. This would remove the need for silly points conversions.

      • Philip Bishop March 7, 2015 / 10:27 pm

        Yes, I completely agree that changing all the GCSEs to the 9–1 system would completely avoid this issue. It’s what I would have done if it was my call.

        While English and Maths are getting a double-weighting (which needs to be taken into account), this is because of they are core, not because they have the new grading system. The fact that they are double-weighted does mean that the new, fairer treated 9–1 GCSEs will make up 50% of most students’ Progress 8 score in 2017 (20% each for Maths and the better English and 10% for the lower attaining English).

        I guess the DfE would argue that this is only a temporary problem while the old GCSEs are phased out. That will probably be scant consolation to the 1.5m students (or, rather their) affected over the three years the changes will take. I agree that some sort of split between the old and new GCSEs, as you suggest, would be a fairer approach.

  3. Steve March 8, 2015 / 8:16 am

    Bearing in mind that only Maths and English have new GCSEs being sat in 2017, and (crucially) that all students sit these 2 subjects and they have to appear in the Attainment8 subjects then couldn’t they just have used 1-8 for G-A* and 1-9 for new grades in 2017? That maintains the 1 grade improvement is equally valuable irrespective of whether at the lower, mid or higher grades?

    The real pinch point comes in the year when we have some new GCSEs in the options baskets where students around the country will have different mixes of old and new GCSEs based on their particular, personal, choice of subjects.

    If the DfE had gone with subject specific Progress 8 calculations (and then summed best 8 of these using the basket rules) as appeared in some versions then the issue of conversion points wouldn’t need to exist and the key principle of fairness would still exist. The consequence is that attainment8 couldn’t have started to be calculated until all new GCSEs exist (which seems a lesser issue as Prog 8 is the key accountability measure anyway).

    The other advantage of summing subject specific progress calculations is that it removes incentives to find “easier” GCSE equivalents to fill the optional basket as we’d not be comparing grades in GCSEs to grades in, say, BTECs.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s