OK, firstly let’s start with the truth. Why not.
Predicting most things is impossible. Accurately forecasting statistical scores for 200+ independent people in your school and aggregating them into an combined score across 8 subjects is impossible. Throw in several more unknown such as curriculum choice, national trends, a change of qualification specifications and a whole furore over an adjustment to legacy GCSE points and you have more than just a panic on your hands.
However, in my opinion it’s still sort of right to do something, even if this is simply to develop internal consistency so that schools can see, within their own school what subjects are performing well, which students are lagging behind etc… but the question remains… what and how?
There a number of hurdles to consider in forecasting P8 in 2017:
- Predicting grades or extrapolating predicted grades from current working at grades.
Right so any forecast needs to attempt this, schools have to do this with any future prediction they make. The fly in the ointment is that in 2017, English and maths are on the new 9-1 grading structure and although there are some signposts from Ofqual, nobody really knows how these grades will pan out and teachers are not at all experienced in knowing what these grades look like in practice.
Right so this hurdle is there, and it is known. The next two hurdles are less easy to even think about, let alone navigate.
2. The adjustment of points in 2017 for legacy GCSE qualifications.
3. The national attainment 8 estimates in 2017.
Right – to explain hurdle 2 – if you didn’t know GCSE points in legacy (A*-U) graded qualifications attract different points in 2017 than in previous years. I’ve spoke about this several times and most recently here: https://dataeducator.wordpress.com/2015/09/30/points-of-view/
Here is the conversion table for GCSEs (there are others for BTECS, AS levels etc):
When forecasting P8 – this matters, because the previously published attainment 8 estimates from starting points (hurdle 3) were created from a points structure as shown in the 2016 points column. This obviously means that the attainment 8 estimates will change and it could be sensible to assume that they will go down in 2017.
Would this be a sensible assumption?
Progress 8 is made up of 8 subjects, English and maths, 3 Ebacc subjects and 3 Other subjects. English and maths are double weighted, so in essence 10 scores make up the progress 8, ergo also the attainment 8 estimate.
In 2017 English and maths are on the new specification 9 to 1 where a grade 9 = 9 points, grade 5 = 5 points etc. The additional grade at the top – the grade 9 will not be attained by many students. Therefore the bulk of the change is in the middle of the structure, where for example the bottom of a C currently which is worth 5 points will become the bottom of the grade 4 (roughly) which of course is equal to 4 points.
Therefore in English and maths, overall students will be attracting fewer points, even though there is a widening of the points scale. English and maths make up at least 40% of the P8 measure.
The legacy GCSEs in 2017, Ebacc and Other, which make up 50 to 60% typically of the measure are also attracting reduced points, with only the A* grade attracting more. The bulk of grades are worth the same or less.
This almost definately (ha!) means that attainment 8 estimates will come down.
So how do we forecast P8 in 2017. We have 6 options:
Option 1 – We don’t. We accept that it’s too difficult, too many unknowns, too little help from the DfE and Ofqual, so we just don’t do it. Ofsted gleefully accept this and we all dance off into the sunset.
Option 2 – We simply plug the known adjusted points scores in 2017 into the known attainment 8 estimates from 2014, 2015 or 2016 when they become available.
Pro – We are dealing with known points scores, and known attainment 8 estimates. So we can say “based on 2017 points and 2016 estimates, we would achieve XYZ”.
Con – The progress 8 forecast would come out unfeasibly low. For many of you plugging this into your various systems at the moment, I know that this is happening – I know that you know that your forecast is low, but unless you get another method of calculation going on – you are stuck with it.
Option 3 – We adjust the points in 2017 for legacy GCSEs to better reflect the gaps between grades in line with 9-1 points. So for instance, we don’t say a grade C is worth 4, we say it’s worth 4.5, or 4.8 or some other number to account for reduction in points scores but still keeping a relationship of sorts with the 9-1 grading structure. i.e. this is to say for example that we accept that a grade C is less than a grade 5 but to account for the unknown attainment 8 estimates we use existing estimates but with increased points values to negate the effect.
Pro – We are sort of keeping a known element here (the attainment 8 estimates and trying to balance off the loss of points in a more realistic way).
Con – It’s incredibly messy to do this. I don’t like this solution even less so since I wrote it down.
Option 4 – We adjust the 2015 or 2016 attainment 8 estimates downwards across the board by a similar factor to the likely reduction in points between 2016 and 2017. Right so this means that instead of saying the attainment 8 starting point from say a fine-level 4.2 is 41 points, we will we reduce it by a factor of say 7% or 11% or some other number that we would have to magic up a statisitcal explanation for. Anyway a reduction so 4.2 = 41, becomes 4.2 = 38 or whatever. We could apply different factors to different starting points.
Pro – We are keeping 2017 points as designated by the DfE, this means we will get the school attainment 8 right – thats a known outcome. IF we get the attainment 8 points estimates reduction right we will end up with a progress 8 score close to what it will actually be.
Con – Again we are introducing an unknown, but everything has some unknowns involved so perhaps not a bad option overall
Option 5 – We don’t adjust the A8 estimates and we don’t adjust the legacy points for GCSEs. i.e. we keep grade C as a 5, grade B as a 6 etc.
Pro – For legacy GCSEs we can have consistency, we can say in 2015 or 2016 – this subject achieved XYZ attainment points and for 2017 forecasts we are comparing on the same scale, therefore we can see if the individual subject is predicting an improvement or not. Also because the exisiting A8 estimates are created on the same scales, the P8 scores should be relatively relative, although obvious Eng. and maths being on a different scale will be having an impact. Essentially though for legacy subjects you are comparing like points with estimates derived from like points.
Con – Your A8 score won’t be right, it will be over inflated.
Option 6 – The DfE do something! How about the DfE just say, “OK OK, the points and that we are keeping them, but what we will do is get some of our statisticians to model the likely impact of grade and points reform on the attainment 8 estimates”. Crucially what they will do under option 6 is come up with some ex-ante estimates (before the event) that they will then stick to after the event. Like they said when they first described progress 8 to us.
Pro – The massive, massive pro here is that schools would be able to ACCURATELY calculate their Progress 8 scores based either on predictions or on on actual results in the summer on results day or shortly after. We wouldn’t need to wait with baited breath for some new estimates to be counted. We would know where we stand in relation to a positive or negative progress 8 score.
Con – It would be extremely unlikely that national would come out at 0.00, so we would have to judge ourselves and be judged in relation to a national figure that is not 0. It might for instance be +0.03 or something. That in turn could move the floor standard to -0.47. This would be an unknown.
I can live with this con, but can the DfE?
Which leads me onto the second con. They will not do this. So forget it.
Option 4 or 5 for me. Will probably have a tinker about with both using the APA Pro solution (plug).
Thanks – comments more than welcome!
EDIT – I’m adding this comment to this blog after the edudatalab report of 23/02/2016 “Don’t try to forecast Progress 8”.
The blog of course is correct, forecasting progress 8 accurately is folly. And I should add that even after 2019, we might get one semi normal year in 2020 before in 2021 we have the removal of KS2 fine-levels to contend with, leaving us with yet another year in limbo.
However, schools sort of need to demonstrate that they are making progress don’t they? They want to be confident that if their students achieve certain grades that they will be making ‘good progress’ compared to similar students.
Expected progress is dead in the water, I imagine those helpful transition matrices a lot of schools use will be discontinued. Now it’s all very well fumbling around in the dark and hoping for a good outcome, but I sort of feel the schools of this great country deserve something a bit better than the major accountability measure being deemed unmeasureable until after the event.
I look back with sadness at the the DfE secondary accountability consultation and it’s findings (particularly the statements on pages 25 & 26).
Schools are between a rock and a hard place, lambasted by some organisations for inaction, criticised for lack of strategic thinking, harpooned by others for being unable to provide accurate forecasts.
I’ve since followed this blog up with a second post: https://dataeducator.wordpress.com/2016/02/28/forecasting-progress-8-part-2/