Forecasting Progress 8 for 2017

OK, firstly let’s start with the truth. Why not.

Predicting most things is impossible. Accurately forecasting statistical scores for 200+ independent people in your school and aggregating them into an combined score across 8 subjects is impossible. Throw in several more unknown such as curriculum choice, national trends, a change of qualification specifications and a whole furore over an adjustment to legacy GCSE points and you have more than just a panic on your hands.

However, in my opinion it’s still sort of right to do something, even if this is simply to develop internal consistency so that schools can see, within their own school what subjects are performing well, which students are lagging behind etc… but the question remains… what and how?

There a number of hurdles to consider in forecasting P8 in 2017:

  1. Predicting grades or extrapolating predicted grades from current working at grades.

Right so any forecast needs to attempt this, schools have to do this with any future prediction they make. The fly in the ointment is that in 2017, English and maths are on the new 9-1 grading structure and although there are some signposts from Ofqual, nobody really knows how these grades will pan out and teachers are not at all experienced in knowing what these grades look like in practice.

Right so this hurdle is there, and it is known. The next two hurdles are less easy to even think about, let alone navigate.

2. The adjustment of points in 2017 for legacy GCSE qualifications.

3. The national attainment 8 estimates in 2017.

Right – to explain hurdle 2 – if you didn’t know GCSE points in legacy (A*-U) graded qualifications attract different points in 2017 than in previous years. I’ve spoke about this several times and most recently here: https://dataeducator.wordpress.com/2015/09/30/points-of-view/

Here is the conversion table for GCSEs (there are others for BTECS, AS levels etc):

pspoints

When forecasting P8 – this matters, because the previously published attainment 8 estimates from starting points (hurdle 3) were created from a points structure as shown in the 2016 points column. This obviously means that the attainment 8 estimates will change and it could be sensible to assume that they will go down in 2017.

Would this be a sensible assumption?

Progress 8 is made up of 8 subjects, English and maths, 3 Ebacc subjects and 3 Other subjects. English and maths are double weighted, so in essence 10 scores make up the progress 8, ergo also the attainment 8 estimate.

In 2017 English and maths are on the new specification 9 to 1 where a grade 9 = 9 points, grade 5 = 5 points etc. The additional grade at the top – the grade 9 will not be attained by many students. Therefore the bulk of the change is in the middle of  the structure, where for example the bottom of a C currently which is worth 5 points will become the bottom of the grade 4 (roughly) which of course is equal to 4 points.

Therefore in English and maths, overall students will be attracting fewer points, even though there is a widening of the points scale. English and maths make up at least 40% of the P8 measure.

The legacy GCSEs in 2017, Ebacc and Other, which make up 50 to 60% typically of the measure are also attracting reduced points, with only the A* grade attracting more. The bulk of grades are worth the same or less.

This almost definately (ha!) means that attainment 8 estimates will come down.

So how do we forecast P8 in 2017. We have 6 options:

Option 1 – We don’t. We accept that it’s too difficult, too many unknowns, too little help from the DfE and Ofqual, so we just don’t do it. Ofsted gleefully accept this and we all dance off into the sunset.

Option 2 – We simply plug the known adjusted points scores in 2017 into the known attainment 8 estimates from 2014, 2015 or 2016 when they become available.

Pro – We are dealing with known points scores, and known attainment 8 estimates. So we can say “based on 2017 points and 2016 estimates, we would achieve XYZ”.

Con – The progress 8 forecast would come out unfeasibly low. For many of you plugging this into your various systems at the moment, I know that this is happening – I know that you know that your forecast is low, but unless you get another method of calculation going on – you are stuck with it.

Option 3 – We adjust the points in 2017 for legacy GCSEs to better reflect the gaps between grades in line with 9-1 points. So for instance, we don’t say a grade C is worth 4, we say it’s worth 4.5, or 4.8 or some other number to account for reduction in points scores but still keeping a relationship of sorts with the 9-1 grading structure. i.e. this is to say for example that we accept that a grade C is less than a grade 5 but to account for the unknown attainment 8 estimates we use existing estimates but with increased points values to negate the effect.

Pro – We are sort of keeping a known element here (the attainment 8 estimates and trying to balance off the loss of points in a more realistic way).

Con – It’s incredibly messy to do this. I don’t like this solution even less so since I wrote it down.

Option 4 – We adjust the 2015 or 2016 attainment 8 estimates downwards across the board by a similar factor to the likely reduction in points between 2016 and 2017. Right so this means that instead of saying the attainment 8 starting point from say a fine-level 4.2 is 41 points, we will we reduce it by a factor of say 7% or 11% or some other number that we would have to magic up a statisitcal explanation for. Anyway a reduction so 4.2 = 41, becomes 4.2 = 38 or whatever. We could apply different factors to different starting points.

Pro – We are keeping 2017 points as designated by the DfE, this means we will get the school attainment 8 right – thats a known outcome. IF we get the attainment 8 points estimates reduction right we will end up with a progress 8 score close to what it will actually be.

Con – Again we are introducing an unknown, but everything has some unknowns involved so perhaps not a bad option overall

Option 5 – We don’t adjust the A8 estimates and we don’t adjust the legacy points for GCSEs. i.e. we keep grade C as a 5, grade B as a 6 etc.

Pro – For legacy GCSEs we can have consistency, we can say in 2015 or 2016 – this subject achieved XYZ attainment points and for 2017 forecasts we are comparing on the same scale, therefore we can see if the individual subject is predicting an improvement or not. Also because the exisiting A8 estimates are created on the same scales, the P8 scores should be relatively relative, although obvious Eng. and maths being on a different scale will be having an impact. Essentially though for legacy subjects you are comparing like points with estimates derived from like points.

Con – Your A8 score won’t be right, it will be over inflated.

Option 6 – The DfE do something! How about the DfE just say, “OK OK, the points and that we are keeping them, but what we will do is get some of our statisticians to model the likely impact of grade and points reform on the attainment 8 estimates”. Crucially what they will do under option 6 is come up with some ex-ante estimates (before the event) that they will then stick to after the event. Like they said when they first described progress 8 to us.

Pro – The massive, massive pro here is that schools would be able to ACCURATELY calculate their Progress 8 scores based either on predictions or on on actual results in the summer on results day or shortly after. We wouldn’t need to wait with baited breath for some new estimates to be counted. We would know where we stand in relation to a positive or negative progress 8 score.

Con – It would be extremely unlikely that national would come out at 0.00, so we would have to judge ourselves and be judged in relation to a national figure that is not 0. It might for instance be +0.03 or something. That in turn could move the floor standard to -0.47. This would be an unknown.

I can live with this con, but can the DfE?

Which leads me onto the second con. They will not do this. So forget it.

 

Option 4 or 5 for me. Will probably have a tinker about with both using the APA Pro solution (plug).

Thanks – comments more than welcome!

————————————————

EDIT – I’m adding this comment to this blog after the edudatalab report of 23/02/2016 “Don’t try to forecast Progress 8”.

The blog of course is correct, forecasting progress 8 accurately is folly. And I should add that even after 2019, we might get one semi normal year in 2020 before in 2021 we have the removal of KS2 fine-levels to contend with, leaving us with yet another year in limbo.

However, schools sort of need to demonstrate that they are making progress don’t they? They want to be confident that if their students achieve certain grades that they will be making ‘good progress’ compared to similar students.

Expected progress is dead in the water, I imagine those helpful transition matrices a lot of schools use will be discontinued. Now it’s all very well fumbling around in the dark and hoping for a good outcome, but I sort of feel the schools of this great country deserve something a bit better than the major accountability measure being deemed unmeasureable until after the event.

I look back with sadness at the the DfE secondary accountability consultation and it’s findings (particularly the statements on pages 25 & 26).

Schools are between a rock and a hard place, lambasted by some organisations for inaction, criticised for lack of strategic thinking, harpooned by others for being unable to provide accurate forecasts.

28/02/2015:

I’ve since followed this blog up with a second post: https://dataeducator.wordpress.com/2016/02/28/forecasting-progress-8-part-2/

 

Advertisements

11 thoughts on “Forecasting Progress 8 for 2017

  1. weeklydouble February 22, 2016 / 10:44 pm

    Fascinating insight into thoughts on P8, A8, new points scores and taking stabs in the dark.

  2. Andy February 23, 2016 / 12:27 pm

    Great article describing different ways of approaching the uncertain times ahead. I am currently going for Option 2, using 2014 Attainment 8 estimates, my reasoning being as follows: The national estimates will likely go down due to decreased grade values as stated above, however this may be offset slightly by schools offering a better (by DfE/P8 values) curriculum. At worst, we will be pleasantly surprised when the 2017 national data is out, rather than the other way around. I personally wouldn’t be comfortable with plucking some projected national estimates out of the air as described in Option 4 – that seems like a lot of responsibility to get it right, and what if it results in over inflated figures?

    Option 6 would be ideal, although it will never happen.

  3. JennyB February 23, 2016 / 2:00 pm

    I have gone for option 4 with one school I am working with that will have it’s first set of KS4 results in 2017 (making life really hard for them as they don’t even have previous years of data to play with!). I took the highest of the ’15 or ’16 estimates for each fine level, then figured out what that equated to per subject in terms of old GCSE and points then converted either to new GCSE grades, or reformed points (40:60 split for 2017). And then rounded up – hopefully meaning I have built in a slight security blanket. It does feel risky though, essentially the estimates drop for anyone below a fine-level of 5.0, but having run the analysis on 3 sets of data so far this year, the results churned out seem reasonable… We shall see!

  4. Tim Leunig, Chief Analyst, DfE February 24, 2016 / 12:20 am

    The Fisher Family Trust’s Chief-Wonks, Becky Allen and Mike Treadaway, have written a blog setting out why they think that neither individuals schools nor DfE should produce these predictions. They offer the following advice to schools:

    “The best way to achieve the best Progress 8 score possible is to ask a series of questions for each pupil.

    At KS4 subject choice time:
    Is this pupil taking the right subjects for them at GCSE?
    Is it reasonable to require them to take an EBacc-aligned mix of subjects?
    Are they filling as many slots as students with similar KS2?

    Throughout KS3 and KS4:
    In each subject are they making good progress and are they on track to achieve good grades?
    What does my tracking or target-setting software tell me is reasonable achievement for this subject, given their KS2 fine grade?”

    You can read their blog here: http://www.educationdatalab.org.uk/Blog/February-2016/Don%E2%80%99t-try-to-forecast-Progress-8!.aspx#.Vsz2ZvmLTcs

  5. Rebecca Allen, Education Datalab February 24, 2016 / 10:38 am

    The link to our blogpost on why schools should not try to forecast Progress 8 is here: http://www.educationdatalab.org.uk/Blog/February-2016/Don%E2%80%99t-try-to-forecast-Progress-8!.aspx#

    I think it is important that we do everything we can to discourage schools, LAs, academy chains and others from trying to do this. If asked, perhaps they should just say their cohort’s forecast Progress 8 is zero. This is the national average each year and I find it hard to believe that many schools can make a more educated guess than this.

  6. NMCK83 May 29, 2016 / 3:37 am

    Hi Peter

    In option 6 you suggest that the DfE could make a reasonable estimate for attainment 8 for 2017 taking into account the changes to the GCSE points / grades system. I agree with this and it was something I asked the DfE about in November last year. I asked if the 2015 attainment 8 estimates could be converted into their estimated 2017 equivalents. I also requested that if they could not do this would they supply the raw data (learners’ fine points scores and GCSE grades) and I would calculate the estimates myself. This was the reply I received.

    “I’m afraid that we won’t be able to provide the national attainment 8 estimates based on 2017 point scores – this is not something we have calculated and it would take a substantial amount of time to do so.

    We also can’t provide you with the raw data as this is based on the results of all schools nationally.

    We will of course provide 2017 attainment 8 estimates when the first reformed GCSEs have been sat.”

    I was wondering have you made a similar request and if so what response did you receive?

    Very shortly the GCSE results for the 2016 cohort will be known and the DfE will be calculating the attainment 8 estimates for this year. This would be an ideal time for them to, in parallel, calculate the 2016 estimates and also shadow 2017 estimates using the 2017 “new legacy” GCSE points scores. I do not know what database software is used at the DfE, or the data structure, but to calculate the 2017 attainment 8 estimates I would expect it to be simply a matter of changing a few values in look up tables so it should not be a massive task.

    I know that these estimates would still not be totally accurate as they would not take account of the new 1-9 GCSE grades or any changes in schools’ curricula and subject entry that occur with 2017 cohort however they would be more useful.

    Do you think there would be any point in requesting this of the DfE?

    • dataeducator May 29, 2016 / 8:39 am

      Hello,

      I have seen similar requests and they have come back with the same answer. They are not going to do it, however helpful it might appear to be to schools. Aside from the time reason, they wouldn’t want to do it in case they got it horribly wrong. They are only dealing in facts.

      Even mentioning predicting progress 8 appears to have become an unspeakable phrase at the moment and the organisations that govern data in education are all singing from the same hymn sheet, “concentrate on attainment 8”, “teach every child as well as you can”. Well… Gee thanks.

      The incredulity of these organisations is what makes me cross to be honest, they find it incredible that a school would wish to have an indication of whether the progress they were making, or have made immediately after results are published, is strong or weak compared to national.

      All of this I accept really, but the indignation of it grinds my gears, because after all this is how Progress 8 was sold to us. We will provide estimates in advance they said, everyone agreed, nodding dogs everywhere. It was blatantly apparent at this point that there would be difficulties with new spec GCSEs, shifting baselines and the like, no matter they said, you will know in advance what your estimates will be.

      A massive mis-sell. It was sugar coated from the start, everyone knew of the difficulties, the changes in the pipeline.

      So back to the question, no. It’s not just the DfE that could do it though, FFT could do it, anyone with access to the national database could do it. But I don’t fancy the chances.

  7. Andrea Whiting November 13, 2016 / 6:39 pm

    Thank you for a really useful blog. At Go 4 Schools we are calculating the P8 for schools using the 2017 points and the 2016 estimates; as you suggest schools are coming back to us as their Progress 8 based on target data is very low; on average FFT20 is -0.4. This is of course a concern to schools as they feel like they understand their cohort and have set challenging targets but the Progress 8 is so low.

    We are trying to advise and support schools; the ASCL advice at their recent conference was extremely useful, so using this as a template we are directing schools to use the Attainment 8 data; which is accurate, looking at the score for this in each of the slots Maths, English, EBACC and Other, then track the residual between their targets and the forecast/current grade.

    I am extremely worried that we have been inundated with schools requesting a Progress 8 score per subject. As your blog options outline the Pro’s can Con’s we looked at all methodology for this and realise that it becomes another statistic that in incorrect and not really helpful to schools and could in fact be totally misleading. The Ebacc slot could be divided by 3 or by the national average of filled slots, neither of which could match the schools curriculum offer.

    As we are in the business of supporting schools to analyse their data and producing National performance measures for schools we have to compete in the market with other providers, but also ensure we are supplying data that is statistically accurate and clearly communicated to schools the coefficients in use.

    We will be adding headline Average Numeric points per subject, so this can be used to again track the residual target vs progress grade. Hopefully this will support data tracking for students to make decisions regarding interventions for focus groups.

    Schools can control their decisions to ensure their student’s outcomes are at the highest level but have no control over National Performance and that of other schools, which is essentially what the P8 is showing. We know the DfE will not produce data prior to the actual examination performance statistics.

    I will be directing schools to your blog and have re-tweeted it. So many SLT in schools are just looking for some type of guidance regarding the changing face of their school data. As your second blog verifies – you have 6 options for the Progress 8 prediction but I have seen many more that are a concern.

    • Robert Taylor November 18, 2016 / 10:01 am

      In our school what we have done to guess at our 2017 Progress 8 number is:
      Align the 2017 points for En and Ma against 2016 points using 1=1, 4=3 5=4 and 7=7 as fixed points and align the 2017 points for all other subjects using the published table.
      We used the 2016 Attainment 8 table for Ma, En, Ebac and Open as a starting point to translate the 2016 Attainment 8 table into 2017 points for Ma, En, Ebac and Open. Then combined the four to make a 2017 Attainment 8 guess.

      The KS2 to 2017 points looks like this:

      KS2 2017 A8 Guess
      1.5 1.148
      2 1.538
      2.5 1.594
      2.8 1.663
      2.9 1.818
      3 1.858
      3.1 1.927
      3.2 2.043
      3.3 2.139
      3.4 2.192
      3.5 2.307
      3.6 2.407
      3.7 2.507
      3.8 2.625
      3.9 2.778
      4 2.883
      4.1 3.018
      4.2 3.187
      4.3 3.380
      4.4 3.525
      4.5 3.716
      4.6 3.961
      4.7 4.233
      4.8 4.556
      4.9 4.773
      5 5.040
      5.1 5.365
      5.2 5.690
      5.3 6.003
      5.4 6.362
      5.5 6.720
      5.6 7.165
      5.7 7.530
      5.8 7.835

      I will be interesting to see just how far away from reality we are come next October.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s