How to set grade boundaries in the new number grade world


Setting grade boundaries is very difficult. In reality, even the actual published grade boundaries for a specific exam paper may not really be valid for a different cohort.

Add in the changes to both the content and the grading systems for most GCSEs, and we have a very complex situation.

Some important points

Internal consistency

If the marking is accurate and has been moderated, it should be internally consistent. I.e. the students who gets the most marks has the best attainment.

This means you will always be able to compare the attainment of individuals and groups within the subject.

External pegging

The problem comes when you try to link a set of marks to an actual external real life GCSE (or other) grade.

There are some assumptions we can make that will help

  1. We know the grade equivalencies. The bottom of a grade 4 is the same as the bottom of a grade C.  the bottom of a 7 is the same as the bottom of an A, and the bottom of a G is the same as the bottom of a 1.
  2. Equivalent outcomes – grade boundaries are being set so similar proportions of students achieve each grade as in previous years
  3. Our marking is internally consistent
  4. Our students are likely to achieve similar outcomes to previous years.

Methods

Ranking

Put all the grades in descending order

Find the attainment percentages for last year in your subject (https://schoolsweek.co.uk/gcse-results-2017-uk-subject-tables/ for national data)

Use the previous attainment data to work out how many students are likely to achieve an A grade or better.

Count down from the top until you reach that number

The mark that student achieved is a good candidate mark for the Grade 7 boundary.

Repeat for the number of student achieving a C or better, to get the Grade 4 boundary.

Repeat again for the number of students achieving a grade, this would be the Grade 1 boundary.

You can then look at individual grade proportions to split out the 2,3,5,6 and 9 boundaries, or make a logical spread throughout the marks.  e.g. if you find that the Grade 4 was at 30 marks and the Grade 7 was at 45 marks, you could put the 6 in at 40 and the Grade 5 at 35.

Some points to consider

This is not very far removed from how the exam boards will set their grades.  They will just have a lot more numbers to play with.

There is no such thing as a completely accurate grade boundary. The best you can do is get a good idea of where a student is compared to the rest of the school and to the exact exam paper.

Grades generated this way would be projections, ie the grade that the test indicates they will achieve in the real exam.  To convert to an actual Working At Grade you will need to raise the boundaries slightly.

 

Use old style exam boundaries

The old exam boundaries are likely to be significantly higher than the new ones due to the changes in challenge presented by the new courses.

For example, in Edexcel Mathematics

Grade 2016 2017
Higher Foundation Higher Foundation
A/7 70% 52%
C/4 35% 71% 17% 51%
G/1 31% 11%

 

English is not as easy to compare as that has shifted from tiered to a single paper, but by looking at the single non-tiered paper 3;

Grade 2016 2017
A/7 90% 65%
C/4 59% 45%
G/1 13% 10%

 

Typically, the percentage of marks needed to achieve a grade have shifted down by over 15 percentage point at most grades.

This is a huge difference compared to the normal drift of grade boundaries shifting each year.  For maths, this has been in a range of about 6 percentage points (eg a C in maths has had a range from 29% to 35% from 2014 to 2016).

This being the case, shifting grade boundaries down by something in the region of 10 percentage points would not be unreasonable.

Make up grade boundaries based on gut feeling

Don’t

In some cases, this may seem like all you can do, but in reality you will effectively be using a ranking system without necessarily doing it formally.  The advantage of doing it formally is that you can discuss how you got to the grade without having to say you made it up.

Hybrid

  1. Set grade boundaries using an adjustment (+/- fudge factor) on previous grade boundaries.
  2. Rank order the students and see if the grades look right; ie the students are getting the sort of grades you expect based on their work in lesson.
  3. Adjust the boundaries until the results look right. NB this is not adjust the boundaries until the results look good
  4. Review the boundaries and makes sure they are not too far away from what has happened previously

 

A hybrid method is probably the best approach as it makes use of all the sources of information and combines them to give you something consistent and explicable.

 

Worked Example for Physics

Old Style GCSE grade boundaries (converted to number grades)

Grade Old GCSE %
8 75
7 60
6 45
5 38
4 31
3 17

 

We are pretty sure they are too high, but how much too high?

An adjustment of 10% seems reasonable based on the maths data

Grade Old GCSE % Adjusted
8 75 65
7 60 50
6 45 35
5 38 28
4 31 21
3 17 7

 

 

So looking at historical grade distribution for our school (we want to do better than this, but it’s a good place to start)

We should have in the region of

Historical % of students achieving that grade Number of Students in current cohort
8 10 4
7 22 9
6 30 13
4 28 12
3 10 4

 

Running through the results in order, the top 4 students should get an 8.  The lowest mark from the top 4 students was 60%.  So that makes a good grade boundary for the grade 8.  Then another 9 students takes us to 42%, so a good boundary for the 7.  And so on to find suitable boundaries for the rest of the grades.

Grade Old GCSE based on similar proportions
8 75 60
7 60 40
6 45 33
5 38 30
4 31 20
3 17 17

 

Taking all those together we can derive an reasonable ‘final’ grade boundary based on both approaches.

Grade Old GCSE based on similar proportions Adjusted Old GCSE Final
8 75 60 65 65
7 60 40 50 45
6 45 33 35 37
5 38 30 28 33
4 31 20 21 25
3 17 17 7 15

 

Plugging these into the class results gives a set of data that looks about right, i.e. the distribution is slightly worse than students achieved last year, so we are probably not making it too easy, but they are not so much worse that we can’t work out who to target.

It is important to remember the point behind grading tests at this stage – we need to work out who we need to work with to get help them achieve their potential.

Advertisements

Are selective schools the end of civilisation as we know it?

First of all, I need to be very clear here, I am horrified that the government is even considering going back 50 year and reintroducing selection.  It goes against every instinct and makes me angry.

That being said there may actually be a way of making it work.

I first started teaching in grammar schools and then moved to the non-selective schools around them, so I have some experience to draw on here, but I am drawing in quite broad strokes with a healthy chunk of generalisation.

What are the advantages of going to grammar school?  As far as I can see they are

  • Better resources – well off areas parents will chuck money at the PTA as it is a cheap alternative to private school
  • Better behaviour – as a rule the kids who work hard to pass the 11+ are not the kids who kick off in lessons, for a huge number of reasons
  • Umm, that’s about it.

Any upgrades in outcomes will tend to flow from the first two points – basically if you can chuck time (less disruption in lessons) and money (more funds to buy cool stuff) at a problem (exam results) the problem gets solved.  Happier teachers, less disruption, students who are used to working and or have parental support makes a winning formula.

At the moment in selective areas grammar schools can outbid anyone else to get specialist teachers in every class.  These teachers don’t tend to be very mobile once in post and will continue to get a good set of grades year after year.

This may not be the best way of making sure that the kids are getting the best teaching.  It may be better than the kids get in the non selective school as those schools are a much more difficult prospect.

It is far more difficult to be effective in the non-selective school because

  • The kids have already been told they are failures
  • There is an increased concentration of difficult students

This leads to rapid staff turnover, which leads to worse outcomes, which leads to staff giving up and leaving and round we go again.

I have to say I am looking forward to seeing Progress 8 based performance tables.

But now, a possible route to make selection work for everybody, and it makes use of the free market economics so beloved of our Tory overlords.

We need to stop the selective schools taking all the best teachers and all the spare cash.

With the collapse of national pay scales and the rise of MATs we can make sure we get the best teachers in the non-selective schools by the simple expedient of paying them more.

Give them all the equivalent of London weighting and you will get staff that want to work there, which means you can recruit better teachers.   If you make selective and non-selective schools work in pairs (or in MATs) you can balance out the pay structure so you have the choice between easy and lower pay scale (selective) or bloody hard work and higher scale (non selective).

Make any PTA/alumini/other donations to the school (well, MAT) go to both,  even if you think you are donating to your posh old boys network.

Take it further – make MATs work locally and have a network of different types of schools for different types of pupil – lets have more UTC type schools in the mix.

You know what, all that is achieving is segregating by class.  But it’s a better idea than just introducing selection for any (all?) schools that want it.

Computer Science reaching for the top of the taxonomy

After a year of feeling like I am bashing my head against a wall… a moment of realisation.

I have been trying to teach new spec GCSE computer science to a group of students who are really struggling.  This has of course led to lessons that are frankly hell on Earth for all of us.  The students are stuck and thus misbehaving, but I have not been able to find the simplification to make it workable.

This afternoon I looked at a slightly different presentation of Blooms taxonomy than the one I had by my desk for years.  Queue Giraffe Moment1.

The top of the tree, the highest level of thought and abstraction is ‘creating’.  This is what we aspire to achieve with our best students, as we move them towards more abstract and independent thought.

Unfortunately for me and my students, it is also the basis of the programming aspects of computer science.

You cannot program unless you can turn a concrete description of a process into a highly abstract representation of that process. You need to assemble/construct/create/design/develop/write an algorithm.

This is a huge ask for anybody, but for a fourteen-year-old who is struggling to get to grips with maths and English it is possibly a step beyond.   Add in the requirement for resilience to deal with the whole ‘fail better’ approach to developing programs and you have a situation that can lead to failing totally.

From where I am sitting there is not much concrete to build up to that creative level from.

Just to take a quick look at learning to write computer programs from a Bloom’s point of view.

Remembering – being able to define key terms such as sequence, selection, iteration, assignment, variables, conditions and so on (probably not many more than that to be honest)

Understanding – being able to explain what the purpose is of those constructs and why you would use them. Being able to understand an algorithm that has been given to you.

Applying – choosing different constructs to solve specific problems e.g. – would you use a for loop or a while loop in this situation.

Analyse – be able to identify different approaches used in an algorithm

Evaluate – be able to justify a choice of approaches used in an algorithm

And then Create as mentioned above

In what other subject are you expected as pretty much the fundamental task to be able to create something as complex as an algorithm to solve a problem.  I can only think of composition in music as any sort of comparison, and even then if its not good it still does something.

I have read previously about the problem of shifting students from pile of Lego stage (knowing there are these different constructs that you can put together to solve problems, see https://wordpress.com/read/feeds/1141878/posts/936233556 )to the creative stage, but it had not sunk in why it was so difficult to make that leap.

Teaching students how to write a loop or another construct is fairly straight forward.  It is just learning a formula that you can apply.  Then we ask students to take their toolkit and try and solve big problems (e.g. write a program to generate a sequence of random maths questions, store the results in a file, have separate files for each class, keep only the top 3 grades per student and be able to present the results alphabetically, by score and by average score…from the 2015 controlled assessment)

If we could just work with understanding algorithms and programs we would be able to build up ideas and go somewhere.  Unfortunately, this is not enough for the computer science GCSE.  With a 20-hour practical exam (the practical exam from hell­2) to fit in and mark before May, that is at least one term down.  So you are left with about 40 weeks to learn to program, oh and to study and learn the other 66% of the content for the exam.

So how are we going to achieve enlightenment?  At the moment I am stuck.  I really don’t think ignoring the non-examined component is any sort of viable route, but I am also really struggling to see a route from concrete to creative and resilient.  A plan is required and anybody who has a good idea… please let me know.

 

1 Giraffe Moment needs some explanation.

Everybody knows what a giraffe looks like.  When you were little you were probably surrounded with cuddly giraffes, cartoon giraffes (and elephants, and lions and tigers and…).   You are so used to the image of a giraffe you have forgotten how utterly strange they are.  Go to the zoo and look at one with fresh eyes.  Look at that neck, the tongue, the mushroom shaped horns.  What a weird beast.  This happens a lot, something that is amazing becomes commonplace and you stop noticing what it really is.  The giraffe moment is when you look at again and realise that it is seriously weird or interesting and it is ripe for a re-examination.

2 The practical exam from hell

A substantial chunk of the new GCSE is the non-examined component, which simply put consists of giving a student a problem, and then telling them to get on and solve it.  They have 20 hours in exam conditions and no access to anything apart from the offline help files in the IDE (if any).  So no help from books, internet, teacher or peers.  That is quite scary for me as a fairly competent programmer, for a slightly lost student it is an absolute terror.

Reconciling with the knowledge based curriculum

For the last few months I have been struggling to remind myself why I want to teach, and what education really is (see ‘remembering stuff as education’).

I think I may have found my way through.

The remembering stuff is the end effect, not the point, of education.

Headguruteacher (https://wordpress.com/read/feeds/2402438/posts/1020119340) has it right when he talks about the mistakes in the mental model.

When it comes down to it, the mental model is the knowledge we want students to have. It is the model that we need to be learning by rote, not the answers.

The word equations for respiration, photosynthesis and combustion are all coming from the same root model – using oxygen to break bonds and release energy. If we have learnt that model, then rest is easy.

There has been a tendency in Physics textbooks to present every derived version of a formula as a whole new chunk of knowledge to be leant. This is a bad idea.  It makes pattern matching easier, and removes the need for understanding.

I struggle a little more to apply this outside of my fields of expertise (how useful is it to learn by rote ‘range of causes of the Renaissance, the principal features and their outcomes’ in year 7, and what sort of mental model you could use to make that easier is beyond me).

So with my thinking reset, the controlled assessment winging its way to the examiner and the sun shining…..time to start working out new ways of making this stuff work.

What does expected progress actually mean?

The arrival of Progress 8 may well have revealed a massive hole in the way we measure students’ progress.

For the past twenty years we have been expecting all students to make 3 levels of progress from KS2 to KS3. This has gradually been stretched to the top students being expected to achieve 4 levels of progress.

At least one academy chain has set a target of 80% of students achieving 3 levels of progress, and 40% achieving 4 levels.

This is all very nice and expects students to do really well.

But what do we mean by ‘expected’

A quick dictionary definition

regard (something) as likely to happen.

So the likely outcome for any given student is to make 3 levels of progress.

Some will make more, some will make less.

80% of students making that level is a pretty high hurdle, but most students achieving what is likely to happen. I can go with that.

40% achieving more than the most likely. That starts looking quite challenging, but let’s go for it.

But hold on, where did this 3 levels of progress being likely come from?

I assume that it was built into the design of levels and GCSE grades, the research took years and costs millions to get those levels set up, so one would hope that they do the job.

Then a big spanner falls into the complexities of guessing how much progress students should make.

Progress8 has shown us the progress that students do actually make (at least the average of the average of their grades with some complexities about what counts where)

What does that tell us then?

 

Average fine level 2015 Attainment 8 estimate Average levels of progress
1.5 13.01 2.0
2 17.57 1.8
2.5 19.14 1.4
2.8 20.11 1.3
2.9 21.62 1.3
3.0 23.08 1.3
3.1 23.65 1.2
3.2 24.16 1.3
3.3 25.88 1.3
3.4 27.42 1.3
3.5 28.27 1.3
3.6 29.95 1.4
3.7 31.81 1.5
3.8 33.33 1.5
3.9 34.98 1.6
4.0 36.12 1.7
4.1 38.45 1.7
4.2 40.30 1.8
4.3 42.24 1.9
4.4 44.15 2.0
4.5 45.52 2.1
4.6 48.20 2.3
4.7 50.43 2.4
4.8 52.48 2.5
4.9 54.77 2.6
5.0 55.88 2.7
5.1 59.07 2.9
5.2 61.28 3.0
5.3 63.70 3.1
5.4 66.12 3.3
5.5 65.46 3.5
5.6 71.42 3.6
5.7 74.00 3.8
5.8 76.14 3.8

 

 

Only those students achieving an average level of a 5 or more at KS2 are making anything like 3 levels of progress on average.

That’s only the HAPs (high Achieving Pupils) making expected progress, only the HAPs achieving ‘what is likely’.

The average student (level 4 at KS2) only makes an average of 2 levels of progress.

Now the same calculations using the recently published (but soon to be abandoned) Value Added coefficients.

 

KS2 Average fine level Levels of Progress
P8 VA
3.1 1.5 1.3
3.2 1.4 1.3
3.3 1.5 1.4
3.4 1.5 1.5
3.5 1.5 1.5
3.6 1.6 1.6
3.7 1.7 1.7
3.8 1.7 1.8
3.9 1.7 1.9
4 1.8 1.9
4.1 1.9 2.0
4.2 1.9 2.0
4.3 2.0 2.1
4.4 2.1 2.2
4.5 2.1 2.2
4.6 2.2 2.3
4.7 2.3 2.4
4.8 2.4 2.5
4.9 2.5 2.5
5 2.6 2.6
5.1 2.8 2.7
5.2 2.9 2.9
5.3 3.0 3.0
5.4 3.2 3.1
5.5 3.0 3.0

Oh look, nowhere near 3 levels of progress unless you start on a level 5.

It’s all very nice to have high expectations of our students, and try to push them top achieve, but this looks more and more like those in charge of education not understanding numbers, they want everybody to be above average.

Talking of which…

A recent discussion with an OFSTED inspector indicates that a P8 score of 0 is always going to be ‘Requires Improvement’. This means every school that is average or below is in trouble.  That is going to be around 50% of schools.

How long is it going to take before Ofsted inspectors get their collective heads around these changes, expecting the average progress of students to be significantly above average makes a mockery of the whole system.

 

DFE replacement for Levels announced

The DFE have finally announced their alternative to levels.

These will be the measures used to track progress from summer 2016 onwards, so we really need to make sure we can back translate our progress from our current system to the new system.

The new system is based on a student’s progress through Age Related Expectations.

Age Related Expectations are given a rating for each year group, and for each subject (see table below)

So a student achieving Year 7 ARE in English will report that grade, whatever chronological school year they are actually in.

For progress grades go in single steps from the colour Magenta to the smell of freshly baked baps.

Attainment grades go in half steps (ie each step is worth half a point of progress) from Blancmange to the sound of a cantankerous Armadillo.

So to take a more definite example

A student in year 8, and has achieved above expectation in Maths. This student is therefore a small green llama, but may only have achieved ‘that slight feeling of nervousness caused by the visit of a distant relative’ in Maths.  Therefore they are not yet making ARE.

I think you will agree this is a far more useful system than has existed previously.

 

Memorising stuff as education

This really worries me.

I realise that this has become the focus of the post Gove education system, but…..is learning and regurgitating a list of stuff of any use to anybody in any way (apart from passing new style GCSEs).

And this is the root of the ‘I never needed anything I learnt at school’ attitude.

My own experience

Give me a list of things to remember, and I am utterly stuffed.

I went to school and did the last year of O levels and the first year of GCSEs. Memorisation was high on the list of desirable outcomes of education.

I could not do it.

At primary school I was treated as some kind of idiot-savant by my teachers. Give me a mathematical problem – it was solved, give me anything that required recall, and I couldn’t do it.

Times tables – not a hope. Embarrassing fact – I can only recite times tables by calculating each step.

I was kept back a year at primary school because I was so slow.

At secondary school the impact of my nice middle class background got me into the A stream despite my inability to recite the kings and queens of England or the longest rivers or the countries in Africa (streaming – dividing the school into 3 groups A who were going to do O levels, B who were going to do CSEs, C who were not).

My salvation was discovering that I could do science by actually understanding what was going on, rather than trying to remember everything. This didn’t always help – I never understood (possibly was never taught) the basics of chemistry, and could never remember enough of the list of facts.

I achieved mediocre GCSEs.

A level was better as understanding and applying stuff became a larger part of the course. There was still a lot of memorising of formulae and the like, but not too bad.

I achieved mediocre A levels, but managed to scrape a place to read Biology at university. This was a bit of a shock for a lot of people as they thought I was thick because I couldn’t memorise stuff.

Doing a degree was easy. You actually needed to understand stuff, and memorising lists was a waste of brain space.

PhD work was even more about understanding – if you needed to know a specific number, you looked it up or wrote it down. What sort of person would rely on their fallible memory to make sure they got their calculations correct?

 

Observations as a teacher

Is the student who can learn by rote without understanding a better product of education than one who can understand but cannot regurgitate rote learning?

I am sure I am not the only teacher who has had many students who worked hard to learn everything for GCSE and achieved excellent grades by being able to recall lists of facts and statements that can be strung together to build answers.

Have you then seen these same students fall apart at A level?

It maybe because I teach Physics, where the amount of stuff to learn is relatively small, but it is all in the application, but I have had GCSE students with a clutch of A and A* grades utterly unable to even approach A level questions, because they don’t know the answer.

What they don’t grasp is you are not supposed to know the answer, you are supposed to work it out.

I used to work in what amounted to a ‘secondary modern’. All the kids had failed the 11+ and been sent to the comprehensive rather than to one of the lovely grammar schools.

The top sets in maths and science were stuffed with students who were not stupid, but could not do the rote learning and literacy tasks in the 11+. They went on to achieve great things at A level and beyond, because they had learnt to understand, not just regurgitate.

 

When I started teaching at the turn of the century I was excited by the changes in education. Students were expected to understand stuff, they had formula sheets to remove the need for parrot learning.  They used computers to plot graphs.  They were tested on how they thought, not just how well they remembered stuff.  This was amazing, and I wanted to be part of it.

I was so excited by the changes in science that meant you needed to actually understand scientific thinking to succeed.

You needed to know why the elements in the periodic table formed certain bonds, rather than being able to list the first 20 elements in order.

In the same way that we improved our education by sharing knowledge in books so each generation didn’t start again from scratch, we had education building on what previous generations had learnt, we had the potential for accelerating understanding of the world. Computers (and books) could do all the memorising and humans could do all the imagination, insight and analysis. The future looked amazing.

And then it didn’t.

I am really worried for our future if the only people who can get decent GCSEs and then A levels and then degrees are those who can memorise lists of stuff.