| Recently Visited | ||||
|
||||
 |
Bouncing tables
2011-03-04 08:03:31
3 March 2011 Last updated at 12:16 Share this page Delicious Digg Facebook reddit StumbleUpon Twitter Email Print Go Figure: The bouncing league table By Michael Blastland GO FIGURE - Seeing stats in a different way Continue reading the main story In today's Magazine The cult of lawn The end of the Asbo era Living with a Nazi name Why are fountain pen sales rising?
League tables that don't include uncertainty do not make it easy to spot a good school, Michael Blastland says in his regular column.
One year a school brims with eager bright sparks; the next, an excess of strugglers and dossers.
The school goes up and down the league. But how good is it really? Ups and downs in pupil ability make underlying standards of teaching hard to see.
Statisticians can actually measure the degree of uncertainty around this underlying quality, though this is seldom reported.
The animation suggests how it might look.
Continue reading the main story #slt { padding-bottom: 10px; } To see the enhanced content on this page, you need to have JavaScript enabled and Adobe Flash installed. (function() { var glow; gloader.load( ["glow", "1", "glow.dom"], { onLoad: function(fetchedGlow) { glow = fetchedGlow; glow.ready(init); } } ); function init() { new glow.embed.Flash( "/news/special/uk/school_league_flash/swf/school_league_blast_464.swf", "div#slt", "9", { width: "465px", height: "490px" }).embed(); } })();Each school is pictured bouncing around within its range of uncertainty to show what league tables might look like if they took uncertainty seriously. Think of it as showing the variety of league-table positions any school might achieve - given constant teaching quality but different batches of pupils.
This range usually includes - somewhere, though we don't know where - the real, underlying performance.*
You might have picked the school that seemed best, but maybe its quality was flattered by a dazzling year group. Maybe when your child came to GCSEs, the school you chose did less well, or maybe better.
Continue reading the main story “Start QuoteIn the real world, there's a lot of fuzziness about - and it's often ignored”
End QuoteIn fact, I've rigged the animation so that any of the five schools here could be the best, or worst. Some are a little more likely to be better than others, but we can't be sure. In the event, someone probably gets lucky with a good intake. Maybe you mistook that luck for quality - which turned out not to be so high.
By the way, we haven't let the schools just bounce around anywhere. Each has a particular uncertainty interval around it.
There's more uncertainty for small schools where a few high/low-achieving pupils can bring a big difference to the results - but make the underlying teaching standard harder to see.
And there's less uncertainty around big schools, which is why Clots Comp (small) bounces around a lot with plenty of uncertainty, and why Brookside (big) doesn't bounce so much. We're a little more confident that the Brookside results are closer to representing the underlying quality of the school.
Exam results are not the only measurementJust to be clear, in our league table, the real quality of each school never varies. We're just not sure what that quality is. And of course, we're only talking about quality in terms of achieving a good position in the leagues. Some will want much more from a school than that.
There are other ways of showing uncertainty graphically, but not many. One is the funnel plot (pdf), while others use error bars.
Some use coloured bands of uncertainty around a central estimate, as the Bank of England does for its inflation and GDP forecasts (pdf).
We've used animation to try to overcome the problem that graphs often make things appear concrete, league tables too, and to take away any fixed point to use as a mental anchor. The idea has its weaknesses, no doubt. But in the real world, there's a lot of fuzziness about - and it's often ignored.
We can make judgements about schools, indeed we have to, and often, quite rightly, we use whatever information we can lay our hands on, exam results and all. But what if we routinely tried to show the fuzziness? What if we made a point of proclaiming uncertainty rather than brushing it aside?
Mind you, you can't ever see it in Premier League football.
*Statisticians talk of 95% confidence intervals, meaning that if we created confidence intervals around a school's results and our method is sound, these intervals will include the true, underlying performance 95% of the time. There is uncertainty around all educational performance tables, whether raw, value-added, or whatever, as there would be around hospital league tables.
A selection of your comments is published below.
As a teacher and mother, I would far rather only see value-added tables for schools - no non-selective school has any say in who is admitted, but it's how they are helped whilst there that counts. What parents do not realise is that if their child is likely to do well academically, they could very well send them to the local 'sink' school (and here I am talking in terms of reputation only) where the pupils will be streamed and the top performing sets are likely to have very few pupils in them - why pay for small classes when they are available free?
We were talking about this at the Cheshire East Association of Governing Bodies committee last night: what everyone forgets that 'achievement' is, in education, learner-centred, not a matter of absolutes. What for one child is a trivial accomplishment is for another a massive achievement, because all our children differ in ability.
This is just the sort of analysis we need to show that stats need to be qualified. Of course this is an inconvenient truth because people want yes/no answers so they don't have to deal with complexity. The paragon of virtue may never drink or smoke and eat endless fruit and veg and die in middle age whereas the red meat eating alcoholic smoker may live to 90. The "best" school "on average" may fail your child and in reality your child is the only pupil you care about!
At last, an article on the stupidity of school league tables. If the DfE published and explained the uncertainty bars, then is would show how meaningless the figures really are. A study of variation from year to year would also show how meaningless they are, as the batches of pupils (cohorts in the jargon) vary. However it suits the government (both this and the last) to have these so-called statistics, and most people are too mathematically ill-informed to see the problems Thank you for pointing them out. Yet again I am reminded of the quote from Andrew Lang: "He used statistics as a drunken man uses a lamp-post - for support rather than illumination"
When my daughter was studying for her GCSEs, her comprehensive's A-level results were okay but not impressive. So, like other parents, we moved her for 6th form studies. Two years later, despite having lost some of their best students, the school's A-level results were their best for some ten years! Why? The school was shaken out of its complacency and took the necessary steps to prevent such pupil loss in the future.
Any teacher has experienced the inescapable fact that each year the one big variable is the student intake. One tear I had a "dream Team - highly intelligent and motivated. Great for me, and fabulous "A" Level results, but I knew I'd be criticised the next year when I knew results would be below average whatever we did. This isn't new, but the emphasis on league tables makes results seem more important despite their statistical doubtfulness. As a paranoid science teacher I just KNEW that: Good results = good students, and twas ever thus. Glad I have retired.
Sounds like a good idea, although it still has a weakness in that it presumes that the pupils that end up at each school is random each year. This assumption breaks down if pupils that are already likely to do well tend towards certain schools - in which case you could get an average school but with a consistently good intake of pupils high in the league tables (at least in absolute grades).
School league tables simply reflect the social background of catchment areas. Schools in wealthy areas do well while schools in less well-off areas struggle in the tables. I think it would be more relevant to compare schools with similar social backgrounds. I would split schools into Divisions according to their social background. So, for example, you could have a division of schools from areas of multiple deprivation or a division of schools of a rural character. Headteachers and schools would benefit from this fairer comparison and they could learn how other schools of a similar nature overcome challenges. The tools currently exist for this kind of Geo-demographic segmentation, so it would be a very cheap exercise to undertake and would deliver significant value to the whole education community.