Evaluation, statistics and OR

Alternatives to a Census

The New Local Government Network want all work on the 2011 Census scrapped as they think they have found a better way to measure population that won't cause errors.  They think that taking a mix of administrative data on GP practice rolls, school places, electoral registers and tax rolls will do the job for them.  Normally I have a lot of respect for local authority research people as they have a lot of knowledge and experience and understand the holes in official data sources.  They are also generally a very dedicated lot and don't move on to political roles, but stay in the background making sure things tick over.  I can only assume mavericks got involved in this.

GP practice rolls are totally fallible and will not solve the Slough problem that the NLGN keeps going on about.  Research has shown that migrant workers, young men aged 18 to 30 and new immigrants are unlikely to register with a GP, but will use A&E services if necessary.  The males aged 18 to 30 issue has always been a problem and was particularly highlighted in the days of the poll tax, when it was looked at as another way of finding "missing" people for tax purposes.  Plus do you trust your local surgery to get all your details right and to not pass on things you don't want the Government to know?

School places - not everyone has children and not all children go to schools that this register would cover.  What about foreign nationals coming to UK schools?  What about the independent sector, who again may not collect or be required to supply all the data required.  Nominal numbers of school places are also an issue, as local authorities get into real hot water if they try to close classes or schools.

Electoral registers - the poll tax debacle caused people to remove themselves from the electoral roll.  I have had periods in my life when I have either been on no electoral rolls or on two or three different ones - none of it by my own volition I hasten to add.  Sharp practices in Birmingham and postal votes have also shown the potential problems with this.

And finally tax records.  Where is the Information Commissioner on this one?  Again, the tax office doesn't collect all the data you need for a useful population count (qualifications for example).  I for one would not be happy that data that I freely give to one source for one purpose is linked to data that I give for another purpose to another source.  People go on about civil liberties, but this sort of project, with implicit requirements for huge amounts of fuzzy matching, exceeds all previous Big Brother attempts.

If you don't like the traditional Census, try again to find something better.  This does not seem adequate to me.

That A level time of year - again!

And I can't not comment!  But once again what do you say?  With over 97% of exams passed, is the A level really the exam you cannot fail?  On the one hand, you don't want to do down those whose results have come out today, on the other hand, what do you say to those of us with low A level grades from over 20 years ago?

Was I really so terminally thick that I failed my English A level back in 1981 - or if the CEM Centre grade inflation is to be believed, my O at A level would actually get me in today's currency a nice little D or maybe even a C?

The government is apparently putting in place measures to make A levels more difficult from September (bad luck those of you starting your A level courses in 2008!), but will this make much difference?  Can you imagine a politician standing up and saying that the pass rate has gone down now exams are more challenging and that this is a good thing?  Not in a million years!

Those in academic circles know that standards are falling in A levels - behind closed doors it is debated, but never published and to use the phrase "dumbing down" gets you bullied, dismissed or your contract not renewed.  The CEM Centre research quoted by the CBI earlier this week will be buried - which is a shame as these guys really know their stuff.

However when there is an attempt to re-base by changing the qualification structure, everyone seems to go to jelly.  Tomlinson was reviled for his outstanding work on qualifications, and if the Diplomas are the back way to implementing his vision, so much the better.

The Daily Telegraph is perhaps not the best place to look for rational and impartial debate on this topic.  I looked at the comment board on the A levels this morning and sincerely hope that Becky who commented at 8:16am as one of this year's breed of all A students wrote firmly tongue in cheek.  Otherwise her examiner and her teachers should be sent to prison for stealing state money - they haven't educated her, she wasn't literate. 

The columnist whose article got ripped apart was trying to make a decent point however.  We can argue all we like about the standards at A level, but what about the depressingly large proportion of 14 year olds who lack literacy and numeracy skills?  What about those who leave school with nothing, or worse leave school with one G at GCSE and are therefore not eligible for some forms of support as they have a (useless) qualification?  If nothing else, doing A level study equips you with a few skills - not least being able to access the Daily Telegraph comments board - but to leave with no skills is a waste of opportunity.

Contextual Value Added - Why bother?

Ofsted has come out of the closet on its views of publishing contextual value added scores and their use - albeit that their views seem to be something of a curate's egg, and need just as much careful interpretation as the actual CVA figures themselves.

Contextual Added Value (CVA) is the latest measure to show how well schools are serving their clients (sorry, pupils), and takes into account prior attainment (added value) and the factors in the local environment that might affect performance but are outside the school's control such as poverty or ethnicity of the local population.  This is all mulched up together (good statistical term that) to give a score normalised around 1000 and with a confidence interval - which is generally not so well published as the basic score.

Ofsted has given out guidance on interpretation, for use by schools, and has said which bits of what is published should be used and which bits are meaningless - this BBC summary is quite good on this.

My gripe is this: government would like parents and guardians to have information available on which they can base their choices for their children, and later in life, similar information available to potential students so they too can choose the best route for their future.  However, with literacy and particularly numeracy levels at a low ebb in England, how are people expected to interpret CVA information?  Yet it is this information that will be supplied to people to make choices on.

In my view, these figures are not fit for public use, and therefore it is questionable whether there is any value in putting them in the public domain.  If an absolute score has a confidence interval of 10 which could take it above or below the national average, all people need to know about an institution is that it is average.  Anything more and the number is being vested with spurious accuracy, giving people lots of opportunities to make the wrong decision, as they don't understand numbers.

There is a similar link to the publication of medical findings that are too statistically complex for 99.9% of the population to understand, so now we are faced with a measles epidemic in the UK due to people panicking.

I think the "Tiger that Isn't" should be compulsory reading for everyone in the country - but that assumes a decent level of literacy too!