Evaluation, statistics and OR

Alternatives to a Census

The New Local Government Network want all work on the 2011 Census scrapped as they think they have found a better way to measure population that won't cause errors.  They think that taking a mix of administrative data on GP practice rolls, school places, electoral registers and tax rolls will do the job for them.  Normally I have a lot of respect for local authority research people as they have a lot of knowledge and experience and understand the holes in official data sources.  They are also generally a very dedicated lot and don't move on to political roles, but stay in the background making sure things tick over.  I can only assume mavericks got involved in this.

GP practice rolls are totally fallible and will not solve the Slough problem that the NLGN keeps going on about.  Research has shown that migrant workers, young men aged 18 to 30 and new immigrants are unlikely to register with a GP, but will use A&E services if necessary.  The males aged 18 to 30 issue has always been a problem and was particularly highlighted in the days of the poll tax, when it was looked at as another way of finding "missing" people for tax purposes.  Plus do you trust your local surgery to get all your details right and to not pass on things you don't want the Government to know?

School places - not everyone has children and not all children go to schools that this register would cover.  What about foreign nationals coming to UK schools?  What about the independent sector, who again may not collect or be required to supply all the data required.  Nominal numbers of school places are also an issue, as local authorities get into real hot water if they try to close classes or schools.

Electoral registers - the poll tax debacle caused people to remove themselves from the electoral roll.  I have had periods in my life when I have either been on no electoral rolls or on two or three different ones - none of it by my own volition I hasten to add.  Sharp practices in Birmingham and postal votes have also shown the potential problems with this.

And finally tax records.  Where is the Information Commissioner on this one?  Again, the tax office doesn't collect all the data you need for a useful population count (qualifications for example).  I for one would not be happy that data that I freely give to one source for one purpose is linked to data that I give for another purpose to another source.  People go on about civil liberties, but this sort of project, with implicit requirements for huge amounts of fuzzy matching, exceeds all previous Big Brother attempts.

If you don't like the traditional Census, try again to find something better.  This does not seem adequate to me.

That A level time of year - again!

And I can't not comment!  But once again what do you say?  With over 97% of exams passed, is the A level really the exam you cannot fail?  On the one hand, you don't want to do down those whose results have come out today, on the other hand, what do you say to those of us with low A level grades from over 20 years ago?

Was I really so terminally thick that I failed my English A level back in 1981 - or if the CEM Centre grade inflation is to be believed, my O at A level would actually get me in today's currency a nice little D or maybe even a C?

The government is apparently putting in place measures to make A levels more difficult from September (bad luck those of you starting your A level courses in 2008!), but will this make much difference?  Can you imagine a politician standing up and saying that the pass rate has gone down now exams are more challenging and that this is a good thing?  Not in a million years!

Those in academic circles know that standards are falling in A levels - behind closed doors it is debated, but never published and to use the phrase "dumbing down" gets you bullied, dismissed or your contract not renewed.  The CEM Centre research quoted by the CBI earlier this week will be buried - which is a shame as these guys really know their stuff.

However when there is an attempt to re-base by changing the qualification structure, everyone seems to go to jelly.  Tomlinson was reviled for his outstanding work on qualifications, and if the Diplomas are the back way to implementing his vision, so much the better.

The Daily Telegraph is perhaps not the best place to look for rational and impartial debate on this topic.  I looked at the comment board on the A levels this morning and sincerely hope that Becky who commented at 8:16am as one of this year's breed of all A students wrote firmly tongue in cheek.  Otherwise her examiner and her teachers should be sent to prison for stealing state money - they haven't educated her, she wasn't literate. 

The columnist whose article got ripped apart was trying to make a decent point however.  We can argue all we like about the standards at A level, but what about the depressingly large proportion of 14 year olds who lack literacy and numeracy skills?  What about those who leave school with nothing, or worse leave school with one G at GCSE and are therefore not eligible for some forms of support as they have a (useless) qualification?  If nothing else, doing A level study equips you with a few skills - not least being able to access the Daily Telegraph comments board - but to leave with no skills is a waste of opportunity.

Contextual Value Added - Why bother?

Ofsted has come out of the closet on its views of publishing contextual value added scores and their use - albeit that their views seem to be something of a curate's egg, and need just as much careful interpretation as the actual CVA figures themselves.

Contextual Added Value (CVA) is the latest measure to show how well schools are serving their clients (sorry, pupils), and takes into account prior attainment (added value) and the factors in the local environment that might affect performance but are outside the school's control such as poverty or ethnicity of the local population.  This is all mulched up together (good statistical term that) to give a score normalised around 1000 and with a confidence interval - which is generally not so well published as the basic score.

Ofsted has given out guidance on interpretation, for use by schools, and has said which bits of what is published should be used and which bits are meaningless - this BBC summary is quite good on this.

My gripe is this: government would like parents and guardians to have information available on which they can base their choices for their children, and later in life, similar information available to potential students so they too can choose the best route for their future.  However, with literacy and particularly numeracy levels at a low ebb in England, how are people expected to interpret CVA information?  Yet it is this information that will be supplied to people to make choices on.

In my view, these figures are not fit for public use, and therefore it is questionable whether there is any value in putting them in the public domain.  If an absolute score has a confidence interval of 10 which could take it above or below the national average, all people need to know about an institution is that it is average.  Anything more and the number is being vested with spurious accuracy, giving people lots of opportunities to make the wrong decision, as they don't understand numbers.

There is a similar link to the publication of medical findings that are too statistically complex for 99.9% of the population to understand, so now we are faced with a measles epidemic in the UK due to people panicking.

I think the "Tiger that Isn't" should be compulsory reading for everyone in the country - but that assumes a decent level of literacy too! 

Getting the unemployed back to work

Both the UK and France have come up with plans for this over the past week, limiting access to benefits for the unemployed. 

The French have gone for a definition of the rights and duties of the unemployed, which link receiving benefits to whether or not the unemployed person rejects a reasonable offer of work.  The reasonable offer is defined and changes with time, so that shortly after being laid off, you have the right to expect at least 95% of your previous salary, but after a year, a reasonable job will pay you the same or more as your benefits rights.  Similarly the distance you have to travel to a suitable offer is defined and does change over time, the reasonable limit being an each way journey of 1 hour by public transport or over 30 kilometres each way, whichever is longer.  This system does bring a certain level of bureaucracy, as one would expect from the French, but the travel limits are interesting in a country where it is assumed you will go home for a full four course lunch.

The British "Work for Benefits" plan was revealed on Monday, and has the support of New Labour and the Conservatives (spooky).  It aims to reduce the number of people receiving benefits and will look at Job-seekers Allowance as well as Incapacity Benefits.  I have a few concerns about work to receive benefits - whose jobs are being taken to do this work?  If people are removed from benefits but can't find work, how do they live? 

While there are undeserving poor out there, there are also deserving poor who need help.  I have a neighbour who is an alcoholic.  He keeps being put in prison, but unless the underlying causes of his alcoholism are addressed, he is incapable of functioning according to the norms of society, so comes out, re-offends and is straight back in prison as a recidivist.

What jobs are we prepared to put people out of, in order that people on benefits can do that work?  This is a similar problem to what do we do with people who are given community orders instead of prison sentences.  Are these people taking jobs that others might find a good market niche and profitable for themselves?

I try not to push Internet sites that are not mainstream, but the Daily Mash had an extreme but amusing take on what the unemployed might do, and it definitely struck a chord with me.  Are we pushing people into meaningless activities to justify their benefits, and if we aren't, have we just displaced a problem?

Who is responsible for pupil behaviour?

Sir Alan Steer has published his review of pupil behaviour, and has said that parents and adults set a bad example for young people.  Bad behaviour is a key reason for teachers leaving the profession that has to be one of the most important to the progress of the country.

For those of us without children, it is relatively easy to put all the blame on parents, but as the review points out, young people are also influenced by what they see around them.  Greedy and aggressive behaviour will have an influence on all those around us, not just those that we are related to.  If what you see around you is the norm for behaviour, and that includes a lot of swearing, throwing V signs at other drivers for real or imagined discourteous behaviour, lack of acknowledgement of the existence of others, lack of basic good manners, and a me-me-me culture, what sort of person do you turn out to be?

So we all have a role to play in the development of young people, but the current norms of society seem to push against us all playing our part.  The "wot you lookin' at" riposte and fear of reprisals if one comments on bad behaviour puts a break on what is left of society trying to regulate itself.  However learning providers who work with difficult young people say that setting boundaries, not swearing back and telling people when they have crossed the line is important, and develops respect.

Indeed it seems to me that society has become a dirty word in the UK and anything that limits personal freedom is resented.  At the same time, we are bombarded with directives from government, health advisers, and a range of "authority" bodies on how we should behave and live our lives, developing an intolerance of the nanny state.  However someone needs to remind us when we overstep the line, as it seems that our personal views of limits of acceptable behaviour have been eroded, and while we all demand respect, we are not inclined to give it.

How far the media and its constant building up and destruction of figures in public life is responsible, how far the liberty of the 60s is to blame - two questions for the sociologists perhaps.  But as individuals we can all make a difference by behaving with integrity, and doing as we would be done by.  If you like being sworn at in the street fine, but if not....

Ingrid Betancourt is finally free

A few months ago, I wrote a blog in this area that was not totally in line with my given subjects, but raised the issue of looking at news stories that don't particularly have a British slant to them.  Prompted by the return of Prince Harry early from a tour of duty in Afghanistan, I wrote about the trials of Ingrid Betancourt, a Colombian presidential candidate who had been held by the FARC for over 6 years, and who'se health was deteriorating.

In London on Wednesday afternoon, the key stories in the London Standard were a drugs raid that Prince William had some peripheral interest in and the hopes of Andy Murray against Raphael Nadal.  At Southampton Airport the following morning, there were no obvious splashes on newspaper front pages of anything of interest.  Once home, my own local paper had it emblazoned across the cover - Ingrid Betancourt Libre.

I still get emotional when I look at the photos supplied on this - follow this link to see for yourself.  Fantastic news!

14-19 Diplomas - the way of the future or the emperor's new clothes?

And to be honest, I am not sure which way I lean on this one.

I am concerned about A levels, in that I don't think they are a consistent currency for measuring educational attainment.  I did three back in 1981, failed one and got two in those days reasonable passes, but passes that would now be considered not good enough.  They were hard, and required interpretation and thought, or so I thought then.  I did another one in 2004 and got straight As, and could see the marks I had gained for each section of the assessment, so where I had done slightly less well and where I had been frankly embarrassingly good.  Now as this was an A level in French and I have never been good at verb endings or gender agreements between adjectives and nouns and have trouble remembering what gender things are anyway, this brilliance was a bit worrying.  I also didn't feel I needed to interpret or analyse anything (no learned debate of Candide or Les Enfants de Siecle).

I am also concerned that A-levels and those that teach them are, with the best will in the world, a bit removed from the work arena.  I can quite see that something is needed for those who don't want to leave school at 16, or whose parents are unlikely to let them leave school at 16, that is a bit more grounded in the real world therefore.  Thus the Diplomas in Manufacturing for example, particularly as they are influenced by the needs of real employers, should be a good grounding in practical knowledge and skills that can assist in employment and also in developing the talents of young people.

I think I agree with the CBI in that Diplomas in Humanities and Languages may therefore not instantly spring out as priority areas for development.  Equally the problem of running a dual system, with private and grammar schools working with A-levels as the gold standard and non-selective schools and colleges running with Diplomas will inevitably bring or reinforce a two-tier system.

On the whole I am not wedded to A-levels, so I would be happy to see them go once and for all, if there was a system with the full range of subjects available for young people to access.  Whether anyone will see the need for a Diploma option in Latin or Ancient Greek, it must be offered in order to provide the full catalogue of options.

What I find unhelpful in the whole debate, are those people and organisations that trash the concept of Diplomas without any real knowledge.  If they continue to get bad press before they start, those young people who do them will be stigmatised, and Diplomas could fail without being given a chance.  The A level isn't a sacrosanct gold standard, any more than the O level was.  Stop sneering and see how young people and employers work with the new qualifications and assess it in 10 years time.

Numeracy and Skills for Life

The National Audit Office has officially congratulated the Government on the achievement of its 2004 and 2007 Skills for Life targets and says it is well on the way to achieving the 2010 target too.  This is good news.

Personally I would thank the Government a bit less and instead heap praise on the LSC, learning providers, community and voluntary groups and individuals who have delivered those achievements.  I would also praise the individuals who overcame barriers of all sorts to improve their basic skills - be that finding the time to do the learning, making that first scary step into a college or provider, or even more daringly, admitting to someone that they needed help with something as fundamental as literacy skills.

I pick out literacy skills because that is where the main achievements have been made and where the most progress and impact has come from.  A further swathe of learners will have achieved English as a Second or Other Language qualifications, enabling them to function better in British society.  Where the policy has almost failed however, is to improve the numeracy skills of the adult population - the scale of the problem is greater, but to date the impact has been less.

In part, this has to be laid at the feet of a society where it is acceptable to say "I'm no good at maths".  I would be mortified to say that but I do remember class mates at school saying that, and the easiest way to get people off a business studies degree is to face them with a pretty basic quantitative methods paper, which many fail because it's numbers, and answers are right or wrong, though you can get the odd mark for getting a method right.

Another key reason is the way maths is often presented as a scary separate subject rather than integral to life as we know it.  Construction courses include maths for calculations for how much plaster or tiles to cover a wall, mechanics courses include maths in terms of quantities of lubricants and mixes of washes and anti-freeze, cookery is all about measurement and proportion, even art and drawing includes geometry and proportion.  The more obvious subjects such as economics, accountancy, management all need a good grasp of numbers and what they mean.  While betting slips, the pools, darts and football all bring a certain need for arithmetic and statistics, God's own game of cricket has inspired some of the most complex mathematical modelling that the man in the street will encounter.

So why is it acceptable to say you are no good at numbers?  Grab life, your leisure time, your finances and your work and take pride in being able to do maths.

Housing Market Correction or .....

the end of the world as we know it (many thanks, R.E.M.)?

Last week I spent quite a bit of time with two economists that I respect enormously, and one topic of conversation was the state of the economy and the indicators not just from the UK, but Europe and America too.  Over the next few weeks I hope to explore a number of these, and today's offering is UK house prices.

As many readers know, I am not the greatest fan of media presentation of statistics and their interpretation, but on one topic I am nothing to my husband who will happily rant on the misleading and contradictory indicators published on house prices.  This is much in the news currently as Nationwide - the latest to publish their monthly overview of the market - signal a price fall of 2.5% in May (interesting as the month doesn't end until Saturday) and somewhere deep in the small print they point out that this is a seasonally adjusted figure.  The Daily Telegraph have picked up this story and run with it, pointing out that taking out the seasonal adjustment (which they seem to think is cheating anyway) the actual fall was 2.8% - a whole 12% different!!

The BBC have also picked up the story as you would expect and have also published a series of expert reactions to the figures and to the current market as a whole.  It is particularly interesting to look at the different views of various players in this field - lenders, estate agents, government departments and professional bodies - definitely worth a read!

One interesting aspect of the coverage in the Telegraph is to point out how the statistics actually differ that are being offered.  Estate agency chains can look at either the prices in their windows or the home valuations for mortgage approvals (though it is not totally clear how they deal with retention issues) - the Nationwide use the latter.  But neither of these are exactly what a piece of real estate actually sells for - so selling values are best observed though the figures produced by the Land Registry.  The Royal Institute of Chartered Surveyors look in addition at transaction volumes to see how the market is moving.  No danger of comparing like with like then, unless you keep an eye on longitudinal data from one or other source!

It seems fairly clear from all sources that house prices are falling on average and are likely to continue to fall for a while yet.  There could well be local and regional variations, which will mean that if you look at a particular area, you might be able to disprove the trend - our old house and its neighbours seems to have remained static in value for the past 4 years.

However I also seem to remember that for the past 7 to 8 years, there have been rumblings about house price inflation being too great and how impossible things are for first time buyers and how the bubble will surely break sometime soon and how driving the economy through house prices is a bad thing and how house prices will remain high as demand outstrips supply.  Well demand still outstrips supply (I assume people haven't just evaporated and there haven't been that many houses built in the past 5 years), and things may get more affordable for first time buyers, but the credit crunch means they also have to have a chunk of savings for a hefty deposit.  Of course with low interest rates, their savings aren't increasing very fast, and while the City has had billions in bonus payments, most share based investment schemes seem to have done badly recently, so that deposit is just as hard to develop.

The market is changing, that is for sure.  Take your pick of the pundits, throw a dice, cut a pack of cards - none of them really know how things are going to develop, as the wider economy is in a very strange place.  Me?  I'm investing in gold!

The Power of Excel

I apologise for the lack of postings recently on this forum, but this is due to a surfeit of work, and my clients must come first.

At 4:30pm on a Friday afternoon, I have finally finished a major data crunching exercise, and am in a position to write it all up.  However before I do that, I need to distance myself from the pain and backache of analysing something in excess of 36 files with over 180 separate data sets in excel (and perhaps close to half a million records).  The back pain and also swollen ankles are due to my ability to sit mesmerised in front of a computer screen for hours on end not moving anything but my mouse hand - obsessive and entirely my own fault.

I also need to distance myself from the mental pain of it all.  Excel is a fine product and I am a major fan of spreadsheets - they are very powerful and when used by skilled people can do amazing things.  In the hands of incompetents, they will produce rubbish - but that is the way with many computer tools, particularly statistical ones.  If you feed garbage in, you get garbage out, albeit with pretty graphs.

However for large data sets, Excel isn't always the answer, and my mental pain comes from the fact that about two thirds of the way through my epic analysis task (which took 4 solid days I may add), I realised that I would have been better off and would have done the work more quickly if I had done it differently. 

If I had spend half a day or a day developing an SPSS file with the Excel data all concatenated into one file (easy enough as it was all in a common format), I could then have spend a happy hour programming SPSS to do the analyses it has taken me so much time to do, and a further day to push the SPSS output into a pretty format for the report I will now spend the weekend writing.

There is a rather painful lesson to be learnt here I think!