Evaluation, statistics and OR

The dilemma of the paid researcher

I was reading Private Eye this morning - it always helps to keep up to date with the seemier side of government and policy I think.  For once I am also reading a current copy - No 1353, 15 November if you want to follow this up - and in the Just Fancy That! section on page 9 I found a very familiar story.

A survey of academics undertaken by the London School of Economics found that academics hired by government to evaluate the effects of its policies are being leaned on to reach suitable conclusions.  Excuse me LSE - it's not just the academics that find that.

This is a challenging dilemma for any researcher - you have valid results, but they don't say what your "employer" wants them to say.  Contracts with Government and many other agencies require consultants (academic or otherwise) to say they sign up to Market Research Society or BERA guidelines or both or an international standard.  These clearly state that the researcher should present the findings appropriately irrespective of whether they tell the story demanded by the sponsor of the research.

So on the one hand, you are required to tell the story demanded by your paymaster, on pain on getting paid and being taken on to do future work, while on the other hand, if you present the required results, you are breaching the relevant researcher code of ethics, which puts you in breach of contract, which could mean you don't get paid or future work (if you get caught).

I have been in this position only once (fortunately), and it wasn't a nice place to be.  I think LSE need to widen their survey to cover non-academics - there are after all loads of sources to find who has been working for government departments.  The major research societies then need to work with their members to see how far this is happening and help their members find a way to maintain their professional dignity and integrity.

 

Training for those new (or relatively new) to Research

It has been an interesting period at Plantagenet Consulting Ltd.  The changes in Government Machinery, the "bonfire of the quangos", and the savage cuts to all and sundry in the public sector have proved to be a challenge for companies like Plantagenet.  However despite that, I have had the immense pleasure of continuing my working relationship with HOST Policy Research, and have in fact now become a partner in HOST.

HOST is a larger organisation, and has a longer track record than Plantagenet, and has a wide range of subject experts to call on to meet client needs.  It can therefore be rather more proactive and outgoing.

In February 2013, HOST will be offering a series of masterclasses at Roffey Park near Horsham.  Dorothy Berry-Lound - the founder and Managing Partner of HOST - will be offering sessions on how to set up impact assessments and evaluations and managing research sub-contractors.  I will be leading sessions on scenario planning and developing a research strategy.  For those that stay over at Roffey Park with us, there will also be extra free sessions in the evening.

You can find more information and how to book at this web page.

 

Better Statistics education for all

The Royal Statistical Society have recently launched a report alongside The Actuarial Profession looking at the quality of teaching of statistics in schools and the role that statistics should have in eduation.  This is so important I am plugging it and their campaign Getstats shamelessly.

I don't know about others who use statistics in their job, but I am fed up with the knowing semi-humorous comments by the likes of Terry Wogan about "another government survey - ho ho" or "lies, damned lies and statistics".  That phrase was more a warning for professionals not to abuse the power that they had over numbers.

Similarly every latest health scare in the press that tells you not to do something that contradicts the last health advice because the likelihood of dying from something has doubled from 1 in 100,000 to 2 in 100,000 really gets on my nerves.  My health statistic?  The likelihood of you dying at some stage is 100% - some may say that there was a chap that didn't do it, and therefore it is less than 100% but I say he had help from another Source and that doesn't count.

So support a worthy cause, and take some time each day to be come a bit more numbers savvy, because otherwise you will be duped at every turn and will only have yourself to blame.

 

Forecasting and Planning for Extreme Events

Pity the planners and forecasters - their names are mud across Northern Europe as snow brings transport, delivery services and health services to their knees.  Why?  Because most of the time, their forecasts and plans save the public purse and private corporations huge sums of money, by not having stocks of materials, staff and plant on hand but not doing anything.  However extreme events show where the gaps are to cover the worst eventualities, and that is what we are facing at the moment.

When I did temperature demand forecasting for British Gas a long time ago, we planned for peak days - a 1 in 20 cold winter had to be accounted for with ease and on long term planning we looked at a 1 in 50 cold winter to ensure capacity, and a 1 in 50 cold day within the year for short term storage capacity (gas holders they used to be called).  We used weather data (daily maximum and minimum temperatures and daily wind speeds) going back to the mid-1920s to model the climate of northern England, and our reference really bad winters were 1928/29, 1946/47, 1962/63 and 1981/82, with 1946 and 1962 the worst we had to play with.  Some weather monitors are calling 2010/11 a 1 in 100 cold winter already, and we aren't half way through yet, so planning is difficult as it would appear to be more extreme than existing records.

Then think about the resources - in order to cope with a 1in 20 event, on a normal day you will have resources sitting in stock (in the case of things) or that can be diverted to other uses (people, but only if that is in their contract and competence), or just sitting around depreciating (plant that can be used for one thing only - snow plough attachments spring to mind for some reason).  Those who watch the money don't like that, and will look at ways to avoid having stocks of stuff that might not be needed (excess salt), people (one of the biggest costs to an organisation) and plant (surely we can borrow someone's snow plough?) sitting idle.

When that 1 in 50 event happens, there is no-one to take your call however urgent, the salt runs out after two weeks and no-one will lend you a snow plough as they are using them themselves or can't get it to you in time.  The result is a knock on to services to the public in general or to customers in particular.  When I worked at British Gas, we were not long out of being a public monopoly and still ran the business from well head to burner, so it was our pride and our obligation (on pain of dismissal) to ensure that there was enough gas in the system to meet all domestic and non-interruptible needs.  As the monopoly went (and after my time I hasten to add), domestic customers did get cut off and in events that were not as extreme as a 1 in 20 winter.  It was no longer an integrated system and cost savings had to be made.

Then add in the human effect - despite warnings to not travel unless absolutely essential, Parisiens decided that Christmas Shopping, visits to beauticians and hairdressers were essential and got stuck in their cars and had to be rescued.  You can warn, but people won't listen because it won't affect them.  Similarly, the most extreme weather event I dealt with at British Gas was in 1991 when we had a foot of snow, but it wasn't that cold.  Gas usage rocketed because of the psychological effect of snow.  If we could have found the records for lying snow in the weather archives going back to the mid-1920s, I am sure a 0/1 variable for snow would have improved our peak day forecasts. 

Nature is chaotic, and we can't plan for every eventuality (Haiti and the Boxing Day Tsunami are good examples), and if we plan for the very worst, there will be resources that are rarely if ever used.  The irony is that this cold snap has come at the start of a major UK austerity drive in public spending - to save money public sector staff will go, but who will drive the gritters and ploughs in future and who will pay for the salt/grit mix?  I'm not sure tax and council tax payers will want to find the extra cash somehow.

 

Adult Learning Figures

The BBC are now reporting the results of the annual NIACE Adult Learning Survey - this is a national survey undertaken on an annual basis with a sample size of about 5,000 respondents allowing for the data to be cut in some ways (social class, level of highest qualification or region) but not in multiple ways (social class and region for example).  The main value of the survey is as a longitudinal barometer of participation in learning among the adult population showing broad trends - the Government criticism that individual years should not be looked at in isolation is fair, but the same can be said of their own survey, NALS (National Adult Learning Survey).

Some of the NIACE findings are wholly unsurprising to those of us who have been looking at learning survey data for the past 15 years.  Those from higher socio-economic groups are more likely to participate in learning after school age than those from lower socio-economic groups.  I know you will always find the example of the bin man who has been doing a degree with the Open University for the past 7 years, but remember this is a generalisation based on majority findings.  However people from higher socio-economic groups include people who are doctors, accountants, lawyers and members of professional institutions, most if not all of which demand continuous professional development to maintain a licence to practise.  I am almost worried that only 53% of those respondents from the higher socio-economic groups have undertaken learning in the past 3 years.

There is concern that there has not been an explosion among those in lower socio-economic groups undertaking learning, given the millions being poured into schemes such as Train to Gain and the adult literacy and numeracy campaigns.  I suspect that even though this work has seen large numbers of new learners, the fall in the availability of free or low cost adult education opportunities has had an effect.  Not least in the mind of the respondent - having just completed a telephone survey myself recently, I can confirm that people don't necessarily remember slightly dull or ordinary business activities, where they do remember attending an external event or something that was fun. 

When training is delivered in the work place, it is highly effective for both employer and employee in general, but it may not be remembered as learning activity by an individual, particularly if it was not clearly badged as a learning event.

Surveys are excellent tools for understanding the broad characteristics of the population, but with adult learning, there is also access to the Learning and Skills Council Statistical First releases, which while complex do give definitive numbers enrolled on publicly funded learning.  NALS was last run in 2005, has had a small sample size in the past (less than 5,000 sometimes), is not undertaken regularly, and while the next round is in the planning phase, is unlikely to show headline results until 2010.

Redundancy Statistics

For those of us who work with or near the public sector, it is important to have facts and statistics to hand and to understand how those statistics are arrived at, so we can place the right level of reliance on them when formulating policy, bidding for funds or evaluating activity to see what works.

Recently I was tendering for a massive piece of work in the north of England to do ongoing monitoring and evaluation of ESF funding.  One of the aims of this evaluation was to be able to ensure that funds were being spent where they were needed and that they were being spent effectively.  The implication being that if they weren't, funding would be stopped and transferred to a more needy cause.

The colleagues that I was working on this with are brilliant, hugely knowledgeable about policy, evaluation methodology, full of ideas for doing things a bit differently, and able to express themselves to any audience.  My part in these things is often to be hideously well informed about statistics, management information and latest developments. 

The night before the final presentation pitch, I spent ages trying to find out recent redundancy figures for the Region in question.  I know these things exist - when I was still "within the system" I used to get regular reports from Jobcentreplus on actual and potential reducndancies by borough or district.  You had to use them with caution, but once information was in the public domain after employees had had their 90 day consultation, you would have thought that somewhere these figures were collated.

Well it seems not.  So if like me, you are outside the system and you need to make judgements on redundancy numbers over a period of time and in a specific location, you are back to relying on the good old British press.  My lesson for the day therefore is, don't go with the headline, read the full report.  There were stories about hundreds of job losses, that when you burrowed right into the story, turned out to be less than 50 for the area of interest, with others spread around the country.  It was no less sad for each individual and no less life changing for each family involved, but for the area of interest, it meant a lot less impact.  You also have to do this for every local paper and for every individual story.

That's a lot of work being repeated I would guess - wouldn't it be nice if those jolly chaps at ONS identified that there is a big issue going on in the labour market and started to monitor it?

Alternatives to a Census

The New Local Government Network want all work on the 2011 Census scrapped as they think they have found a better way to measure population that won't cause errors.  They think that taking a mix of administrative data on GP practice rolls, school places, electoral registers and tax rolls will do the job for them.  Normally I have a lot of respect for local authority research people as they have a lot of knowledge and experience and understand the holes in official data sources.  They are also generally a very dedicated lot and don't move on to political roles, but stay in the background making sure things tick over.  I can only assume mavericks got involved in this.

GP practice rolls are totally fallible and will not solve the Slough problem that the NLGN keeps going on about.  Research has shown that migrant workers, young men aged 18 to 30 and new immigrants are unlikely to register with a GP, but will use A&E services if necessary.  The males aged 18 to 30 issue has always been a problem and was particularly highlighted in the days of the poll tax, when it was looked at as another way of finding "missing" people for tax purposes.  Plus do you trust your local surgery to get all your details right and to not pass on things you don't want the Government to know?

School places - not everyone has children and not all children go to schools that this register would cover.  What about foreign nationals coming to UK schools?  What about the independent sector, who again may not collect or be required to supply all the data required.  Nominal numbers of school places are also an issue, as local authorities get into real hot water if they try to close classes or schools.

Electoral registers - the poll tax debacle caused people to remove themselves from the electoral roll.  I have had periods in my life when I have either been on no electoral rolls or on two or three different ones - none of it by my own volition I hasten to add.  Sharp practices in Birmingham and postal votes have also shown the potential problems with this.

And finally tax records.  Where is the Information Commissioner on this one?  Again, the tax office doesn't collect all the data you need for a useful population count (qualifications for example).  I for one would not be happy that data that I freely give to one source for one purpose is linked to data that I give for another purpose to another source.  People go on about civil liberties, but this sort of project, with implicit requirements for huge amounts of fuzzy matching, exceeds all previous Big Brother attempts.

If you don't like the traditional Census, try again to find something better.  This does not seem adequate to me.

The difference between audit and evaluation

At Plantagenet I am currently involved in a large number of evaluations - mainly through other consultancy companies.  The reason - apart from the fact that I am good at it - is that a large chunk of European Funding comes to an end in June 2008, and funding and delivery agencies are having to make sure they have evaluated how they have spent the money.  I don't know if it is the same across Europe, but in the UK they are keen to tick all the boxes and demonstrate they have done what is required.

These final evaluations are interesting as they show how the project or programme developed and how well the aims and vision of the work was delivered.  Ideally they are the moment to sit back and reflect that the lives of so many people have been affected for the better in some way.  In addition they can identify what worked best in delivery terms so that next time there is a purchasing round, people know what to look for.

I prefer this in many ways to the formative and interim evaluations where you are following a project as it runs, identifying what is going well and trying to get things that are not going so well sorted out.

Where things get a bit scary is where determining the difference between a final evaluation and an audit.  From the audits that have been done to me, it seems that you are looking for the things that don't add up, the things that weren't valid and need to be repaid, tracking the missing paperwork, identifying the gaps.  My brother used to do some of this in his role as a loss adjuster, and said it could be fascinating when you knew there was a problem, but dull if done for too long.

With an evaluation, I like to be able to follow the process that a project or programme has taken - follow the story if you like.  I also like to be able to delve into the figures on inputs and outputs to put a numeric story to it - and this is where my skills are mainly focused and is what gets close to auditing I suspect.  But for me the bit that makes evaluation a nice type of project, is the chance to go out and speak to delivery agents and similar and hear the stories of where things have worked.  I can't give examples here - mostly these things are very close to the project's aims and very personal - but they do give you a warm and fuzzy feeling when you are writing up the final report.

Do auditors feel warm and fuzzy when they have found a mistake or when they can find no issues with a project?  I guess that depends on the auditor.

Happiness versus marketing and the media

Today's e-mailed Expat Telegraph brought me a story with the headline "We're wealthier than in 1987, but no happier", and said that between 1973 and 2006, people's satisfaction levels have hovered around an average of 86%.  Wow - that tells me a lot.

While ONS who came up with the stats behind this headline talk about an Easterlin Paradox, in which the relationship between income and happiness declines after a certain level of wealth is reached, isn't there a lot more to it?

There is the old adage that "Money can't buy you happiness", although I guess it makes being miserable a lot more comfortable.

The media seem to rule our lives nowadays - or perhaps I am just getting older - and they very rarely seem to focus in on good news stories (apart from Carla Bruni's legs perhaps).  If a positive statistic is mentioned, who ever came up with it is obviously lying; if someone is good, we find a way to shoot them down; if it is a nice sunny day, it is down to global warming; if food has taste, it will give us cancer; if I enjoy a glass of wine, I am obviously a lush heading for every ill on the planet.  Against all this doom and gloom, you have to be a bit special to keep happy.

Then there is advertising, blatant or covert, which is designed to make us feel less than happy with our lot.  There are many models, but I will go back to Simon and March who have a cycle of purchasing that works for most things - you are dissatisfied with the status quo, so you do something to improve it (buy a consultancy project was the option in my research, but it could be buy a new car), once you have gone through that process, you are happier for a while, however a parallel process is going on where you also find that your aspirations have been raised as well, so your satisfaction with the new status quo is lessen (that car isn't quite as fab as the Jones') and round the cycle you go again.

That new car with better fuel consumption, that bigger house with the extra bedroom that you will never use and the garage you won't put your car in, that new TV or computer or MP3 player that is out of date within a week - you are always being told that something better has arrived.  If you don't listen to the marketing chaps, then they take away what you like (Windows 98 for so many people) and you have to find an alternative.

So in many ways, I am not surprised that we aren't all happier than in 1987, when all we really had to worry about was the awfulness of the mullet haircut and what would the new Micheal Jackson video be like.

But am I 86% happy?  If I were materialistic, I'd be surprised if I were that happy, but I don't much care about how new my car is, how many functions my mobile phone has that I can't use, so I hope the marketing has failed on me!

What is normal - Gales in March?

This week the UK and northern France have been battered by gales with gusts of up to 82 miles per hour in the UK and up to 160 kph off the Breton coast.  There is already talk of the demise of global warming, as these gales are evidence that it isn't getting hotter - or are they?

First off I take exception to the phrase global warming, as there are all sorts of arguments about some places getting warmer and some getting colder (looking at its latitude, the UK is already unnaturally warm anyway).  I prefer to talk about climate change.

Secondly, I think it is fair to say that the weather is a chaotic system and does not conform to nice norms calculated over 50 or 100 or 150 years as being the average for the time of year.  In any case to get an average, some things have to be higher or more or stronger and other things less.

Chaotic systems are subject to variation and odd extremes.  You could say that our recent gales are part of that.  Or you could say it is part of the weather pattern of Northern Europe and is actually quite normal to have abnormal events.

When I was a child I was given a tiny book by Maurice Sendak called Chicken Soup with Rice, which had a poem a month about Chicken Soup - the entry for March went thus:

"In March the wind blows down the door and spills my soup upon the floor.  It laps it up and roars for more.  Blowing once, blowing twice, blowing chicken soup with rice."

I think something like that reminds us that gales in March, particularly in the run up to the equinox and tied in to high tides is actually normal, yucky weather for March, and that things are rather nice and normal!!