Evaluation, statistics and OR

Unemployment Measures

Today in France, the domestic headlines are dominated by what appears to be good news - the unemployment rate in February fell to the lowest level seen in 24 years, 8.4%.

You would think that joy would be unconfined and that M. Borloo who has overseen this steady fall for the past 3 years would be being feted as a national hero.  Sadly, as we are in the run up to the Presidential elections in France, and M. Borloo is firmly in the Sarkozy camp, 11 other candidates and interest groups, not to mention the Unions are all kicking up.  Their argument is that the figures do not represent reality.

Now the agreed standard method for calculating unemployment across Europe is by survey methods - hence the UK having two main measures - the Government preferred based on the Labour Force Survey, and the Claimant Count, which is based on numbers signing on.  The difference is that the survey method includes those who are actively looking for work, but are not entitled to benefits or who have not signed up for benefits.

All surveys include some level of error, and potentially include some bias as well.  The Labour Force Survey has been criticised for accessing those that are easy to get to, and missing out others such as the homeless, or those who travel a lot for work, but at a national level, the error rate introduced in this way is small and not significant.

The argument in France currently, is that while unemployment (counted as those actively seeking work of any type, including temping or part time) is going down, and the check figure of employment is also going up, the quality of the jobs is not what it was.  It is argued that many are doing jobs that earn less than the minimum wage or are not full time (35 hours per week only in France).  In particular, the CGT - the most militant of the unions it seems - say that there are not enough jobs being created, apart from in Construction and my personal favourite, "the nebulous area of personal services".

France is not really ready for the 21st Century at times - in the developed world, job creation will be in services and particularly the personal services, as manufacturing can be done more cheaply elsewhere.  However the strangely Seventies figure of Bernard Thibault at the head of the CGT will not go down without a fight.  That is one of the biggest problems facing France today.

Ken Bowen

This morning, when checking my e-mails, I received a note from the OR Society letting members know that Ken Bowen had died - I can't speak for the rest of the Society, but for me the OR world is now diminished.

Ken was my external supervisor for my PhD at Hatfield Polytechnic between 1986 and 1989 and he was an inspiration for my working life.  I am proud to have a copy of his PhD on my bookshelves and to have worked with him twice.

His knowledge was wide ranging and profound.  Obviously he was an amazing mathematician, had experience of defence issues, teaching methods and all aspects of OR, but in addition he played cricket until quite recently, was a humanist, and took a lively interest in art and architecture.

He taught me to look for models and heuristic rules in the world around me and to keep thinking in all situations.  One example was when he drove round the M25 from Guildford to Hatfield on one of his regular visits - it was a windy day, and once he arrived I took him to the Senior Common Room for coffee.  There he described to me how he had developed a model for identifying when he was going to face the worst gusts of wind and to be prepared for them, based on topology and lorry numbers.  It wasn't sophisticated, but the lesson was clear in terms of the way one should look at the world around.

Another instance was when I went round to see him at Royal Holloway and Bedford New College at Egham.  We spent the morning discussing how my research was progressing and how I could best gather soft intelligence in a structured manner.  Then at lunch time he took me to the mansion, so I could see the red brick version of Chambord, as he knew my love of France and Europe, and finally to see the hoard of fine art that the College had, including pictures by the "other James Holland".  This was not my father but the Victorian landscape painter, as I had once told Ken that I had not seen any of this other Holland's work.

As a teacher and mentor, in my view he was second to none.  I am sure that other former students and colleagues will feel that a wise and cheerful guide and support has been taken away, and that OR may not be quite as rich in the future as it has been in the past.

Recognising the importance of learning languages

What fabulous news to come out earlier this week - modern foreign languages are to be compulsory in primary schools in England by 2010!

As I have said before in my entry on "I want to communicate!", the power of learning languages and being able to communicate with others who do not share your mother tongue is immense. 

I was e-mailed privately after that blog by a friend who lives in France.  Her deaf niece from England was visiting them, and the children went out to play with French neighbours, and the niece came back speaking French, despite her communication difficulties.  My friend couldn't reiterate strongly enough the importance of the desire to communicate and the pleasure that comes from making that breakthrough in another language.

In my village in France, I have joined the walking group that meets on a Sunday for a 7 to 10 kilometre hike in the local area.  At first the group leader and the other walkers were a bit wary of speaking to me, and either did so very loudly (that endearing thought that the louder you speak, the more likely you are to be understood), or in very simple phrases or words (Look! Deer! Mud!).  Now they have got used to me, and realise that I speak French, and with some I hold quite intricate conversations and jokes, while others realise that I am not a swamp monster, just because I am not French.  It makes for a wonderful feeling of community, and I feel much more part of the village.

While there is a generation of young people in English schools that have been let down but being allowed to drop or never learn languages, future generations will be much better prepared.  Like I said at the start - fabulous news!

The importance of Evaluation

Keeping an eye, as I do, on public sector tenders that are being put out to competition among consultants such as me, it seems there is a major push on evaluation currently.  This should not be a surprise, as for the public and not for profit sectors, evaluation should be a key tool for a number of reasons:

• ensuring the project is doing what was intended
• monitoring what inputs are going into the project and what outputs are being generated by those inputs
• determining the impact of those outputs
• demonstrating the sound deployment of scarce public resource for the good of the target community

In the major public funding rounds, and I can talk from experience of ESF, ERDF, SRB and NRF, from the very moment of winning these funds, there is a clear expectation that the application of the funds will be monitored and evaluated.  (Apologies for the acronyms - the first two are funds from Europe, SRB or the Single Regeneration Budget is mostly complete now, and NRF is the Neighbourhood Renewal Fund).  It is also true that the best of the funding recipients build the evaluation process into their projects - allocating funding up front, being clear on what inputs, outputs and impacts will look like, and setting up internal or external processes and resources to ensuring the evaluation can happen.

For a consultant like me, it is great to work with such organisations, as they are clear in what is required, and with the evaluation included in early stages of the project, there are no nasty surprises at the end, when it comes to justifying spend.  It is also more cost effective to the public purse, in my view, with evaluator and project group working together through the process to ensure data is collected in a timely fashion, that something can be done if the project is going off track, and there is no panic at the end to prove the benefits.

The evaluations that, if I am honest, make consultants a chunk of money, are those that are tagged on at the end of the project in a panic, once the funds have been dished out, and key players have moved on to the next big thing.  The ones where a project manager is reading through their notes of the programme, and gets that awful sinking feeling that there was a final stage to go through - sounds familiar?

Such add-on evaluations, in my view, are never as robust or insightful as evaluations that are part of the programme, but should be able to show the success or otherwise of the funding.  Finding the required data and outputs can be challenging and time consuming, with long programmes, the initial ambitions and aims may be difficult to unearth as players and practises move on, and there is always the excuse that the essential information can no longer be found due to filing errors.

Best practise is to build in evaluation from the start - save time, money, resource and worry by doing that; or be prepared for a painful process at the end of the project!