Evaluation, statistics and OR

The difference between audit and evaluation

At Plantagenet I am currently involved in a large number of evaluations - mainly through other consultancy companies.  The reason - apart from the fact that I am good at it - is that a large chunk of European Funding comes to an end in June 2008, and funding and delivery agencies are having to make sure they have evaluated how they have spent the money.  I don't know if it is the same across Europe, but in the UK they are keen to tick all the boxes and demonstrate they have done what is required.

These final evaluations are interesting as they show how the project or programme developed and how well the aims and vision of the work was delivered.  Ideally they are the moment to sit back and reflect that the lives of so many people have been affected for the better in some way.  In addition they can identify what worked best in delivery terms so that next time there is a purchasing round, people know what to look for.

I prefer this in many ways to the formative and interim evaluations where you are following a project as it runs, identifying what is going well and trying to get things that are not going so well sorted out.

Where things get a bit scary is where determining the difference between a final evaluation and an audit.  From the audits that have been done to me, it seems that you are looking for the things that don't add up, the things that weren't valid and need to be repaid, tracking the missing paperwork, identifying the gaps.  My brother used to do some of this in his role as a loss adjuster, and said it could be fascinating when you knew there was a problem, but dull if done for too long.

With an evaluation, I like to be able to follow the process that a project or programme has taken - follow the story if you like.  I also like to be able to delve into the figures on inputs and outputs to put a numeric story to it - and this is where my skills are mainly focused and is what gets close to auditing I suspect.  But for me the bit that makes evaluation a nice type of project, is the chance to go out and speak to delivery agents and similar and hear the stories of where things have worked.  I can't give examples here - mostly these things are very close to the project's aims and very personal - but they do give you a warm and fuzzy feeling when you are writing up the final report.

Do auditors feel warm and fuzzy when they have found a mistake or when they can find no issues with a project?  I guess that depends on the auditor.

Happiness versus marketing and the media

Today's e-mailed Expat Telegraph brought me a story with the headline "We're wealthier than in 1987, but no happier", and said that between 1973 and 2006, people's satisfaction levels have hovered around an average of 86%.  Wow - that tells me a lot.

While ONS who came up with the stats behind this headline talk about an Easterlin Paradox, in which the relationship between income and happiness declines after a certain level of wealth is reached, isn't there a lot more to it?

There is the old adage that "Money can't buy you happiness", although I guess it makes being miserable a lot more comfortable.

The media seem to rule our lives nowadays - or perhaps I am just getting older - and they very rarely seem to focus in on good news stories (apart from Carla Bruni's legs perhaps).  If a positive statistic is mentioned, who ever came up with it is obviously lying; if someone is good, we find a way to shoot them down; if it is a nice sunny day, it is down to global warming; if food has taste, it will give us cancer; if I enjoy a glass of wine, I am obviously a lush heading for every ill on the planet.  Against all this doom and gloom, you have to be a bit special to keep happy.

Then there is advertising, blatant or covert, which is designed to make us feel less than happy with our lot.  There are many models, but I will go back to Simon and March who have a cycle of purchasing that works for most things - you are dissatisfied with the status quo, so you do something to improve it (buy a consultancy project was the option in my research, but it could be buy a new car), once you have gone through that process, you are happier for a while, however a parallel process is going on where you also find that your aspirations have been raised as well, so your satisfaction with the new status quo is lessen (that car isn't quite as fab as the Jones') and round the cycle you go again.

That new car with better fuel consumption, that bigger house with the extra bedroom that you will never use and the garage you won't put your car in, that new TV or computer or MP3 player that is out of date within a week - you are always being told that something better has arrived.  If you don't listen to the marketing chaps, then they take away what you like (Windows 98 for so many people) and you have to find an alternative.

So in many ways, I am not surprised that we aren't all happier than in 1987, when all we really had to worry about was the awfulness of the mullet haircut and what would the new Micheal Jackson video be like.

But am I 86% happy?  If I were materialistic, I'd be surprised if I were that happy, but I don't much care about how new my car is, how many functions my mobile phone has that I can't use, so I hope the marketing has failed on me!

Should children be in school?

Yesterday (Friday 28th March) the airwaves were full of debates about whether children should be kept in school or not.  The intention of the debate was looking at school food and how to prevent children eating fast food every lunchtime rather than the healthy options offered by the school canteen.

There seemed to be a certain level of horror at forcing children aged between 11 and 16 to be in school during the lunch hour - it was against their human rights, it was preventing them making adult decisions about what to do in their free time and what they wanted to eat.

What utter tosh!  You can't have it both ways.  You can't say that children have to be driven to school as the streets aren't safe and they might be abducted or attacked or damaged by traffic, and then say that they can go out at lunchtime when all these things are just as likely to happen.  You can't say that childhood doesn't last long enough and then expect 11 to 14 year olds to make decisions about what to do in their lunchtimes and roam the streets, albeit that 15 and 16 year olds might start to take that responsibility.

In addition, lunchtimes are an ideal time to get involved with sport, hobbies, music, drama or catch up with homework.  When I was at school, breaks weren't long enough for games of hopscotch, huge sessions of french cricket, the music club, rehearsals for official or less official music or drama events, time in the library and getting that last bit of homework finished or that last bit of preparation for a vocab test done.  Has all this gone nowadays in the search for the best chippie?

Say no to young people from time to time.  Say that they must stay in school (there will still be truants, but not a mass outbreak).  Tell them what to eat - self-discipline is something that comes with age and how many middle aged middle class can say no to that glass of wine at the end of the working day?  Kids shouldn't be left to go to hell in their own way, until you have shown them how life can be otherwise.

What is normal - Gales in March?

This week the UK and northern France have been battered by gales with gusts of up to 82 miles per hour in the UK and up to 160 kph off the Breton coast.  There is already talk of the demise of global warming, as these gales are evidence that it isn't getting hotter - or are they?

First off I take exception to the phrase global warming, as there are all sorts of arguments about some places getting warmer and some getting colder (looking at its latitude, the UK is already unnaturally warm anyway).  I prefer to talk about climate change.

Secondly, I think it is fair to say that the weather is a chaotic system and does not conform to nice norms calculated over 50 or 100 or 150 years as being the average for the time of year.  In any case to get an average, some things have to be higher or more or stronger and other things less.

Chaotic systems are subject to variation and odd extremes.  You could say that our recent gales are part of that.  Or you could say it is part of the weather pattern of Northern Europe and is actually quite normal to have abnormal events.

When I was a child I was given a tiny book by Maurice Sendak called Chicken Soup with Rice, which had a poem a month about Chicken Soup - the entry for March went thus:

"In March the wind blows down the door and spills my soup upon the floor.  It laps it up and roars for more.  Blowing once, blowing twice, blowing chicken soup with rice."

I think something like that reminds us that gales in March, particularly in the run up to the equinox and tied in to high tides is actually normal, yucky weather for March, and that things are rather nice and normal!!

Be a World Citizen

I have just come back from the UK, and spent quite a bit of yesterday reading about a young man who tried to fight for the freedom and democratic rights of others.  Unfortunately, due to a couple of media plunders, that young man had to give up his front line work, but will probably do good things behind the scenes in future.  Bad luck they spotted you, Harry!

At the weekend, there was an incursion into Ecuador, and the second in command of the FARC was killed - there was a bit about that in the papers, but not nearly so much - after all, it is a continent away and nothing to do with us.

The week before the FARC released some of the hostages it has been holding for years, who brought with them out of the Columbian jungle news of another hostage.  She is called Ingrid Betancourt, she has been a hostage of the FARC for over 2000 days, she is being treated appallingly, and it is reported that she has hepatitis and may be dying.

Up in the United States, we are being kept enthralled by a contest for who will stand for President between a potential first woman President and a potential first black President.  It is a momentous moment for the world, but what does it have to do with Ingrid?  At the time she was taken hostage, she was campaigning to be President of Columbia, and 6 days before she was taken, she was filmed with the 2nd in command of FARC, demanding an end to hostilities, to hostage taking and to trying to pervert the course of democracy.

If Hilary Clinton or Barak Obama were to be kidnapped before the next bout of primaries, there would be an international outcry about democracy and law and order.  If they were held for 6 years, the country suspected of hiding them would be bombed to smithereens (think of Afghanistan or Iraq).  What would happen if they died in captivity?

Think back to the release of John McCarthy, Terry Waite and Brian Keenan, and then be a world citizen and sign the petition to release Ingrid Betancourt.

Construction Skills and the Olympics

The 2012 Olympics and the bespoke buildings and resources required for them will mean a need for more skilled construction workers - about 182,000 of them according to the Construction Skills Network.

It is good to see this sort of statement from the industry, although all sorts of other projects at both local and national level will also drive needs like this.  I have been involved in trying to forecast the extra numbers needed for a variety of construction projects in the Coventry and Warwickshire sub-region as well as for the major regeneration of Hastings.  Organisations in the South East have had similar headaches over projects such as Terminal 5 and the Thames Gateway.

What makes this one a bit different?  The world's eyes are on us for the 2012 Olympics - can we do what we said we would do in our bid?  Elements of the media are already talking down the Olympics, in much the same way as they talked down the Millennium Dome after a few editors were inconvenienced when it opened.  It is right or trendy or clever to start saying the whole thing will be a fiasco now, before we have got to the opening ceremony?  That shows very little faith in our construction industry, the people who will take part in the opening ceremony and the business growth that should come from the Games, assuming it isn't sabotaged by the Whittering Classes (Chattering seems far too kind a terminology).

So, why not help the Olympic dream by encouraging young people to learn to be skilled craftspeople, with abilities that will ensure they are employable beyond 2012?  If you are mobile and flexible and skilled, there will always be jobs, as our need for major projects to be undertaken never seems to end.

Pass a language, don't speak it!

Today's rumour on the BBC education website that GCSE languages could be assessed without an oral exam goes beyond dumbing down, and if true, would turn language teaching in England into a bad joke.

It appears that the oral examination may be dropped and some form of continuous assessment brought in as the oral examination is a bit stressful, and that stress might stop people taking up languages.  Now I might be a bit of a freak, but the best memories I have of my O-levels (French, German, Spoken English), A-level (AS and A2 French), Degree German and PhD were the oral examinations, where you got a chance to show off to someone outside your circle of teachers and could demonstrate that you could do it!  I remember in my O-level French oral, rather patronisingly explaining to the examiner that many of my class mates might be having trouble, as we had all only recently returned from our German exchange trip - I got an A for the oral.

Setting aside my rather perverse view on the fun of oral examinations, the real test of learning a language is being able to have some sort of spoken interaction with a native speaker of that language (probably why Latin orals never really got fashionable).  That other person shouldn't really be the person who taught you the language in the first place as they will have preconceptions about your ability and also are already prepared for your accent and way of expressing yourself.  They may not be able to be wholly objective therefore.

If you find yourself using a foreign language in anger that you have had to learn in school, it is also likely that your experience will be stressful - be it that first attempt at buying a croissant or a beer or a coffee, trying to find a loo, trying to find your way round the Paris metro or how to pay for 200 miles on the Autoroute.  Even more stressfully, you could be trying to work out what that nice policeman with the speed camera is trying to say about your ability to stick to the speed limit, and respond to that politely.

Stress is an inevitable part of our lives and learning to cope with stressful situations is also part of the education process, so stopping a testing process at age 15 or 16 because it is a bit stressful can really be nothing more than dumbing down and certainly not in the interests of a young person.  It would be nice if someone saw sense on this, but I really doubt that will happen.

Today's rumour on the BBC education website that GCSE languages could be assessed without an oral exam goes beyond dumbing down, and if true, would turn language teaching in England into a bad joke.

It appears that the oral examination may be dropped and some form of continuous assessment brought in as the oral examination is a bit stressful, and that stress might stop people taking up languages.  Now I might be a bit of a freak, but the best memories I have of my O-levels (French, German, Spoken English), A-level (AS and A2 French), Degree German and PhD were the oral examinations, where you got a chance to show off to someone outside your circle of teachers and could demonstrate that you could do it!  I remember in my O-level French oral, rather patronisingly explaining to the examiner that many of my class mates might be having trouble, as we had all only recently returned from our German exchange trip - I got an A for the oral.

Setting aside my rather perverse view on the fun of oral examinations, the real test of learning a language is being able to have some sort of spoken interaction with a native speaker of that language (probably why Latin orals never really got fashionable).  That other person shouldn't really be the person who taught you the language in the first place as they will have preconceptions about your ability and also are already prepared for your accent and way of expressing yourself.  They may not be able to be wholly objective therefore.

If you find yourself using a foreign language in anger that you have had to learn in school, it is also likely that your experience will be stressful - be it that first attempt at buying a croissant or a beer or a coffee, trying to find a loo, trying to find your way round the Paris metro or how to pay for 200 miles on the Autoroute.  Even more stressfully, you could be trying to work out what that nice policeman with the speed camera is trying to say about your ability to stick to the speed limit, and respond to that politely.

Stress is an inevitable part of our lives and learning to cope with stressful situations is also part of the education process, so stopping a testing process at age 15 or 16 because it is a bit stressful can really be nothing more than dumbing down and certainly not in the interests of a young person.  It would be nice if someone saw sense on this, but I really doubt that will happen.

Business Survival Rates

On Monday I went with my husband to the local Chambre de Commerces for a half day introductory course on setting up in business in France.  Having done a similar thing in the UK a number of year's ago with enterprise agencies, we were pretty familiar with a lot of what was presented.  We were slightly shocked at the confusion as to which Chambre you need to register with, which depends on what you intend to do, and the fact that you could need to dual register, but that is a totally different story.

The session started with some statistics from INSEE on business survival rates - with the usual scare of perhaps half making it through to 3 years trading and only about 40% surviving to 5 years.  The lady running the course did comment that it was widely recognised that survival rates were favourably influenced by start ups receiving help from their adviser at the relevant Chambre.  Given that fact, another member of the course asked whether age affected survival rates, but that was not available.

Now that set me thinking - as you would expect.  There must be information collected on the age and gender of business start ups, not least through HMRC and Companies House, but how to look at the factors that make a successful start up?  If it is a one person business, that is relatively easy, provided the data is captured.  If that person takes on an employee, then it is also possible to allocate success to the manager (it may not be fair, but it is possible).

The difficulty is where you start up with some one or some many.  Can you allocate equal shares of someone's personal characteristics?  How do you measure influence?  This is what gets me to a certain extent about women's business support as you lose eligibility as soon as you work with a man, and yet men make up about half the population.

So, an interesting question, and one to which I don't have an answer, but if anyone has a nice big data set and a bit of money, I am more than happy to spend some time working up a few statistical models to see if I can sort out some determinants of business success based on age and on gender.

How can Geography be boring?

Ofsted has reported on the quality of geography teaching in schools based on its inspections between 2004 and 2007, finding that it was the worst taught subject in schools and that pupils found it boring.  As a suggestion, Ofsted recommended covering global issues in classes including climate change and sustainable development as well as trade disputes.  The Inspectorate feels that geography needs to be made more relevant.

Words fail me - well almost!!  I did Geography O and A levels and also sat the supplementary S level paper you could do at the time (1981), and it was the most interesting and most relevant subject I did.  Even (perish the thought) 27 years ago we studied the greenhouse effect and climate change, and debated whether it would make life warmer or colder in the UK.  Field trips and questions about soil and river formations and glaciation made any trip out with the school or my family more interesting as you kept a good eye out for a hanging valley or a fault formation.

My inspirational teacher - Daphne Swann - had travelled widely in school holidays and worked with international aid agencies in Africa as well as teacher exchanges in Europe.  Consequently any discussion of third world countries could be enlivened by slide shows and stories from Nigeria or China, while our A level field trip was to Denmark.  That included port surveys, dairy farming, retail developments, coastal and dune formation and flora, and glacial deposits via land forms (the end of the last Ice Age ice sheet makes the hills in the centre of Jutland) and the soil and stone forms in those land forms.

And this is perhaps what it comes down to - any subject can be killed by a bad teacher, and any subject can be inspirational with a good teacher.  My nascent interest in French aged 10 was killed off for a good 20 years by a poor teacher.  My abilities in Maths were limited by having 10 teachers in 2 years and it wasn't until Polytechnic and John Hobson that I remembered I could actually do Maths.  I don't think I would have passed Chemistry O level without a fairly scary but amazing year taught by Dilwyn Thomas, and Miss Polemounter enabled me to get a Latin O level and still be able to understand Private Eye citations and church inscriptions even today.

Geography should be one of the easiest to teach and inspire if a teacher has any ability and any interest in the world they live in - do we give potential teachers the right messages?

School League Tables

It's that time of year again - the school league tables for GCSE and A-levels are out.  As usual schools, education authorities and other interested parties are concerned that the tables are misleading, don't represent their achievements, put undue pressure on staff and pupils, etc.

It is one of the "joys" of performance statistics that however much you come up with what you consider a clean number to describe performance, someone will quibble with it or declare it unfair.  Currently GCSE passes are unfair because they favour girls, some independent schools do international qualifications that don't count for league tables, they don't take into account the prior performance of the pupils or the socio-economic context of the school.

Value added indicators have now been added - you hear less about them being unfair, but my quibble with them is that they are tortuous to calculate and how do you describe them to the average parent at the end of a hard day at work?

Then you get all the issues about branding young people failures if they don't meet the current base threshold (5 GCSE A* to C passes including Maths and English), and how schools fiddle the figures by not letting some poorly performing pupils take any exams if they will bring down the school performance figure.  Plus consternation from educationalists as to the validity of some of the qualifications that count towards that 5 good passes total.

Let's face it - there won't be a measure that pleases everyone.  But that minimum standard - 5 good GCSE or equivalent passes - drives public spending and policy on learning and training.  It should be the aim of society to furnish every young person that is capable with that minimum standard of education that will make the rest of their lives that bit easier, and the best time to do that is by the age of 16.