Evaluation, statistics and OR

The dilemma of the paid researcher

I was reading Private Eye this morning - it always helps to keep up to date with the seemier side of government and policy I think.  For once I am also reading a current copy - No 1353, 15 November if you want to follow this up - and in the Just Fancy That! section on page 9 I found a very familiar story.

A survey of academics undertaken by the London School of Economics found that academics hired by government to evaluate the effects of its policies are being leaned on to reach suitable conclusions.  Excuse me LSE - it's not just the academics that find that.

This is a challenging dilemma for any researcher - you have valid results, but they don't say what your "employer" wants them to say.  Contracts with Government and many other agencies require consultants (academic or otherwise) to say they sign up to Market Research Society or BERA guidelines or both or an international standard.  These clearly state that the researcher should present the findings appropriately irrespective of whether they tell the story demanded by the sponsor of the research.

So on the one hand, you are required to tell the story demanded by your paymaster, on pain on getting paid and being taken on to do future work, while on the other hand, if you present the required results, you are breaching the relevant researcher code of ethics, which puts you in breach of contract, which could mean you don't get paid or future work (if you get caught).

I have been in this position only once (fortunately), and it wasn't a nice place to be.  I think LSE need to widen their survey to cover non-academics - there are after all loads of sources to find who has been working for government departments.  The major research societies then need to work with their members to see how far this is happening and help their members find a way to maintain their professional dignity and integrity.

 

Training for those new (or relatively new) to Research

It has been an interesting period at Plantagenet Consulting Ltd.  The changes in Government Machinery, the "bonfire of the quangos", and the savage cuts to all and sundry in the public sector have proved to be a challenge for companies like Plantagenet.  However despite that, I have had the immense pleasure of continuing my working relationship with HOST Policy Research, and have in fact now become a partner in HOST.

HOST is a larger organisation, and has a longer track record than Plantagenet, and has a wide range of subject experts to call on to meet client needs.  It can therefore be rather more proactive and outgoing.

In February 2013, HOST will be offering a series of masterclasses at Roffey Park near Horsham.  Dorothy Berry-Lound - the founder and Managing Partner of HOST - will be offering sessions on how to set up impact assessments and evaluations and managing research sub-contractors.  I will be leading sessions on scenario planning and developing a research strategy.  For those that stay over at Roffey Park with us, there will also be extra free sessions in the evening.

You can find more information and how to book at this web page.

 

Better Statistics education for all

The Royal Statistical Society have recently launched a report alongside The Actuarial Profession looking at the quality of teaching of statistics in schools and the role that statistics should have in eduation.  This is so important I am plugging it and their campaign Getstats shamelessly.

I don't know about others who use statistics in their job, but I am fed up with the knowing semi-humorous comments by the likes of Terry Wogan about "another government survey - ho ho" or "lies, damned lies and statistics".  That phrase was more a warning for professionals not to abuse the power that they had over numbers.

Similarly every latest health scare in the press that tells you not to do something that contradicts the last health advice because the likelihood of dying from something has doubled from 1 in 100,000 to 2 in 100,000 really gets on my nerves.  My health statistic?  The likelihood of you dying at some stage is 100% - some may say that there was a chap that didn't do it, and therefore it is less than 100% but I say he had help from another Source and that doesn't count.

So support a worthy cause, and take some time each day to be come a bit more numbers savvy, because otherwise you will be duped at every turn and will only have yourself to blame.

 

Evaluation - what is the purpose?

As budgets in the public sector are constrained by Government policy, one area that seems to be being squeezed is evaluation.  Recently a discussion with a former client over the progress with their new project provoked the following exchange:

- I gather the new project is proving interesting?

- O yes, very much so, but challenging too.

- How will you be evaluating it?

- We don't need to do that, as the requirement for evaluation by the funders has been dropped.

If evaluation was only to prove to funding agencies that work is being done to the schedule and cost that is set out in the contract, then it would be called audit.  There is a real purpose to evaluation, or rather a number of purposes.

Most importantly, formative evaluation, undertaken during the life of the project or programme is there to help those delivering the project ensure that things are going to plan, that there are no perverse effects to beneficiaries or to the market place, and that the systems they employ to manage the project are adequate and not overly onerous.  Lessons can be learnt and changes implemented to ensure that the full final benefits of the project are achieved.

Summative evaluation, at its most basic, provides evidence to all that the project achieved all (and hopefully more) that it set out to at the start, demonstrating the quality of the project manager or organisation in being able to deliver successfully and to time.  Better evaluation will also set out the impact of the project on its beneficiaries and to the wider community or economy to provide evidence for future planning and policy development.  It also should provide an opportunity for a full 360 review of how the project worked to inform future work patterns and personal development plans.

Evaluation is not a nice to have add on to a project, it is a key component of a project, without which wheels will be reinvented, or perhaps TECs and Business Links - who knows?

 

Evidence based policy?

Those of us in the consultancy game, particularly around evaluation, like to think that when we write conclusions and recommendations, someone listens to them and does something about them.

I have experienced the glow from time to time of having my findings reflected back to me - most recently on a wonderful project called Business Brokers in North Staffordshire.  There, with a colleague and working under the Inspira Consulting banner, we did an extensive mid-term evaluation of the programme to help them meet their contracted targets and also help them think about the future beyond their funding contract and how they could ensure the sustainability of the service.  In the last three months, we have gone back and done the final evaluation of the funded programme, looking at contract performance and also at how our recommendations were followed up.  They had done extremely well in achieving their contracted outputs and have found ways to maintain the service, albeit at a lower level, but then at the moment, they have less funding.  Whether it was polishing our egos or not, they did say more than once, "following your advice we looked at this" or "we have done this".

However in education matters, this is often far less the case.  My impression has been that decisions are made as knee-jerk reactions, without letting the previous set of changes embed themselves and demonstrate worth, and due to the loudest set of shouting in the media.  I am not totally sure how often Apprenticeships have been re-branded since they were reintroduced in the late 90s, but the myriad brands have not helped employers or prospective employers identify with the product.  Similarly, new qualifications are brought in and then dropped at the drop of a hat - who remembers GNVQs?  And if you have one, how do you explain it to people now?

It seems that my impressions are not totally without foundation for once as well.  This article on the BBC website heralds a report that says just that - politicians do not look at the evidence when changing education policy and practice.  For those of us who have recently been following the mantra of "evidence based policy making", this will be something of a shock, but then - are politicians going to take any notice of this report either?

Participation in Learning: half full or half empty?

Such is the volume of news reporting and issuing of statements, that two stories were published by the BBC within 24 hours of each other that give the average reader no hope of understanding how well the UK is doing in terms of skills.

First was what should be an extremely good news story - more than 1 in 4 of the entire population of England is in education and training.  While that will underestimate the number of people actually learning, it is a tremendous statistic.  I could spend a happy while picking holes in it, with particular reference to the Unique Learner Number, but that would be nitpicking.  The expansion in participation in education and training is a marvellous thing and some of the statistics quoted tell a marvellous story.  Read the story for yourself here - but if you don't have time, know that in the late 1950s about 10% of school leavers achieved 5 O-levels while today the minimum performance level for a school is 30% of 16 year olds achieving 5 A*to C GCSEs, with some Grammar schools achieving 99% or 100% of this and many bog standard schools achieving at least 50%.

The writing of the story tries hard to make this a bad news story by asking how all this education is going to be paid for, but "as any fule kno" gaining skills and qualifications pays for itself both for the individual and for the economy as a whole, so stop asking silly questions.

Where does the downer come from then?  A day later the University and College Union points out that in their analysis the UK is falling being in the qualifications race and that there is not enough investment in education and skills.  While the UK has made tremendous strides (according to the first article), any progress is trashed as inadequate and lower than our competitors in the second.  We are allegedly behind such economic dragons as Portugal and Greece in the qualifications race and must do better.

Two things strike me about some of this latter report - is the quality of education the same, are they comparing like with like, is graduateness in the UK as good, better or worse than graduateness in Greece or Portugal or the USA.  At one time, it used to be said a UK degree was as intellectually challenging as a USA masters, while the West Germans used to make the same unfavourable comparison between a German degree and a UK degree (I never believed that however).  Secondly, if Portugal and Greece have such better education and qualification rates than the UK, why are the PIGS countries of Europe regarded as the drag on European economic growth?  (PIGS by the way - Portugal, Italy, Greece, Spain.)

Another happy thought is that the financial services companies that sparked off the global slowdown employed some of the brightest mathematical PhD students from the best universities - and look where that took us all!

Social equality and education

"Professions reserved for rich" is the headline and the social mobility report seems to indicate thatit is ever more difficult to get into the professions - medicine, law and journalism being those cited.  Shock, horror - large swathes of judges and senior civil servants were independently educated!  So were quite a lot of politicians of all persuasions.  Leading universities must take account of the social background of pupils, particularly those at low-achieving schools when looking at admissions, they say.

Well, that has got me going again, I have to say.  Firstly, journalism, however good, is not one of the professions, and the current state of the British media puts it firmly below advertising copywriters in my opinion.

Secondly, is this bashing the public schools again?  Some of them are very good, many put time into pupils who need that bit of extra tuition as they are not all that bright, so what people are paying for is smaller class sizes and better discipline.  When I was at school, Eton was often described as a fee paying comprehensive, as selection was limited to those winning scholarships.  What they do seem to give young people are social skills, producing vibrant and articulate young people who are not afraid to follow a less popular course (being clever, good at sport or good at classical music).

However how do you determine social background accurately?  One of the best determinants of a child's academic performance is the academic performance of its parents, but universities are unlikely to get away with asking prospective candidates about the grades their parents got at A level and degree.  It is also not related to the social class of the parents, which is generally determined by occupation.  The Standard Occupation Classification puts Managers in a high SOC group (1, 2 and 3 - Managers, professionals and associate professionals respectively) but a manager can cover a multitude of sins, while associate professionals can also cover some occupations that do not require a degree.  When I was applying for Polytechnic, my parents' occupations were respectively retired and cook, and if one takes the application of social grouping to those occupations, I would have been down in the lower brackets, even though my father was an artist (up there in social terms) and my mother was a cordon bleu chef.  Occupation doesn't really do it therefore.

If we look at educational attainment, that is far more likely to be a good determinant, but until there is a good retrospective database of qualifications, it is likely to be skewed.  We are frequently told that people lie about their qualifications on CVs and job applications.  In any case, it won't cope with the City types for example who made shed-loads of money without having qualifications, but can afford to pay for public schools.

And what about all those exceptions - those who made good despite a "poor" background and ill-educated parents, or those who didn't benefit from their fortune of having wealthy or well educated parents?

A yong person at age 17 or 18 wants to stand for something themselves - forget their parents and the money - look at their grades, their ambition and the quality of the school they went to and its performance.  High grades at a high performing school is expected and the young person deserves well, but high grades and a poor performing school and the young person deserves the best.

Whither DIUS?

There is a lot going on in politics and I try to ignore most of it, as the media driven personality frenzy is just not my thing.  I will admit to having been wowed by the Obama story, but that is more interesting to me than expenses scandals and whether Gordon Brown single-handedly brought down theglobal economy (he didn't - in my humble opinion - no one person could).

I get interested when something happens that affects those that I work for and with, and which will have an effect on employment and skills however.  So who spotted that sneaky move last Friday that in a single stroke, removed the Department that is responsible for Innovation, Universities and Skills?  Well so far, it seems only the Universities sector has bothered to speak out, highly upset that they are put with anything so practical as business.

Once again FE and skills at level 3 and below is the poor relation and forgotten.  With DIUS, I gather the Minister took a real interest in what was happening and read everything he was sent on issues in the skills world.  He worked closely with DWP to ensure that the Integrated Employment and Skills concept got off the ground and in the West Midlands at least, is starting to be observed.  He could balance the needs of FE and HE nicely as he had two powerful lobbies to work with, but they were at least related in their objective to improve knowledge and skills.

I mean no disrespect to Lord Mandelson, but to mix HE and FE with a third powerful and indeed stroppy and diverse group (business) and you start to get a monster department with very differing needs and egos to match.  He will have his work cut out to serve and work with three such lobbies (remembering that BERR has also such things as Trading Standards under its wing) and to the detail that John Denham brought to the role.

It is also unhelpful that this occurs just as Machinery of Government is dismantling the LSC and putting together a number of new agencies to take its place - the Youth Funding Agency will be fine with DCSF but the Adult Funding Agency may not find itself a priority for the new department.  It is also not helpful when new initiatives between DIUS and DWP are being brought in to assist those facing or made redundant in the current economic downturn.  If Government will not save LDV and over 800 jobs are lost (excluding supply chain), but puts money into supporting redundant workers - that is one silly department, as now it is all together.

What is the future for Skills in the UK?  Who knows - currently it is not a political priority.

So farewell, then...

Mark Heysom, formerly Chief Executive of the Learning and Skills Council.

I left the LSC about 6 months after he joined I guess, but even in that time he had made an impact on staff.  While that impact wasn't always regarded as a good thing, at least you knew he was somewhere in the organisation.  In many ways he was a more sympathetic character than his predecessor, known by many in the LSC as Monty Burns.  I met "Monty" once and introduced him to my colleagues on a show and tell style visit - I was impressed at his lack of charisma, and his ability to wind up my colleagues by being vague and uninformed.

I was not completely Mark Heysom's number one fan - the move from local to regional focus was not a good one in my view and the processes to move staff around meant that there was real hiatus and impact on learners and on projects.  This has been recognised in evaluations of ESF programmes at regional level and in more than one region too.  It did also seem that this was a reorganisation too far, being the third since the LSC's inception in 2001.

I did feel sorry for him when Gordon Brown sprung the split of the Department for Education and Skills and its delivery agencies on him and I gather with only 20 minutes to half an hour warning of the public announcement.  That has not helped performance or cohesion in the LSC in the past 18 months, and the next year promises to be as chaotic as the move from local to regional organisations.

However I have to admit to being very impressed that he has fallen on his sword, so to speak, over the issue of college rebuilding funds and related problems.  It seems a long time since anyone did the honorable thing, without ensuring a golden parachute, and Mark should be commended for that.  Good luck to you in whatever you chose to do next, sir!

Measuring the benefits of Mentoring

The Centre for Policy Studies has published a report concluding that the benefits of mentoring are exaggerated and unproven.  It seems that the Prince's Trust and I disagree!

It seems that the Centre for Policy Studies does not think that there is enough evidence to prove the case for mentors, particularly children and young adults.  Well, au contraire say I and the Prince's Trust and from totally different bases too.

I have recently been involved in a number of evaluations of projects funded by the European Social Fund - the 2000 to 2006 monies have been spent and now need to be accounted for.  Those evaluations have been in the West Midlands and in the South West regions - quite different areas with quite different problems, yet both demonstrated benefits of having mentoring support, particularly for vulnerable people.

The projects ranged from support for ex-offenders and really hard to reach young people and encouraging them into learning or employment, encouraging the work-less back into employment, to helping people with mental health issues into learning and employment.  In all cases, the one to one mentoring gave the beneficiary (technical term for the person having the project done to them) the extra support they needed to face up to and overcome barriers, real or perceived, to learning or employment.

The mentor support included all sorts of things from the relatively mundane provision of bus travel tokens and assistance in completing application forms, through exploring an individual's career history to identify transferable skills, through helping young gang members to cross postcode boundaries, to providing the boundaries of acceptable behaviour and helping young people in particular identify what is unacceptable as well.

Those mentors that I met were special people as well.  They could fearlessly work with those that I would run away from.  They could encourage, empathise with and support people with very different backgrounds to their own.  They could inspire some of the most lost people in society to reconnect with the world around them.  They could inspire friendship and companionship with and among people with no real knowledge of how to relate to those around them appropriately.  And at the end of the day, the mentors could go home and lead a normal family life, and not be hounded with the sorts of self-doubt and worries that I for one would have after an intense day.

If, for example, just one or five of the several hundred young people in one of the West Midlands projects went on to get a job, and didn't commit any more crime, the substantial funds put into that project would have had a positive payback.  The cost of supporting someone on benefits or in prison for a substantial amount of time, plus the insurance costs, the social costs, the human cost to the victims of crime would outweigh the cost of that project.  In fact nearly 50 people gained jobs, and many more were living crime free existences at the end of the project.

To me, that payback is priceless, and I am a fervent supporter of mentoring programmes.  I suspect that my ESF funded projects evaluations would not cross the Centre for Policy Studies radar however, so they will never see the value.