Evaluation, statistics and OR

When your browser decides something is bad

This is my weblog about work related things.  I have no cookies on it, I don't particularly allow comments and I haven't put in any links for a while, so I am not sure why my browser thinks the site is evil and the source of potential world destruction.

I could understand if I had posted here my rather pungent views on Brexit and its stupidity, but I haven't (to date).

Apologies if you are trying to reach me via this and failing!

 

What is formative evaluation?

So much for good resolutions (see previous post), but it has been a busy time.

I am involved in several evaluations at present, some of programmes that are just starting, some of things that are coming to an end and some that are still running and what an external view of how they are doing.

How about a bit of jargon here? For those that are starting or still going, the evaluation should be formative – i.e. looking at how things are being done, and what can be improved. For those that are coming to an end, it is summative evaluation – summing up what has been achieved, and ideally how well that process has gone as well.

For me, formative evaluations that focus too much on demonstrating achievement, and don’t look carefully at process and what can be improved in how things are done are a bit of a waste of time and money. You should have management information that helps you to keep an eye on how well you are doing – if you don’t you should be worried. The formative evaluation should look at how things are working, digging behind the performance figures to see if you are doing well despite poor systems, ineffective communication and lack of partner buy-in (in which case how much better could you do with those things fixed). Or if you are doing badly, what is it that is causing that? Systems, people, relationships, the environment you are working in, or a poor concept initially?

I think if things are not going to profile, people are far more open to formative evaluation, but if performance is good or above profile, they tend to sit back on their laurels and not worry about how that performance is being achieved. And that is a mistake in my view. Either there are lessons that can be learned by others, including those funding your project, because you have got it right, or things that can be made better, and help save resources or push activity further and do more good.

Do you know why you are doing well?

A change for the New Year

I don't really do New Year's Resolutions, but sometimes the change to a new calendar year makes it easier to make a break with the past.

During last Friday's management team meeting (held at the pub as ever), I commented that my blog profile was tired and no longer relevant.  I've not worked in the realm of skills for a number of years, and I do not comment professionally about the relationship between the UK and Europe.  What I generally do nowadays is a bit of research and research support and a lot of evaluation.  I need a refresh.

So welcome to the new face of Plantagenet Consulting blogging - once a month I will post here about evaluation, statistics and soft OR techniques that I have been or plan to use.  I also promise to blog once a week at one of my other places on all things nature, rural, community, heritage.

Happy, healthy and prosperous 2018 to anyone who reads this!

 

The dilemma of the paid researcher

I was reading Private Eye this morning - it always helps to keep up to date with the seemier side of government and policy I think.  For once I am also reading a current copy - No 1353, 15 November if you want to follow this up - and in the Just Fancy That! section on page 9 I found a very familiar story.

A survey of academics undertaken by the London School of Economics found that academics hired by government to evaluate the effects of its policies are being leaned on to reach suitable conclusions.  Excuse me LSE - it's not just the academics that find that.

This is a challenging dilemma for any researcher - you have valid results, but they don't say what your "employer" wants them to say.  Contracts with Government and many other agencies require consultants (academic or otherwise) to say they sign up to Market Research Society or BERA guidelines or both or an international standard.  These clearly state that the researcher should present the findings appropriately irrespective of whether they tell the story demanded by the sponsor of the research.

So on the one hand, you are required to tell the story demanded by your paymaster, on pain on getting paid and being taken on to do future work, while on the other hand, if you present the required results, you are breaching the relevant researcher code of ethics, which puts you in breach of contract, which could mean you don't get paid or future work (if you get caught).

I have been in this position only once (fortunately), and it wasn't a nice place to be.  I think LSE need to widen their survey to cover non-academics - there are after all loads of sources to find who has been working for government departments.  The major research societies then need to work with their members to see how far this is happening and help their members find a way to maintain their professional dignity and integrity.

 

Training for those new (or relatively new) to Research

It has been an interesting period at Plantagenet Consulting Ltd.  The changes in Government Machinery, the "bonfire of the quangos", and the savage cuts to all and sundry in the public sector have proved to be a challenge for companies like Plantagenet.  However despite that, I have had the immense pleasure of continuing my working relationship with HOST Policy Research, and have in fact now become a partner in HOST.

HOST is a larger organisation, and has a longer track record than Plantagenet, and has a wide range of subject experts to call on to meet client needs.  It can therefore be rather more proactive and outgoing.

In February 2013, HOST will be offering a series of masterclasses at Roffey Park near Horsham.  Dorothy Berry-Lound - the founder and Managing Partner of HOST - will be offering sessions on how to set up impact assessments and evaluations and managing research sub-contractors.  I will be leading sessions on scenario planning and developing a research strategy.  For those that stay over at Roffey Park with us, there will also be extra free sessions in the evening.

You can find more information and how to book at this web page.

 

Better Statistics education for all

The Royal Statistical Society have recently launched a report alongside The Actuarial Profession looking at the quality of teaching of statistics in schools and the role that statistics should have in eduation.  This is so important I am plugging it and their campaign Getstats shamelessly.

I don't know about others who use statistics in their job, but I am fed up with the knowing semi-humorous comments by the likes of Terry Wogan about "another government survey - ho ho" or "lies, damned lies and statistics".  That phrase was more a warning for professionals not to abuse the power that they had over numbers.

Similarly every latest health scare in the press that tells you not to do something that contradicts the last health advice because the likelihood of dying from something has doubled from 1 in 100,000 to 2 in 100,000 really gets on my nerves.  My health statistic?  The likelihood of you dying at some stage is 100% - some may say that there was a chap that didn't do it, and therefore it is less than 100% but I say he had help from another Source and that doesn't count.

So support a worthy cause, and take some time each day to be come a bit more numbers savvy, because otherwise you will be duped at every turn and will only have yourself to blame.

 

Evaluation - what is the purpose?

As budgets in the public sector are constrained by Government policy, one area that seems to be being squeezed is evaluation.  Recently a discussion with a former client over the progress with their new project provoked the following exchange:

- I gather the new project is proving interesting?

- O yes, very much so, but challenging too.

- How will you be evaluating it?

- We don't need to do that, as the requirement for evaluation by the funders has been dropped.

If evaluation was only to prove to funding agencies that work is being done to the schedule and cost that is set out in the contract, then it would be called audit.  There is a real purpose to evaluation, or rather a number of purposes.

Most importantly, formative evaluation, undertaken during the life of the project or programme is there to help those delivering the project ensure that things are going to plan, that there are no perverse effects to beneficiaries or to the market place, and that the systems they employ to manage the project are adequate and not overly onerous.  Lessons can be learnt and changes implemented to ensure that the full final benefits of the project are achieved.

Summative evaluation, at its most basic, provides evidence to all that the project achieved all (and hopefully more) that it set out to at the start, demonstrating the quality of the project manager or organisation in being able to deliver successfully and to time.  Better evaluation will also set out the impact of the project on its beneficiaries and to the wider community or economy to provide evidence for future planning and policy development.  It also should provide an opportunity for a full 360 review of how the project worked to inform future work patterns and personal development plans.

Evaluation is not a nice to have add on to a project, it is a key component of a project, without which wheels will be reinvented, or perhaps TECs and Business Links - who knows?

 

Forecasting and Planning for Extreme Events

Pity the planners and forecasters - their names are mud across Northern Europe as snow brings transport, delivery services and health services to their knees.  Why?  Because most of the time, their forecasts and plans save the public purse and private corporations huge sums of money, by not having stocks of materials, staff and plant on hand but not doing anything.  However extreme events show where the gaps are to cover the worst eventualities, and that is what we are facing at the moment.

When I did temperature demand forecasting for British Gas a long time ago, we planned for peak days - a 1 in 20 cold winter had to be accounted for with ease and on long term planning we looked at a 1 in 50 cold winter to ensure capacity, and a 1 in 50 cold day within the year for short term storage capacity (gas holders they used to be called).  We used weather data (daily maximum and minimum temperatures and daily wind speeds) going back to the mid-1920s to model the climate of northern England, and our reference really bad winters were 1928/29, 1946/47, 1962/63 and 1981/82, with 1946 and 1962 the worst we had to play with.  Some weather monitors are calling 2010/11 a 1 in 100 cold winter already, and we aren't half way through yet, so planning is difficult as it would appear to be more extreme than existing records.

Then think about the resources - in order to cope with a 1in 20 event, on a normal day you will have resources sitting in stock (in the case of things) or that can be diverted to other uses (people, but only if that is in their contract and competence), or just sitting around depreciating (plant that can be used for one thing only - snow plough attachments spring to mind for some reason).  Those who watch the money don't like that, and will look at ways to avoid having stocks of stuff that might not be needed (excess salt), people (one of the biggest costs to an organisation) and plant (surely we can borrow someone's snow plough?) sitting idle.

When that 1 in 50 event happens, there is no-one to take your call however urgent, the salt runs out after two weeks and no-one will lend you a snow plough as they are using them themselves or can't get it to you in time.  The result is a knock on to services to the public in general or to customers in particular.  When I worked at British Gas, we were not long out of being a public monopoly and still ran the business from well head to burner, so it was our pride and our obligation (on pain of dismissal) to ensure that there was enough gas in the system to meet all domestic and non-interruptible needs.  As the monopoly went (and after my time I hasten to add), domestic customers did get cut off and in events that were not as extreme as a 1 in 20 winter.  It was no longer an integrated system and cost savings had to be made.

Then add in the human effect - despite warnings to not travel unless absolutely essential, Parisiens decided that Christmas Shopping, visits to beauticians and hairdressers were essential and got stuck in their cars and had to be rescued.  You can warn, but people won't listen because it won't affect them.  Similarly, the most extreme weather event I dealt with at British Gas was in 1991 when we had a foot of snow, but it wasn't that cold.  Gas usage rocketed because of the psychological effect of snow.  If we could have found the records for lying snow in the weather archives going back to the mid-1920s, I am sure a 0/1 variable for snow would have improved our peak day forecasts. 

Nature is chaotic, and we can't plan for every eventuality (Haiti and the Boxing Day Tsunami are good examples), and if we plan for the very worst, there will be resources that are rarely if ever used.  The irony is that this cold snap has come at the start of a major UK austerity drive in public spending - to save money public sector staff will go, but who will drive the gritters and ploughs in future and who will pay for the salt/grit mix?  I'm not sure tax and council tax payers will want to find the extra cash somehow.

 

Welcome to my world, Mr Cameron!

"We was robbed" seemed to be the general opinion on the lack of success of the bid to host the World Cup 2018.  There were also comments about the unfairness of producing the bid with the best marks but not getting the job, and how the voting process should be changed, and the relative waste of spending money on producing materials as requested, when Britain was out of the process after the first bidding round.  It was all wrong and frightfully unfair!

You would think a politician would understand the vagaries of a voting system, but I doubt our Prime Minister, Prince William or David Beckham have had much experience in the last three years of public tendering processes.  This is why I say "welcome to my world!"

There are very few projects nowadays that are offered to external organisations (builders, consultants, advertising agencies to name a few) without some sort of tendering process, and while not of the scale of a World Cup bid, actually for a micro-enterprise like Plantagenet Consulting, the tendering process is a major expense.  First there is finding out about the tendering opportunity - BIS kindly provides the Supply 2 Gov website, but at about £250 per year subscription for anything other than the most basic view, this is not cheap (other portals are available I hasten to add). 

Relevant opportunities are either one off contracts or framework agreements.  For a business the size of Plantagenet Consulting, you can forget framework agreements - you can never get beyond the tickbox element for the various policies you have to have, as the fact that you have fewer than 5 employees, so you don't need them, is not a valid excuse in general.

So you are looking at an invitation to tender for a single contract.  If you are lucky, the potential client does not have a tendering portal such as Bravosolutions (again other portals are available) to battle against, and does not want a tender in hard copy (don't get me started on the challenges of the Mail system and courriers), but will accept an e-mail document. 

In order to produce a good tender, that meets the requirements of the potential client and represents you well, I estimate the minimum time required is 2 days, and the maximum viable time is two weeks, depending on the potential value of the project.  For a micro business that is between 2 and 10 days, and for an individual consultant, allowing for admin and such things we reckon there are 15 billable days in a month, but for a bigger project, you can see that every extra day is an opportunity cost of whatever your day rate happens to be.

If you are incredibly lucky, you get the project and away you go.  If you are "lucky" you are shortlisted, and get to go to all the additional expense of attending a beauty parade with a presentation of how you and yours would best meet the customer's needs.  You are never recompensed for attending these parades, even if it becomes clear that you are only making up numbers.  Sometimes your pitch wins, sometimes it doesn't.  And in at least 50% of cases, your excellent tender will be rejected (and realistically it will often be a higher per cent of cases), and sometimes you will get useful feedback and sometimes you won't.  If you do, it must be valued and pondered over.

Such is the world of the consultant, occasional highs leading to valuable work and rewarding new relationships, but so often counting the not inconsiderable cost of being one of those who didn't win.  Often as well, you feel that actually you did a good job, and deserved a better hearing, but as with the World Cup bid, it just didn't happen.

Mr Cameron, if you have read this, consider the amount of wasted effort and money in ALL unsuccessful bids around the country in a single year.  Your bid's failure was a bit more public than most, but you expect public suppliers to go through that self same process and beauty pagent, and invest proportionate costs.  In your failure, you found a peer group to relate to!

Evidence based policy?

Those of us in the consultancy game, particularly around evaluation, like to think that when we write conclusions and recommendations, someone listens to them and does something about them.

I have experienced the glow from time to time of having my findings reflected back to me - most recently on a wonderful project called Business Brokers in North Staffordshire.  There, with a colleague and working under the Inspira Consulting banner, we did an extensive mid-term evaluation of the programme to help them meet their contracted targets and also help them think about the future beyond their funding contract and how they could ensure the sustainability of the service.  In the last three months, we have gone back and done the final evaluation of the funded programme, looking at contract performance and also at how our recommendations were followed up.  They had done extremely well in achieving their contracted outputs and have found ways to maintain the service, albeit at a lower level, but then at the moment, they have less funding.  Whether it was polishing our egos or not, they did say more than once, "following your advice we looked at this" or "we have done this".

However in education matters, this is often far less the case.  My impression has been that decisions are made as knee-jerk reactions, without letting the previous set of changes embed themselves and demonstrate worth, and due to the loudest set of shouting in the media.  I am not totally sure how often Apprenticeships have been re-branded since they were reintroduced in the late 90s, but the myriad brands have not helped employers or prospective employers identify with the product.  Similarly, new qualifications are brought in and then dropped at the drop of a hat - who remembers GNVQs?  And if you have one, how do you explain it to people now?

It seems that my impressions are not totally without foundation for once as well.  This article on the BBC website heralds a report that says just that - politicians do not look at the evidence when changing education policy and practice.  For those of us who have recently been following the mantra of "evidence based policy making", this will be something of a shock, but then - are politicians going to take any notice of this report either?