Does coaching work? Should coaches prove a return on investment (ROI)? Or is it enough that they justify their interventions on the basis of plausibility and reputation alone?
It is tempting to assume that if a leadership programme deploys best practice, and has the stamp of approval from a respected supplier, then it must be doing some good. Yet this may not be the case.
The March issue of Harvard Business Review contained a case study where a leadership development policy had such a negative effect on the business that it threatened its survival1.
The company (given the codename ProductCo) had adopted a policy of job rotation for high-potential individuals. High-flyers were moved every 16 months or so in line with a best practice policy implemented on the advice of a leading management guru. But the company relied on long-term project management.
Analysis of the practice of rotation found that it “destabilised core processes… and prevented the development of deep technical expertise”. There was an “implosion” of quality that annually cost the company more than $1 billion, the study concluded.
It’s a sobering thought for those involved in rolling out top-level leadership development initiatives.
However, it is irresponsible to be unconcerned about the impact of coaching on a business. As Neil Twogood, chief operating officer of Performance Consultants and a former finance director, says: “With coaching, if the company is paying the fee, it is completely reasonable for it to want to know the value.”
Research by Alison Carter, principal research fellow at the Institute of Employment Studies, suggests that “listening only to what the coaching clients say can paint too rosy a picture of the coaching programme” (see “How to evaluate coaching”, Coaching at Work, vol 4, issue 2).
Once it is accepted that the impact of coaching should be assessed, the next question is: How?
Quality vs quantity
Traditionally, the two camps have been those who favour an ROI type of analysis, and those who favour qualitative assessment. Experts interviewed for this article indicate that the dividing line does not matter. The more significant difference is between those who link coaching interventions to business objectives, with clear, continual assessment of impact, and those whose coaching is more ad hoc.
Qualitative and quantitative techniques are complementary; neither is capable of producing the full picture. The Kirkpatrick model sets four levels for assessment of a training intervention: individual experience, learning, behaviour, and business impact. Coach Brian Wishart, BP’s former head of organisational and individual learning for Europe, comments: “A lot of people use Kirkpatrick but are only at level one. People are pleased if they turn up on time and cover what they are supposed to cover. It has to be challenging to move up to the fourth level: was it transformational? Did the business change?”
There has been considerable progress in developing objective methods of assessing business impact, but also increasing awareness of their limits.
Haig Nalbantian, partner at Mercer and co-author of the HBR article, has spent 15 years developing and refining methods of statistical analysis to assess the impact of people management decisions. He says management should engage far more in assessing the facts of the matter on the ground.
Show me the evidence
But he stresses that management can never be a precise science. “I use the term ‘sleuthing’. When you use data mining, you never make a recommendation based purely on a statistical relationship. You need theory, experience and statistical relationships. It’s like detective work.”
Others concur that aspects of management are inherently unmeasurable. Tom Smith, head of organisational development at Lane4, says: “There is a tendency to think that on a five-point scale, a score of ‘five’ is five times bigger than one – that it’s a linear scale, for example, in a performance appraisal. And you know that is not the case. What we’re trying to look at is a multiple method approach, using three or four ways to measure the same thing.”
Many aspects in management are non-linear, ambiguous or hypothetical. You cannot measure what might have been. For example, coaching might have helped an individual to handle a key business relationship in such a way that it appears to have led to retention of a major contract, but it is impossible to be certain that it would not have been the case anyway. So the ROI for the coaching may be either millions of pounds – or nothing.
Starting point
The fullest understanding can come from continual enquiry as part of the coaching process. Carol Wilson, managing director of Performance Coach Training, says: “Someone in a coaching session tells me: ‘I’ve started getting reports in on time, for the first time ever.’ I might say: ‘What does that mean for the company – what are the benefits?’ And the individual may say: ‘It means the sales force is better prepared.’ And I say: ‘How will that affect profits?’ – ‘Profits will go up by a percentage’, and I ask: ‘Could you give an estimate?’”
As Margaret Chapman, coaching psychologist and author of The Emotional Intelligence Pocketbook, wrote in the aforementioned Coaching at Work article with Alison Carter: “Numbers can convince stakeholders who are paying/sponsoring the coaching, while individual success stories can easily be remembered and used as illustrations.”
The challenge, however, as Chapman told peers at the First European Coaching Psychology Conference in December, is that “despite the interest in coaching ROI and cost benefit kinds of outcomes, well-designed and well-documented ROI evaluation efforts are rare”.
It is also important to measure a baseline. It is not possible to assess progress, after all, without knowing the starting point. Twogood says: “You have to articulate what it is you are trying to do, so that at the end of the intervention, what will be different? And why will that make a difference to the business?”
Yet coaching outcomes can be unpredictable, he adds. “I coached an executive [who] had a real problem with presentations. The company came to the conclusion that it wasn’t worth him carrying on in that role. He moved sideways, to a role in which he was much, much happier. If we had specified an outcome, it could have been ‘to be much better at presentations’; but what they have is an employee they value, making better use of their strengths.”
With coaching, it is common for interventions to have grown in an unco-ordinated way. Wishart says such a pattern had built up at BP: “There were lots of coaching suppliers on different rates at different business streams and no attempt to work out if they were adding value or whether they were accredited… Some were trained over the weekend; others had PhDs.”
Hard line
He put together a coaching framework in conjunction with procurement, to give it a business focus. “Procurement people were used to dealing with large bits of machinery. They took a hard line. But I think that was required.”
Sandra Hilton, consultant at Sheppard Moscow, argues that individual and business objectives have to be tied in together. “I never do coaching as a remedial action. I don’t believe coaching is a suitable intervention for fixing people. For me, one of the first questions would be: what is the purpose of the coaching? What are you expecting to be the return? What does success look like? Objective-setting becomes really important. I coach in keeping with a contract all the way through.”
Coaching at a senior level should have an unashamedly commercial focus, she adds. “It is helping senior people really keep focus on the bigger picture. You are challenging that as a coach. If you are senior and working with a coach, you are signalling that you are willing to be challenged.”
Tiffany Gaskell, executive director at Performance Consultants International, is a former investment banker at Credit Suisse. As such, she is comfortable with multi-method approaches of measurement, including a quantitative element.
Systems thinking
The subjective viewpoint is valid, she says, as long as you are honest in the approach. “What you are measuring [in coaching] is the behaviour change and the impact on the business. We make sure that the answers are adjusted for probability. So we ask: ‘On a scale of one to 10, how confident are you that it was the coaching that made the difference?’ This is the way we work with clients; they’re happy with this.”
With this approach, quality assurance and ROI assessments are bound in together. “For example, one managing director wanted to develop his leadership skills.” Learning how to delegate led to one of his team generating a £7.5 million return. Of overriding importance in evaluation and ROI is systems thinking – planning coaching in line with business strategy, and continually assessing the impact.
The approaches of Nalbantian – who uses advanced statistical analysis – and other coaches interviewed here – who use more qualitative assessments – may look like polar opposites. In fact, they share the following concepts:
• People management programmes affect the business, and may affect the whole organisation.
• These effects are complex, with intended and unintended results.
• The purpose of the HR investment should be to help the organisation as well as the individual(s) being invested in.
• It is the duty of the coach and HR managers to continually monitor the impact of the intervention on the organisation, and to amend it where appropriate.
The important dividing line is not between the statistical measurers and those that provide a qualitative description. Rather, it is between those that think strategically and those that do not; those that link coaching to organisational development and those that are more piecemeal.
Assessment of the business impact of a coaching intervention is both necessary and possible. Ideally, it combines quantitative results with qualitative analysis, rather than pretending that a precise return can be calculated.
Reference
1 H Nalbantian and R Guzzo, “Making Mobility Matter”, in Harvard Business Review, March 2009.
Case study 1: Sainsbury’s
Background and objectives
In the 1990s, Sainsbury’s lost its crown as market-leading food retailer in the UK to Tesco. In October 2004, Justin King, the supermarket’s new chief executive, launched the “Making Sainsbury’s Great Again’” programme. Part of the plan was to encourage and nurture the engagement of all 150,000 employees.
The programme
The company hired Lane4 to help with the delivery of the programme, which involved developing 326 leaders, and teaching coaching skills throughout the business. Sainsbury’s also introduced a masters-level Coaching Accreditation Programme, accredited by Middlesex University.
Evaluation and ROI
A multi-method approach was used. Qualitative responses to questionnaires on the Tough Love programme show a mean score of 5.86 out of 7 on a range of capabilities covering improvement of leadership and teamwork skills. On the masters programme, there was a 6.3 mean response.
What were the business benefits? One respondent described how the coaching had helped the store to re-open and begin trading swiftly after a £1.5 million refit: “We are the only store in the region to come out of the refit programme and be profit-making… We were the only store to land sales budgets from day one re-opening after a refit. Before coaching, we weren’t as high-performing a team.”
Measuring the impact of coaching
1 Assess the baseline – it’s impossible to gauge progress unless you know what the starting point is.
2 Link the intervention to defined organisational objectives.
3 Measure the results over time, and continually.
4 Commit to systems thinking: all leadership development interventions will have an impact on the organisation – potentially a significant one.
5 Where resources allow, use a mix of measures – sales (where relevant), profits, statistical analysis, employee feedback and participants’ views – to build up a broad picture of organisational dynamics. Quantitative measures often indicate the “what”; qualitative measures indicate the “how” and “why”.
6 Be proportionate – do not overcomplicate or engage in advanced measurement where the cost of measurement would outweigh the business benefit.
Source: interviews for this article
Case study 2: Multinational business-to-business and retailer
Background and objectives
The incoming global head of learning and development at a multinational business-to-business and retail firm hired Performance Coach Training to design a programme to address problems of staff retention, absenteeism, lack of motivation and a ‘telling’ rather than coaching style of management.
The programme
Performance Coach Training carried out a survey designed to ask staff and managers about their personal values compared with the values they felt the organisation exhibited and those they wanted to see in the workplace. This uncovered challenges concerning motivation and practical operational matters, which focus groups followed up.
The coaching intervention included: one-to-one coaching for the CEO and board members; training in coaching skills at all levels from team leaders to the CEO; training of internal coaches: and a “train the trainer” programme for learning and development managers.
Evaluation and ROI
Evaluation was through business results and qualitative information from questionnaires. At the end of the first year, various surveys were run.
The results included:
• 18 per cent reduction in absenteeism;
• 26 per cent improvement in retention;
• out of the 10 current organisation values in a survey on values, positive values rose from four to five;
• profits up between 4 per cent and 11 per cent across the UK.
Carol Wilson, managing director of Performance Coach Training, comments: “To be effective, ROI must be a continual, ongoing process approached from all possible perspectives including formal and informal means and, wherever possible, the results estimated by individual identified managers should be published.”