Jessica Jarvis, CIPD adviser for training, learning and development, says “There is not much out there on evaluation, and companies are struggling. People understand that evaluation is important, but the word seems to strike fear into practitioners as they think it is hard.” But, as Professor David Lane of Middlesex University asks, “What if the main ingredient for the success of coaching is that managers sit down and listen to employees? Why bother with coaching then?” Lane has worked with Jarvis in researching the soon to be published The Case for Coaching: Making Evidence based Decisions on Coaching, co-written with Annette Filery-Travis. They looked at the perception of coaching in 100 companies, reviewed 170 case studies in the current literature, and examined 30 companies directly as case studies. The literature review showed more questions needed to be asked, says Lane. “As with any emerging profession, there is plenty of research on proprietary systems. People are coming up with unsurprising results that coaching is wonderful. What we need is to ask whether interventions really have an impact, and whether some of the measures are just perceptions.
“We’ve asked practitioners to point to things that have actually changed, and asked what seem to be the factors affecting those changes. We’ve then compared the results with our own analyses of various cases. But although this is a reasonable starting point, it is not research at a level that would be accepted in other professions.” Jarvis and Lane think organisations need to take the bull by the horns if they are to get the most out of coaching. Over the past year or two, coaching has become more widely entrenched and companies believe they are seeing definite benefits. Now they need hard facts to back this up.“Practitioners have to take this on board as much as academics if they are to persuade the sceptics, benchmark against other organisations and make improvements to coaching,” says Jarvis. Lane and Jarvis’s research suggests that coaching is useful for some skills work, performance and development issues, but not for major systemic issues and for people who are resisting change.
Measuring the effectiveness of coaching was certainly a hot topic among 20 senior HR professionals from organisations such as Unilever, HSBC and KPMG, who gathered at a symposium last August hosted by the London-based School of Coaching. Tricia Bey, managing director at the School of Coaching, says that there was much talk about HR increasingly facing questions from their colleagues in finance about the return on investment (ROI) of coaching. But all agreed that there was no single, simple way of measuring value.
“The outcomes of one-to-one coaching programmes with external coaches are difficult enough to measure. So finding a satisfactory way of measuring the ROI of developing and using coaching skills as part of everyday management and leadership was thought to be even more elusive. But the links are strong,” says Bey. Carole Gaskell, managing director and founder of coaching firm Full Potential Group, recommends putting in evaluation measures up front. “You tend to get what you measure, so it’s important to do ROI. We find organisations are much more likely to achieve benefits if they work it all out at the beginning,” she says.
In its assessment of ROI, coaching is ahead of other HR interventions, she adds. “Coaching is leading the march in being focused on ROI perhaps because it is high profile and because it can have such stark dramatic impacts. As coaching can have such direct impact on the bottom line, it is a no brainer to measure it, and it is easier to measure than many other HR interventions.” As for measuring perceptions, a growing body of research shows that coaching is seen as beneficial.
In a survey carried out in August of 36 executives receiving coaching in South Africa, 97 per cent of respondents said they believed coaching delivered ROI to their employer. And 18 per cent felt coaching had increased bottom-line profitability, according to the report by Jenny Hogarth, an executive coach based in South Africa and co-founder of South African coach development and standards body Comensa. There was an overwhelming belief that coaching is getting results. Of the 18 per cent who felt it had increased bottom-line profitability, most were senior executives, whereas the others tended to be middle managers who might not be able to perceive the impact on the bottom line,” she says. More than 78 per cent of respondents perceived that coaching had improved relationships at work and ability to handle conflicts. Seventy-eight per cent felt their relationships at work had improved, 63 per cent reported more satisfaction with their career and 60 per cent reported productivity rises.
Another piece of research showing benefits from coaching is “Maximising the impact of executive coaching”,1 which found a perceived ROI of almost six times the initial investment. The research posits that coaching translates into doing, that this doing translates into an impact on the business, and that this impact can be quantified and maximised. It draws on Kirkpatrick’s paradigm for evaluating the effectiveness of development programmes,2 which examines four levels of impact:
- reaction to the programme and planned action
- learning
- behavioural change
- business results
It then adds a fifth level ROI.
The research looked at the coaching provided to 100 US executives who rated their level of satisfaction with the coaching process, described their goals for coaching, and rated how effectively they had achieved these. It then averaged the ROI for the 43 participants who had estimated it. The result was 5.7 times the initial investment in coaching (see “Calculating ROI for coaching”).
Keep evaluation simple
However, evaluation needs to be approached with care, say the experts. At the School of Coaching’s symposium, there were warnings against over complicating the evaluation process, searching for too scientific a tool, or creating an expectation of instant value. And both David Lane and Jessica Jarvis stress that organisations should be realistic and pragmatic. “Start somewhere, and measure as and when you have time and resources, gradually building up an evidence base,” says Jarvis. One simple approach, she says, is to ask internal managers to come up with three examples of how coaching has helped them to do their job better. “These can be used alongside harder forms to persuade senior people of the power of coaching.”Where someone is undertaking coaching training, she suggests asking their manager what improvements they have seen in softer skills. These might include motivation or managing the team. These criteria should be picked carefully “look for what senior managers really want to know, and link measures to that” (see “What to measure” panel, above). Jarvis also suggests taking the previous spend on development activities and comparing this with coaching. “Some of the case studies for our book said that if they had put people through a business programme instead of having an executive coach for six months, it would have been more expensive and less enjoyable for the executive. So organisations can do cost-benefit analyses without doing extra work,” she says.“They can also look at changes in employee attitude surveys and feedback from people on courses. They will get an overall picture emerging.”
Some companies are beginning to carry out internal trials comparing results among coached employees with those among non-coached staff. This process can be relatively easy, says Lane. “Often coaching is phased in, so it is possible without much effort to run comparison trials.” He says companies should address three key questions: what the issues are, what process brings about change, and whether the change is beneficial in a broad or narrow sense. “Inner benefits” are a useful focus, says Mel Leedham, director of research and programme design at the Oxford School of Coaching and Mentoring. “A lot of coaching is about working on inner benefits to improve outer benefits, so looking at improving areas such as clarity, focus and confidence will help to improve performance,” he says. Effective evaluation should not be a one-off exercise at the end of a coaching relationship or contract, he adds. Instead, it “should be a continual process designed to help the organisation better understand the relationship – how it has had an impact on the participants, colleagues, peers and customers as well as on the business results, and how it has been affected by other internal and external factors”. Leedham suggests concentrating on measurable factors such as the qualities of the coach, the feelings and behaviours of the coachee, and their improved performance and capability. He believes that if organisations do this, they will not need to measure the impact of coaching directly on the business results, because this impact would be an inevitable consequence.
Before beginning
Jarvis recommends planning how the organisation intends to evaluate coaching before setting out. HR should work with the line manager and the coach early on to establish realistic ways of monitoring progress and success, she says.
- Carole Gaskell recommends firms choose three or four key areas to measure (see “ROI business measures” panel, facing page). Jarvis suggests focusing on areas such as:
- the performance of the coach
- feedback to improve the organisation’s management
- administration of coaching processes
- the individual’s and line manager’s satisfaction with the coaching intervention
- the degree of behaviour change
- development achieved by the individual
- the impact on business results
- 360-degree feedback
Carrying out 180 or 360-degree assessments at the outset of coaching, and regularly during and after, can be used to flag up any improvements in softer areas such as communication.
This approach is favoured by Jarvis, Gaskell and Professor David Clutterbuck, who runs consultancy and research firm Clutterbuck Associates. Clutterbuck recommends carrying out 360-degree questionnaires among coachees’ colleagues and requiring coaches to complete a specific set of questions on relationship quality, outcomes for the individual, and outcomes for the organisation. Seventy per cent of firms use feedback from coaching participants to measure effectiveness of the intervention, according to CIPD research (see “Measures used to assess effectiveness of coaching” box).
Employee attitude/climate surveys
The Scottish Highlands and Islands Enterprise network recently introduced an anonymous climate questionnaire to help it to evaluate the impact of its coaching focused team-leadership programme. The questionnaire is sent out to all staff every four to six months, and includes questions about senior management’s performance and leadership approaches.
The completed questionnaires are returned to external provider Genesis Consulting and the data collated in quarterly reports for senior management.
The questionnaire has pinpointed one area for concern. Susan Myles, HR director at Scottish Highlands and Islands Enterprise, says: “Already one of the problems identified is that managers do not listen sufficiently. This data will help identify improvements and problems.”
David Lane stresses that where possible, organisations should compare feedback mechanisms such as climate surveys or 360-degree feedback between coached and non-coached employee groups. And he thinks much can be improved in the way data is collected and analysed. “Most companies use unsophisticated surveys. They need to understand which factors influence other factors. This is easy with modelling, changing spreadsheets into statistical analysis,” he says. The idea of setting up control groups within retail bank branches, or within sales forces for different regions, and observing the differences between those whose managers used coaching skills in their managerial style and those who didn’t, appealed to HR professionals at the School of Coaching symposium. It would help to test the attribution weighting and refine the scorecard, which could then be used more widely, they felt.
Scorecard approach
The symposium delegates recognised the difficulties in directly attributing coaching interventions to measurable outcomes. But they thought a scorecard approach was a useful way of tracking effectiveness.
Tricia Bey suggests using a combination of tangible performance-related outcomes and more subjective outcomes. She says it should be possible to attribute business value to each scorecard element and then to estimate the degree to which coaching had contributed to the outcome expressing this as an “attribution weighting”. By combining the various elements and their weightings, a broad measure of business value could be derived.
In evaluating coaching, other areas offer lessons, says Lane. “We should learn from management, psychotherapy and psychology in general.” Meanwhile, perhaps, we should not tie ourselves in too many knots. John Hoerner, former chief executive of UK clothing and international sourcing at Tesco, believes coaching has helped the retailer’s clothing business to grow. But he says: “I think the benefits are ‘soft’ rather than ‘hard’. I could tell coaching was making a difference by talking to the people themselves – they wanted to tell me about their positive experiences. Second, in more than half of the people in the first and second rounds of the programme, I myself saw positive differences in their business and behaviour.”
What to measure
- Individual and line manager satisfaction with coaching
- Improved appraisal and performance
- Impact on business performance indicators
- Comparison of pre and post coaching
- 360-degree feedback ratings
- Achievement of objectives set at the start of the coaching assignment
- Feedback from the coach and coachee
- Staff turnover rates, particularly among key staff
- Performance of the coach and coachee
- Employee attitude and climate surveys
- Customer satisfaction levels
Calculating ROI for coaching
- Estimate value of resolving issues (£x)
- Percentage of improvement attributed to coaching (x%)
- Multiply step 1 by step 2 to arrive at savings due to coaching (£x)
- Your confidence in the savings in step 3 as a percentage (x%)
- Multiply step 3 by step 4 (£x)
- Cost of coaching (£x)
- Subtract step 6 from step 5 (£x)
- Divide step 7 by step 6 (£x)
- Multiply step 8 by 100 for ROI expressed as a percentage (%)