The measure of you

The number of organisations using coaching is steadily rising, yet its true value is still not being assessed. The CIPD’s John McGurk shares his practitioner guide to real-world coaching evaluation

I was amazed when a colleague told me that the energy company she works for doesn’t use coaching. After all, it’s now part of normal management practices for most organisations, as a string of Chartered Institute of Personnel and Development (CIPD) surveys have shown.

Coaching and mentoring are powerful and enabling tools for raising performance, aligning people and their goals to the organisation, and cementing learning and skills1. Coaching is also a powerful agent for driving cultural change and agility – organisations use it linked with organisational development2.

However, coaching has an Achilles heel. Evaluation is largely neglected and this mustn’t continue, particularly in the current climate when every item and line of expenditure is being acutely scrutinised.

While the number of people who report the use of coaching is steadily rising, its real value is still not being captured, as we established in our 2010 Learning & Talent Development survey.

Make the link

Well under half of respondents used approaches such as linking coaching outcomes to key performance indicators (KPIs) or more quantitative and mixed-method approaches, such as return on expectation (ROE) and return on investment (ROI). Few linked coaching evaluation to performance and only 13 per cent frequently discussed evaluation at management meetings. Only about 20 per cent frequently collected and analysed data on the impact of coaching.

Clear and present danger

In difficult times, anything that cannot prove its value will be increasingly vulnerable. Coaching cannot claim a unique contribution to organisational performance and impact if its practitioners and champions assume its value rather than prove it. We need to build a convincing evaluation narrative, yet many organisations are failing to do this.

We need to move towards a systematic approach based on a thorough review of the coaching process. The CIPD sees evaluation as a cornerstone of effective coaching and we want to assist practitioners in developing best evaluation practice. So what’s getting in the way? Perhaps it’s our focus on delivery.

Delivery focus

Many practitioners think that developing and delivering coaching is what they are there to do. This can lead people to believe that simply introducing coaching is enough. As Jarvis et al pointed out in The Case For Coaching (2006), there is often an assumption that time spent in any learning activity such as coaching always has a positive payback. The authors also suggest that evaluation may not be addressed because we might uncover negative results that could threaten coaching. Although we know from our surveys that coaching is being used primarily for performance management and leadership development, we know less about its impact on those areas.

There are other issues too:

  • An overuse of the Kirkpatrick model, even the augmented versions. Despite valiant attempts by Kirkpatrick and his successors to update the model, many use the least sophisticated version based on reactions and anecdote. But we need to look for broader and richer approaches to evaluation.
  • An obsession with a very narrow view of ROI. This generally subtracts the costs from the benefits of a coaching assignment and expresses that as a percentage. This is meaningless without the context of the coaching and evaluation of other activities. Philips and Philips (2008), provide a much more robust and systematic ROI approach, which is detailed in the report. As evaluation expert Paul Kearns argues, ROI without a baseline is next to useless.
  • Concern that evaluation is not a favourite activity of L&TD practitioners. We explain in the report that MBTI type ENFP is well over-represented in the coaching and L&TD community, so we have to be more mindful about developing evaluation. L&TD practitioners work best with delivery and collaboration over learning issues. Getting down to the data may not be their favourite task.
  • Our use of the softer data around coaching is not systematic. For example, coaching conversations are a source of rich data about the progress of coaching. With an appropriate and proportionate approach to confidentiality we can use basic tools to capture the nature of the conversation.
  • A lack of systematic collation of the sources of data to inform coaching. For example, psychometrics test pre-employment, manager reports, 360s and employee engagement scores, all provide valuable data.

Knowing and doing

So what should we do? Evaluation starts with delivery and what people want from us.

Stakeholders won’t expect us to produce a spreadsheet with scenario forecasts for coaching and ROI. They are more likely to be convinced if we can tell them how many people are coached, how much we spend on external coaches, the length of assignments and data on impact: perhaps engagement scores before or after coaching, or maybe anonymous 360 feedback on people’s ability to complete projects.

If we are also generous about interventions introduced by other departments and we can apportion some of the effect to coaching, we will have compelling evidence. That’s not happening enough.

We use the lift conversation with the finance director to illustrate how poor evaluation can undermine the resources available for coaching in testing times. According to our survey, roughly seven out of every ten lift conversations would not go well.

What next?

The CIPD believes that the best method of evaluation for any L&TD intervention is to take a holistic approach – the Value of Learning approach we refer to in our research with Portsmouth University (2007). We should shift from this narrow ROI to ROE approach. What did we expect the coaching intervention to deliver? Which behaviours or skills do we wish to see? What improvements?

This raises the issue of alignment. Aligning coaching interventions to the goals of the business is key. The basics are simple. Make it relevant, align what you are doing and measure it.

We provide a simplified graphic model of the approach in Figure 2 above. The RAM (relevance, alignment and measurement) approach is useful for all learning and talent interventions and keeps us focused on the outcome, not the process.

Finally, an integrated approach is vital. If, for example, we are unaware of the sponsorship and ownership issues within the organisation, we won’t get a clear view. If we are not conversant with the positioning and purpose of coaching we cannot design evaluation effectively. If we haven’t given a great deal of thought to how coaching is resourced and paid for, including issues like the role of external support and consultancy, the use of internal coaches and the training of line managers, we will not be able to evaluate effectively from the start.

The CIPD has developed the OPRA model (Figure 1), which helps practitioners think about coaching from the point of view of Ownership, Positioning, Resourcing, Procurement, Assessment and Evaluation. This thinking tool can help us to provide an effective space for evaluation.

What the CIPD surveys revealed

  • Coaching is not being effectively evaluated (CIPD Learning & Talent Development surveys)
  • Only 36 per cent of organisations evaluate coaching (CIPD L&D Survey 2010)
  • The minority that do evaluate, focus on qualitative assessment, such as reaction and stories and testimony
  • Where quantitative evaluation is used, it is often a crude use of return on investment (ROI)
  • Evaluation is not being grounded in a capability perspective linking it to the organisation and people plan
  • Good practice is out there and a more systematic mindset would deliver a step change in evaluation performance
  • A wide range of data can be used for coaching evaluation, but practitioners need to access these data streams

Source: CIPD’s Real World Coaching Evaluation Project (autumn 2010)

Figure 1 The OPRA model
Figure 2 The RAM model

References and further information

  • 1 CIPD 2011 Learning & Talent Development survey
  • 2 CIPD Sustainable Organisation Performance: What Really Makes the Difference? Shaping the Future final report (January 2011)
  • CIPD, Taking the Temperature of Coaching, 2009
  • CIPD, Developing Coaching Capability in Organisations, 2008
  • CIPD, Learning & Talent Development surveys’ Training and Development surveys, 2005-2010
  • CIPD, Value of Learning: From Return on Investment to Return on Expectation, 2007
  • J Jarvis, D Lane and A Fillery Travis, The Case for Coaching: Making Evidence-Based Decisions, CIPD, 2006
  • P Kearns, How Accurate or Necessary is ROI for Learning and Talent Development? (Presentation to CIPD HRD Conference 6-7 April, 2011)
  • J Passmore (ed), Psychometrics in Coaching, London: Kogan Page, 2009
  • J Phillips and P Phillips, ROI in Action Case Book, Pfeiffer, 2008
  • J Phillips and P Phillips, Show Me the Money: How to Determine ROI in People, Projects and Programs, San Francisco: Berrett Koehler, 2007
  • J Pfeffer and R Sutton, The Knowing-Doing Gap, Harvard Business School, 2004

Coaching at Work, Volume 6, Issue 4

Let’s get engaged

Learning, engagement and networking are Linda Aspey’s passions, and she has found the perfect way to use them. The chair of BACP’s new Coaching division tells Liz Hall why she is ‘absolutely thrilled’ about her new role Linda Aspey, chair of the British Association for Counselling & Psychotherapy (BACP)’s new Coaching division, is passionate about […]

Altered images

How can we assess the organisational impact of executive coaching interventions? By asking third parties to share their perceptions, say Erik de Haan & Christiane Niess Billions of dollars are spent every year on executive coaching. As far back as November 2004, Harvard Business Review estimated the global market to be worth around US $1bn. […]

Keeping council

Most in local government are braced for difficult times ahead. But not North East Lincolnshire Council and it’s all down to the performance coaching programme it put in place three years ago for its managers. Liz Hall finds out why Britain’s public sector, and the rest of the nation, has been left reeling in the […]

Chinese whispers

How is coaching growing in China? Slowly, says Dinah Gardener, of a culture that views the Western concept with suspicion. But a combination of best practice and Chinese wisdom could be about to change all that Coaching in China is beginning to take off. Twenty per cent of delegates at the inaugural Asia Pacific Coaching […]

Are you there?

Fans of the work of Ingmar Bergman, will appreciate the importance of silence in human interaction. In coaching too, silence can be golden. But just how does that work in a telephone session? John Charlton explains Silence is a really important part of good coaching. A good coach knows silence indicates reflection,” says independent coach […]

A fair cop

With 60 per cent of its short-term offenders returning to jail, Hull had a huge and costly problem. That is, until ex-chief inspector, Russ Waterman, came up with a liberating solution. Liz Hall investigates Russ Waterman used to be a “typical cop”. He went to work, did his paperwork and went home. But after a […]

Adapt and survive

Senior managers are increasingly being asked to work across cultures on a global scale. To do so they need to let go of their ingrained beliefs and become cultural chameleons. Jenny Plaister-Ten explains why this ‘unlearning’ is the jewel in the crown of global coaching In today’s global economy, executives are increasingly required to be […]

If the hat fits

Much has been published on how to improve leadership performance, yet its practice remains uneven. Could coaching hold the key to transferring learning to the workplace, asks Jonathan Passmore There is no shortage of books on leadership, many of which claim to offer a unique model or a shortcut to greatness. Yet despite such wide […]

On song

What has music to do with coaching? Rachel Ellison brought her iPod along to a session with a high-level executive. His personal transformation proved to her she was on the right track Go and get your pads on,take your place in the sun From Bangalore to Kingston, the age of revolution… from The Age of […]