Research matters
How can we measure return on investment in coaching? It’s tempting to use anecdotal research but we need to be much more clear and consistent in our approach, say Paul Stokes and Ruth Garrett of Sheffield Hallam University
One of the current preoccupations of coaching research is return on investment (ROI). This is noticeable in the rise in conference offerings and articles on the subject. Meanwhile, a search on Google for coaching and ROI brings up 1,480,000 hits. That said, this does not seem to have translated into many published research-based articles on the subject.
At the Coaching & Mentoring Research Unit at Sheffield Hallam University, we are constantly on the look-out for published research on ROI in coaching to supplement our primary research and coaching experiences. However, we were disappointed to find in some of these studies a number of flaws.
First, we looked at McGovern et al1. The authors talk about coaching being “underutilised”, despite its popularity. This is “because the paucity of empirical research into its effectiveness leaves the field open to speculation and subjective opinion”. This contrasts sharply with their methodology, which involves inviting interviewees to provide estimates of the benefits of coaching. This is “conservatively” adjusted based on the interviewee’s confidence in their own estimate. Using the costs of coaching, the estimate and the confidence factor, the researchers claimed that for their sample of 100 interviewees, the average ROI was 5.7 times the original estimate. Our concerns about this are that:
- using financial data in this way gives off an aura of objectivity and precision, but these .fiures are essentially based on the subjective view of interviewees;
- the authors draw heavily on personal anecdotes from interviewees to support their claims, which seems to contrast with the initial call for a move away from “speculation and subjective opinion”;
- all of these stories are positive and include no negative impact of coaching.
- We then looked at Parker-Wilkins2. Again, interviewees were asked to estimate their own monetary value for each benefit they felt they had accrued from the process. This was again adjusted for confidence of estimate and estimated confidence of contribution. We were concerned here with:
- such a precise claim for ROI based on such estimates – ie, 689 per cent;
- a lack of an attempt to control for self-reported benefits;
- a use of anecdotes to support and explain financial data.
These approaches are useful in that they draw attention to the benefits of coaching, but they seem freely to mix the authority that comes from financial data with the richness that stories about coaching can provide. Unfortunately, such research ends up being neither fish nor fowl. On the one hand, there is a lack of rigour when applying traditional research protocols to ROI and coaching; on the other, the narrow focus on hard measures leaves little space for the richness of a properly executed narrative analysis. It is worth recalling the advice of Alison Carter3: “Academics are well aware that problems of causality can be viewed as a barrier to evaluation, especially for employers that might be seeking to ‘prove’ something. There is no logical causation between improved business results and the fact there has been coaching and such links are interpretative.”
In other words, it is tempting for those of us who are fans of coaching to make strong claims about its usefulness. But we need to be clear and consistent in terms of the basis for these claims.
References
1. J McGovern, M Lindemann, M Vergara, S Murphy, L Barker and R Warrenfelz, “Maximising the impact of executive coaching: behaviour change, organizational outcomes and return on investment”, The Manchester Review, vol 6, no 1, pages 1-9, 2001.
2. V Parker-Wilkins, “Business impact of executive coaching: demonstrating monetary value”, Industrial & Commercial Training, vol 38, no 3, pages 122-127, 2006.
3. A Carter, “Practical methods for evaluating coaching”, Institute for Employment Studies, Report 430, 2006.