Are coaches today’s ‘snake oil’ merchants? Coaching continues to operate on limited evidence, but it’s time this changed, argue Andrew J T George and Jonathan Passmore
The middle of the 19th century many people made their fortunes on patent medicines, often called ‘snake oil’, that were marketed as a panacea for all ills.
These had various formulations. One contained mineral oil, callow, turpentine, pepper and camphor, while some contained cocaine, amphetamine or opium, no doubt increasing repeat sales! They all had in common extravagant claims but no scientific evidence that they worked (Young, 1961; Anderson, 2000).
So why did people buy them? Apart from the slick marketing, or persuasive charms of the salesperson, one main reason was that people who took ‘snake oil’ often got better. In many cases there would have been a clear association between using ‘snake oil’ and an improvement in the person’s condition. People would therefore, quite reasonably, believe that the medicine was doing them good.
This might seem contradictory, after all, looking back today, the contents of these ‘remedies’ was more likely to make you sick than better. However, there are three reasons for the ‘positive effects’ experienced by users that scientists have identified as the ‘placebo effect’, ‘regression to the mean’ and the impact of ‘concurrent events’ (Goldacre, 2009). Let’s look at these and their implications for coaching.
This effect is a well-known and studied phenomenon (Benedetti, 2008; Price et al, 2008; Finniss, 2018) and is down to the patient’s belief in the treatment. Its magnitude can be influenced by factors including the nature of the intervention (such as a pill’s size and/or colour, or whether the procedure involves an injection), the price paid and the perceived status of the therapist (Meissner & Linde, 2018), and the language used by the professional and the relationship between them and the patient (Glare et al, 2018; Wampold, 2018). In medicine, the placebo effect must be considered in studies determining whether new treatments work (George et al, 2016).
Regression to the mean
This is a relatively simple statistical principle (Kahneman, 2011; Farnam Street, 2019). If data is randomly distributed over time around a mean (average) value, and if you pick a moment in time when the data is far from the mean, next time you look at the data it’s likely to be closer (regress) to the mean just by chance (there are more data points close to the mean than far from the mean). For example, if you’re monitoring exam results in a school and one year it gets much better results than normal, you’d typically expect lower results the next year (assuming there was no underlying change in performance).
In medicine, patients are often treated when their symptoms are at their worst. If they have a fluctuating, non-progressive, disease you’d expect them (on average) to improve even if the treatment is ineffective.
Impact of concurrent events
Nothing happens in isolation. Most illnesses get better of their own accord. People will be taking other treatments for their condition (other medicines, bed rest or tender loving care). If you were to take ‘snake oil’, and your condition was to improve either because of the passage of time or something else you were doing, you could falsely attribute this improvement to the ‘snake oil’.
The magic treatment
The above factors can mislead any professional (and their client) into thinking what they’re doing is effective. The belief that a treatment or intervention, including coaching, has to be effective because people who receive it improve, is a cognitive bias that’s deeply rooted and difficult to overcome.
In conventional medicine, these factors have often led to adoption of an ineffective treatment, for example, arthroscopic lavage and debridement of the knee, which uses minimally invasive surgery to ‘clean out’ the knee joint. As patients with osteoarthritis tended to improve following this procedure, it was widely adopted. However, randomised control trials showed that patients receiving a placebo treatment also reported similar levels of improvement (Moseley et al, 2002), presumably due to the factors described.
While many coaches believe their coaching is transformational, the coaching profession is in the uncomfortable position that the evidence for coaching, as well as particular coaching techniques, can only be described as weak when considered alongside standards set for medical research. There are few properly controlled trials looking at the benefit of coaching (eg, Grant, 2013; Jones & Underhill, 2019) and these are often small-scale, or have question marks regarding methodology (Passmore & Lai, 2019), while the cognitive bias outlined earlier is likely to have been playing out.
Let’s consider coaching in the light of the three factors described earlier. In terms of the placebo effect, the hiring (and payment) of an expert, the nature of the meeting, the expectations of the client and the HR department, the time that both coach and client spend on the task, these all contribute to an expectation of success. Of course, in coaching, this placebo effect is something that can, and should, be deliberately employed by the coach to improve the outcome for their client, by enabling clients’ confidence in the coach and the coaching process.
In terms of regression to mean, coaching is often sought when the client is experiencing something that’s not normal for them. If, for example, they’re having unusual difficulties with their line manager, the quality of this relationship is likely to fluctuate over time. So, if coaching takes place when the relationship is worse than normal, the client is likely to experience a regression to the mean and see an improvement, even with no external intervention.
Further, coaching does not happen in isolation. The situation will often resolve with time or as a result of other interventions, for example, the client who makes a transition to a new position will gradually find they’re less anxious, more aware of who to connect with and how to work in their new role. It’s tempting to attribute reaped benefits to coaching, rather than to factors such as time passing.
Coaching: the modern-day ‘snake oil’?
Coaching continues to operate on a limited evidence base. We too often rely on extrapolation from other fields such as psychotherapy, or a small number of quantitative studies, but most of all on personal anecdote from our clients and our recollections as coaches, expressed through case studies and qualitative research on felt experience.
This isn’t to say that qualitative research has no value – it has a vital role to play – but coaching needs to evolve and look more to quantitative research if it is to build the same standard of evidence base as other psychological or scientific interventions. It is no longer acceptable to argue that coaching is an art, coaching is different, or I know from what my clients tell me just how coaching has transformed their lives.
One might argue that this does not matter if clients find that coaching improves outcomes. However, as professionals we should be deeply concerned if the only impact we can demonstrate is placebo. Without robust evidence it becomes harder to justify our fees or argue the case that coaching should be an essential component of HR strategies.
We believe the current position is not sustainable. When coaching was a relatively small and young enterprise it was acceptable to explain away the lack of evidence, as we did back in 2007 (Passmore & Gibbes, 2007).
However, given the amount of money that individuals and businesses are now investing in coaching, the lack of evidence poses long-term risks to the professionalisation of coaching.
If we’re serious about coaching, the time for change is now. We need to join in a quest for the scientific standard of evidence. Organisations must be active by commissioning research, academics must look to broaden their research to create a balance between qualitative and quantitative research methods, coaches and professional bodies must participate in the research and publishers must share the output of this work in articles and make the associated data freely available
The gold standard in scientific clinical research, the randomised controlled trial, should form the foundations of quantitative research. Such studies will allow us to better understand coaching’s effect, compare the impact of different approaches for different presenting issues and explore the active ingredients of coaching: for example, the importance of empathy compared to asking open questions.
If we continue for another 20 years to dance around the edges of evidence-based practice, we will be little different from yesterday’s ‘snake oil’ sellers, relying on a silver tongue and anecdotal stories of past successes, instead of proving a compelling evidence-based practice informed by high-quality studies.
The time for serious research has arrived, and to deliver it requires us all getting involved in large-scale studies and collaborative research.
l Andrew J T George MBE is an executive coach and consultant. He was deputy vice chancellor at Brunel University London, where he is emeritus professor and has contributed to biomedical research, both as an academic and as founding chair of the National Research Ethics Advisors’ Panel. He is a director of the NHS Health Research Authority.
- Jonathan Passmore is professor of coaching and behavioural change at Henley Business School and director of the Henley Centre for Coaching. He has published widely over the past two decades including 30 books and over 100 scientific papers.
- A Anderson, Snake Oil, Hustlers and Hambones: the American Medicine Show, Jefferson, NC: McFarland & Co Inc., 2000
- F Benedetti, ‘Mechanisms of placebo and placebo-related effects across diseases and treatments’, in Annual Review of Pharmacology and Toxicology, 48, 33-60, 2008
- Farnam Street, Regression toward the Mean: An Introduction with Examples [Online]. Available: https://fs.blog/2015/07/regression-to-the-mean/ [Accessed 11 July 2019]
- D G Finniss, ‘Placebo effects: Historical and modern evaluation’, in International Review of Neurobiology, 139, 1-27, 2018
- A J T George, C Collett, A Carr, S Holm, C Bale, S Burton, M Campbell, A Coles, G Gottlieb, K Muir, S Parroy, J Price, A S C Rice, J Sinden, C Stephenson, K Wartolowska and H Whittall, ‘When should placebo surgery as a control in clinical trials be carried out?’, in Bulletin of the Royal College of Surgeons, 98, 75-79, 2016
- P Glare, I Fridman and C E Ashton-James, ‘Choose your words wisely:
The impact of message framing on patients’ responses to treatment advice’,
in International Review of Neurobiology, 139, 159-190, 2018
- B Goldacre, Bad Science, London: Fourth Estate, 2009
- A M Grant, ‘The efficacy of coaching’, in Passmore, J, Peterson, D B & Freire, T (eds.) The Wiley-Blackwell Handbook of the Psychology of Coaching and Mentoring, Chichester, UK: Wiley-Blackwell, 2013
- R J Jones and B O Underhill, ‘Mastering evaluation’, in Passmore, J, Underhill, B O & Goldsmith, M (eds.), Mastering Executive Coaching, London: Routledge, 2019
- D Kahneman, Thinking Fast and Slow, London: Penguin Group, 2011
- K Meissner and K Linde, ‘Are blue pills better than green? How treatment features modulate placebo effects’, in International Review of Neurobiology, 139, 357-378, 2018
- J B Moseley, K O’Malley, N J Petersen, T J Menke, B A Brody, D H Kuykendall, J C Hollingsworth, C M Ashton and N P Wray, ‘A controlled trial of arthroscopic surgery for osteoarthritis of the knee’, in New England Journal of Medicine, 347, 81-88, 2002
- J Passmore, and C Gibbes, ‘The state of executive coaching research: What does the current literature tell us and what’s next for coaching research?’, in International Coaching Psychology Review, 2, 116-128, 2007
- J Passmore and Y-L Lai, ‘Coaching definitions’, in International Coaching Psychology Review, 14, 69-83, 2019
- D D Price, D G Finniss and F Benedetti, ‘A comprehensive review of the placebo effect: Recent advances and current thought’, in Annual Review of Psychology, 59, 565-90, 2008
- B E Wampold, ‘The therapeutic value of the relationship for placebo effects and other healing practices’, in International Review of Neurobiology, 139, 191-210, 2018
- J H Young, The Toadstool Millionaires: A Social History of Patent Medicines
in America before Federal Regulation, Princeton, NJ: Princeton University