Research in Coaching: It Is No Longer a Luxury, It Is Essential

In the middle of the nineteenth century many made their fortunes on patent medicines, often called ‘snake oil’, that were marketed as a panacea for all ills. These had various formulations. One contained mineral oil, callow, turpentine, pepper and camphor, while some contained cocaine, amphetamine or opium, no doubt increasing repeat sales! They all had in common extravagant claims but no scientific evidence that they worked.

Why did people buy them? Apart from the slick marketing, or persuasive charms of the salesperson, one main reason was that people who took snake oil often got better. In many cases there would have been a clear association between using ‘snake oil’ and an improvement in the person’s condition. People would therefore, quite reasonably, believe that the medicine was doing them good.

This might seem contradictory, after all looking back today, the contents of these ‘remedies’ was more likely to make you sick than better. However, there are three reasons for the ‘positive effects’ experienced by users, what scientists have identified as the ‘placebo effect’, ‘regression to the mean’ and the impact of ‘concurrent events’. Let’s look at these and their implications for coaching.

Placebo

This effect is a well-known and studied phenomenon and is down to the patient’s belief in the treatment. Its magnitude can be influenced by factors including the nature of the intervention (such as a pill’s size and/or color, or whether the procedure involves an injection), the price paid and the perceived status of the therapist, and the language used by the professional and the relationship between them and the patient. In medicine, the placebo effect must be considered in studies determining whether new treatments work.

Regression to the mean

This is a relatively simple statistical principle. If data is randomly distributed over time around a mean (average) value, and if you pick a moment in time when the data is far from the mean, next time you look at the data it’s likely to be closer (regress) to the mean just by chance (there are more data points close to the mean than far from the mean). For example, if you’re monitoring exam results in a school and one year it gets much better results than normal, you’d typically expect lower results the next year.

In medicine, patients are often treated when their symptoms are at their worst. If they have a fluctuating, non-progressive, disease you’d expect them (on average) to improve even if the treatment is ineffective.

Impact of concurrent events

Nothing happens in isolation. Most illnesses get better of their own accord. People will be taking other treatments for their condition (other medicines, bed rest or tender loving care). If you were to take snake oil, and your condition was to improve either because of the passage of time or something else you were doing, you could falsely attribute this improvement to the snake oil.

The magic treatment

The above factors can mislead any professional (and their client) into thinking what they’re doing is effective. The belief that a treatment or intervention, including coaching, has to be effective because people who receive it improve is a cognitive bias that’s deeply rooted and difficult to overcome.

In conventional medicine, these factors have often led to adoption of an ineffective treatment, for example, arthroscopic lavage and debridement of the knee which uses minimally invasive surgery to ‘clean out’ the knee joint. As patients with osteoarthritis tended to improve following this procedure, it was widely adopted. However, randomized control trials showed that patients receiving a placebo treatment also reported similar levels of improvement, presumably due to the factors described.

Implications for coaching

While many coaches believe their coaching is transformational, the coaching profession is in the uncomfortable position that the evidence for coaching, as well as particular coaching techniques can only be described as weak when considered alongside standards set for medical research. There are few properly controlled trials looking at the benefit of coaching and these are often small scale, or have question marks regarding methodology, while the cognitive bias outlined earlier is likely to have been playing out.

Let’s consider coaching in the light of the three factors described earlier. In terms of the placebo effect, the hiring (and payment) of an expert, the nature of the meeting, the expectations of the client and the HR department, the time that both coach and client spend on the task all contribute to an expectation of success. Of course, in coaching this placebo effect is something that can (and should) be deliberately employed by the coach to improve the outcome for their client, by enabling clients’ confidence in the coach and the coaching process.

In terms of regression to mean, coaching is often sought when the client is experiencing something that’s not normal for them. If for example, they’re having unusual difficulties with their line manager, the quality of this relationship is likely to fluctuate over time. So, if coaching takes place when the relationship is worse than normal, the client is likely to experience a regression to the mean and see an improvement, even with no external intervention.

Further, coaching does not happen in isolation. The situation will often resolve with time or as a result of other interventions, for example the client who makes a transition to a new position will gradually find they’re less anxious, more aware of who to connect with and how to work in their new role. It‘s tempting to attribute reaped benefits to coaching, rather than to factors such as time passing.

Coaching: the modern-day snake oil?

Coaching continues to operate on a limited evidence base. We too often rely on extrapolation from other fields such as psychotherapy, or a small number of quantitative studies, but most of all on personal anecdotes from our clients and our recollections as coaches, expressed through case studies and qualitative research on felt experience. This isn’t to say that qualitative research has no value, in fact it has a vital role to play, but coaching needs to evolve and look more to quantitative research if it is to build the same standard of evidence base as other psychological or scientific interventions. It is no longer acceptable to argue, coaching is an art, coaching is different, or I know from what my clients let me just how coaching has transformed their lives.

One might argue that this does not matter if clients find coaching improves outcomes. However, as professionals we should be deeply concerned if the only impact we can demonstrate is placebo. Without robust evidence it becomes harder to justify our fees or argue the case that coaching should be an essential component of HR strategies.

We believe the current position is not sustainable. When coaching was a relatively small and young enterprise it was acceptable to explain away the lack of evidence, as we did back in 2007. However, given the amount of money that individuals and businesses are now investing in coaching, the lack of evidence poses long-term risks to the professionalization of coaching.

If we’re serious about coaching, the time for change is now. We need to join in a quest for the scientific standard evidence. Organizations must be active by commissioning research, academics must look to broaden their research to create a balance between qualitative and quantitative research methods, coaches and professional bodies must participate in the research and publishers must share the output of this work in articles and make the associated data freely available for review. 

The gold standard in scientific clinical research, the randomized controlled trial, should form the foundations of quantitative research. Such studies will allow us to better understand coaching’s effect, compare the impact of different approaches for different presenting issues and explore the active ingredients of coaching: for example the importance of empathy compared to asking open questions. If we continue for another 20 years to dance around the edges of evidenced-based practice, we will be little different from yesterday’s snake oil sellers, relying on a silver tongue and anecdotal stories of past successes, instead of proving a compelling evidence based practice informed by high quality studies.

The time for serious research has arrived, and to deliver it requires us all getting involved in large-scale collaborative research.