One of the most obvious questions in any type of healthcare is ‘does this treatment work?’ If we want to answer it reliably, we need clinical trials. In such studies, patients are typically treated with two different therapies, for instance a drug and a placebo, and subsequently the outcomes of both groups are compared. This sounds simple enough, but investigations of this nature are far from easy to do:
- They can be expensive
- They require diverse types of expertise
- They can take many months
- They rely on the cooperation of patients
Therefore, it seems prudent that the conduct of clinical trials is tightly controlled. We need to make sure that our limited resources are not being wasted. When embarking on a clinical trial, investigators have to pass through a series of checks; for instance, they must write a protocol, obtain ethics approval for it, submit it to a funding agency, secure access to a sufficiently large patient population, conduct the study under the watchful eye of peers, demonstrate that they adhere to the protocol, record the findings, evaluate the results, write up the final report when the study is finished, submit it to a journal, pass through peer review, and finally answer to criticism once the paper is published. In other words, all aspects in the planning, execution, evaluation and publication of a clinical trial are rigorously controlled — all except one: the primary investigator him/herself.
This issue is important in all types of research but nowhere more so than in alternative medicine. In this realm, we are often dealing with primary investigators (PIs) who are believers in the very therapy they plan to test, and who have little or no solid grounding in science. The aim of these PIs frequently is not to employ science to falsify a hypothesis, but to confirm their own belief.
During the planning, execution, evaluation and publication of a trial, PIs have to make a myriad of decisions. They must decide, for instance, on the precise research question, aspects of the trial design such as the nature of the control group, the inclusion/exclusion criteria, the external expertise that needs recruiting, the statistical approach, and the journal for publication. Crucially, they decide how to solve the plethora of problems that emerge daily while the trial is running. This position enables PIs to exert considerable influence over the study and its outcome.
Thus, the PI is a key factor in any study, and we may well ask how reliable, impartial and honest he or she is. Clearly this is a most delicate question, and the published paper of a clinical study rarely offers any information to answer it. What else do we have to go by?
We could investigate a PI’s track record by assessing all the studies published by him or her and by determining the percentage of studies that have reached a positive conclusion. If we do this in the realm of alternative medicine, we soon identify some serious concerns.
Alternative treatments (acupuncture, chiropractic, homeopathy, osteopathy, shiatsu, etc) are, of course, highly diverse but they share two important features: they lack plausibility and their efficacy is largely unproven. It would therefore be most unreasonable to expect every single trial of an alternative therapy to show that the treatment in question is efficacious. If we nevertheless can identify investigators whose conclusions are invariably positive, we might suspect that something is not quite right.
One does not need to search for long to find PIs who fall into this category. Let me give you just one real life example. Below is a list of conclusions from the 10 most recent Medline-listed clinical trials by one specific PI. (I have removed the name of the treatment and replaced it with ‘XY’).
XY may sustain the effects and serve as a cost-effective therapy for individuals with neck arthritis pain.
XY had greater improvement in hand function as measured by a digital hand exerciser following the first session and across the four-week period.
XY led to increased preterm infant weight gain…
The XY therapy group had less pain and perceived greater grip strength…
XY therapy groups versus the control group had a greater decrease on depression, anxiety and back and leg pain…
The group who received XY showed a greater decrease in depression, depressed affect and somatic-vegetative symptom scores…
The infants who received XY therapy exhibited lower increases in heart rate…
…not only mood states but also relationships improve mutually when depressed pregnant women [receive] XY…
Preterm infants who received XY not only showed greater weight gain but also a greater increase in serum insulin and IGF-1 levels…
…XY has pacifying or stress reducing effects…
If you think this is a rare exception, you are mistaken. In the realm of alternative medicine, one has no difficulties in finding other PIs with similar track records of producing exclusively or predominantly positive results. In fact, on my blog, I cite several further examples under the deliberately satirical heading of ‘Alt Med Hall of Fame’.
It is obvious that false positive results and conclusions do untold damage: not only do they waste our scarce research funds, but they mislead fellow scientists and misguide clinicians as well as patients into making wrong therapeutic decisions. So how can this damage be minimised? Insisting on a solid background in science for all PIs would be an important first step. A further measure could be to investigate the track record of researchers applying for research funds. If we identify applicants who have previously produced nothing but positive results in areas where this is less than plausible, we might think twice about supporting his or her work. In other words, investigating the investigators might prevent harm.
Edzard Ernst, emeritus professor at the University of Exeter, is the author of Homeopathy: The Undiluted Facts and the awardee of the John Maddox Prize 2015 for standing up for science. He blogs at edzardernst.com.