Beware the believers: why it’s a problem if researchers only ever get positive results

One of the most obvious questions in any type of healthcare is ‘does this treatment work?’ If we want to answer it reliably, we need clinical trials. In such studies, patients are typically treated with two different therapies, for instance a drug and a placebo, and subsequently the outcomes of both groups are compared. This sounds simple enough, but investigations of this nature are far from easy to do:

  • They can be expensive
  • They require diverse types of expertise
  • They can take many months
  • They rely on the cooperation of patients

Therefore, it seems prudent that the conduct of clinical trials is tightly controlled. We need to make sure that our limited resources are not being wasted. When embarking on a clinical trial, investigators have to pass through a series of checks; for instance, they must write a protocol, obtain ethics approval for it, submit it to a funding agency, secure access to a sufficiently large patient population, conduct the study under the watchful eye of peers, demonstrate that they adhere to the protocol, record the findings, evaluate the results, write up the final report when the study is finished, submit it to a journal, pass through peer review, and finally answer to criticism once the paper is published. In other words, all aspects in the planning, execution, evaluation and publication of a clinical trial are rigorously controlled — all except one: the primary investigator him/herself.

This issue is important in all types of research but nowhere more so than in alternative medicine. In this realm, we are often dealing with primary investigators (PIs) who are believers in the very therapy they plan to test, and who have little or no solid grounding in science. The aim of these PIs frequently is not to employ science to falsify a hypothesis, but to confirm their own belief.

During the planning, execution, evaluation and publication of a trial, PIs have to make a myriad of decisions. They must decide, for instance, on the precise research question, aspects of the trial design such as the nature of the control group, the inclusion/exclusion criteria, the external expertise that needs recruiting, the statistical approach, and the journal for publication. Crucially, they decide how to solve the plethora of problems that emerge daily while the trial is running. This position enables PIs to exert considerable influence over the study and its outcome.

Thus, the PI is a key factor in any study, and we may well ask how reliable, impartial and honest he or she is. Clearly this is a most delicate question, and the published paper of a clinical study rarely offers any information to answer it. What else do we have to go by?

We could investigate a PI’s track record by assessing all the studies published by him or her and by determining the percentage of studies that have reached a positive conclusion. If we do this in the realm of alternative medicine, we soon identify some serious concerns.

Alternative treatments (acupuncture, chiropractic, homeopathy, osteopathy, shiatsu, etc) are, of course, highly diverse but they share two important features: they lack plausibility and their efficacy is largely unproven. It would therefore be most unreasonable to expect every single trial of an alternative therapy to show that the treatment in question is efficacious. If we nevertheless can identify investigators whose conclusions are invariably positive, we might suspect that something is not quite right.

One does not need to search for long to find PIs who fall into this category. Let me give you just one real life example. Below is a list of conclusions from the 10 most recent Medline-listed clinical trials by one specific PI. (I have removed the name of the treatment and replaced it with ‘XY’).

XY may sustain the effects and serve as a cost-effective therapy for individuals with neck arthritis pain.
XY had greater improvement in hand function as measured by a digital hand exerciser following the first session and across the four-week period.
XY led to increased preterm infant weight gain…
The XY therapy group had less pain and perceived greater grip strength…
XY therapy groups versus the control group had a greater decrease on depression, anxiety and back and leg pain…
The group who received XY showed a greater decrease in depression, depressed affect and somatic-vegetative symptom scores…
The infants who received XY therapy exhibited lower increases in heart rate…
…not only mood states but also relationships improve mutually when depressed pregnant women [receive] XY…
Preterm infants who received XY not only showed greater weight gain but also a greater increase in serum insulin and IGF-1 levels…
…XY has pacifying or stress reducing effects…

If you think this is a rare exception, you are mistaken. In the realm of alternative medicine, one has no difficulties in finding other PIs with similar track records of producing exclusively or predominantly positive results. In fact, on my blog, I cite several further examples under the deliberately satirical heading of ‘Alt Med Hall of Fame’.

It is obvious that false positive results and conclusions do untold damage: not only do they waste our scarce research funds, but they mislead fellow scientists and misguide clinicians as well as patients into making wrong therapeutic decisions. So how can this damage be minimised? Insisting on a solid background in science for all PIs would be an important first step. A further measure could be to investigate the track record of researchers applying for research funds. If we identify applicants who have previously produced nothing but positive results in areas where this is less than plausible, we might think twice about supporting his or her work. In other words, investigating the investigators might prevent harm.

Edzard Ernst, emeritus professor at the University of Exeter, is the author of Homeopathy: The Undiluted Facts and the awardee of the John Maddox Prize 2015 for standing up for science. He blogs at edzardernst.com.


  • It is more than ridiculous that ANY alternative medical practice is tolerated by the NHS. If people want to deal with quacks and charlatans, they should pay for the privilege themselves and suffer the consequences. We KNOW THESE ‘TREATMENTS’ are BOGUS. They don’t just lack plausibility; they are complete nonsense. Homoeopathy? Twaddle. There is a hundred times more p|ss in my local swimming pool than active ingredient in any homoeopathy medicine.

  • M P Jones

    “they lack plausibility”

    While I agree with the conclusions of the article this initial expression of prejudice rather does detract from its quality.

    That aside, why would acupuncture and chiropractic ab initio lack plausibility?

    • What initial expression of prejudice?

      • M P Jones

        The one I quoted – that the disciplines under examination can a priori be assumed to lack plausibility. That is a serious bias.

        • Do you think they don’t lack plausibility?

          • M P Jones

            Having suffered from back problems since the age of 14 the chiropractor has helped me many times where standard medical care has failed – mostly cases of being carried in and walking out. I have no opinion about the other alternative practices.

          • That says nothing about plausibility, does it?

          • M P Jones

            Plausibility needs to be based on something. Based on my personal experience I find it plausible that chiropractic works as claimed in some cases. That is a subjective evaluation and not scientifically significant. So, in order to arrive at an objective measurement of plausibility you need to have already done some of the research, enough to carry statistical significance. The ‘based on’ part of a statement about plausibility should not be omitted.

          • M P Jones said:

            “Based on my personal experience I find it plausible that chiropractic works as claimed in some cases.”

            No. All you done is assume it does work because of your personal anecdote. That says nothing about its plausibility.

          • M P Jones

            We are running in circles here. Amusingly, a dictionary example of the use of ‘plausibility’ reads: “he offers no support for the plausibility of his theory”, implying that the usage requires the ‘based on’ or ‘because’ to follow. So, I stand by what I wrote in my reply to you.

          • I understand the word plausibility to mean plausibility of action.and I think that’s what Prof Ernst also understand by it Your own experience could be considered one datum of evidence that chiropractic has a beneficial effect (but with all the caveats about anecdotes, particularly post hoc ergo propter hoc) but because it seems to have worked does not mean that the mechanism by which it is claimed to work is plausible. Chiropractic (although there is now a spectrum) was founded on the tenet that displacements of the vertebrae impinged on nerves causing pain – and diseases – with some claiming these nerves control every cell in the body and therefore all function. We know this is not correct and therefore the purported mechanism of action (adjusting those vertebrae to remove the impingement) that many chiros base their diagnoses and treatments is pseudo science and not plausible.

            The same applies to the likes of homeopathy and acupuncture. It is entirely possible that some treatment might have beneficial effects even if its mechanism of action is either unknown or implausible. That wouldn’t necessarily stop us using that treatment, but, as in the case of chiropractic, homeopathy and acupuncture, the best evidence says they don’t work – they have no specific effects over placebo. But with a mechanism of action that has a low plausibility, it would rightly take a significant volume of robust evidence to overcome that low or non-existent prior plausibility.

          • M P Jones

            I don’t disagree with anything you say about the specifics concerning alternative treatments. My only point is that when used in science the word ‘plausible’ must have solid (mathematical) legs to stand on.

          • I’m just going to focus on bits of your comment. Specifically the “works as claimed” in “Based on my personal experience I find it plausible that chiropractic works as claimed in some cases”.

            This isn’t a criticism of your position. I don’t doubt that you feel you got a benefit from whatever treatment you got, but I suspect that this is one of those cases where undue credit was given to the profession.

            In broad strokes the claim made in Chiropractic is that there is an “innate intelligence” in the body that basically keeps it working by adapting “universal forces” and matter for use in the body. This “innate intelligence” flows through nerves and any “blockage” prevents the signals getting through. These blockages are what they call “subluxations” and these have never been successfully measured. The theory though is that adjusting the spine removes the subluxations allowing the innate energy to flow allowing the body to take care of itself.

            The problem here is that none of this makes any sense from any scientific field. The “energy” described has never been measured. The innate intelligence is little more than a poorly defined concept and, as far as I’m aware, has never been demonstrated in a reliable rigorous manner to even exist. Similarly, subluxations have never been demonstrated to exist also.

            So this is how chiropractic “claims” to work.

            Having said all that, there are 2 types of Chiro. The straights and the mixers. Straights adhere to the original teachings of Palmer. The mixers are taking a more modern approach and are trying to filter out the unproven things and keeping the things that they can test and validate. The mixers are basically evolving into physiotherapists. They already perform physiotherapy but what they bring from their origins is not helping them.

            You would be better off with a physiotherapist. You get the treatment that works without the pseudoscientific underpinning which is still woven into the treatment.

          • M P Jones

            Thanks for your kind interest. I have never read the pseudoscientific parts of chiropractic practitioners’ claims, only that certain types of manipulation of body parts can lead to alleviation of some types of complaints. In this case (my own) I don’t care much about the mumbo-jumbo, only about the results – and yes, I have tried numerous physiotherapists and specialist doctors who have not been able to produce the outcome a few chiropractors have produced in three or four sessions. Whether this has been caused by methodical issues or particular competence or incompetence in the various instances I have no way of knowing.

            I should probably add that my first port of call in case of a medical issue is the local GP, not a practitioner of alternative methods, and also that in one case a normal medical specialist in a hospital significantly improved my back problems and provided what turned out to be a lasting cure, something eluding the chiropractors I previously consulted. And with this: no more about me. I am not an interesting subject.

            What you say about the ‘mixers’ is probably true. Around twenty years ago I consulted a chiropractor in a clinic comprising both chiropractors and physiotherapists. The chiropractor performed these somewhat violent manipulations which repositioned whatever pieces that were out of alignment whereas the physiotherapists prescribed exercises to strengthen parts of the musculature surrounding the spine in order to subsequently keep it in order. This collaboration worked and I have used the exercises ever since.

            Of course, if people were only allowed to use methods they understood it would be the end of democracy – and car mechanics ;-).

    • why would acupuncture and chiropractic ab initio lack plausibility

      A better question would be why would acupuncture and chiropractic have plausibility?

      Both acupuncture and chiropractic were developed in a time(acupuncture), or by a person(chiropractic), where the nature of biology was relatively unknown and made many assumptions about what they thought they saw. Time has marched on and our knowledge base has vastly improved.

      The supposed mechanisms claimed by acupuncture and chiropractic have been demonstrated to not be real.

      Qi has never been measured. There is no biological structure that relates to Qi, the meridians or the chakras.

      Similarly, subluxations have never been measured. For a treatment that claims to adjust the flow of energy you would think that this would be measurable. Otherwise, how can you be sure your “adjustment” has succeeded in doing anything.

      Given the current state of knowledge today on biology, chemistry and physics if someone was to come up with the ideas of acupuncture and chiropractic today they would be dismissed as quacks or charlatans because there is no plausibility from the outset.

      • M P Jones

        Your logic presumes that the current state of medicine and anatomy is so perfect that further fundamental discoveries can be ruled out. There is also a semantic issue here: throughout scientific evolution many concepts have been renamed and their definitions refined in step with new knowledge. That the ancient Chinese and Tibetan acupuncturists used different descriptive concepts is not surprising (compare ‘the four elements’). That we can find no physiological chi or chakras is thus not surprising. I do not know whether or not acupuncture objectively works as claimed or whether the effects reported by patients are placebo effects – my doctorate is not in medicine. The plausibility issue should not be based on lay descriptions but determined by medical research – and perhaps it has. Not my field. I just reacted to the expression of the assumption of lack of plausibility as a scientific fallacy.

        • Your logic presumes that the current state of medicine and anatomy is so perfect that further fundamental discoveries can be ruled out.

          Ah… no. That may be your reading of it but that is not the intent. I would suggest not reading more into my words than is actually written.

          I make no claim about the quality of our current knowledge base. Just that it is superior to what we used to have because scientific advancement tends to build on what we already know. This also means discarding things that are demonstrated to be false.

          The plausibility issue should not be based on lay descriptions but on medical research – and perhaps it has. Not my field. I just reacted to the expression of the assumption of lack of plausibility as a scientific fallacy.

          Ah. Fair enough.

          There’s a regular group of us that lurk in these topics and respond to this sort of thing when people start defending or promoting pseudoscience. In this case the medical research is fairly solidly in the camp of “this isn’t a real thing” so we likely responded in a reactive way also.

          In this case there isn’t an assumption of lack of plausibility. The good quality*, replicated research in the area is pretty clear unfortunately.

          * Good quality in this context is that which adheres as closely as possible to the principals of the Scientific Method. Good blinding, randomisation, good numbers of participants, replication, peer-review, published in respected sources etc.

          • M P Jones

            Thanks for the clarification :). In these days of dumbing down, jumping to conclusions, confusing covariance with causality, etc, I have become a bit touchy when it comes to defending scientific method :-).

          • Welcome to the club 🙂