This week there has been enormously wide coverage in the press for one of the worst papers on acupuncture that I’ve come across. As so often, the paper showed the opposite of what its title and press release claimed. Presumably the wide coverage was a result of the hyped-up press release issued by the journal, BMJ Acupuncture in Medicine. That is not the BMJ of course, but it is, bafflingly, published by the BMJ Press group, and if you subscribe to press releases from the real BMJ, you also get them from Acupuncture in Medicine. The BMJ group should not be mixing up press releases about real medicine with press releases about quackery. Especially as there seems to be something about quackery that’s clickbait for the media.
As so often, the press release was shockingly misleading. It said:
Acupuncture may alleviate babies’ excessive crying
Needling twice weekly for 2 weeks reduced crying time significantly
This is totally untrue. Here’s why.
The paper made the most elementary of all statistical mistakes. It failed to make allowance for the jelly bean problem.
The paper listed 24 tests of statistical significance — but focused attention on three that happened to give a P value (just) less than 0.05, and so were declared to be ‘statistically significant’. If you do enough tests, some are bound to come out “statistically significant” by chance. They are false positives, and the conclusions are as meaningless as ‘green jelly beans cause acne’ in the cartoon. This is called P-hacking and it’s a well-known cause of problems.
It was evidently beyond the wit of the referees to notice this naive mistake. It’s very doubtful whether there is anything happening but random variability.
And that’s before you even get to the problem of the weakness of the evidence provided by P values close to 0.05. There’s at least a 30 per cent chance of such values being false positives, even if it were not for the jelly bean problem, and a lot more than 30 per cent if the hypothesis being tested is implausible. I leave it to the reader to assess the plausibility of the hypothesis that a good way to stop a baby crying is to stick needles into the poor baby.
On the Today programme, I was interviewed by the formidable John Humphrys, along with the mandatory member of the flat-earth society who the BBC seems to feel obliged to invite along for ‘balance’. In this case it was professional acupuncturist Mike Cummings, an associate editor of the journal in which the paper appeared. Perhaps he had read the Science Media Centre’s assessment before he came on, because he said, quite rightly, that
‘in technical terms the study is negative … the primary outcome did not turn out to be statistically significant’
to which Humphrys retorted, reasonably enough, ‘So it doesn’t work.’ Cummings’s response to this was a lot of bluster about how unfair it was for NICE to expect a treatment to perform better than placebo. It was fascinating to hear Cummings admit that the press release by his own journal was simply wrong.
Another obvious flaw of the study is the nature of the control group. It is not stated very clearly but it seems that the baby was left alone with the acupuncturist for 10 minutes. A far better control would have been to have the baby cuddled by its mother, or by a nurse. That’s what was used by Olafsdottir et al in a study that showed cuddling worked just as well as another form of quackery, chiropractic, to stop babies crying.
Manufactured doubt is a potent weapon of the alternative medicine industry. It’s the same tactic as was used by the tobacco industry. You scrape together a few lousy papers like this one and use them to pretend that there’s a controversy. For years the tobacco industry used this tactic to try to persuade people that cigarettes didn’t give you cancer, and that nicotine wasn’t addictive. The mainstream media obligingly invite the representatives of the industry who convey to the reader/listener that there is a controversy, when there isn’t.
Acupuncture is no longer controversial. It just doesn’t work. Try to imagine a pill that had been subjected to well over 3,000 trials without anyone producing convincing evidence for a clinically useful effect. It would have been abandoned years ago. But by manufacturing doubt, the acupuncture industry has managed to keep its product in the news. Every paper on the subject ends with the words ‘more research is needed’. No it isn’t.
Acupuncture is a pre-scientific idea that was moribund everywhere, even in China, until it was revived by Mao Zedong as part of the appalling Cultural Revolution. Now it is big business in China, and 100 per cent of the clinical trials that come from there are positive.
If you believe them, you’ll truly believe anything.