With Covid studies, the quality of the evidence matters | David Spiegelhalter and Anthony Masters

In The Adventure of the Copper Beeches, Sherlock Holmes says: “Data! Data! Data! I can’t make bricks without clay.” Recent claims of massive benefits from wearing masks and using ivermectin against Covid-19 depended on mainly low-quality clay.

Meta-analysis is a technique for pooling the results from many studies, but it cannot make silk purses out of sows’ ears. A recent British Medical Journal review looked at six, fairly porcine, studies concerning mask-wearing and estimated an impressive 53% reduction in risk. But the single randomised controlled trial estimated the smallest effect: a reduction of about 18% (-23% to 46%) in Sars-CoV-2 infections. The “heaviest” studies, an analysis of US states and a survey of about 8,000 Chinese adults in early 2020, observed rather than experimented and its editorial highlights the risks of confounding variables influencing both wearing masks and infections and the impossibility of disentangling the effects of measures fluctuating simultaneously. Indeed, this review found an identical 53% reduction from handwashing.

To be fair, it is challenging to imagine rigorous randomised trials of the effects of mask-wearing or any other behaviour. But these are considered both feasible and essential for evaluating medical treatments and so in principle it should be easier to evaluate ivermectin, a cheap, anti-parasitic drug, heavily promoted as an overlooked treatment for Covid-19. One meta-analysis originally found ivermectin reduced Covid-19 mortality by about 56%, a huge effect. But then a preprint server withdrew the largest study and there have been further concerns over major errors and fraud in some ivermectin analyses. After removing studies with a high risk of bias, the estimated effect dissipated: the estimated decreased mortality was about 10% but with a wide uncertainty interval (- 42% to 43%), so no firm conclusions can be drawn.

Huge effort has been expended on scrutinising ivermectin research, even leading to the intriguing observation the drug appears to work in countries with a high prevalence of roundworm, whereas what is needed are the kind of good-quality studies that finally laid the claims about hydroxychloroquine to rest. Fortunately, these are now under way for ivermectin.