Revised report on impact of Covid lockdowns leaves unanswered questions

The overwhelming majority of academic studies have one chance to make a splash. Once that moment has passed – which tends to be when the paper is published – the spotlight moves on in the relentless search for new material.

But not all studies adhere to that trend. Some return time after time. And it must come as no surprise that this happens most with reports that tackle questions of global importance, or that reach controversial conclusions, or manage to achieve both at once. As the Covid inquiry opens, the value of lockdowns is about as important as questions can get.

Last February, a trio of researchers posted a working paper online that reviewed published work on the impact of Covid lockdowns on mortality rates. In short, it looked for evidence that lockdowns saved lives. The findings were arresting: the authors concluded that lockdowns prevented only 0.2% of US and European deaths in the first wave of the pandemic.

For all the headlines that followed, the report and its authors drew flak. The researchers were economists not epidemiologists or public health experts, namely Prof Steve Hanke from Johns Hopkins University and the libertarian thinktank, the Cato Institute; Jonas Herby, a consultant at the Centre for Political Studies in Copenhagen, and Lars Jonung, emeritus professor at Lund University in Sweden, a country famed for its looser restrictions on the pandemic.

The review raised eyebrows among many experts. It focused on 34 studies, about a third coming from other economists, but excluded important epidemiological studies. It didn’t seem to take account of the timing of lockdowns. And it defined “lockdown” as any government policy consisting of at least one nonpharmaceutical intervention (NPI), where NPIs meant measures such as closing schools or businesses, but also more minor things such as mandating face masks. The implication was that a requirement to wear face coverings alone, or to stay home while infected, would qualify as a lockdown. On reading the paper, Adam Kucharski, professor of infectious disease epidemiology at the London School of Hygiene & Tropical Medicine, spoke of “half-baked methods”.

At the time, Dr Joshua Sharfstein, vice dean of the Johns Hopkins School of Public Health, distanced the school from the work, saying it was not a peer-reviewed scientific study and that “serious questions” had been raised about its methodology. He also corrected a potential misunderstanding: the study did not compare lockdowns with doing nothing. Instead, it compared legally enforced interventions with interventions not required by law. As the study said: “We do not look at the effect of voluntary behavioural change.” That would include people choosing to keep themselves safe because there was a pandemic under way.

The response to the paper fuelled a rewrite. Version two appeared, again online, in May last year. The authors dropped some studies they decided were no longer eligible and changed some of their calculations. This time, instead of claiming lockdowns prevented only 0.2% of US and European deaths in the first wave, the figure became 3.2%, a 16-fold increase.

In normal circumstances, scientists would wait for the research to be peer-reviewed and published in a scientific journal to read the final word from the authors. But the latest headline-grabbing version, “Did lockdowns work?” has come out as a book – a “revised and extended” version of the May working paper. Its publisher is the neoliberal thinktank, the Institute of Economic Affairs, credited with coming up with many of Liz Truss and Kwasi Kwarteng’s free market policies.

The book maintains that lockdowns – as defined by the authors – prevented 3.2% of US and EU deaths in the first wave of the pandemic. But it notes that based on nine specific NPIs, lockdowns in Europe and the US reduced mortality by 10.7% in the spring of 2020 – about 23,000 in Europe and 16,000 in the US.

Kucharski said much of the report is identical to the May 2022 pre-print, “so all the sensible criticisms made back then, from a lack of accounting for epidemic dynamics to performing a ‘meta-analysis’ on datasets that aren’t independent, still hold.” But he said it was “a shame the report added so little insight “because we do need rigorous analysis of how effective different measures were during dynamic epidemic waves – lockdowns were a blunt, last-ditch tool and countries need to find a better alternative for future pandemics.”

Prof Rowland Kao, an epidemiologist at Edinburgh University, said specific details about the UK’s lockdown matter too. In the first lockdown, care homes suffered brutal, disproportionate death rates because, despite the restrictions, residents were not adequately protected from infected patients arriving from hospitals. “As it stood, lockdown without protection of people in care homes wasn’t in a position to achieve one of its most important potential outcomes.”

Prof Devi Sridhar, a public health expert at Edinburgh University said: “The real questions are whether these severe restrictions, and the harm they cause to employment, mental health, society, are worth the lives being saved and also whether these places would have anyway been closed or starved of business because people voluntarily choose not to put themselves and their loved ones at risk.” She cited the example of Florida, where governor, Ron DeSantis, adopted a “freedom” approach, yet businesses introduced their own guidelines for running safely, alongside mask mandates that were often more stringent than in Britain, for example, requiring children as young as two to wear masks. Many businesses still had to close because of sick employees, or workers and customers staying away out of fear.

On the release of the latest version of the report, Sharfstein told the Guardian: “In every country, Sweden included, the interventions themselves saved many, many lives. But in every country, Sweden included, there were major economic impacts.”

“Whether the interventions should have been required by law or purely voluntary depends a lot on local circumstances, among other factors,” he added. “But it’s important to make sure people don’t get confused and think that we would all have been fine just living our lives as usual in the spring of 2020. That would have been catastrophic.”