[Editor’s note: With managing editor Dr. Gorski on a rare holiday this week, we have economist and lawyer Eric Kramer providing a guest post in his stead, with a complementary post to John Howard’s from Friday.]
Spiked recently published a critique of New Zealand’s zero COVID policy by Jay Bhattacharya, an author of the Great Barrington Declaration. Here is his conclusion:
Ultimately, New Zealand’s Zero Covid strategy was immoral, incoherent and a grand failure.
Bhattacharya’s most compelling argument is that the zero COVID policy led to an increase in non-COVID mortality that substantially offset the COVID deaths the policy averted. If this were true, it would indeed be a powerful criticism of New Zealand’s policy. However, the best available estimates suggest that excess non-COVID mortality in New Zealand was negative during the pandemic – fewer people died than we would have expected based on historical data. New Zealand’s COVID policy appears to have prevented thousands of COVID deaths, with no offsetting increase in non-COVID mortality.
Bhattacharya got this wrong for a simple reason: he did not attempt to compare COVID deaths averted to estimates of excess non-COVID mortality. Instead, he used statistical innuendo to create the impression that the policy backfired. He also seems to have been misled by a chart of excess deaths based on a biased methodology, a mistake that would have become apparent to him if he had bothered to compare excess non-COVID mortality to COVID deaths avoided.
COVID deaths and excess mortality in New Zealand
As Bhattacharya explains, New Zealand closed its borders to most international travelers early in 2020, and then used internal lockdowns on an as-needed basis to keep the prevalence of COVID low throughout most of the epidemic.
This policy response clearly prevented many COVID deaths. Through July 24, 2022, New Zealand recorded 1,455 COVID deaths. If it had opened its borders and allowed the virus to spread freely the number of deaths would have been much higher.
How much higher? If New Zealand had a COVID death rate equal to that of the United States, roughly 14,000 additional COVID deaths would have occurred. If we compare New Zealand to U.K. the result is similar; if we use the European Union average COVID death rate, 11,000 additional deaths would have occurred. This calculation is based on the difference in confirmed deaths per million as tallied in Our World in Data, as of July 24, 2022.
It is conceivable that the zero COVID policy could have led to an increase in non-COVID deaths that substantially or fully offset the COVID deaths avoided. To determine if this occurred, we need to compare actual mortality in New Zealand during the pandemic to mortality we would have expected based on recent historical experience. Our World in Data publishes three estimates of excess mortality; their preferred estimate is based on work by Karlinsky and Kobak, and it suggests that excess mortality through July 24, 2022 was negative: -1,104. Furthermore, the actual deaths used in this excess mortality calculation includes the 1,455 COVID deaths noted above, so non-COVID mortality appears to have been reduced by roughly 2,500 relative to pre-pandemic expectations. Karlinsky and Kobak suggest that the reduction in mortality in New Zealand is due to reduced influenza mortality caused by lockdowns and social distancing. Recent estimates of life expectancy (not yet peer reviewed) which account for changes in the age structure of the population show life expectancy decreasing in the United States and England/Wales during the pandemic, but increasing slightly in New Zealand. This tends to corroborate the estimate of negative excess mortality in New Zealand.
The upshot is that the zero COVID policy appears to have avoided something like 10,000 to 14,000 deaths, and it modestly reduced non-COVID mortality as well.
Bhattacharya’s analysis
Bhattacharya acknowledges that New Zealand’s policy prevented many COVID deaths, but he strongly implies that excess non-COVID mortality offset reduced mortality from COVID (my bold):
On the negative side of the ledger is the tremendous burden on the New Zealand population that has come from being isolated from the rest of the world for such a long time, and from the intermittent lockdowns the government imposed on the population. All-cause excess deaths – below baseline levels in 2020 – shot up in 2021 and in 2022. Some weeks, recorded death rates were at 32 per cent above the norm. New Zealand delayed and reduced COVID deaths at the expense of increasing deaths from other causes. And then, of course, there are the enormous economic, psychological and additional health costs of lockdown that the population will pay out for over the coming years.
Bhattacharya’s suggestion that excess non-COVID deaths offset COVID deaths avoided is not supported by the available evidence: it appears that excess non-COVID deaths were negative. Furthermore, Bhattacharya has no basis for making this apparently false claim, since he does not present estimates of either COVID deaths averted or excess non-COVID mortality.
Instead of comparing lives saved and lost due to zero COVID, Bhattacharya uses statistical innuendo to cast doubt on the efficacy of New Zealand’s approach to COVID.
Consider Bhattacharya’s statement that excess deaths “shot up” in 2021. Bhattacharya links to a graph that shows the percent deviation of the actual weekly death rate from the death rate that would be expected based on historical experience (so-called “p-scores”). The graph seems to show that excess mortality was positive in 2021. It turns out that this is probably not true; Bhattacharya has linked to the wrong set of estimates. I will return to this point below. But even taking the chart Bhattacharya cites at face value, it does not support his claim that the zero COVID policy backfired for several reasons. First, eyeballing a chart of p-scores does not tell us how many excess deaths occurred. A small increase in excess deaths would not significantly support the claim that excess deaths offset lives saved by the zero COVID policy. Second, Bhattacharya does not adjust excess mortality by removing COVID deaths. Third, Bhattacharya focuses on the increase in excess mortality from 2020 to 2021 (deaths “shot up”) but ignores the negative excess mortality that the chart suggests occurred in 2020.
Turning briefly to 2022, the same problems arise. It is true that excess mortality was positive in 2022 (even using the better set of estimates). However, to evaluate New Zealand’s policy we need to know how many excess non-COVID deaths occurred, so we need to start with total excess deaths and remove COVID deaths. It turns out that there were 1,669 excess deaths from January 2 through July 24 (using the preferred set of estimates). However, over this same period there were 1,406 COVID deaths (see here), so almost all the excess deaths in 2022 were from COVID, they were not non-COVID deaths inadvertently caused by the zero COVID policy.
The link Bhattacharya supplies to support his statement that excess deaths rose in 2022 takes us to an article about COVID deaths, not excess mortality, so it’s just there for decoration. (Remember that COVID deaths need to be removed from overall excess mortality before comparing to COVID deaths averted.) And Bhattacharya’s claim that excess death rates were “32 per cent above the norm” for “some weeks” in 2022 is true (putting aside the fact that he is using the wrong set of estimated p-scores), but he is just cherry picking the worst weekly numbers off a chart that bounces around considerably from week to week, and distracting attention from the real issue, which is whether excess non-COVID deaths were high enough to offset a substantial fraction of the COVID deaths averted.
Conflicting estimates of excess mortality
Our World in Data originally published excess mortality figures that used a simple average of historical mortality to estimate expected deaths during the pandemic. This methodology can lead to misleading results due to seasonality and changes in mortality due to population aging. For example, if mortality is rising due to population aging, using historical mortality rates can lead to inflated estimates of excess mortality. To address these problems, Karlinsky and Kobak changed their estimation method to adjust for seasonality and secular trends in mortality rates. Our World in Data now emphasizes these estimates, but it has continued to publish p-scores based on the old methodology; see the discussion here. Bhattacharya linked to a chart of p-scores based on the now-outdated estimation methodology. A set of p-scores based on the newer methodology is here, and it suggests that excess mortality remained negative in 2021, before becoming slightly positive in 2022.
Bhattacharya may have concluded that the zero COVID policy led to a substantial increase in non-COVID mortality because he was misled by the p-scores he saw. Recall that the p-score chart Bhattacharya links to seems to show that excess mortality was positive in 2021 as well as 2022. However, as I emphasized above, this p-score chart does not justify Bhattacharya’s criticism of New Zealand’s approach to COVID, because it does not tell us the number of excess non-COVID deaths and compare this number to an estimate of the number of COVID deaths averted (recall he never estimates this number). His discussion of excess mortality is just innuendo. Furthermore, if Bhattacharya had attempted to check if the apparent rise in non-COVID deaths suggested by the likely biased p-score estimates was large enough to offset the thousands of COVID deaths avoided, he would have quickly realized that there was a problem with the p-score data he cited, because Our World in Data only publishes counts of excess deaths based on the revised estimates that show negative excess mortality in 2021, which is inconsistent with the p-score series he cites.
A short restatement of the problem
Bhattacharya suggests that New Zealand’s zero COVID policy led to a substantial offsetting increase in non-COVID mortality. This is simply not supported by the available evidence. Zero COIVID saved a substantial number of lives, and it appears to have reduced non-COVID mortality as well, perhaps by reducing influenza deaths. On its face, the policy seems to have substantially reduced mortality.
Instead of doing a straightforward comparison of COVID deaths averted and excess non-COVID mortality, Bhattacharya pointed to estimated p-scores that appear to show that excess mortality was positive in 2021 and 2022. However, these estimates were based on a methodology that is believed to overstate excess mortality, and, more important, he failed to look up and report the number of estimated excess deaths, to remove COVID deaths, and to compare the number of excess non-COVID deaths to lives saved.
The fact that New Zealand’s policy appears to have saved thousands of lives does not prove that the zero COVID policy was justified, because it did have real costs, but Bhattacharya cannot declare the policy a failure without making a real effort to balance these costs against the very substantial number of lives saved.
Making the perfect the enemy of the good
Bhattacharya does point to some potentially valid criticisms of New Zealand’s response to COVID. He complains that the government rollout of vaccines was unnecessarily slow, and that the government erred by delaying the end of its zero COVID policy until most Kiwis were vaccinated, rather than opening up as soon as the most vulnerable people got their shots. It is also conceivable that excess mortality will rise in the future, and that some number of COVID deaths were merely delayed, not completely averted, possibilities that Bhattacharya gestures towards but does not seriously evaluate.
These are debatable points; for example, according to this article, several countries that successfully contained COVID, including New Zealand, intentionally delayed their vaccination programs to learn about adverse events from experience in other countries. This may have been a mistake, but Bhattacharya needs to argue for this, not just assert it.
Similarly, a reasonable person can argue that New Zealand should have ended its zero COVID policy earlier, after vaccinating its most vulnerable residents. But this is not self-evident. Even people at lower risk die from COVID; why not delay the reopening a month or two to get more people vaccinated? Rather than making an effort to estimate costs and lives saved, Bhattacharya claims that early reopening was justified because “[b]oosting with additional vaccine doses – especially in the Omicron era – also does not prevent infection”. However, the paper he cites does not support this conclusion, and in any event vaccination clearly prevents COVID deaths.
More fundamentally, even if we agree that the government mishandled some aspects of the pandemic response – and no doubt it did – that hardly justifies Bhattacharya’s confident declaration that the policy was “a grand failure”. This conclusion seems to reflect his strong commitment to avoiding lockdowns and achieving herd immunity as quickly as possible, not a careful consideration of what New Zealanders gained and lost from pursuing a zero COVID strategy.
Bhattacharya owes his readers a retraction
Throughout the pandemic, Bhattacharya has criticized the media and his professional colleagues for misleading the public about COVID. He can set an example by retracting his misleading essay on New Zealand’s COVID policy. Bhattacharya is certainly entitled to criticize New Zealand’s approach to COVID, but he is not entitled to use statistical innuendo to create the impression that New Zealand’s policy backfired and led to a substantial increase in non-COVID mortality when the data suggest this is false. He also should acknowledge that he cited estimates of excess mortality rates that many believe overstate excess deaths.