Today, the death toll for COVID-19 in the US crossed 100000, a staggering number that makes this virus the third leading cause of death nationwide after heart disease and cancer since this outbreak started. On this tragic occasion, I was reminded of some of the warnings from early on in the outbreak. For example, the following quote comes from an interview by Dr. Deborah Birx on the Today show on March 30 (at this point there had been a mere 4066 deaths).
“So in the flu models, the worst case scenario is between 1.6 million and 2.2 million deaths. That’s the projection if you do nothing. So, we’ve never really done all of these things that we are doing. We’ve put them into a model. We’ve looked at the Italy data with their self-isolation. And that is where we come up with, if we do things together well, almost perfectly, we could get in the range of 100000 to 200000 fatalities.”
https://www.nbcnews.com/news/us-news/dr-deborah-birx-predicts-200-000-deaths-if-we-do-n1171876 (See the video around the 3 minute mark)
Sadly, the model she references, which seems to be an ensemble model that involves aggregating the results from a variety of different models, appears to have been correct. We have already surpassed the number of fatalities from the most optimistic forecast. To make matters worse, there were an average of 1118 deaths per day over the last week, and it is probable, particularly if there is a second wave, that we will surpass 200000 deaths sometime in the fall. This too is consistent with Birx’s explanation which stated
“[This] best-case scenario would be 100% of Americans doing precisely what is required. But, we’re not sure based on the data that you are sharing from around the world and seeing these pictures, that all of America is responding in a uniform way to protect one another. So we also have to factor that in.”
In other words, she believed that the assumptions about participation in social distancing used to generate the more optimistic forecasts of 100000 were inconsistent with how people were responding and that the outcome would likely be worse.
This is far from the only example of models making COVID-19 forecasts that ultimately proved correct. On March 16-17 a research group from the University of Massachucetts surveyed 18 leading epidemiologists and asked them to use their models to predict the course of COVID-19. Thus far the consensus predictions have either proven correct (within the 80% uncertainty interval) or the results are still uncertain but the predictions appear to be on track.
Despite the fact that the data suggests that many of the models (particularly consensus models) produce accurate predictions, there are many people who continue to peddle the false narrative that the models have overblown the threat of this virus and led us astray. For example, in a May 21 (just 5 days ago) op-ed in the USA Today, Senator Rand Paul and Representative Andy Biggs state:
“Fauci and company have relied on models that were later found to be deficient. He even has suggested that he can’t rely on any of the models, especially if the underlying assumptions are wrong. Yet, Fauci persists in advocating policies that have emasculated the medical care system and ruined the economy.”
I found this to piece to be remarkably misleading and surprising coming from a politician that I once held in high regard.
For one, the Paul and Biggs reference only one model (the IHME model) that was deficient and yet seem to hint that all of the models were deficient. As I discussed in a previous post, concerns about this particular model are warranted. However, this is just one of many models that the CDC considers. One deficient model does not make all models deficient.
Secondly, they misrepresent Fauci’s claim by saying that he admitted that he can’t rely on any of the models. Fauci’s original statement said that models are “only as good and as accurate as your assumptions”. This is an acknowledgement that bad assumptions would produce bad predictions. Fauci goes on to say that
“whenever the modelers come in, they give a worst-case scenario and a best-case scenario. Generally, the reality is somewhere in the middle.”
He is claiming that the assumptions used to predict the worst case scenario (2.2 million deaths) are probably wrong just as the assumptions used to predict the best case scenario (only 100 thousand deaths) are wrong. The reality is somewhere in between and we do not know how close we will come to either of those extreme scenarios. That is a far cry from saying that models are unreliable. On the contrary, it suggests that the models give us a way to determine a plausible range of outcomes and that we need to prepared to refine our assumptions and rerun the models as more data becomes available if we want to generate accurate predictions.
Two months later, we have a great deal more data and both Fauci’s and Birx’s statements and the models that were used to generate their estimates have proven to be remarkably consistent with the data. There is still a reasonable debate to be had about the trade-offs between the various policy alternatives, and about how those policies have affected and will continue to affect human behavior. These are difficult issues even when we agree on the facts, but when people misrepresent those facts, it makes genuine debate nearly impossible. Given the success (or as Eugene Wigner puts it: “the unreasonable effectiveness”) of the models thus far, perhaps it is time to give the models and the modelers a break. Sure, all models are wrong sometimes, but this time the models actually got it right.