"The United States has done really well." - Donald J. Trump (September 10, 2020)
This past week the president made the claim that the US has handled the pandemic quite well. Trump backed up his comments by pointing out that cases in Europe were rising while they were falling in the US. These remarks were in stark contrast to recent media coverage which has characterized the US response as "the worst", "embarrassing", riddled with "mistakes", "failing" and "catastrophic" (try searching for "US coronavirus response" on google news) amid recent reports that Trump misled the public about the pandemic by downplaying it in public while admitting privately that the virus was "deadly stuff."
Meanwhile the death toll continues to climb and is likely to exceed 200,000 within the next week and trust in the CDC and NIH and experts such as Dr. Anthony Fauci continues to fall. So who is telling the truth? Is the president right? Has the US handled it well? Are we doing better than other countries? Or is the narrative that the US response has been inadequate correct? As always, let's look at the data.
Current trends
The first thing to note is that cases are falling in the US while they are rising in countries like France, Spain and the UK. These countries have all seen a significant uptick in new cases over the last few weeks recently surpassing the peak levels from earlier in the spring. Meanwhile, the average number of new US cases per day has decreased by over 50% in the last 6 weeks. Deaths are also inching downward having dropped below 1000 per day after a rise in August. So if you are following the trend lines, then the US trending in the right direction while a number of European nations are not.
On the other hand, using this to argue that US has handled the pandemic better glosses over a few important details. The US cases are dropping because in late July we reached record highs of over 70000 new cases per day. This was more that twice the peak number recorded in April. Meanwhile the number of new cases in these European countries were in the hundreds. So, although it appears that Europe is experiencing a second wave now, it seems a bit disingenuous to point to our downward trend without acknowledging that this is the result of a massive upward trend that Europe managed to avoid over the summer. So, his observations are correct but they do not tell the full story.
How do you measure who is doing well?
What about his more general claim that the US has handled the pandemic "really well"? This is a bit harder to evaluate because it is so unspecific. If you look at the total number of confirmed cases and deaths, then we can definitively say that "we're number one!" (as in the most cases and deaths of any country on earth). So, from that perspective his claim seems dubious at best.
However, raw totals are not the best way to evaluate our response. Maybe early on when the number of cases was still growing exponentially that made sense, but not anymore. Now that the virus has spread across the world, it is important to consider not just how far the virus has spread, but also how badly it has hit each country. The US is the 3rd most populous country in the world behind China and India, so we should have more cases and deaths. We have more people who can get sick! A better way to measure how badly we have been affected is to look at deaths per capita which is the total number of deaths divided by the population. There the US fares slightly better with only the 12th most deaths per capita. This is still not great, but at least we are not the worst.
Comparison to peers
How do we compare to our peers? The short answer is: not well. Here is a table comparing the totals for the US to our neighbor to the north and the EU.
The death rate per capita (or the functionally equivalent deaths per million) for the US is 85% higher than the EU and 136% higher than Canada. For perspective, this means that if the US had the death rate for the EU, we would have around 103,000 deaths instead of the 192,000 we have right now. If you focus on recent trends, we look even worse. Our death rate over the last month is over 8 times that of the EU and 18 times that of Canada.
Exploring other factors
Deaths per capita is a better measure, but it also ignores some important factors. For example, certain countries may be more vulnerable to the virus due to factors like high population density (which makes it easier to spread), an older population (which increases the death rate) or a high poverty rate (which may mean low quality medical care and little ability to survive economic disruption). So, if you really want to determine who is doing well, you should adjust for these factors. So, naturally, this is what I set out to do. Spoiler alert: This doesn't change much so if you don't care about models and visualizations, feel free to skip to the conclusions.
I aggregated data describing the populations, population densities, GDP per capita (a measure of economic productivity per person), and percentage of the population over 65 years of age by country from the World Bank, and then combined it with data containing the number of COVID cases and deaths from Johns Hopkins.
To see how well countries are doing, I graphed the deaths per capita up to this point, a measure of how badly the country was hit overall, against the deaths per capita over the last month. This allows us to see if countries that were hit hard early on are still experiencing large numbers of deaths. Try clicking on the plot below to see an interactive version.
There is a lot of data in this plot to parse. Each bubble represents a country. Countries in the top right were both hit hard early and are still dealing with large numbers of deaths relative to their population. That is NOT where you want to be. The size of each bubble is determined by the population. So populous countries like China, India, US, Brazil and Indonesia are represented by larger bubbles. The color of each bubble is determined by the GDP per capita right now, but there is a dropdown menu in the interactive version where you can change the variable to use population density or the percentage of the population over 65 years old. If you hover over a point, you can see more details about each bubble including the country name, and other statistics. There is also a menu at the top that you can use to circle a cluster of European countries as well as the US to make them easier to find.
So, what does this tell us? There are a few things that jump out.
China, the big dark bubble in the bottom left, has suspiciously low death rates. It seems a bit too good to be true. I don't really trust their data, but if it is indeed correct, then that is what "doing really good" would look like.
Most European countries are clumped together in the upper middle of the plot. This means that they were hit hard early on and have high death totals relative to their population, but they haven't been doing nearly as badly in terms of the number of deaths lately. These countries all tend to have elderly populations and moderately high population densities so they are likely to be especially vulnerable to the pandemic. However their high GDPs might indicate that they were better positioned to withstand economic shutdowns which could explain why their deaths are low right now.
The US is the moderately large, light colored bubble in the upper right. This means we have had both a high number of total deaths and a large number of recent deaths relative to our population. This is despite the fact that our population is younger than most of the European countries, our population density is much lower and we have a higher GDP. All those things would point to us having a much lower death rate (at least recently), but alas that is not the case.
Feel free to explore the rest of the data if you are curious. If anything interesting jumps out at you, please feel free to share by commenting below!
Predicting death rates
When you look at where the US lies in the plots above and the countries that are nearby, something seems amiss. As the saying goes, "one of these things is not like the others." The other countries that have such high death rates both overall and recently tend to be poorer developing countries in Latin America. Most of the developed nations seem to be much better off. So, intuition suggests that the US is doing worse than one might expect and not better. But how could we actually test this?
I decided to build a model that tries to predict the death rate over the last month based on the relevant characteristics of the country. In this case, I tried to account for:
Population density
Percent of the population aged 65 and over
GDP per capita
Deaths and cases per capita by mid May (a measure of how bad each country was hit early)
Location
The model generates a predicted death rate for each country by
Finding the death rates of the most similar countries according to these criteria
Averaging their death rates weighted toward the more similar countries.
For example, based on this criteria the five most similar countries to the US are: Canada, Ireland, Panama, Switzerland, and Mexico and the next five are Bahamas, Sweden, Ecuador, France, and the Dominican Republic.
For the technically inclined, this is called a K-Nearest Neighbors regression model. To see how the model did, I compared the predictions to the observed death rates and plotted them below (for 5 neighbors). Note: Again if you click on the plot it will take you to an interactive version.
Countries that fall below the line (prediction>observed) are doing better than their peers and better than we would expect based on their characteristics whereas countries that are above the line (prediction<observed) are doing worse than expected. Here the size again represents population and I have marked the US in red to make it easy to spot.
Note that the US is well above the line, meaning that the US is doing significantly worse that expected. Meanwhile Europe is closer to the line. The predictions can explain about 45% of the variability in the observed values. So knowing the location, GDP, population, and age distribution of each country, etc. does not tell you exactly how well they should be doing right now, but it can give often give you a pretty good estimate for most countries. However, the US seems to be one of the exceptions. According to this model the US death rate over the last month is between 4 and 8 times larger than we would expect based on its location, demographic characteristics and the trends seen in similar countries. Similarly, if you instead use the model to predict the overall death rate, you find that the US death rate is 2-3 times higher than expected which is right in line with what we saw when comparing to hand-picked peers of Canada and the EU. If anything, adjusting for demographic factors makes the US look even worse.
Conclusions
There are a couple of possible interpretations for this:
There could be another important factor that I did not consider that explains why certain countries (including the US) are more vulnerable to the virus than others. If any of my readers has an idea of another important factor to include, I am open to suggestions.
The US has simply not handled the crisis well. If the US death rate was at the predicted rate, then we would have around 100 thousand deaths right now instead of nearly 200 thousand. Sadly, this would mean that the failures of our pandemic response may have caused around 100 thousand unnecessary deaths. That is a sobering total.
So, in summary, the data provides little evidence that the US has done "really well". If anything, we are doing much worse than expected for a country with our demographic makeup and resources. This raises the question: Why? But that question will have to wait for another post.
Given the current trends it is possible that things will start to look better for us over the next couple of months. Maybe the current trends will continue taking us from our still high case totals to a more acceptable level. Hopefully the death totals and death rates will continue to decline. But then again, maybe this is just the calm before the storm.
“We've been through this before. Don't ever, ever underestimate the potential of the pandemic. And don't try and look at the rosy side of things...We need to hunker down and get through this fall and winter because it’s not going to be easy.”
If he is right, then we need to temper our expectations for what this fall is going to look like.
So, who can we trust? Is it the president, who prefers rose colored glasses and who has repeatedly told us that he has done a "great job", that this is "under control" and that it will soon (magically) "disappear?" Or is it the media who has pulled no punches in their characterization of his administration's handling of the crisis? Unfortunately, this analysis cannot fully answer this question. What it does tell us is that the US has been hit hard by this pandemic, much harder than similar countries. So, sadly, you can only claim that we have done "really well" if you are cherry picking the facts or if you have no interest in telling the truth.