“Cases are going up in the U.S. because we are testing far more than any other country, and ever expanding. With smaller testing we would show fewer cases!” - President Donald J. Trump via Twitter
Regardless of whether you are a fan of the 45th president of the United States, you have to admit: He is not wrong! The U.S. has been gradually increasing its testing capacity since March and we are now testing over 500,000 people per day. We have administered around 10 million more tests than any other country. If we had tested less people, we would indeed “show” fewer cases. In fact, if we tested no one at all, we would “show” no cases.
On the other hand, if we did less testing we would have thousands of unexplained illnesses and deaths, because there is an important difference between “showing” cases and actually having cases and although testing affects the former, it has no bearing on the latter.
Setting aside the issue of whether this was a joke as some have claimed, his statement is a perfect illustration of a phenomenon known as Goodhart’s law which can be summarized as follows:
“When a measure becomes a target, it ceases to be a good measure.”
In other words, once people start judging their success based on a particular metric (like number of positive tests) and change their behavior in order to optimize that metric (like reducing the number of tests), then the metric itself is no longer a good way of measuring success. This phenomenon can have deleterious effects in a variety of contexts. Basing school funding on standardized test performance may sound like a good idea, but if teachers respond by teaching to the test instead of teaching valuable critical thinking skills, then it may actually make our schools worse and not better. Wells Fargo (a bank and financial services company) suffered some major recent setbacks due to Goodhart’s law. The company evaluated bankers’ performance using the number of new accounts they opened. So, how did the banker’s respond? Many started opening accounts that customers did not want through questionable means (sometimes even without the customer’s knowledge). This hurt both the business (to the tune of a $3 billion dollar fine) and the customers.
The point is that we need to be careful about optimizing for the right things. A metric is just that, a metric. It is a proxy for a real-world goal and it may not always measure the right things. If we get too caught up in the numbers and forget about the true objective, then we may end up shooting ourselves in the foot. In the context of COVID-19, the true goal is to minimize the impact of the virus, and one way to measure success is to compute the number of people who test positive. You would think that keeping that number low would be a good thing. However, as I pointed out earlier, focusing on minimizing that metric would result in the absurd strategy of testing no one in order to avoid positive tests. Is that what the president is suggesting? For all of our sake, let’s hope not.
While most of us know better than to think that no testing would be a good way to fight a pandemic, there are implications of comments like this that can be problematic, but not so obviously absurd. For example, some have argued that the reason why the number of cases is increasing is due to the increase in testing and not the spread of the virus. This notion seems to be gaining some traction in certain circles.
As you may have heard the number of new COVID cases is increasing again. Here is a nice graphic from CNBC to illustrate this:
We saw a first peak in early April where the average number of new cases exceeded 30000. However, those totals dropped to just over 20000 by the beginning of June. However, in the last two weeks, this has started to rise sharply. On June 19, we crossed 30000 cases in a single day for the first time since May 1. This has happened in four out of the last five days since then. Some have suggested that this rise of is just the result of higher testing. That seems to be the implication of the president’s comments. But is that really the case? How can we tell whether those tests are signs of a worsening pandemic or just an artifact of the higher testing rates?
The key is to look at the positive test rates. We know that the number of tests is going up. If the outbreak is not getting worse, then this would mean more and more healthy people are getting tested, so the percentage of tests that come back positive should decrease. Is that what we are seeing?
Some of my new colleagues at JHU put together a nice dashboard for answering questions like this. Here we see the number of new cases broken down by state along with the number of tests and the percent of tests that came back positive. What it reveals is that there are a number of states (such as Connecticut, District of Columbia, Massachusetts, New York, New Jersey, Vermont, and Rhode Island) where the percentage of positive tests is quite low and seems to be decreasing. This suggests these states are doing well. Testing has caught up with the spread of the virus and the number of new cases is declining despite the fact that they are testing more people. That is what recovery looks like. It does not mean that they will not be hit by a second wave. The virus still only infected a relatively small percentage (<20%) of the population, so there are lots of susceptible people who could still get sick. However, the first wave seems to be subsiding. Notice that these states were hit hard early on, and many of them were slow to reopen.
Unfortunately, there are also quite a few states (Arizona, Florida, Georgia, Nevada, South Carolina and Texas being the most notable examples) where the number of cases is rising sharply AND the positive test rate is increasing as well. So, even though more people are getting tested, larger and larger fractions of those people are sick. That is a sign that the virus is spreading in those areas. It is worth noting that most of these states were not hit hard early. Many of them also have mobility patterns that have returned to close to normal levels.
There are also quite a few states where the positive test rate has remained relatively constant despite rising test numbers, suggesting the number of cases is growing, but doing to relatively slowly.
So what do we make of this? Is it a second wave? Is it the result of increased testing? The data suggests that the increased number of tests can only account for a small part of the rise in cases. The sharp rise we have seen in the last two weeks has been accompanied by a rise in the positive test rate. This means that the virus is still spreading, and it is doing so faster than the number of tests. This is not a second wave, because the places where the virus is spreading now are different than the places where it was spreading initially. This suggests that this is just the delayed arrival of the first wave. The adoption of significant social distancing measures is like pressing the pause button on the growth of the outbreak. It is unlikely to get rid of the virus, but it can stop it in its tracks. As a result, the states who had already been hit hard saw flattened curves and the states who had not yet been hit avoided a significant wave (at first). However, now that these measures are being abandoned around the country, the spread of the virus has resumed and we are seeing new outbreaks in a variety of states. And sadly, some states (Alaska, Delaware, Hawaii, Idaho, Kansas, Michigan, Montana and Washington) that seemed to have passed the peak are starting to show signs of a rebound.
Unfortunately, as much as we all want this to be over, it is not. Until the virus is eradicated from our shores (which may never happen), herd immunity is reached (a long way off), or a vaccine is found and distributed (still looks to be at least 6 months away), we cannot let our guard down and act like the pandemic is over. In times like these, I am reminded of the words of the great mathematician and philosopher, Ian Malcolm.
“[They] were so preoccupied with whether or not they could, that they didn't stop to think if they should.”
He may have been talking about scientists, but it applies to all of us. Now that mandatory lockdowns have largely ended, there are many things that we CAN do, but that doesn’t mean that we SHOULD do them. You can go dine it at a restaurant in many states. That doesn't mean it is a good idea. Take out anyone? You can hold larger gatherings now. But is that really a wise choice?
Precautions like hand-washing, wearing masks, keeping a distance, and avoiding large gatherings are still effective and necessary tools for preventing the spread of this virus. We don’t need to stop living our lives, but instead of doing things the way we have always done them, it is worth asking if there are things we can do to reduce the risk of catching or spreading the virus:
Could I move the gathering outside instead of inside?
Could I do it remotely instead of face to face?
Could I stagger arrivals to avoid crowds?
Could I rearrange the layout to make it easier for people to keep a distance?
What other small steps could I take to make this activity safer?
These things may not be convenient, but they are a way of taking care of one another. I suspect that if we were willing to be creative, we could find ways to do most of the activities that we want to do with relatively minor adjustments, especially now that summer has arrived. Isn’t that worth it if it means lowering the risk to the vulnerable, showing respect to the fearful, and showing concern for all?
P.S. The good news is that the death rate seems to be decreasing. This may be due to better medication (there are a couple of promising drugs that seem to have some real evidence behind their effectiveness), hospitals that are better prepared for surges, and even better care as we learn more about how the virus affects people. Whatever the reason, this is a positive sign that gives me some measure of hope that the worst is behind us!
Comments