You may have noticed that this blog has been largely silent for a while. After sharing a couple of posts per week for 6 months, I haven’t shared anything for almost 2 months. There are a couple of reasons for that:
1. Life got busy. Researching and writing these posts takes a lot of time and new responsibilities at work have cut into some of the time that I would have devoted to this blog.
2. We all needed a break. This election season was one of the most heated in recent memory. The last thing any of us needed during that time was to be bombarded with more “information” clamoring for our attention. Taking a step back seemed like the right thing to do.
3. There wasn’t much to say. I started this blog as an attempt to use whatever limited abilities that I have to help during a national crisis. In March and April, that meant trying to synthesize all of the information about COVID to explain why it was something that needed to be taken seriously even if you did not agree with all of the mandates and lockdowns. In the summer that meant trying to evaluate some of the rumors floating around in order to see see if they were actually consistent with the data. However, as the fall rolled around it became clearer and clearer that the time for a careful examination of the evidence and respectful discussion about how to interpret it had mostly passed. By this point, people had made up their minds and retreated into their cozy echo chambers. Further discussion only seemed to promote further anger and division.
So, I found other ways to help the fight against COVID, and in the meantime, I made an extra effort to watch and listen to what other people including my friends, family and acquaintances were talking about. This time of listening revealed that the problems in our society run far deeper and have spread much wider than I ever could have imagined. Not a day goes by where I do not wrestle with the question of what it means to pursue truth and to encourage others to do the same in a time when truth has become virtually irrelevant to our public discourse. Perhaps I will write more about this at some point, but right now I have more questions than answers.
Amidst all of this, I started to receive messages from readers who miss the blog and have questions about recent events. So, I thought I would take some time to share some of those questions and answers through the next couple of posts.
What do you make of this Danish Mask Study? Is it credible? - E from Michigan
Here is a link to the study in question:
The short answer is yes, the study is credible. The source is a well-respected medical journal and the paper is well-written and honest about its methods and conclusions.
The long answers is yes, but we need to be careful to interpret it properly and I have little faith that people will do that.
In case you missed it, in the study, Danish researchers did a controlled experiment in which subjects were either asked to wear masks (2392 in all) or not wear masks (2470 in all) and then they observed the proportion of participants that contracted COVID over the next month. In all 42 of the mask wearers (1.8%) got COVID and 53 of the control group got COVID (2.1%). In other words, mask wearers were 18% less likely to get COVID, but this difference was not statistically significant. This raises a few questions:
1. What does "the difference is not statistically significant" mean?
It means that the study was inconclusive. The result did not demonstrate that masks reduce your risk of contracting COVID as some (probably including the researchers) might have expected nor did it demonstrate that masks fail to reduce your risk. The result of their statistical analysis was a confidence interval, which is a range of plausible values, for the difference between the percentages of mask-wearers and non-mask wearer that would be infected in repeated experiments. They found that range was -1.2 to 0.4. In other words, based on their experiment it is plausible that the percentages of masked participants who got sick could be anywhere between 1.2 % lower than the percentage of unmasked participants and 0.4% higher. Although the most likely outcomes is that the gap was around 0.3% (2.1%-1.8%), a difference of 0% was also plausible, they can’t rule out the possibility that there is no difference. Note: Saying that the rate was reduced from 2.1% to 1.8% is equivalent to saying that masks reduced the infection rate by 18% (when you use all of the relevant decimal places).
This sort of inconclusive result is quite common. Suppose you were given a coin and asked to check if it was fair (50-50). If you flipped the coin 5 times and got heads twice and tails three times (40% vs 60%) , you would of course end up with an inconclusive result. You can’t conclude the coin is unfair but you also cannot conclude that it is fair. On the other hand, if you flip the coin 1000 times and got 400 heads, then that would be statistically significant evidence that tails comes up more often for that particular coin. Notice that an inconclusive result is precisely what you would expect in the first case (with 5 flips) even if the true proportion of heads was 0.4. The problem was of course that the experiment did not include enough trials!
The same is true of the Danish mask study. The study either did not have enough participants or it did not last long enough to see enough infections to detect the sorts of differences between mask wearers and non-mask wearers that were observed. This is NOT the same thing as saying that there was no difference. In fact, in the study the proportion of masked participants who got COVID was lower than the proportion of unmasked participants. If they had done a study with 4 times as many participants or they had waited until 4 times as many participants got COVID and obtained the exact same result (18% lower infection rate for mask wearers), then the result would have been statistically significant.
So unfortunately there is not much we can conclude from the study about the effectiveness of masks in protecting the wearer. The best estimate we could make based on the study would be that masks reduce the wearers infection risk by 18%, but that estimate is too uncertain to draw a definitive conclusion.
2. How does this sort of inconclusive result happen?
This sort of thing happens for a handful of reasons. First of all, experiments are expensive so researchers have to cap the number of participants somewhere. This means there is always some risk of an inconclusive result even if your hypothesis is correct. In this case, the researchers estimated the number of participants they would need so that if the masks reduced your risk by half, then 80% of the time their study would achieve a significant result. In other words, they knew going into the study that even if their hypothesis was spot on, there was a one in five chance that their results would be inconclusive.
Secondly, choosing the right experiment size requires estimating the size of the effect you are studying. They designed the study estimating that masks would reduce the infection rate by 50%. In their sample, the infection rate was reduced by 18%. So, its possible that they were overly optimistic about masks. Its also possible that masks do reduce the infection rate by 50% and they just caught a bad break due to the randomness of the sample.
3. Should this study have been published if the results were inconclusive?
This is complicated. On the one hand, it is inevitable that this study is going to be misrepresented. A quick search for “Danish Mask Study” revealed five links on the first page of results alone claiming that it proved that masks don’t work (from sources like the Federalist and the Washington Times) which I should reiterate is NOT WHAT THE STUDY SHOWS! It’s not clear whether the authors of these pieces don’t understand statistics or whether they are being deliberately obtuse, but either way this is no surprise.
If this study does persuade people not to wear masks, it could cost lives. This goes beyond any potential benefits to the wearer (which based on the study we might hypothesize are real but modest). There is another part of the equation the study did not address: whether masks protect the people around the wearer. A variety of studies have demonstrated that masks are effective at filtering out a large percentage of respiratory droplets which are believed to be the primary mode of transmission for COVID-19 (such as this one: Low-cost measurement of face mask efficacy for filtering expelled droplets during speech | Science Advances (sciencemag.org)). Sadly, we live in a society where most people make a decision and then look for evidence to justify it rather than the other way around, so that is unlikely to prevent people from using this study as ammunition against the use of masks.
On the other hand, this is precisely why it is important to publish even negative results. If journals never publish negative results then researchers will just keep looking until they find positive results leading to “discoveries” of patterns that aren’t really there. This is a recipe for reaching false positives (See for example this comic strip xkcd: Significant in which researchers “discover” that green jelly beans cause acne by running a test that gives false positives 1 out of 20 times 20 times, hence the cover photo). This would further erode public trust in scientists and would hinder their ability to inform our responses in crises like this one where this sort of expertise is vital.
The scientific method relies on following the evidence regardless of whether it leads where we expected or where we want it to lead. The credibility of scientists depends on a willingness to do that. So, in that sense I am glad the study was published. It is an example of science doing exactly what is supposed to do: generating hypotheses, testing them, analyzing them objectively, and then refining them in light of the evidence. Hopefully this will lead to further (conclusive) studies into mask effectiveness so that the next time we face a global pandemic of this scale (which I hope won’t occur for quite a while!), we will have a better understanding of how to mitigate it.
4. So should we wear masks?
This study cannot answer that question. I hope that rather than reading misleading headlines and concluding that “masks don’t work”, people will look a littler closer and see a more nuanced picture, one that doesn’t answer our questions about masks, but that does show scientists doing exactly what they are supposed to do: following the evidence.
However, there are plenty of other studies that suggest that masks do make a difference under certain circumstances (a mask is probably not doing anything when you are walking your dog, but if you are sitting in a crowded room that is a different ball game). It is clear that they don’t prevent infection entirely, but there is quite a bit of evidence that they do reduce the risk of infection somewhat and even small risk reductions can have an out-sized impact during an epidemic especially when combined with other mitigation measures. So I wear as mask when I am indoors and out of the house or when I am outdoors with other people around. The mask is inconvenient and annoying at times, but I see that as a small price to pay to reduce the risk of being exposed to or spreading the virus and as a way to show respect and concern for others.
If you have a question about math and current events. Feel free to send me an email (see the contact page above). Coming soon: Questions about excess deaths and lockdowns.