CLINICAL MATTER

The Covid-19 pandemic was accompanied by a parallel pandemic of misinformation and disinformation. While sinister forces are usually at work when it comes to disinformation, some misinformation can occur when the scientific evidence changes and outdated information is spread.
Sometimes, scientific findings from similar studies can be contradictory, and what needs to be done is to synthesize the evidence to come up with a cogent recommendation. A prestigious organization called the Cochrane Library is the premier source of these summative reviews, called meta-analysis, which seek to combine study results to make sense of the myriad findings. The Cochrane library publishes a series of tools and computer software that standardizes this process and generates a Forrest plot, a graph of whether the overall evidence is beneficial, harmful, or makes no difference to standard of care. These graphs also reflect the level of uncertainty inherent in the recommendation. The closer the results of the studies included, the smaller the uncertainty. Conversely, disparate results will result in a wide range of uncertainty. The nature of a meta-analysis is that the quality of the conclusions is highly dependent on the quality of studies included in the analysis. In addition, there is the issue of heterogeneity, where studies may be measuring different outcomes or different diseases and they may not be combinable or comparable.
When a Cochrane review ([https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD006207.pub6/full](https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD006207.pub6/full)) recently concluded there was no clear utility to wearing masks based on a meta-analysis of 78 Randomized Controlled Trials (RCTs), many conspiracy theorists took this as a definitive statement that masks don’t work. Mainstream and social media quickly took up this erroneous interpretation and one of the authors of the meta-analysis doubled down by stating that, “There is just no evidence that they (masks) make any difference. Full stop.” Looking closely at the paper itself, the authors themselves stated that: “The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions. There were additional RCTs during the pandemic related to physical interventions but a relative paucity given the importance of the question of masking and its relative effectiveness and the concomitant measures of mask adherence, which would be highly relevant to the measurement of effectiveness, especially in the elderly and in young children.”
This is hardly a “full stop” and explicitly comments on how inadequate the state of the evidence is, particularly when it comes to low adherence to masking. Adherence to interventions is a major factor in the effectiveness of any preventive measure. For instance, in the prevention of HIV it is important that condoms be worn “correctly and consistently.” Otherwise, they don’t work well or work at all. There are very good studies that correlate consistent mask wearing and a significantly lowered risk of acquiring Covid-19. A recent study in California ([https://www.cdc.gov/mmwr/volumes/71/wr/mm7106e1.htm](https://www.cdc.gov/mmwr/volumes/71/wr/mm7106e1.htm)) found that while a consistently worn N95 respirator performed better than a consistently worn surgical masks, an N95 respirator used inconsistently offers much less protection than a properly and consistently worn surgical mask.
Looking further at the findings of the Cochrane paper, the authors state: “There is uncertainty about the effects of face masks. The low to moderate certainty of evidence means our confidence in the effect estimate is limited, and that the true effect may be different from the observed estimate of the effect. The pooled results of RCTs did not show a clear reduction in respiratory viral infection with the use of medical/surgical masks. There were no clear differences between the use of medical/surgical masks compared with N95/P2 respirators in healthcare workers when used in routine care to reduce respiratory viral infection.”
Perhaps one of the hardest concepts to explain in science is that the absence of evidence does not mean absence of an effect. It is entirely possible that an intervention works but the correct study hasn’t been done. Therefore, the conclusion, especially with the uncertainty of the pandemic, should not have been that masks don’t work. The conclusion should instead have been that there is uncertainty in the evidence when looking at how masks have performed in the past with respiratory diseases other than Covid-19. The consequences of not using that intervention and belatedly finding an effect, however, are much more severe than just going ahead and using the intervention.
Because of the controversy the paper generated, the Cochrane editors released their opinion on the matter ([https://www.cochrane.org/news/statement-physical-interventions-interrupt-or-reduce-spread-respiratory-viruses-review](https://www.cochrane.org/news/statement-physical-interventions-interrupt-or-reduce-spread-respiratory-viruses-review)) and even wrote an editorial to address this absence of evidence vis-à-vis the decision to recommend masking. The bottom line is that doing nothing would have been more harmful: “Many commentators have claimed that a recently-updated Cochrane Review shows that ‘masks don’t work,’ which is an inaccurate and misleading interpretation. It would be accurate to say that the review examined whether interventions to promote mask wearing help to slow the spread of respiratory viruses, and that the results were inconclusive. Given the limitations in the primary evidence, the review is not able to address the question of whether mask-wearing itself reduces people’s risk of contracting or spreading respiratory viruses.”
Finally, the idea of peer-review among scientists with a similar knowledge base is extremely important since peers are in a better position to understand the nuances of the data. Insisting that we only use interventions, especially low-risk ones, that have high quality RCTs backing them up is a nihilistic approach and is not realistic.
A great example is the use of parachutes when jumping out airplanes. No one in their right mind would do a study where half of the participants would willingly jump out of an airplane without a parachute just to prove that parachutes work, right? Surprisingly, it turns out that somebody actually did do that to prove a point ([https://www.bmj.com/content/363/bmj.k5094](https://www.bmj.com/content/363/bmj.k5094)).
The huge caveat in this study is that the participants without parachutes only agreed to jump out of the plane if it was still on the ground and not while it was in the air. Predictably, the results of the study showed that there was no difference in deaths between jumping out of an airplane with or without a parachute—as long as it was on the ground. Half-jokingly, the authors strongly cautioned against extrapolating this finding to higher altitudes. This is exactly the same kind of caution we should exercise against this flawed meta-analysis, especially when dealing with something as deadly as Covid-19.
As more and more information become available and new studies come to light, properly curating the correct and scientifically-sound content is of paramount importance. If even scientifically rigorous organizations like Cochrane can be misunderstood and respected scientists occasionally go rogue, getting the correct health information can be quite challenging. The best way to ensure good quality information is to get it from trusted sources like the WHO and the CDC. When in doubt, sitting down with your family doctor is always a good idea.