The Burnet Institute is defending a COVID-19 mask mandate study published by its researchers, despite a scientific journal marking it with a notice of concern after a review identified multiple "weaknesses" in its findings.
The work by Burnet researchers, published in mid-2021, focused on the relationship between mask use and COVID-19 transmission during Victoria's second wave after a mask mandate was first introduced in Melbourne in July 2020.
The study's methodology relied on a model that identified a "hinge day" eight days after the mask mandate was introduced when it was expected the effects of the policy were evident.
"We used two forms of data to assess change in mask usage: images of people wearing masks in public places obtained from a major media outlet and population-based survey data," the study said.
Researchers said they also considered potential confounding factors such as daily COVID test and COVID cases among "population subsets affected differentially by the mask policy" such as healthcare workers.
It concluded "the mandatory mask use policy substantially increased public use of masks and was associated with a significant decline in new COVID-19 cases after introduction of the policy".
"This study strongly supports the use of masks for controlling epidemics in the broader community," the study stated.
After an investigation sparked by reader feedback, scientific journal PLOS ONE marked the Burnet Institute study about the effect of mask-wearing on a reduction in COVID-19 cases with an Expression of Concern (EOC).
'Number of weaknesses' identified
The notice, published late last week, highlighted issues including the use of survey and photographic data about mask use.
The note stated a statistical expert and two members of the journal's editorial board identified "a number of weaknesses" associated with the study design.
Among the concerns was the study's use of 44 photographs from a digital newspaper taken in Melbourne before and after the mandate came into effect, to assess changes in behaviour.
"Expert advice provided to the PLOS ONE Editors in follow up indicated that the sample of photographs taken for purposes other than this study may not be representative, particularly in view of the timing and location differences; however, this was considered a valid approach to acquire data about mask usage," the EOC reads.
Similar concerns were raised about the study's reliance on self-reported mask use data — called Survey of COVID-19 Responses to Understand Behaviour (SCRUB) — with 704 survey respondents from Victoria before the mask mandate came into effect, and 104 after.
"The PLOS ONE Editors received expert input that the much smaller sample size for the SCRUB survey data in the post-mandate period reduces the probability of the post-mask mandate estimates being fair estimates of true mask use," EOC reads.
The third main area of concern outlined in the journal related to the study design's inability to "exclude the possibility of contributions from unmeasured confounding variables".
These include restrictions introduced within days of the mask mandate, including a curfew and limit on movement, as well as the closure of childcare facilities, schools and non-essential businesses.
The EOC notes that in view of this, "conclusions about causation cannot be drawn".
An EOC is similar to an editor's note and is designed to alert readers to serious concerns about a published work's compliance with the journal's policies.
Epidemiologist says paper 'caused some reaction' when published
Deakin University epidemiology chair Catherine Bennett said it was "helpful" that the EOC had been added, but argued the paper itself should be revised as well to avoid people reading the original study without that context.
"I would have really liked to have seen a revised paper," she said.
"Then you've got an updated version of the paper that actually includes the concerns and the authors' response to it."
She said the nature of the pandemic was difficult for researchers.
"We've had a push of a lot of complex community-based studies being done in the course of the COVID pandemic, and I think we need to be really mindful about how reliable those results are," Professor Bennett said.
"And it's not to say that they're not useful. But it's important to put the caveats around how we interpret results."
Professor Bennett said when the Burnet paper was first published in PLOS ONE, it had "caused some reaction in terms of the way data were collected and analysed, and also how the data were interpreted".
"The authors themselves have acknowledged the limitations, ... they talk about the problems in using these kinds of data.
"Some of the concern is that that perhaps wasn't as well-reflected in the final conclusions, as it should have been, to make it clear that this study wasn't about and wasn't able to look for a correlation between mask mandates and infection rates in the community.
"It was looking for the association between the mandates and some measures of mask-wearing, and also then looking for a correspondence ... with what was generally happening in the community, recognising that a whole lot of other things were in place at the same time."
Pinpointing the effects of mask-wearing has always been "an impossible task", according to Professor Bennett.
She said beyond the protective filtration qualities masks themselves provided, more people wearing them could have other effects on behaviour, such as limiting people's mobility.
"It was possible that one of the ways masks actually really did contribute was by limiting people's behaviour, like how much time they spent out of the home, reminding them about the importance of other things like social distancing and hand hygiene because things had escalated to a point where you had not only to wear a mask, but you had this physical reminder that you're in a pandemic," Professor Bennett said.
She said the Burnet study should have "talked through" such complexities.
In the study, mobility was treated as a potential confounding factor, but Professor Bennett said it should have been treated as something mask-wearing may effect.
She said any of the "indirect effects" of wearing a mask were likely to lessen or change as the novelty wore off.
"In Victoria we saw even the compliance with mandates fall away dramatically after the extended second wave response," Professor Bennett said.
"So what may have been shown here in relation to mask potential contribution might not apply beyond the first significant wave."
Burnet Institute says 'main point' was around correlation of masks and case reduction
But the Burnet Institute released a statement defending the study.
"The authors stand by the results in the paper," it said.
The statement said a paper on the same topic by another team led by Monash University's James Trauer, which was published in a different journal, "came to similar conclusions".
"The main point of our paper is that the timing of the introduction of a mask mandate correlated with a statistically significant reduction in cases," the statement said.
"We extended the work from this primary finding to provide evidence that mask usage increased following the introduction of the mask mandate, supporting the role of mask wearing in case reduction.
"However, we agree that correlation does not mean causation and we state in the manuscript that 'care should be taken in ascribing causation' and discuss other potential indirect effects that might explain the effect of the mandate."
Author of similar study says he is surprised by EOC
James Trauer, who heads up epidemiological modelling at Monash University's school of public health and preventative medicine, said he was surprised by the addition of an EOC on the Burnet Institute paper.
"The way I would characterise the study is that it looked at the question of, if we can consider that there was an important effect of face coverings on the second wave in 2020, what was the strength of that effect," Dr Trauer said.
"I think the Burnet study does go some way to answering that question and had some methodological limitations. But I thought [it] was an important contributor to the general discussions around those issues."
He led the study the Burnet Institute referenced in its statement, published in the journal Nature Communications, which stated in its abstract that "face coverings were likely particularly important" among multi-faceted interventions that "led to the dramatic reversal in the epidemic trajectory".
The study also focused on the actions taken in the wake of Victoria's second COVID wave.
"The second wave in Victoria was an example of a setting in which we could see a pretty clear effect from face coverings and then in other settings, we haven't seen that effect at a population level," Dr Trauer said.
Dr Trauer said the Burnet study researchers "did go in slightly with the assumption that there had to be some effects attributed to face coverings".
"That was a starting point for their analysis, that there has to be some effect and we're going to then look at the strength of their effect," he said.
"But given that I thought it was a reasonable analysis and does make some contribution to the debate, so I was quite surprised to see they were forced to publish a statement of concern."
