Why It’s Time To Adopt A ‘Red Team’ Approach In Science

While tens of thousands are dying from the Covid-19 virus and hundreds of millions of people around the world are facing the loss of their jobs and livelihoods, the scientific community is deeply divided over the nature, spread, and health risks of this new coronavirus.

The evident divisions and contradictory results published in thousands of new studies in recent weeks (and the conflicting scientific advice provided to governments) are causing growing confusion, anger, and disarray both within the scientific community and the general public.

Scientific models and predictions based on widely differing assumptions are exposed as fatally flawed as never before.

As a result, institutional science is hemorrhaging trust around the world while the way research is conducted and published is facing an existential crisis.

In many ways, the coronavirus crisis has triggered the biggest crisis of science in modern history.

In light of this evident disarray, calls for a radical reform of quality control of scientific methods and claims and the introduction of institutional Red Teaming are gaining ground.

In a compelling article in the journal Nature, Professor Daniël Lakens sets out the arguments for a radically new way to conduct quality-control of scientific research and its methods (see below).

The introduction of institutional red teams into the way science is organized and funded in open societies should be the top priority of a scientific reformation after the end of the Covid-19 crisis (see e.g. Red Teams Can Save Climate Science From Itself).

This kind of scientific paradigm shift will be absolutely essential if we want to learn the biggest lesson of the coronavirus disaster.

It would also help to ensure that free nations can avoid repeating similar catastrophic mistakes and disastrous policy decisions based on fallacious modeling and flawed predictions.

To guard against rushed and sloppy science, build pressure testing into your research.

Daniël Lakens

As researchers rush to find the best ways to quell the COVID-19 crisis, they want to get results out ultra-fast. Preprints — public but unvetted studies — are getting lots of attention. But even their advocates are seeing a problem. To keep up the speed of research and reduce sloppiness, scientists must find ways to build criticism into the process.

Finding ways to prove ourselves wrong is a scientific ideal, but it is rarely scientific practice. Openness to critiques is nowhere near as widespread as researchers like to think. Scientists rarely implement procedures to receive and incorporate pushback. Most formal mechanisms are tied to the peer-review and publishing system. With preprints, the boldest peers will still criticize the work, but only after mistakes are made and, often, widely disseminated. […]

It is time to adopt a ‘red team’ approach in science that integrates criticism into each step of the research process. A red team is a designated ‘devil’s advocate’ charged to find holes and errors in ongoing work and to challenge dominant assumptions, with the goal of improving project quality. The team has a role similar to that of ‘white-hat hackers’ hired in the software industry to identify security flaws before they can be discovered and exploited by malefactors. Similarly, teams of scientists should engage with red teams at each phase of a research project and incorporate their criticism. The logic is similar to the Registered Report publication system — in which protocols are reviewed before the results are known — except that criticism is not organized by journals. Ideally, there is a larger amount of speedier communication between researchers and their red team than peer review allows, resulting in higher-quality preprints and submissions for publication.

Even scientists who invite criticism from a red team acknowledge that it is difficult not to become defensive. The best time for scrutiny is before you have fallen in love with your results. And the more important the claims, the more scrutiny they deserve. The scientific process needs to incorporate methods to include ‘severe’ tests that will prove us wrong when we really are wrong. […]

This shows that assembling a red team isn’t enough: research teams need to commit to addressing criticism from the outset. Sometimes, this is straightforward — items on checklists are absent from a proposal, or an independent statistical analysis yields different results, for example. Usually, it will be less clear whether criticism merits changing a protocol or including a caveat. The key is that, when results are presented, the team transparently communicates the criticism that the red team raised. (Perhaps incorporated criticism could be listed in the methods section of a paper, and unincorporated criticism in the limitations.) This will show how severely a claim has been tested. […]

With research moving faster than ever, scientists should invest in reducing their own bias and allowing others to transparently evaluate how much pushback their ideas have been subjected to. A scientific claim is as reliable as only the most severe criticism it has been able to withstand.

Read more at The GWPF