Reliable science takes time. But the current system rewards speed

Source: The Conversation (Au and NZ) – By Jason Chin, Senior Lecturer, College of Law, Australian National University

Lately, there have been many headlines on scientific fraud and journal article retractions. If this trend continues, it represents a serious threat to public trust in science.

One way to tackle this problem – and ensure public trust in science remains high – may be to slow it down. We sometimes refer to this philosophy as “slow science”. Akin to the slow food movement, slow science prioritises quality over speed and seeks to buck incentive structures that promote mass production.

Slow science may not represent an obvious way to improve science because we often equate science with progress, and slowing down progress does not sound very appealing. However, progress is not just about speed, but about basing important societal decisions on strong scientific foundations. And this takes time.

Unfortunately, the pressures and incentives modern scientists face are almost universally against slow science. Secure, permanent university jobs are scarce, and with budget cuts, this appears to be getting worse.

As a result, the pressure to publish has never been higher. Indeed, in my yearly performance meetings, I am asked how many articles I’ve published and what is the status of the journals I published in. I am not asked how robust my methods are and how discerning my peer reviewers were.

The problems with fast science

Our current “fast science” approach has produced a host of problems.

Much as with fast food, scientists are incentivised to produce as much science as possible in as little time as possible. This can mean cutting corners. We know, for instance, that larger samples lead to more trustworthy results because they are more likely to be representative of the relevant population. However, collecting large samples takes time and resources.

Fast science is also associated with gaming the system. As a hypothetical example, an educational scientist might collect data to find evidence for their theory that a new teaching style promotes better learning. Then, they look at the data and realise the intervention did not quite improve learning. But if you squint at it, there might be a trend if you drop a couple of pesky outliers that didn’t see a benefit. So, they do just that.

This an example of what’s known as a “questionable research practice”, because it’s not considered outright fraud by conventional standards. Surveys in many fields suggest these practices are widespread, with about 50% of scientists saying they have engaged in them at least once.

Fast science is also associated with more obviously unethical practices.

Reports of fabricated data are likely due, in part, to scientists trying to publish as quickly as possible. An industry has even sprung up around scientific fraud – what are known as “paper mills”. These organisations produce articles around fabricated data and then sell authorship to those papers.

Why trustworthy science takes time

So, what does slow science look like and how can it help?

The late English statistician Douglas Altman provided one of the most famous descriptions of the slow science mantra: “We need less research, better research, and research done for the right reasons”.

In many ways, it is the opposite of fast science: large samples and careful, well-documented, transparent practices.

Recall the hypothetical example of the scientists testing a new education practice. Rather than immediately jumping into data collection, the slow practice would be to first write a “registered report”. In other words, scientists would write out their theory and how they propose to test that theory, and send that out for peer review prior to collecting data.

The journal would then follow the normal process of soliciting peer reviews and allowing the scientists to revise their report in response to those reviews. Then, the authors would collect data, with publication in the journal being assured as long as they follow the agreed upon methods.

There are two major benefits to registered reports: it allows for peer feedback while it is still possible to improve the study and it removes an incentive to engage in questionable or fraudulent practices. Using the registered report format can take longer. But it is associated with more credible findings.

Two other slow practices worth mentioning are conducting research in a way that is reproducible and correcting errors in the existing body of research.

In theory, all science should be reproducible. That is, scientists should share their methods and data such that other scientists can both verify that work and build on it (developing new recipes, to continue the analogy to slow food).

Similarly, cleaning up the scientific record is incredibly important. For the same reasons that chef Gordon Ramsay likes to a clean a kitchen out before improving it, science needs to get a handle on what existing findings are reliable before we can build on them.

This means carefully going through existing publications to find studies that show indications of being fabricated or otherwise unreliable. This sleuthing is rare among university scientists because it does not typically result in publications. But it is highly important.

Slow science is slowly gaining steam

Currently, it requires bravery to engage in slow science.

Universities are keen to move up the university rankings lists. Those rankings are driven by publishing. So, universities hire, promote and retain their scientists based on their publications. This makes it risky to slow down.

There are, however, some reasons to hope. Movements are afoot to redefine research quality to take into account more aspects of slow science.

The Declaration on Research Assessment is a worldwide initiative to move away from ranking systems that ignore the principles of slow science.

Grassroots organisations are also creating platforms for more open and exacting peer review.

And advocates for more careful research practices have recently been appointed to important positions, such as with research funders and academic journals.

These developments are worth following and building upon because society does not need heaps of low-quality science. It needs science that deserves trust.

Jason Chin is affiliated with the Association for Interdisciplinary Metaresearch and Open Science (AIMOS), a charity that promotes the study and improvement of research methods. AIMOS is a co-founder of the open peer-review platform, MetaROR.

– ref. Reliable science takes time. But the current system rewards speed – https://theconversation.com/reliable-science-takes-time-but-the-current-system-rewards-speed-249497