Consumer Research Reports, Science & Technology

Reproducing a Scientific Result: Easier Said than Done

Everyone says it is the crisis! The expression “reproducibility crisis” in French has its Wikipedia article since the end of 2016 and the equivalent in English ( reproducibility crisis or replication crisis ) since the beginning of 2015. A quick search for the term reproducibility crisis in Google trends shows that this expression has become popular since the first half of the 2010s.

The journal Nature has made it one of its recurring editorial subjects in recent years. In particular, a survey in the form of a questionnaire of several hundred scientists from all fields was published in 2016 and has since been taken up by all the articles evoking the subject: yes, according to the majority of the scientists questioned, there is an inability to reproduce scientific experiments published, and yes, they say it is a crisis, implying not only that the matter is serious, but also that it is new.

Here is a crisis of which the scientific institution, in search of legitimacy near the political authorities (in climate sciences) or administrative (for clinical trials) or quite simply of the general public (is glyphosate toxic, yes or no?), would be fine.

What is reproducibility?

Although reproducibility is often considered in principle to be the least of things in science (if not what confidence in scientific results?), In practice, it is quite different.

From the history of science, reproducibility is a complex question: the diversity of terms ( verification, replicability, repeatability, checking, robustness, etc.) and their polysemy characterize it. Reproducibility by whom (yourself, a colleague, a competitor, a proofreader, a verification body)? For what (to validate, to contradict, to interpret)? How (same instrumentation, same protocol, the same conclusion by other means)? And besides, what exactly should be the same (the same measures, similar patterns, compatible conclusions from different results)? And when do we need to be the same (to demonstrate, to invalidate or contradict, to generalize)?

In general, the literature in the history of science shows that if reproducibility leads to more reliability in science, this is one of the means among others, not always sufficient, not necessary.

Diversity of sciences

Reproducibility is also experienced in different ways depending on the scientific field. The questions of reproducibility do not arise in the same way depending on the experimental situations: do we seek to detect the signal of gravitational waves in the noise of vibrations of a few millimeters in amplitude on metal arms several kilometers long? Are we trying to make mouse populations as significant as possible, but do they follow the same diet as in the animal facility of another laboratory? The science philosopher Sabina Leonelli offers several categories of scientific activities for which reproducibility does not necessarily have the same meaning or equal importance.

For example, certain fields work on rare or perishable objects of study. What is the meaning of reproducibility in archeology where the activity consists of finding something new at each excavation? How to define the reproducibility in certain astronomical observations of infrequent events? Reproducibility is, therefore, multifaceted and context-dependent, but it is also different depending on the scientific field. Different on how it is perceived and implemented but also different on its importance as a scientific standard.

In anthropology, for example, one of the epistemic principles at the basis of its reliability is reflexivity rather than reproducibility: the questioning of the relationship between the researcher and his object of study to allow a relevant analysis of his research conditions and results. This is, moreover, an epistemic principle from which many sciences could draw inspiration (where they are content with a sometimes unconvincing declaration of absence of conflicts of interest).

Why the crisis?

And yet, the crisis, it installed almost simultaneously in a wide variety of scientific fields despite this diversity.

On the one hand, there is a link or at least a concomitance with the movement of open access (open access). The oligopolies of scientific publishers tend to make scientific literature inaccessible to ordinary people (even ordinary researchers): one of the workhorses of open access is the demand for more transparency in science. By extension, the open science movement(open science) requires that scientific experiments can be reproducible, within the framework of this transparency. Reproducibility is claimed as the absolute standard, the standard that allows confidence in scientific activity, confidence on the part of the research community themselves but also for scientific funding institutions, and citizens. The link between publication, transparency and reproducibility is particularly significant in the criticism of peer review, which accompanies the open access movement.

However, the analysis we have just made of the diversity of scientific fields implies that the requirement of reproducibility (as a means of obtaining reliability) poses a problem for the vitality of many scientific fields for which this requirement can be without an object, even counterproductive. This is not to deny that the best possible research practices are desirable.

In the computational domain, for example, a domain present in many sciences, the interdependence of computer libraries is a puzzle for the reproducibility of computer programs, which is the object of more and more attention. But the reference to the requirement of reproducibility as if it were a universal value applicable everywhere, in the same way, is more of moral panic than a reflection on the sciences in their diversity.

Leave a Comment

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.