IELTS Reading: Summary completion

Whenever a scientific result is shown to be wrong, a familiar reassurance follows: this is science working as intended, because science is self-correcting. The phrase does a great deal of work. It is the reason a reasonable person is asked to trust a body of knowledge produced by fallible people with careers to build, and it converts the discovery of an error from an embarrassment into a demonstration of health. What is rarely examined is whether the claim is true, and if so, on what timescale, in which fields, and through whose labour.

The question stopped being abstract about a decade ago. A collaboration of several hundred researchers took a hundred studies published in leading psychology journals and repeated them as faithfully as the original reports allowed, with larger samples and analysis plans fixed in advance. Rather more than a third produced a statistically significant result in the same direction as the original, and the average size of the effects was roughly half. Comparable exercises in preclinical cancer biology and in experimental economics returned figures that were better in one case and worse in the other, but nobody has yet run such an exercise and found that most of the literature held up.

The causes are structural rather than personal, and this matters, because a problem produced by incentives is not solved by exhorting people to be honest. Journals have historically published findings and not their absence, so a study that finds nothing tends to remain in a drawer, and the published literature is therefore a biased sample of the research actually done. Samples have often been too small to detect the effects being looked for, which guarantees that whatever does reach significance overstates the size of the effect. And the analysis of a data set involves dozens of defensible choices, each of which can be made after looking at the result, so that a determined and entirely sincere researcher can arrive at a publishable finding from data that contain nothing.

A good deal has been done about this. Preregistration requires a researcher to state the hypothesis and the analysis before collecting data, which removes the flexibility that made the problem possible. Registered reports go further: the journal reviews the design and commits to publishing whatever the study finds, which removes the incentive to find something. The effect is measurable. Where conventional literatures report null results in a small minority of papers, registered reports in the same fields return them around sixty per cent of the time, which is a change in what gets published rather than in what is true.

Not everyone is convinced that the reform movement has diagnosed the illness correctly. Teodor Vasiliev, who works on the methodology of the social sciences, argues that a failed replication is far less informative than it is treated as being. Human behaviour is sensitive to context, and a study conducted in a different country, a decade later, with a different population, is not the same study; where the original effect depended on a condition nobody thought to record, its absence the second time is expected rather than damning. He adds, pointedly, that the reformers now have careers and funding of their own, and that a movement whose product is criticism has its own reasons to find a great deal to criticise.

The first half of this is a serious argument and the second is not. That a critic benefits from criticising says nothing about whether the criticism is correct, and the same logic would dispose of the original findings just as efficiently. The point about hidden conditions, though, deserves an answer, and the answer is that it is only an argument at all when the condition is specified in advance. A moderator proposed after a replication fails, to explain why it failed, can be invented for any result whatever, and a theory protected by such moves has stopped being a theory. The remedy is exactly what preregistration asks for: say beforehand what the effect depends on, and let the replication test that.

Where I think the reassurance genuinely misleads is in its timescale. Over decades and across large questions, science plainly does correct itself: the wrong theories of heat, of disease, of continental fixity all gave way, and none of them gave way because their proponents changed their minds. But the individual finding is a different matter. A clinician deciding next month, a teacher choosing a method for September, a minister drafting a policy this year, all need to know whether a particular result is reliable now, and for that purpose the correction mechanism is far too slow. Retractions take years. A discredited finding continues to be cited, often approvingly, long after the retraction, and the citation rate barely moves.

My own conclusion is that the adjective is wrong. Science is not self-correcting; it is correctable, which is a different and less comfortable property. Correction does not happen automatically, as a consequence of the scientific method operating in the background. It happens when a particular person spends two years and a grant repeating somebody else's experiment, and it does not happen when nobody is willing to fund that, when no journal will print the result and when the work counts for nothing at the next promotion round. The reforms that matter, therefore, are the boring institutional ones, because they decide whether checking is a career or a hobby.