The journal Cortex launched an innovation in scientific publishing called a Registered Report. Unlike conventional publishing models, Registered Reports split the review process into two stages. Initially, experimental methods and proposed analyses are pre-registered and reviewed before data are collected. Then, if peer reviews are favorable, we offer authors “in-principle acceptance” of their paper. This guarantees publication of their future results providing that they adhere precisely to their registered protocol. Once their experiment is complete, authors then resubmit their full manuscript for final consideration.
Why should we want to review papers before data collection? The reason is simple: because the editorial process is too easily biased by the appearance of data. Rather than valuing innovative hypotheses or careful procedures, too often we find ourselves applauding impressive results or being bored by non-significant effects. For most journals, issues such as statistical power and technical rigor are outshone by novelty and originality of findings.
By venerating findings that are eye-catching, we incentivize the outcome of science over the process itself, forcing aside other vital issues. One of these sacrificial lambs is statistical power – the likelihood of detecting a genuine effect in a sample of data. Several studies in neuroscience suffer from insufficient statistical power, so – driven by the need to publish – scientists inevitably mine their under-powered data sets for statistically significant results. Many will p-hack, cherry pick, and even reinvent study hypotheses to ‘predict’ unexpected findings. Such practices cause predictable phenomena in the literature, such as poor repeatability of results, a prevalence of studies that support stated hypotheses, and a preponderance of articles in which obtained p values fall just below the significance threshold. Furthermore, an anonymous survey recently showed that these behaviors are not the actions of a naughty minority – in psychology and neuroscience they are the norm. We ourselves are guilty.
Registered Reports will help minimize these practices by making the outcome of experiments almost irrelevant in reaching editorial decisions. Cortex is the first journal to adopt this approach, but our underlying philosophy is as old as the scientific method itself: If our aim is to advance knowledge then editorial decisions must be based on the strength of the experimental design and the likelihood of a study revealing definitive results – and never on how the results themselves appeared. We know that other journals are watching Cortex to gauge the success of Registered Reports. Will the format be popular with authors? Will peer reviewers be engaged and motivated? Will the published articles be influential? We have good reasons to be optimistic.
In the lead-up to Registered Reports, many scientists have told us that they look forward to letting go of the toxic incentives that drive questionable research practices. And our strict peer review will ensure that our published findings are among the most definitive in cognitive neuroscience.
Leave a Reply