Pre-registration in a nutshell

Separate the generation and confirmation of hypotheses:

   Come up with an exciting research question   

   Write a paper proposal without confirmatory experiments   

   After the paper is accepted, run the experiments and report your results   

What does science get?

  • A healthy mix of positive and negative results
  • Reasonable ideas that don’t work still get published, avoiding wasteful replications
  • Papers are evaluated on the basis of scientific interest, not whether they achieve the best results

What do you get?

  • It's easier to plan your research: get feedback before investing in lengthy experiments
  • Your research is stronger: results have increased credibility
  • Convince people that they will learn something even if the result is negative

Important practicalities

→ There are two phases. The first one involves the presentation of paper proposals and will conclude with NeurIPS 2021 workshop day. The experiment phase will start after the acceptance of the proposals and will continue after the day of the workshop.
→ The deadline for the proposal is September the 19th. The deadline for the results will be towards the end of April, beginning of May 2022 (TBD).
→ The proposal is non-archival and will be included in the workshop proceedings. The full papers, formed as the collation of proposal and results, will each published as a journal in a PMLR volume.
→ Please read our tutorial before submitting, which you can find here. The paper structure and general rationale is different to what you may be used to. Even if the proposals must not include experimental results, it is important to carefully design and describe the experimental protocol, with the aim of eventually obtaining conclusive results.
→ You can submit here via CMT. For your proposals, please use our modified NeurIPS template.
→ For the proposal, the maximum length is five pages (references excluded). For the full papers there is not a strict limit, although we recommend to limit the experiments to a maximum extra four pages with respect to the proposal.
→ For inspiration, have a look at the other sections of this website, which provide further information. Moreover, you can find all the videos and proposals of last year edition here, and the PMLR volume with the full papers from 2020 workshop here.

Important Dates

1) Proposal phase:

Selection of pre-registered papers for the NeurIPS Workshop

 19th Sept 2021  Paper submission (authors)
 20th Sept to 27th Sept 2021  Review period (reviewers)
 1st Oct to 8th Oct 2021  Rebuttal period (authors)
 22nd Oct 2020  Final decisions

2) Results phase:

Selection of results papers for PMLR journal

 April/May 2021 (TBD)  Full paper (with results) submission (authors)

Preregistration in more detail

What is pre-registration and how does it improve peer-review? Benchmarks on popular datasets have played a key role in the considerable measurable progress that machine learning has made in the last few years. But reviewers can be tempted to prioritise incremental improvements in benchmarks to the detriment of other scientific criteria, destroying many good ideas in their infancy. Authors can also feel obligated to make orthogonal improvements in order to “beat the state-of-the-art”, making the main contribution hard to assess.

Pre-registration changes the incentives by reviewing and accepting a paper before experiments are conducted. The emphasis of peer-review will be on whether the experiment plan can adequately prove or disprove one (or more) hypotheses. Some results will be negative, and this is welcomed. This way, good ideas that do not work will get published, instead of filed away and wastefully replicated many times by different groups. Finally, the clear separation between hypothesizing and confirmation (absent in the current review model) will raise the statistical significance of the results.

Call for Papers: We are inviting submissions on the range of topics covered at NeurIPS! Pre-registered papers will be published at the workshop, which is non-archival. After NeurIPS, authors will have the opportunity to submit the results paper to the Proceedings of Machine Learning Research (PMLR), a sister publication to the Journal of Machine Learning Research (JMLR). (you can find last year's proceedings here). The review process for this second stage will aim to ensure that the authors have performed a good-faith attempt to complete the experiments described in their proposal paper.

FAQs

Organisers

Samuel Albanie
Samuel Albanie
University of Cambridge
João F. Henriques
João F. Henriques
University of
Oxford
Alex Hernandez-García
Alex Hernández-García
Mila (Quebec AI Institute) and Université de Montréal
Hazel Doughty
Hazel Doughty
University of Amsterdam
Gül Varol
Gül Varol
École des Ponts ParisTech

Previous editions

This is the third edition of the workshop. Below you can find previous years' pages.
In particular, here you can find the PMLR proceedings of the final papers (proposal+experiments) from last year's edition.

Questions?