Speakers

Joelle Pineau

Joelle Pineau

FAIR, MILA, McGill

Francis Bach

Francis Bach

INRIA

Jessica Zosa Forde

Jessica Zosa Forde

Brown University

Pre-registration in a nutshell

Separate the generation and confirmation of hypotheses:

   Come up with an exciting research question   

   Write a paper proposal without confirmatory experiments   

   After the paper is accepted, run the experiments and report your results   

What does science get?

  • A healthy mix of positive and negative results
  • Reasonable ideas that don’t work still get published, avoiding wasteful replications
  • Papers are evaluated on the basis of scientific interest, not whether they achieve the best results

What do you get?

  • It's easier to plan your research: get feedback before investing in lengthy experiments
  • Your research is stronger: results have increased credibility
  • Convince people that they will learn something even if the result is negative

Call for Papers

What is pre-registration and how does it improve peer review? Benchmarks on popular datasets have played a key role in the considerable measurable progress that machine learning has made in the last few years. But reviewers can be tempted to prioritize incremental improvements in benchmarks to the detriment of other scientific criteria, destroying many good ideas in their infancy. Authors can also feel obligated to make orthogonal improvements in order to “beat the state-of-the-art”, making the main contribution hard to assess.

Pre-registration changes the incentives by reviewing and accepting a paper before experiments are conducted. The emphasis of peer-review will be on whether the experiment plan can adequately prove or disprove one (or more) hypotheses. Some results will be negative, and this is welcomed. This way, good ideas that do not work will get published, instead of filed away and wastefully replicated many times by different groups. Finally, the clear separation between hypothesizing and confirmation (absent in the current review model) will raise the statistical significance of the results.

We are inviting submissions on the broad range of topics covered at NeurIPS! The paper template is structured like a mini-tutorial on the pre-registration process to get you started quickly. Pre-registered papers will be published at the workshop. The final results will be published in the Proceedings of Machine Learning Research (PMLR), a sister publication to the Journal of Machine Learning Research (JMLR).

Important info and dates

The review cycle for a pre-registered study consists of two stages: the proposal paper and the results paper. These stages reflect the exploratory (hypothesis generation) and confirmatory (hypothesis testing) phases of research.

Proposal paper

  • Read our mini-tutorial/template (PDF) — it serves as a paper template, describes the submission process and the intended spirit of a pre-registration.
  • Submit your paper anonymously to CMT. Differently from traditional submissions, the experimental section must only contain a description of experiments and protocol, and what conclusions can be drawn in different cases, without the results themselves. The pre-registration proposal should use the paper template. We recommend 4 pages, but we allow up to 5 pages (excluding references) for the pre-registration proposal. Note that, for some venues, only papers up to 4 pages (without references) are not considered as 'prior submission', see e.g. CVPR. For others, e.g. NeurIPS, non-archival workshops like ours do not count as dual submissions. The deadline for submissions is October 7th.
  • Besides quality and potential impact of the idea, reviewers will also assess: (1) Are the experiments appropriate for validating the core hypothesis of the work? (2) Is the experimental protocol description sufficient to allow reproduction of the experiments? You will then have a rebuttal period (until October 27th) to address the comments of the reviewers, by writing (up to) a single page response.
  • Decisions will be sent to authors by October 30th.
  • On the day of the workop (December 11th or 12th, remote), authors will present their proposals and (optionally) their preliminary results.

Results paper

  • Authors carry out the experimental protocols proposed in their accepted proposal papers.
  • The results will be presented in second document known as the results paper. This will be appended to the proposal paper to form the complete document. The deadline for the results paper is going to be in April, 2021 (tentative).
  • We will then support and encourage the final results to be published at the PMLR in combination with the pre-registered paper in April 2021 (tentative).
  • In case of interest, we will also organise a second virtual meeting at the end of April 2021 to discuss the experimental results and the lesssons learned.

FAQs

Organisers

João F. Henriques

João F. Henriques

University of Oxford

Samuel Albanie

Samuel Albanie

University of Oxford

Michela Paganini

Michela Paganini

Facebook AI Research

Gül Varol

Gül
Varol

University of Oxford

Questions?