|→ There are two phases. The first one involves the presentation of paper proposals and will conclude with NeurIPS 2021 workshop day. The experiment phase will start after the acceptance of the proposals and will continue after the day of the workshop.|
|→ The deadline for the proposal is September the 19th. The deadline for the results will be towards the end of April, beginning of May 2022 (TBD).|
|→ The proposal is non-archival and will be included in the workshop proceedings. The full papers, formed as the collation of proposal and results, will each published as a journal in a PMLR volume.|
|→ Please read our tutorial before submitting, which you can find here. The paper structure and general rationale is different to what you may be used to. Even if the proposals must not include experimental results, it is important to carefully design and describe the experimental protocol, with the aim of eventually obtaining conclusive results.|
|→ You can submit here via CMT. For your proposals, please use our modified NeurIPS template.|
|→ For the proposal, the maximum length is five pages (references excluded). For the full papers there is not a strict limit, although we recommend to limit the experiments to a maximum extra four pages with respect to the proposal.|
|→ For inspiration, have a look at the other sections of this website, which provide further information. Moreover, you can find all the videos and proposals of last year edition here, and the PMLR volume with the full papers from 2020 workshop here.|
1) Proposal phase:
Selection of pre-registered papers for the NeurIPS Workshop
|19th Sept 2021||Paper submission (authors)|
|20th Sept to 27th Sept 2021||Review period (reviewers)|
|1st Oct to 8th Oct 2021||Rebuttal period (authors)|
|22nd Oct 2020||Final decisions||
2) Results phase:
Selection of results papers for PMLR journal
|April/May 2021 (TBD)||Full paper (with results) submission (authors)|
What is pre-registration and how does it improve peer-review? Benchmarks on popular datasets have played a key role in the considerable measurable progress that machine learning has made in the last few years. But reviewers can be tempted to prioritise incremental improvements in benchmarks to the detriment of other scientific criteria, destroying many good ideas in their infancy. Authors can also feel obligated to make orthogonal improvements in order to “beat the state-of-the-art”, making the main contribution hard to assess.
Pre-registration changes the incentives by reviewing and accepting a paper before experiments are conducted. The emphasis of peer-review will be on whether the experiment plan can adequately prove or disprove one (or more) hypotheses. Some results will be negative, and this is welcomed. This way, good ideas that do not work will get published, instead of filed away and wastefully replicated many times by different groups. Finally, the clear separation between hypothesizing and confirmation (absent in the current review model) will raise the statistical significance of the results.
Call for Papers: We are inviting submissions on the range of topics covered at NeurIPS! Pre-registered papers will be published at the workshop, which is non-archival. After NeurIPS, authors will have the opportunity to submit the results paper to the Proceedings of Machine Learning Research (PMLR), a sister publication to the Journal of Machine Learning Research (JMLR). (you can find last year's proceedings here). The review process for this second stage will aim to ensure that the authors have performed a good-faith attempt to complete the experiments described in their proposal paper.