Confirmatory Reports at IRSP: Guidelines for Authors

Confirmatory Reports (CRs; previously and elsewhere Registered Reports; see https://cos.io/rr/) is a form of empirical article in which the methods and proposed analyses are reviewed and pre-registered prior to the research being conducted. This format is designed to minimise bias in deductive science, while also allowing flexibility to conduct and report exploratory (unregistered) analyses.

The philosophy of the editorial team with respect to CRs rests on continuous exchanges and discussions with the authors. We are aware that not all social psychologists are familiar with this publication format and we are certainly happy to help in this process. One of the ways we plan to reduce the workload for authors, editors, and reviewers, is by letting our editors create a project on the Open Science Framework (OSF) after a first-page overview submitted to the journal. This project will include the hypotheses, data, and scripts to analyze the data. This will allow authors and reviewers to work more efficiently by adopting a transparent “research workflow”.  All reviews and editorial letters will be stored and will be open to our readers.

Initial submissions will be triaged by the editorial team for suitability in Stage 1. For this stage, authors are requested to send a one-page, bullet-pointed overview prior to submitting their CR for full review. We will invite authors of proposals that pass triage to submit a full proposal for in-depth peer review (Stage 2). Following review, the proposal will be rejected outright, offered the opportunity to revise, or accepted in principle for publication. Following in principle acceptance (IPA), the authors will then proceed to conduct the study, adhering exactly to the peer-reviewed procedures. When the study is complete the authors will submit their completed manuscript for re-review (Stage 3). At this stage we will also ask authors to  upload their raw data, pre- and post-data collection analytic code for all analyses presented in the manuscript, digital study materials, and laboratory log to a publicly accessible file-sharing service. Pending quality checks and a sensible interpretation of the findings, the manuscript will be published regardless of the results.

Confirmatory Report Editors:
Hans IJzerman, Université Grenoble Alpes
Lorne Campbell, Western University

Confirmatory Reports Editorial Review Board:
Ivan Ropovik, University of Presov, Slovakia
Sarah Stanton, The University of Edinburgh, Scotland
Nick Brown, University of Groningen, the Netherlands
Fieke Wagemans, Universität Duisburg-Essen, Germany
Janis Zickfeld, University of Oslo, Norway

Stage 1: One-page proposal of the CR

Authors are requested to submit a one-page, bullet-pointed overview to the journal prior to submitting their CR for full review. One editor (Lorne Campbell or Hans IJzerman) will provide a quick turnaround on suitability of the proposed CR for the journal. If the proposal is deemed suitable for the journal, one of the editors will create an Open Science Framework (OSF) component where the author(s) can prepare their Stage 2 manuscript. Only after approval of Stage 1 can authors move on to Stage 2.

Page overview preparation guidelines – Stage 1

The one page proposal should be (maximum) ¾ of a page with bullet points about the research and one paragraph making a brief case for consideration. Replication studies are welcome in addition to novel studies. If a replication study is proposed, the potential value of the replication should be argued (e.g., “the original study is highly cited, yet has never been replicated”). Authors should consider whether they will have the necessary resources to conduct the research, as this is a criterion at Stage 2. The one-page proposal should be e-mailed to rips-irsp@ulb.ac.be.

Stage 2: Initial manuscript submission and review

Stage 2 submissions will be prepared on the OSF in a component created by the editors (details below) and should include a brief cover letter. Authors should only start their Stage 2 submission after the editors have given Stage 1 approval and after the editors have created the OSF project. To submit the OSF project, the authors should prepare a brief cover letter for the journal submission system that includes their “view only” link to their finished project page. The cover letter should be submitted to https://www.rips-irsp.com/submit/start/. The editorial board will not consider Stage 3 manuscripts for in-depth review until a complete Stage 2 submission has been approved.

The cover letter should include:

• - A statement confirming that all necessary support (e.g., funding, facilities) are in place for the proposed research. Note that manuscripts will be generally considered only for studies that are able to commence soon after acceptance; however, authors with alternative plans are encouraged to contact the journal office for advice (note that some grants are available for pre-registered research).
• - An anticipated timeline for completing the study if the initial submission is accepted.
• - A statement confirming that the authors agree to share their raw data, any digital study materials, and analysis code as appropriate, and where they will share this.
• - A statement confirming that, following Stage 2 in principle acceptance, the authors agree to register their approved protocol on the Open Science Framework (https://osf.io/), either publicly or under private embargo until submission of the Stage 3 manuscript. Please note that you should not register your OSF project prior to feedback from and acceptance by the journal.
• - A statement confirming that if the authors later withdraw their paper, they agree to the International Review of Social Psychology publishing a short summary of the pre-registered study under a section Withdrawn Registrations.

Project preparation guidelines – Stage 2

In contrast to most journals, we will not require a separate document to submit the CR. Instead, the editors will create an OSF project for the author. This project, the initial Stage 2 submission, will include the following OSF components:

Component 1: Theoretical Rationale and Hypotheses

• - A concise review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. While we do not impose word limits, conciseness will be judged positively in the review process. Please note that following IPA, hypotheses cannot be changed. Theoretical rationale can only include addition of new literature that has come to light during the data collection process, but cannot change the hypotheses. All changes should be clearly marked for ease of review.

Component 2: Pilot Data

• - Optional. Can be included to establish proof of concept, or feasibility of proposed design, and to verify psychometric properties of methods. Any pilot experiments will be published with the final version of the manuscript and will be clearly distinguished from data obtained for the pre-registered experiment(s).

Component 3: Methods, Procedures, and Scales

• - Full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g., outlier removal). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced. These details may also be limited to the analysis script, in case it is well commented.
• - A description of study procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly in the subsequent experiments or any Stage 2 manuscript can be rejected.
• - Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, a sufficient proportion of valid observations, positive controls, or other quality checks that are orthogonal to the experimental hypotheses. Note that a positive control refers to an additional measurement with a well-defined effect size (e.g., the gain/loss framing study by Tversky and Kahneman, 1981). This measurement can be included as a quality check to better understand participant responding. Authors are encouraged to look at the various ManyLabs projects for successfully replicated effects that can serve as positive controls.

Component 4: Planned Analyses

• - Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and duly followed. Planned analyses can be restricted to a well-written and well-commented script and do not have to be described separately in the Wiki of the component. Only pre-planned analyses can be reported in the main Results section of Stage 3 submissions. However, unplanned exploratory analyses will be welcome in a separate section of the Results called Exploratory Results (see below).
• - Studies involving Null Hypothesis Significance Testing (NHST) must include a statistical power analysis. Estimated effect sizes could be justified with reference to the existing literature. However, since publication bias is known to inflate published estimates of effect size, it is usually more desirable to base one’s power analysis on the theoretically smallest meaningful estimate of the effect size (see Perugini et al. [2018] for helpful pointers). For frequentist analysis plans, the a priori power must be 0.8 or higher for all proposed hypothesis tests. One-tailed tests, if properly justified by the hypothesis, are encouraged. In the case of highly uncertain effect sizes, the use of a variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined. In cases where large samples are practically impossible, authors should contact the editors beforehand outlining their strategy.
• - Methods involving Bayesian hypothesis testing are encouraged. For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified. For example, will you use a uniform up to some specified maximum, or a normal/half-normal to represent a likely effect size, or a JZS/Cauchy with a specified scaling constant. For inference by Bayes factors, authors must be able to guarantee data collection until the data is at least 6 times more likely under the alternative hypothesis as opposed to the null hypothesis (or vice versa). Authors with resource limitations are permitted to specify a maximum feasible sample size at which data collection must cease regardless of the Bayes factor; however, to be eligible for advance acceptance, this number must be sufficiently large that inconclusive results at this sample size would nevertheless be an important message for the field. For further advice on Bayes factors or Bayesian sampling methods, prospective authors are encouraged to read this key article by Schönbrodt and Wagenmakers.
• - If described in the Wiki, any description of prospective methods or analysis plans should be written in future tense.

Component 5: Reviewer Component

Reviewers will be asked to use the commenting function on the OSF to give precise and concise concrete feedback on the individual components. This feedback will be used to help with suggestions for improving the study design. Reviewers will also be asked to upload a file to this component that contains more general and abstract feedback on the project, giving higher-level feedback on the research. All reviews will be stored and open to our readers.

Stage 2 General Comments

Proposals should include a timeline for completion of the study and a proposed resubmission date if Stage 1 review is successful. Extensions to this deadline can be discussed with the Confirmatory Reports editor.

Secondary Registrations

The journal welcomes submissions proposing secondary analyses of existing data sets, provided authors can supply sufficient evidence (e.g., letter from independent gatekeeper) to confirm that they have had no prior access to the data in question.

Language

Submissions can only be in English. We encourage native speakers to proofread their submissions for clarity.

For Stage 1 Proposals that are deemed suitable for the journal, an OSF project dedicated to the proposed research will be created by the editors. Stage 2 Submissions that are judged by the editorial board to be of sufficient quality and scientific importance will be sent for in-depth peer review. In considering papers at the registration stage, reviewers will be asked to provide a brief overview of their evaluation using the comments function in component 5 on the OSF, and upload a file to this component that contains more detailed comments on the following:

1. The importance of the research question(s).
2. The logic, rationale, and plausibility of the proposed hypotheses.
3. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
4. Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.
5. Whether the authors have pre-specified sufficient outcome-neutral tests for ensuring that the results obtained are able to test the stated hypotheses, including positive controls and quality checks.

Following Stage 2 peer review, proposals will be rejected outright, sent back for revision, or accepted. In Principle Acceptance (IPA) indicates that the article will be published pending completion of the approved methods and analytic procedures, passing of all pre-specified quality checks, and a defensible interpretation of the results. Stage 1 protocols are not published following IPA. Instead they are maintained on the OSF. After acceptance, authors will be asked to pre-register their project on the OSF.  Following acceptance, authors are asked to copy Components 1-3 into a document and track changes to evaluate changes between Stage 2 and 3 submissions. Component 4 - if only scripts are included - should be described in words in the planned analysis section of the Stage 3 submission.

Authors are reminded that deviations from the stated experimental procedures that may seriously threaten the validity of the inferences/conclusions can lead to rejection at Stage 3. In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g., change of equipment or unanticipated technical error), the authors must consult the editorial board immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted according to editorial discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 3 submission. If the authors wish to alter the experimental procedures more substantially following IPA but still wish to publish their article as a Confirmatory Report then the manuscript must be withdrawn and resubmitted as a new Stage 2 submission. Note that all registered analyses must be undertaken and reported in the Stage 3 manuscript, but additional unregistered analyses may also be included in a final manuscript (see below).

Stage 3: Full manuscript review

Once the study is complete, authors prepare and submit their manuscript for full review. The author(s) are requested to copy their Components 1-3 into a manuscript (that follows APA style) and track changes to show any changes that may have been made to the content. The final manuscript should be submitted via https://www.rips-irsp.com/submit/start/.They should then make the following additions:

Cover letter. The Stage 3 cover letter must confirm

• - That the manuscript includes a link to the public archive containing anonymized study data, digital materials/code and the laboratory log. The cover letter should state the page number in the final manuscript that lists the URL.
• - That the manuscript contains a link to the approved Stage 2 protocol on the OSF. The cover letter should state the page number in the manuscript that lists the URL.
• - That no data for any pre-registered study (other than pilot data included at Stage 2) were collected prior to the date of IPA.

Submission of anonymised raw data, digital study materials, and laboratory log

• - Anonymised raw data and digital study materials must be made freely available in a public repository/archive with a link provided within the Stage 3 manuscript. Authors are free to use any repository that renders data and materials freely and publicly accessible and provides a digital object identifier (DOI) to ensure that the data remain persistent, unique and citable. Potential repositories include (but are not limited to), the OSF, Figshare, Harvard Dataverse, and Dryad. For a comprehensive list of available data repositories, see http://www.re3data.org/
• - Data files should be appropriately time stamped to show that data were collected after IPA and not before. Other than pre-registered and approved pilot data, no data acquired prior to the date of IPA is admissible in the Stage 3 submission. Raw data must be accompanied by guidance notes (i.e., meta-data), where required, to assist other scientists in replicating the analysis pipeline. Authors are required to upload any relevant analysis scripts and other digital experimental materials that would assist in replication. In addition, authors should specify which variables were removed for the purpose of anonymisation.
• - Any supplementary figures, tables, or other text (such as supplementary methods) can either be included as standard supplementary information that accompanies the paper, or they can be archived together with the data. Please note that the raw data itself should be archived (see above) rather than submitted to the journal as supplementary material.
• - A basic laboratory log must also be provided outlining the range of dates during which data collection took place. This log should be uploaded to the same public archive as the data and materials.
• - The Stage 3 manuscript must also contain a link to the registered protocol (deposited following IPA) on the OSF or other recognised repository.
• - Note that in rare cases, exceptions can be made to the guidelines above. Any exceptions need to be approved by the editors prior to Stage 2 acceptance.

Background, Rationale, and Methods

• - The hypotheses cannot be altered from the approved Stage 1 submission. At Stage 3, any descriptions in future tense within the Stage 2 manuscript should be changed to past tense. Authors are asked to copy-and-paste their text from Components 1-3 into the Background, Rationale, and Methods. Component 4 - if only scripts are included - should be described in words in the planned analysis section of the Stage 3 submission. Any textual changes to the Introduction or Methods (e.g. correction of typographic errors) must be clearly marked (e.g., with track changes) in the Stage 3 submission. Any relevant literature that appeared following the date of IPA should be covered in the Discussion.

Confirmatory Results & Discussion

• - The outcome of all registered analyses must be reported in the manuscript, except in instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but included with justification in a footnote in the “Confirmatory Results”.
• - It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory Analyses”. Authors should be careful not to base their conclusions on the outcome of statistically significant post hoc analyses.
• - Authors reporting null hypothesis significance tests are required to report exact values to three decimal places and effect sizes and their confidence intervals for all inferential analyses.

The resubmission will most likely be considered by the same reviewers as in Stage 2, but could also be assessed by new reviewers. In considering papers at Stage 3, reviewers will be asked to decide:

1. Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls).
2. Whether the stated hypotheses are the same as the approved Stage 2 submission (required).
3. Whether the authors adhered precisely to the registered experimental procedures.
4. Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative.
5. Whether the authors’ conclusions are justified given the data.

Reviewers are informed that editorial decisions will not be based on the perceived importance, novelty, or conclusiveness of the results. Thus, while reviewers are free to enter such comments on the record, they will not influence editorial decisions.

Manuscript Withdrawal and Withdrawn Registrations

It is possible that authors with IPA may wish to withdraw their manuscript following or during data collection. Possible reasons could include major technical error, an inability to complete the study due to other unforeseen circumstances, or the desire to submit the results to a different journal. In all such cases, manuscripts can be withdrawn at the author's’ discretion. Note, however, that given the time and energy devoted by the editorial staff to processing Stage 2 submissions, we would deeply regret such a decision. In the event that a manuscript is withdrawn the journal will publicly record this in a section called Withdrawn Registrations. This section will include the authors’ names, the proposed title, the abstract from the approved Stage 1 submission, and brief reason(s) for the failure to complete the study. Partial withdrawals are not possible; that is, authors cannot publish part of a registered study by selectively withdrawing one of the planned experiments. Such cases must lead to withdrawal of the entire paper. Studies that are not completed by the agreed Stage 3 submission deadline (which can be extended by negotiation with the editorial office) will be considered withdrawn and will be subject to a Withdrawn Registration.

Tips for Avoiding Desk Rejection at Stage 2

Many Confirmatory Report submissions are desk rejected at Stage 2, prior to in-depth review, for failing to sufficiently meet the Stage 1 editorial criteria. In many such cases, authors are invited to resubmit once specific shortcomings are addressed, although major problems can lead to outright rejection. To help minimize the chances of authors’ submissions being desk rejected, we list below the top nine reasons why Stage 2 submissions are rejected prior to review.

1. Cover letter doesn’t make necessary statements concerning ethics, data archiving, and so forth (see above).
2. The protocol contains insufficient methodological detail to enable replication and prevent researcher degrees of freedom. One commonly neglected area is the criteria for excluding data, both at the level of participants and at the level of data within participants. In the interests of clarity, we recommend listing these criteria systematically rather than presenting them in prose.
3. Lack of correspondence between the scientific hypotheses and the pre-registered statistical tests. This is a common problem and severe cases are likely to be desk rejected outright. To maximize clarity of correspondence between predictions and analyses, authors are encouraged to number their hypotheses in the Introduction and then number the proposed analyses in the Methods to make clear which analysis tests which prediction. Ensure also that the power analysis, where applicable, is based on the actual test procedures that will be employed to test those hypotheses; for example, don’t propose a power analysis based on an ANOVA but then suggest a linear mixed effects model to test the hypothesis.
4. Target sample size yields an estimated power that is lower than the minimum power level.
5. Power analysis is over-optimistic (e.g., based on previous literature but not taking into account publication bias) or insufficiently justified (e.g., based on a single point estimate from a pilot experiment or previous study). Proposals should be powered to detect the smallest effect that is plausible and of theoretical value. Pilot data can help inform this estimate but is unlikely to form an acceptable basis, alone, for choosing the target effect size.
6. Intention to infer support for the null hypothesis from statistically non-significant results, without proposing use of Bayes factors or frequentist equivalence testing.
7. Failure to clearly distinguish work that has already been done from work that is planned. Where a proposal contains a mixture of pilot work that has already been undertaken and a proposal for work not yet undertaken, authors should use the past tense for pilot work but the future tense for the proposed work. At Stage 3, all descriptions shift to past tense.
8. Lack of pre-specified quality checks, positive controls, or an appropriate justification for their absence.