By Jeremy Eberle, M.A., University of Virginia
Preregistration of hypotheses and analysis plans on a public, time-stamped registry (e.g., Open Science Framework; OSF) before observing research outcomes promotes transparency and distinguishes planned from unplanned analyses, thereby guarding against selective reporting, improving validity of inferences for planned analyses, and enabling calibration of confidence for unplanned analyses (Nosek et al., 2018; Nosek et al., 2019). Further, preregistration can benefit individual researchers by enabling credit for planned analyses (e.g., in-principle acceptance of Registered Reports, even for null results), signaling rigor (improving publishability), revealing design flaws early, aiding recall of study details and plans, and improving communication with collaborators (Logg & Dorison, 2021; Simmons et al., 2021; Wagenmakers & Dutilh, 2016). Clinical psychologists may be familiar with registering clinical trial procedures (e.g., eligibility criteria, conditions, outcomes) on websites such as ClincalTrials.gov; however, procedures are registered for only a fraction of trials (Nutu et al., 2019), and where registration is required (e.g., by U.S. law), detailed analysis plans (beyond listing outcomes) are not yet required (Cybulski et al., 2016; Nosek et al., 2018). Preregistering detailed hypotheses and analysis plans can be difficult (Nosek et al., 2019); the present article addresses calls for more training (Berenbaum et al., 2021) by presenting strategies for overcoming challenges in clinical science (for resources on creating a preregistration and applying key strategies, see Table S1 at https://doi.org/kbv8).
Planning Analyses and Navigating Unknowns
To start, in much research (including clinical science), it can be difficult to anticipate the decision points that may arise during data collection and analysis (Nosek et al., 2019). Several strategies may aid planning, including (a) using preregistration templates (see Table S1), which range from more structured (e.g., Bowman et al., 2020) to less structured (e.g., Wharton Credibility Lab, 2022); (b) practicing analyses on simulated data; (c) drafting Method and placeholder Results sections in advance; and (d) submitting a Registered Report for peer review of the plan. Strategies for navigating unknown aspects of the data include (e) using sequential preregistration, in which an initial preregistration specifies the plan for evaluating model assumptions, for example, and a later preregistration specifies the plan for testing the chosen model; (f) evaluating model assumptions on a masked dataset, whose key features (e.g., condition codes) are removed or scrambled, prior to registering a given model; (g) preregistering a decision tree that states how intermediate outcomes (e.g., model assumptions) will influence later analysis choices; (h) adopting a field’s standard operating procedures for analysis; and (i) using cross-validation, where the data are split into exploratory and confirmatory subsets and the tests on the confirmatory set are registered after exploration of the exploratory set. For details on these strategies and others, see Nosek et al. (2018), Nosek et al. (2019), and Srivastava (2018).
Updating Plans and Reporting Deviations
It can be especially difficult to anticipate contingencies that may arise in clinical research (e.g., issues recruiting clinical samples and variable task performance tied to clinical difficulties; Tackett et al., 2017; unstructured data from clinical records). Fortunately, “preregistration is a plan, not a prison” (Nosek et al., 2019, p. 817), and deviations from the plan that improve the methodology are common and can be pursued with transparent reporting and explanation. If data collection has not begun, the original preregistration can be withdrawn and replaced with a new version, and if data collection is underway, the original preregistration can be retained alongside an updated version posted before analysis (DeHaven, 2017; see Table S1). If analysis is underway, deviations can still be reported (see Table S1); strategies for transparently doing so (Nosek et al., 2019) include (a) logging decisions made during analysis (perhaps using version control tools [e.g., Git and GitHub; see Eberle, 2022] to track all steps of an analysis); (b) noting all planned analyses and reporting and explaining all deviations (see Eberle et al., 2023, p. 18, for an example reference to deviations in online supplemental material); and (c) using supplements to report results of preregistered analyses alongside results of revised analyses, especially where the different approaches reflect judgment calls and serve as sensitivity analyses. Multiverse analyses, in which all possible approaches are run and reported, can at times be useful (Srivastava, 2018).
Using Existing Data and Conducting Exploratory Analyses
Clinical psychologists’ collection of large archival and ongoing longitudinal datasets presents an additional challenge to preregistration in that difficult-to-collect data may already exist, and researchers may already know some outcomes; such datasets can also generate new hypotheses over time, making it difficult to specify all hypotheses prior to data collection or analysis (Tackett et al., 2017). Although in its idealized form, preregistration occurs (a) prior to data collection, it can also occur (b) during data collection but before any analysis or (c) after an analysis that differs from the one at hand (Benning et al., 2019, who term [b] coregistration and [c] postregistration). In such cases, researchers can document what they already know and do not know about the data prior to registering hypotheses and analyses for their main research question (Nosek et al., 2018) using templates for secondary data registrations (e.g., van den Akker et al., 2021; see Table S1). Clinical research may also be descriptive or correlational (e.g., trends in temperament and cognitive ability over 13 years in > 1 million participants; see Tackett et al., 2020), with fewer “focal hypotheses” than experimental studies (Tackett et al., 2017). In these cases, researchers can preregister any planned exploratory analyses (i.e., neither directional nor nondirectional a priori hypotheses have been formulated, but analyses are still planned), which have greater credibility than unplanned analyses because the preregistration specifies the number of statistical tests to be conducted (Nosek et al., 2019). The templates above have sections for planned exploratory analyses, and results for planned versus unplanned analyses should be separated in manuscripts. Planned and unplanned analyses are both useful; preregistration helps distinguish them (Nosek et al., 2019).
Conclusion
Preregistration can be difficult, especially when doing it for the first time, but researchers can embrace incrementalism and improve with practice (Nosek et al., 2019). Preregistrations can also underpin the Introduction and Method sections of manuscripts (and can be written in these formats; Benning et al., 2019), effectively moving much inevitable decision-making and writing to an earlier stage of the research process (vs. requiring only additional work). Preregistrations also provide public timestamps for one’s ideas and plans; alternatively, researchers can embargo their preregistrations on OSF for up to 4 years, create anonymized “view-only” links to them for peer-review, and make them public when ready (see Table S1). Although clinical psychology and related fields (e.g., personality, developmental, and health psychology; Tackett et al., 2020) present some challenges for preregistration, various strategies can mitigate or overcome these challenges and thereby improve the statistical inferences at the foundation of our science.
___________________________________________________________
References
Benning, S. D., Bachrach, R. L., Smith, E. A., Freeman, A. J., & Wright, A. G. (2019). The registration continuum in clinical science: A guide toward transparent practices. Journal of Abnormal Psychology, 128(6), 528. https://psycnet.apa.org/doi/10.1037/abn0000451
Berenbaum, H., Washburn, J. J., Sbarra, D., Reardon, K. W., Schuler, T., Teachman, B. A., Hollon, S. D., Atkins, M. S., Hamilton, J. L., Hetrick, W. P., Tackett, J. L., Cody, M. W., Klepac, R. K., & Lee, S. S. (2021). Accelerating the rate of progress in reducing mental health burdens: Recommendations for training the next generation of clinical psychologists. Clinical Psychology: Science and Practice, 28(2), 107–123. https://doi.org/10.1037/cps0000007
Bowman, S., Dehaven, A. C., Errington, T. M., Hardwicke, T. E., Mellor, D. T., Nosek, B. A., & Soderberg, C. K. (2020). OSF Prereg Template. PsyArXiv. https://doi.org/10.31222/osf.io/epgjd
Cybulski, L., May-Wilson, E., & Grant, S. (2016). Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals. Journal of Consulting and Clinical Psychology, 84(9), 753-767. https://psycnet.apa.org/doi/10.1037/ccp0000115
DeHaven, A. (2017, May 23). Preregistration: A plan, not a prison. Center for Open Science. https://www.cos.io/blog/preregistration-plan-not-prison
Eberle, J. W. (2022). Improving the computational reproducibility of clinical science: Tools for open data and code. Newsletter of the Psychological Clinical Science Accreditation System (8th ed.). https://www.pcsasnews.org/improving-the-computational-reproducibility-of-clinical-science-tools
Eberle, J. W., Boukhechba, M. O., Sun, J., Zhang, D., Funk, D. H., Barnes, L. E., & Teachman, B. A. (2023). Shifting episodic prediction with online cognitive bias modification: A randomized controlled trial. Clinical Psychological Science, 11(5), 819-840. https://doi.org/jw42
Logg, J. M., & Dorison, C. A. (2021). Pre-registration: Weighing costs and benefits for researchers. Organizational Behavior and Human Decision Processes, 167, 18-21. https://doi.org/10.1016/j.obhdp.2021.05.006
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., … & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23(10), 815-818. https://doi.org/10.1016/j.tics.2019.07.009
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606. https://doi.org/10.1073/pnas.1708274114
Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A., (2019). Open science practices in clinical psychology journals: An audit study. Journal of Abnormal Psychology, 128(6), 510-516.
https://psycnet.apa.org/doi/10.1037/abn0000414
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2021). Pre-registration: Why and how. Journal of Consumer Psychology, 31(1), 151-162. https://doi.org/10.1002/jcpy.1208
Srivastava, S. (2018). Sound inference in complicated research: A multi-strategy approach. PsyArXiv. https://doi.org/10.31234/osf.io/bwr48
Tackett, J. L., Brandes, C. M., Dworak, E. M., & Shields, A. N. (2020). Bringing the (pre)registration revolution to graduate training. Canadian Psychology / Psychologie canadienne, 61(4), 299–309. https://doi.org/10.1037/cap0000221
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017). It’s time to broaden the replicability conversation: Thoughts for and from clinical psychological science. Perspectives on Psychological Science, 12(5), 742–756. https://doi.org/10.1177/1745691617690042
van den Akker, O. R., Weston, S. J., Campbell, L., Chopik, W. J., Ioana Damian, R., Davis-Kean, P. E., Hall, A. N., Kosie, J. E., Kruse, E., Olsen, J., Ritchie, S. J., Valentine, K. D., van ‘t Veer, A. E., & Bakker, M. (2021). Preregistration of secondary data analysis: A template and tutorial. Meta-Psychology, 5. https://doi.org/10.15626/MP.2020.2625
Wagenmakers, E., & Dutilh, G. (2016). Seven selfish reasons for preregistration. APS Observer, 29(9), 13-14. https://www.psychologicalscience.org/observer/seven-selfish-reasons-for-preregistration
Wharton Credibility Lab. (2022, December 21). AsPredicted. https://aspredicted.org/
Disclaimer: The views and opinions expressed in this newsletter are those of the authors alone and do not necessarily reflect the official policy or position of the Psychological Clinical Science Accreditation System (PCSAS).