Search
GESIS Training Courses
user_jsdisabled

Wiss. Koordination

Verena Kunz

Administrative Koordination

Claudia O'Donovan-Bellante

Please wait...

Multiverse Analysis in R – Applications for Survey, Observational, and CSS Data

About
Location:
Online via Zoom
Course times:
10:00-17:00 (CET)
 
General Topics:
Data Analysis, Computational Social Science
Course Level:
Advanced
Format:
Workshop
Software used:
R
Duration:
2 days
Language:
English
Fees:
Students: 220 €
Academics: 330 €
Commercial: 660 €
Keywords
Additional links
Lecturer(s): Maximilian Linde, Paul Balluff, Chung-hong Chan

About the lecturer - Maximilian Linde

About the lecturer - Paul Balluff

About the lecturer - Chung-hong Chan

Course description

Registration deadline: 06.11.2026  
Registration after this date might be possible, but is subject to sufficient demand and availability of places.
 
Scientific conclusions depend on a series of decisions, such as data preprocessing, model specification, variable operationalization and transformation, and data/outlier exclusion (Simmons et al., 2011), resulting in a large universe of possible decision combinations. Oftentimes, results for only one of these decision combinations are reported in a scientific paper, while the pool of alternative decision combinations is undocumented and not further considered. This often leads researchers to put too much confidence in their findings and thus exacerbated claims about reality. Multiverse analysis (e.g., Steegen et al., 2016) is a framework that attempts to remedy this by first making these different and defensible decisions explicit and then systematically examining how results vary across all defensible decision combinations. This allows researchers to obtain a more complete, transparent, robust, and modest understanding of the research question and results at hand. Furthermore, it is possible to get insights on boundary conditions, for instance, by observing that the effect is found in the outcome of interest when it is operationalized in a certain way, but not when it is operationalized in another way.
 
Even though multiverse analysis is a promising tool for more fully understanding the research question and results, users of this approach also face certain challenges. For example, it is often unclear how to report and visualize a large collection of results. This is typically done using a specification curve (Simonsohn et al., 2020), but this approach is not feasible in all scenarios. Furthermore, as the multiverse becomes more complex, the topic of computational resources (and environmental impact) becomes more pressing. Parallel computing can mitigate the issue of exploding runtime. Lastly, the motivation of researchers to use multiverse analysis is at stake. After all, a multiverse analysis might reveal that the findings are not as robust as one might have initially hoped. Nevertheless, the transparency and thoroughness offered by multiverse analysis are key elements for advancing (open) science.
This hands-on workshop equips researchers with the conceptual foundation and practical tools to design, implement, and interpret multiverse analyses in their own research. Participants will learn how analytic flexibility contributes to uncertainty in scientific results and how multiverse designs help reduce hidden researcher degrees of freedom. Drawing on empirical examples from survey research, text analysis, and recent computational and LLM-based studies, the workshop situates multiverse analysis within broader debates on research transparency and reproducibility. Participants will work with their own data or structured case studies provided by the lecturers and start implementing a complete multiverse workflow in R. This includes data preparation, formal parameterization of analytical choices, parallelized model fitting, and visualization using specification curves and related techniques. The workshop concludes with a collaborative hackathon phase in which participants prototype or refine a multiverse design for their own research. By the end, participants will be able to design, implement, and critically evaluate multiverse analyses that strengthen robustness, transparency, and theoretical precision in their empirical work.
 
 
Organisational structure of the course
Classroom instruction
The lecturers will introduce the foundations of multiverse analysis, including its methodological rationale, its role in addressing analytical flexibility, and its implications for research transparency and robustness. Further sessions will cover study design considerations, feasibility and scope of multiverse projects, pre-registration strategies, and computational implementation in R (including parameterization, parallelization, and visualization). In addition, selected empirical case studies of natural experiments, survey research, text analysis, and analysis of digital behavioral data will be discussed to illustrate different types of multiverse designs and specific challenges thereof.
 
Exercises, case studies, and project-based work
A central component of the course is the development and execution of a multiverse design. Participants will work either (a) on a pre-selected empirical study provided by the lecturers, which they will conceptually “turn into” a multiverse study, or (b) on their own ongoing research project. On Day 1, participants will work individually or in small groups to identify analytically defensible choices, define the multiverse dimensions, and outline a feasible implementation strategy. On Day 2, each group will present a short project pitch (~10 minutes), outlining their multiverse design, its theoretical justification, and anticipated challenges. In addition, participants will complete guided hands-on exercises in R, implementing a full multiverse workflow from data preparation to visualization.
 
Support during lab sessions
Throughout all exercise and lab sessions, the lecturers will be available for consultation, methodological feedback, and technical support. This includes assistance with R implementation, feedback on project feasibility and scope, and moderation of group discussions. Participants who wish to discuss their own projects in greater depth are encouraged to contact the lecturers in advance and provide a short description of their research context and specific questions.


Target group

You will find the course useful if:
  • You work with quantitative data in fields such as sociology, psychology, communication science, computational social science, political science, or related disciplines.
  • You are currently designing, analyzing, or planning an empirical study and want to strengthen the transparency, robustness, and theoretical clarity of your research.
  • You use data such as survey data, observational data, text data, digital trace data, or other computational data sources in your research.
  • You face multiple defensible choices in modeling or data preprocessing and want structured ways to deal with them.
  • You have familiarity with statistical modeling and experience with R.
  • You care about improving transparency and credibility in research.
  • You want to systematically evaluate how alternative analytical decisions influence your substantive conclusions.


Learning objectives

By the end of the course, you will:
  • Be familiar with the conceptual foundations of multiverse analysis. You will understand the methodological rationale of multiverse analysis, its relationship to analytical flexibility, robustness, and reproducibility, and its distinction from conventional robustness checks.
  • Be able to design a feasible multiverse study. You will learn how to identify theoretically defensible analytical choices, define multiverse dimensions, assess feasibility and scope, and integrate multiverse logic into study design and pre-registration.
  • Be able to implement multiverse workflows in R. You will acquire practical skills in data preparation, formal parameterization of analytical decisions, parallelized model estimation, and reproducible workflow management.
  • Be able to summarize, visualize, interpret, and communicate multiverse results using specification curves and related visualization techniques, and to interpret patterns of robustness and boundary conditions.
  • Be able to critically evaluate and apply multiverse analysis to your own research. You will have developed the ability to critically assess published multiverse studies and to design, implement, and refine a multiverse analysis tailored to your own empirical projects.


Prerequisites

  • Solid foundation in statistical methods (e.g., multilevel regression, generalized linear model).
  • Experience processing and analyzing quantitative data (e.g., filtering, transforming, formulas).
  • Working knowledge of R (data handling, regression modeling).
    • Software Requirements
    You will require R (https://cran.r-project.org/) and a graphical user interface (e.g., RStudio; https://posit.co/download/rstudio-desktop/). Furthermore, you need the following R packages:
  • ggplot2
  • multiverse
  • specr
    • Please install this software prior to the workshop.
    Optional R packages are:
  • purrr
  • furrr


    • Schedule

    Recommended readings