GESIS Training Courses
user_jsdisabled
Search

Scientific Coordination

Dr.
André Ernst
Tel: +49 221 4703736

Administrative Coordination

Claudia O'Donovan-Bellante
Tel: +49 621 1246-221

Applied Multiverse Analysis

About
Location:
Online via Zoom
 
General Topics:
Course Level:
Format:
Software used:
Duration:
Language:
Fees:
Students: 200 €
Academics: 300 €
Commercial: 600 €
Keywords
Additional links
Lecturer(s): Reinhard Schunck, Nora Huth-Stöckle

About the lecturer - Reinhard Schunck

About the lecturer - Nora Huth-Stöckle

Course description

Any data analysis is based on a large number of decisions. These decisions relate to, among other things, study design, data preparation, and the selection and specification of statistical models (Rijnhart et al. 2021). Therefore, a single analysis represents only one possibility among a larger set of alternatives. This leads to the question of how much the analysis results depend on the often undocumented choices.
The relatively new approach of multiverse analysis (Steegen et al. 2016) addresses two fundamental problems in research: the lack of transparency and the dependence of analysis results on data-analytic decisions (Young 2018). The idea of multiverse analysis is to conduct not just one, but ideally all (meaningful) analyses and present the results in summary form. This can make the impact of data analytic decisions on the results transparent and assess whether the conclusions are robust to alternative (modeling) decisions.
However, multiverse analysis can be time-consuming and resource-intensive, and it introduces new questions and challenges. This refers especially to the comparison of the (many) results. On the other hand, it also makes aspects of the scientific process easier, as it relieves the researcher from the burden to "sell" the best possible story.
In the course, the basics of multiverse analysis are explained step by step, applied to a real data example in Stata, and the results are presented using a so-called specification curve (Simonsohn et al. 2020). Participants will gain practical experience in conducting a multiverse analysis. The focus is on the application of a multiverse analysis for collected data.  


Target group

Participants will find the course useful if:
  • they are interested in how to improve the reliability of inferences in quantitative research.
  • they are interested in how to conduct research openly and transparently.
  • they are interested in how to apply multiverse analysis in Stata.
  • plan to conduct a multiverse analysis in their own research.


  • Learning objectives

    By the end of the course participants will:
  • have learned and discussed the features, typical applications, advantages, and shortcomings of multiverse analysis.
  • have acquired practical insights into the steps that are needed to set up multiverse analysis, to conduct them, and how    to interpret their results.
  • be able to conduct a multiverse analysis on their own in Stata.
  •  
    Organizational structure of the course
    The course will feature about three hours of classroom instruction and three hours of hands-on exercises and/or group work each day.
     
    Classroom instruction: The instructors will provide an overview of different aspects of multiverse analysis, including the methodological background, the implementation in Stata, practical challenges, will explain the tasks for the hands-on exercises, and discuss the results of the exercises.
     
    Individual exercises: In each exercise, participants are expected to work on assignments in groups. The instructors are available during the exercises to assist or to provide guidance.  
     
    If time permits, participants may consult instructors for more extensive help and guidance on their personal projects. Participants interested in individual consultations concerning their ongoing projects are encouraged to contact the lecturers before the course and to provide a short description of the issues they would like to discuss.


    Prerequisites

  • solid basic understanding of statistical methods.
  • experience in the analysis of quantitative data.
  • solid basic knowledge of Stata.
  •  
    Software requirements
    Participants need a laptop/desktop computer that enables them to access the internet and smoothly work with Stata. Participants will be provided with access to Stata licenses by GESIS but must install the software prior to the course on their own devices.
     
    Stata ados used during the course (and, ideally, installed before the course) include:
  • mrobust
  • multivrs
  • moremata
  • valuesof
  • reghdfe
  • ftools
  • blindschemes
  • estout
  • speccurve (https://raw.githubusercontent.com/martin-andresen/speccurve/master)
  • specurve (https://github.com/mgao6767/specurve)
  • specc


  • Schedule

    Recommended readings