GESIS Training Courses
user_jsdisabled
Search

Scientific Coordination

Alisa Remizova

Administrative Coordination

Janina Götsche

Aggregating Evidence Across Multiple Studies

About
Location:
Online via Zoom
 
General Topics:
Course Level:
Format:
Software used:
Duration:
Language:
Fees:
Students: 330 €
Academics: 495 €
Commercial: 990 €
 
Keywords
Additional links
Lecturer(s): Jessica Daikeler, Rebecca Kuiper

About the lecturer - Jessica Daikeler

About the lecturer - Rebecca Kuiper

Course description

This workshop teaches you how to systematically review and aggregate results from multiple studies to evaluate your hypotheses using advanced methods that go beyond traditional meta-analysis.
In social and behavioral science, the gold standard for scientific evidence is finding results that are consistent across studies. When aggregating results, studies can be generally categorized into two types:
  • i) studies that use homogeneous designs, meaning that the operationalization (and therewith the number) of variables and the statistical model are the same. These studies can include exact replications of an original study, where the methodology is closely replicated to validate findings;
  • ii) studies with heterogeneous/diverse designs. These studies can include conceptual replications of an original study, where different operationalizations and statistical models are used to test the same central theory/hypothesis.
  • Exact replication can be feasible for experimental research, which offers a high degree of experimenter control, but may be hard to publish because of a presumed lack of originality. For other types, like observational studies, exact replication is hardly realizable, and such studies are typically only conceptually replicated.
    Unfortunately, statistical tools for aggregating evidence across conceptual replications or other studies with heterogeneous designs are not commonly used yet, leaving researchers to rely on exact replications. If meta-analysis, the current approach for summarizing results, would be used to aggregate results from heterogeneous studies, the conclusions would be fundamentally unreliable because incomparable estimates are being averaged. While meta-regression can address design variability to some extent, it does so by aggregating estimates for studies only with the same design, providing separate overall estimates for each design type rather than an integrated conclusion.
    In conceptual replications and other studies with heterogeneous designs, the focus is on evaluating the central theory/hypothesis. In these cases, the so-called 'evidence synthesis' or 'support aggregation' offers a powerful approach to combine the evidence for a hypothesis of interest, which we will discuss in this workshop. This approach is not limited to heterogeneous studies-it can also be applied to homogeneous studies and exact replications.
     
    Day 1: Collecting Primary Studies
    The first step in conducting a meta-analysis or evidence synthesis is collecting primary studies, including replication studies and other studies investigating the same phenomenon or hypothesis. On Day 1, we will cover how to systematically review and collect these studies. This includes formulating a research question, defining eligibility criteria for including and excluding studies, conducting the literature search, screening studies, and performing study coding. We will here also provide an overview on available text mining tools in assistance to these steps.
     
    Day 2: Hypothesis Evaluation
    On Day 2, we will focus on evaluating hypotheses using model selection methods. The focus lies on theory-based / informative hypotheses (alternative to null hypothesis). For example, one can evaluate if Medicine A is more effective (e.g., increases happiness) than Medicine B, which in turn is more effective than a placebo (in an ANOVA model: µA > µB > µPlacebo). Alternatively, one can examine if the number of children is a stronger predictor of happiness than income and age (in a regression model with standardized parameters: βNoC > {βInc, βAge}). Two information-theoretical criteria for evaluating such hypotheses are GORIC and GORICA. We will introduce you to these methods, which provide evidence for hypotheses.
     
    Day 3: Evidence Synthesis using GORIC(A)
    Day 3 will cover evidence synthesis using GORIC(A). When multiple studies, regardless of their design, examine the same central hypothesis, the evidence from these studies can be combined. This process, known as evidence synthesis or support aggregation, is similar to meta-analysis for studies with different designs. We will demonstrate how to perform evidence synthesis using GORIC(A).
     
    Note: This workshop does not cover traditional meta-analysis (i.e., fixed or random effects models) or Bayesian meta-analysis. Instead, it will teach you an alternative method to aggregate results from multiple studies by combining evidence for a hypothesis from various studies.
     
    Organizational structure of the course
    The workshop itself comprises lectures with hands-on parts and a lab meeting. During the lab meeting, you will work on exercises provided by the lecture. This can be done individually but also in groups. If of interest, you can also apply the learned methods to your own data/projects.
     
    The lecturers will be available for questions, both regarding the provided exercises and the application to your own data. Depending on wishes and time, the lecture can also discuss one or more exercises and/or applications in a plenary fashion.


    Target group

    The course is suited for you:
  • if you want to combine / aggregate / synthesize results from multiple studies (which may be conceptual replications).
  • if you have one or more theory-based hypotheses (or want to come up with them) and want to evaluate them.


  • Learning objectives

    By the end of the course, you will:
  • know how to perform a systematic review.
  • know how to evaluate one or more theory-based hypotheses (as opposed to null hypothesis testing).
  • know how to interpret the output for evaluating theory-based hypotheses.
  • know how to combine / aggregate / synthesize results from multiple studies.
  • know how to interpret the output for evidence synthesis.


  • Prerequisites

    The workshop itself comprises lectures with hands-on parts and a lab meeting using R.
    No prior experience with R is assumed, but some familiarity with R would be very useful.
     
    Software and hardware requirements
    During the workshop, we will work with R and Rstudio. You should install the latest version of R (https://cran.r-project.org/) and RStudio (https://posit.co/download/rstudio-desktop/) before the start of the workshop.
     
    We will work with the goric and evsyn function from the R package 'restriktor', the newest one; possibly, a version not yet available on CRAN but on github. To install and load restriktor, please use the following R code:
    if (!require("restriktor")) install.packages("restriktor")
    library(restriktor)
     
    If you want to use restriktor from github:
    if (!require("devtools")) install.packages("devtools")
    library(devtools)
    install_github("LeonardV/restriktor")
    library(restriktor)


    Schedule

    Recommended readings