Scientific Coordination

Simulation studies are computer experiments that generate data by pseudo-random sampling. The aim is typically to understand the statistical properties of methods and to compare different methods. Simulation studies are particularly useful when analytic results are not available. As such, simulation studies are a key tool for both methodological and applied quantitative researchers. To produce a meaningful simulation study, careful thought is required. This two-day workshop outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting, and presentation. The course will be practically focused, so concepts will be described in terms of examples and tested out, and there will be opportunities for critical discussion of published simulation studies or those being planned by participants. The course is 'bilingual', with computer code to support R and Stata users, with one lecturer expert in each.

Organizational structure of the course

Each morning and afternoon will involve a lecture followed by a practical session.

All lectures will be delivered 'live' and are interactive: we welcome participants asking questions or discussing examples they have seen, and we will stop for polls and discussions.

Practical sessions will be a mix of discussion-based and computer-based sessions. One of the main practical tasks will be for participants to design, code, execute, analyze, and report back on a simple simulation study that the lecturers will introduce. We will ask participants to work together in small groups and to critique each other's ideas. The lecturers will be on hand to facilitate discussions, discuss participants' general questions, and for individual consultation during these sessions. Participants are invited to send examples of simulation studies that are published or that they are working on, to be discussed during the lectures.

Target group

Participants will find the course useful if:

They are methodological or applied quantitative researchers who need to evaluate the statistical properties of one or more methods

They are PhD students using simulation studies

They are readers of methodology articles, which evaluate methods by simulation

Learning objectives

By the end of the course, participants will:

Understand the rationale for simulation studies and their limitations

Appreciate the importance of careful planning of simulation studies

Have the knowledge to write and debug simple simulation studies in R or Stata

Know how to analyze simulation studies

Be able to present and critique methods and results of simulation studies to prepare for publication

Prerequisites

Training in quantitative science.

Familiarity with R and/or Stata to the level of being able to:

run regression commands and access results.

use loops.

write a simple program (Stata) or function (R).

produce simple graphics.

Software requirements

Participants using their own laptops will need either Stata (version 15 or later) or R (version 4.3.0 or later) installed. If you need Stata, please let us know the latest two weeks in advance of the course start so that we can take care of it.

Schedule

Recommended readings

The 'paper of the course' is Morris, White and Crowther (2019). This paper was an expanded version of an early version of the course. As such, this is the key reference (but it is not necessary to have read it prior to the course).

Morris, TP, White, IR, Crowther, MJ. Using simulation studies to evaluate statistical methods. Statistics in Medicine. 2019; 38: 2074- 2102. https://doi.org/10.1002/sim.8086

Additional recommended literature

Some additional literature is provided below if of interest. It is by no means required for the course.

A.-L. Boulesteix, H. Binder, M. Abrahamowicz, andW. Sauerbrei, “On the necessity and design of studies comparing statistical methods,” Biometrical Journal, vol. 60, no. 1, pp. 216-218, 2017. doi: 10.1002/bimj.201700129. [Online]. Available: https://doi.org/10.1002/bimj.201700129.
R. P. Chalmers and M. C. Adkins, “Writing effective and reliable monte carlo simulations with the SimDesign package,” The Quantitative Methods for Psychology, vol. 16, no. 4, pp. 248-280, 2020. doi: 10.20982/tqmp.16.4.p248. [Online]. Available: https://doi.org/10.20982/tqmp.16.4.p248.
J. Ensor, D. L. Burke, K. I. E. Snell, K. Hemming, and R. D. Riley, “Simulation-based power calculations for planning a two-stage individual participant data meta-analysis,” BMC Medical Research Methodology, vol. 18, no. 1, 2018. doi: 10.1186/s12874-018-0492-z. [Online]. Available: https://doi.org/10.1186/s12874-018-0492-z.
A. H. Feiveson, “Power by simulation,” The Stata Journal, vol. 2, no. 2, pp. 107-124, 2002. doi:10 . 1177 / 1536867x0200200201. [Online]. Available: https://doi.org/10.1177/1536867x0200200201.
A. Gasparini, “Rsimsum: Summarise results from Monte Carlo simulation studies,” Journal of Open Source Software, vol. 3, no. 26, p. 739, 2018. doi: 10.21105/joss.00739. [Online]. Available: https://doi.org/10.21105/joss.00739.
A. Gasparini, T. P. Morris, and M. J. Crowther, “INTEREST: INteractive tool for exploring REsults from simulation sTudies,” Journal of Data Science, Statistics, and Visualisation, vol. 1, no. 4, 2021. doi: 10.52933/jdssv.v1i4.9. [Online]. Available: https://doi.org/10.52933/jdssv.v1i4.9.
G. Heinze, A.-L. Boulesteix, M. Kammer, T. P.Morris, and I. R.White, “Phases of methodological research in biostatistics-building the evidence base for new methods,” Biometrical Journal, 2023. doi:10.1002/bimj.202200222. [Online]. Available: https://doi.org/10.1002/bimj.202200222.
E. B. E. Koehler and S. J.-P. A.Haneuse, “On the assessment of Monte Carlo error in simulation based statistical analyses,” The American Statistician, vol. 63, pp. 155-162, 2009. doi: 10.1198/tast.2009.0030. [Online]. Available: http://dx.doi.org/10.1198/tast.2009.0030.
K. Luijken, A. Lohmann, U. Alter, J. C. Gonzalez, F. J. Clouth, J. L. Fossum, L. Hesen, A. H. J. Huizing, J. Ketelaar, A. K. Montoya, L. Nab, R. C. C. Nijman, B. B. L. P. de Vries, T. D. Tibbe, Y. A. Wang, and R. H. H. Groenwold, Replicability of simulation studies for the investigation of statistical methods: The replisims project, 2023. arXiv:2307.02052 [stat.ME].
G. Maldonado and S. Greenland, “The importance of critically interpreting simulation studies.,” Epidemiology, vol. 8, pp. 453-456, 4 1997, issn: 1044-3983.
H. I. Oberman and G. Vink, “Toward a standardized evaluation of imputation methodology,” Biometrical Journal, 2023. doi: 10.1002/bimj.202200107. [Online]. Available: https://doi.org/10.1002/bimj.202200107.
S. Pawel, L. Kook, and K. Reeve, “Pitfalls and potentials in simulation studies,” arXiv, 2022. arXiv: 2203.13076 [stat.ME]. [Online]. Available: http://arxiv.org/pdf/2203.13076v1:PDF.
J. E. Rudolph, J. K. Edwards, A. I. Naimi, and D. J. Westreich, “Simulation in practice: The balancing intercept,” American Journal of Epidemiology, vol. 190, no. 8, pp. 1696-1698, 2021. doi:10.1093/aje/kwab039. [Online]. Available: https://doi.org/10.1093/aje/kwab039.
G. Rücker and G. Schwarzer, “Presenting simulation results in a nested loop plot,” BMC Medical Research Methodology, vol. 14, no. 1, pp. 129+, 2014, issn: 1471-2288. doi: 10.1186/1471-2288-14-129. [Online]. Available: http://dx.doi.org/10.1186/1471-2288-14-129.
I. R. White, “simsum: Analyses of simulation studies including monte carlo error,” The Stata Journal, vol. 10, no. 3, pp. 369-385, 2010. doi: 10.1177/1536867x1001000305. [Online]. Available: https://doi.org/10.1177/1536867x1001000305.
I. R. White, T. M. Pham, M. Quartagno, and T. P. Morris, “How to check a simulation study,” International Journal of Epidemiology, 2023. doi:10.1093/ije/dyad134. [Online]. Available: https://doi.org/10.1093/ije/dyad134.

Thursday, 10.10.
10:00-13:00	Morning Session Introduction to the course Lecture 1: Principles of simulation studies Practical exercise 1: Plan your simulation study
13:00-14:00	Lunch Break
14:00-17:00	Afternoon Session Lecture 2: Coding a simulation study Practical exercise 2: Code up your simulation study
Friday, 11.10.
10:00-13:00	Morning Session Lecture 3: Analyzing a simulation study Practical exercise 3: - Analyze your simulation study - Prepare to present results
13:00-14:00	Lunch Break
14:00-17:00	Afternoon Session Group session: report back results of your simulation study Lecture 4: Reporting simulation studies Practical 4: Read and critique a published simulation study