Scientific Coordination

This course offers an introduction to conjoint survey experiments, enabling students to conduct their own conjoint survey-experimental research including design, implementation, and analysis.

Survey experiments have become central in the social science methods toolbox, promising the dual benefit of an experimental design (for causal inference) on population-representative samples (for external validity). In addition, conjoint survey experiments promise a causal understanding of complex, multidimensional decision problems citizens and politicians regularly face in political decision-making-like voting for parties or deciding between policy options. This course introduces students to this survey experimental method, outlining the basic logic, summarizing current topical debates on promises and pitfalls, and giving hands-on advice on design, implementation, and analysis in applied work. We expect and encourage students to bring their own conceptual ideas for a survey-experimental research question that they want to develop further during the course.

For additional details on the course and a day-to-day schedule, please download the full-length syllabus.

Organizational structure of the course

We will spend the afternoon sessions with lab sessions and group work. We will prepare replication code (in R and Stata) allowing everyone to replicate an existing published study. Other lab sessions will focus on the practical set-up of conjoints or will be based on group discussions (see detailed description in the day-to-day schedule of the syllabus). The group work sessions will allow you to work on your own research projects and collaborate with others who plan similar research projects.

Target group

You will find the course useful if:

you are interested in designing, implementing, and analyzing a conjoint.

Learning objectives

By the end of the course, you will understand the basics of conjoint experimental design, implementation, and analysis.

Design: know how to set up and evaluate a conjoint survey-experimental research project, including ethical implications, preregistration, and power analysis.

Implementation: comprehend the dos and don'ts of conjoint implementations in applied practice.

Analysis: be able to analyze, visualize and interpret conjoint survey-experimental data.

Prerequisites

This course is most helpful for researchers at earlier stages of a survey experimental project; ideally, they already have a research question in mind and can use the week-long course to delve into the specifics.

The tutorial sessions will focus on R, but we aim at offering a Stata alternative if needed. Basic R knowledge is very helpful, but not a prerequisite.

Most importantly, we expect students to have read the literature listed under required before course days. Optional literature is additionally marked with a star (*).

Software and hardware requirements

You will need to bring a laptop computer to successfully participate in this course.

Before the start of the course, please install either

R (https://cran.r-project.org/) and RStudio (https://posit.co/download/rstudio-desktop/) on your device. These are free and open source.

a recent version of Stata (15 or higher). GESIS will provide you with short term licenses for Stata for the duration of the course if needed.

For an introduction or refresher in R programming, you might consider enrolling in GESIS's two-day onsite course, “Introduction to R for Data Analysis” held in the first week of the Summer School in Cologne, or the four-day online workshop, “ Introduction to R ” offered in May.

For an introduction or refresher in Stata programming, you might consider enrolling in GESIS's two-day hybrid (onsite in Cologne/online via Zoom) course, “ Introduction to Stata for Data Management and Analysis ” held in the first week of the Summer School.

Schedule

L: Lecture

B: Lab

G: Group Work

H: Individual office hours

Day 1 (10:00-18:00)

Intro to conjoints

L: Welcome & why survey experiments

L: Rubin Causal Model & foundations and assumptions of the binary forced-choice conjoint design

B: Try out different conjoint experiments

G: Academic speed-dating; find group of like-minded researchers; work on individual projects in groups (task: present your core idea to the others)

H: Individual office hours

Literature:

Required

Grady, T. (2022). 10 Things to Know About Survey Experiments. https://egap.org/resource/10-things-to-know-about-survey-experiments/

Stantcheva, S. (2023). How to run surveys: A guide to creating your own identifying variation and revealing the invisible. Annual Review of Economics, 15, 205-234.

Optional

* Cunningham, S. (2021). Causal inference: The mixtape. Yale university press. Ch. 4 https://mixtape.scunning.com/04-potential_outcomes.html

* Cornesse, C., et al. (2020). A review of conceptual approaches and empirical evidence on probability and nonprobability sample survey research. Journal of Survey Statistics and Methodology, 8.1, 4-36.

Day 2 (09:30-17:15)

Designing a conjoint

L: Set-up of conjoint survey experiments

L: Working with external survey providers

B: Setting up surveys yourself online (Qualtrics) or offline (Word/Excel)

G: Work on individual projects in groups (task: improve design of your own project)

H: Individual office hours

Literature:

Required

Bansak, K., Hainmueller, J., Hopkins, D., & Yamamoto, T. (2021). Conjoint Survey Experiments. In J. Druckman & D. Green (Eds.), Advances in Experimental Political Science (pp. 19-41). Cambridge:

Cambridge University Press.

Bowen, T., Goldfien, M. A., & Graham, M. H. (2023). Public opinion and nuclear use: Evidence from factorial experiments. The Journal of Politics, 85(1), 345-350.

Mutz, D. C. (2011). Population-based survey experiments. Princeton University Press. Chapters 4, 7.

Optional

* Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014). Causal inference in conjoint analysis:Understanding multidimensional choices via stated preference experiments. Political Analysis, 22(1), 1-

30.

* Hainmueller, J., & Hopkins, D. J. (2015). The hidden American immigration consensus: A conjoint analysis of attitudes toward immigrants. American Journal of Political Science, 59(3), 529-548.

Day 3 (09:30-17:15)

Analyzing a conjoint

L: AMCEs and beyond

L: How to analyze conjoint experiments? Discuss code and interpret results

B: Practice code on how to analyze a conjoint experiment

G: Work on individual projects in groups (task: improve design)

H: Individual office hours

Literature

Required

AMCE and beyond:

Leeper, T. J., Hobolt, S. B., & Tilley, J. (2020). Measuring subgroup preferences in conjoint experiments. Political Analysis, 28(2), 207-221.

Optional

* Abramson, Scott F., Korhan Kocak, und Asya Magazinnik (2022). „What Do We Learn about Voter Preferences from Conjoint Experiments?“ American Journal of Political Science, 66(4), 1008-20.

* Bansak, Kirk, Jens Hainmueller, Daniel J. Hopkins, und Teppei Yamamoto (2022). „Using Conjoint Experiments to Analyze Election Outcomes: The Essential Role of the Average Marginal Component

Effect“. Political Analysis, 31(4), 1-19.

Alternative Designs :

* Clifford, S., Leeper, T. J., Rainey, C. (2024). Generalizing Survey Experiments Using Topic Sampling: An Application to Party Cues. Political Behavior, 46(2), 1233-56.

* Offer-Westort et al (2021) - Adaptive Experimental Design: Prospects and Applications in Political Science. Americal Journal of Political Science, 65(4), 826-844.

* Vecchiato, A. & Munger, K. (2024). Introducing the Visual Conjoint, with an Application to Candidate Evaluation on Social Media. Journal of Experimental Political Science, First View, 1-12.

Day 4 (09:30-17:15)

Estimation and extensions

L: Preregistrations, statistical power, ethics

L: Calculating power

B: Discussing individual research projects

G: Prepare reading and presentation for Friday (group work)

H: Individual office hours

Literature:

Required

Preregistrations:

Alrababa'h, A., Williamson, S., Dillon, A., Hainmueller, J., Hangartner, D., Hotard, M., . . . Weinstein, J. (2023). Learning from Null Effects: A Bottom-Up Approach. Political Analysis, 31(3), 448-456.

doi:10.1017/pan.2021.51

Optional

* Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Percie du Sert, N., ... & Ioannidis, J. (2017). A manifesto for reproducible science. Nature human behaviour, 1(1), 1-9.

Required

Statistical Power:

Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), 33267.

Optional

* Rainey, Carlisle (2024). „Power Rules - Practical Statistical Power Calculations“. https://github.com/carlislerainey/power-rules/blob/main/power-rules.pdf.

* Schuessler, J., & Freitag, M. (2020). Power analysis for conjoint experiments. https://osf.io/preprints/socarxiv/9yuhp/

Implementation / Ethics:

* Mutz, D. C. (2011). Population-based survey experiments. Princeton University Press. Chapter 6.

* Slough, T. (2024). Making a Difference: The Consequences of Electoral Experiments. Political

Analysis, 32(4), 384-400.

Day 5 (09:30-14:30)

Current debates

L: Current debates in conjoint survey experiments (Groups present one paper from the list; ensure short and concise presentation)

L: Dos and don'ts

B: Discuss open questions (link to your own projects if you want)

Literature:

Required

Dos and Don'ts:

Kane, John V. (2024). More than meets the ITT: A guide for anticipating and investigating non-significant results in survey experiments. Journal of Experimental Political Science, 1-16.

Optional

* Stantcheva, S. (2023). How to run surveys: A guide to creating your own identifying variation and revealing the invisible. Annual Review of Economics, 15, 205-234.

One reading required (groups choose on day 1/2 which to read) to be presented on Friday by the groups

Current Issues:

Berinsky, A. J., Frydman, A., Margolis, M. F., Sances, M. W., & Valerio, D. C. (2024). Measuring Attentiveness in Self-Administered Surveys. Public Opinion Quarterly, 88(1), 214-241.

Clayton, K., Porter, E., Velez, Y., & Wood, T. J. (2024). Improving debriefing practices for participants in social science experiments. PNAS nexus, 3(12), page 502.

Clayton, Katherine, Yusaku Horiuchi, Aaron R. Kaufman, Gary King, and Mayya Komisarchik (2023). Correcting Measurement Error Bias in Conjoint Survey Experiments. Unpublished working paper. Available at: https://gking.harvard.edu/conjointE (also links to R package) (13.01.2025).

Dafoe, Allan, Baobao Zhang, und Devin Caughey (2018). Information Equivalence in Survey Experiments. Political Analysis 26(4), 399-416.

Liu, G., & Shiraito, Y. (2023). Multiple Hypothesis Testing in Conjoint Analysis. Political Analysis, 31(3), 380-395.

Mummolo, J., & Peterson, E. (2019). Demand effects in survey experiments: An empirical assessment. American Political Science Review, 113(2), 517-529.

Porter, E., & Velez, Y. R. (2022). Placebo selection in survey experiments: An agnostic approach. Political Analysis, 30(4), 481-494.

Bonus options (if groups see better fit):

Andersen, D. J., & Ditonto, T. (2018). Information and its presentation: Treatment effects in low-information vs. high-information experiments. Political Analysis, 26(4), 379-398.

Brutger, R., Kertzer, J. D., Renshon, J., Tingley, D., & Weiss, C. M. (2023). Abstraction and detail in experimental design. American Journal of Political Science, 67(4), 979-995.

Egami, N., & Hartman, E. (2023). Elements of External Validity: Framework, Design, and Analysis. American Political Science Review, 117(3).

Fong, C., & Grimmer, J. (2023). Causal inference with latent treatments. American Journal of Political Science, 67(2), 374-389.

Horiuchi, Y., Markovich, Z., & Yamamoto, T. (2022). Does conjoint analysis mitigate social desirability bias?. Political Analysis, 30(4), 535-549.

Jenke, L., Bansak, K., Hainmueller, J., & Hangartner, D. (2021). Using eye-tracking to understand decision-making in conjoint experiments. Political Analysis, 29(1), 75-101.

Kane, John V., and Mia Costa (2024). Being Careful with Conjoints: Accounting for Inattentiveness in Conjoint Experiments. Unpublished working paper. Available at: https://preprints.apsanet.org/engage/apsa/article-details/674614a15a82cea2fafcb991 (13.01.2025).

Montgomery, Jacob M, and Erin L Rossiter (2020). So Many Questions, So Little Time: Integrating Adaptive Inventories into Public Opinion Research. Journal of Survey Statistics and Methodology 8(4), 667-90.

Robinson, Thomas S., and Raymond M. Duch (2024): How to detect heterogeneity in conjoint experiments. Journal of Politics, 86(2), 412-427.

Rudolph, L., Freitag, M., & Thurner, P. (2024). Ordering Effects Vs. Cognitive Burden-How Should We Structure Attributes in Conjoint Experiments. Public Opinion Quarterly, 88(3), 991-1016.