Search
GESIS Training Courses
user_jsdisabled

Wiss. Koordination

Alisa Remizova

Administrative Koordination

Claudia O'Donovan-Bellante

Please wait...

Coding Open-Ended Survey Responses: From Codebooks to LLMs

About
Location:
Online via Zoom
General Topics:
Data Analysis, Computational Social Science, Survey Methodology
Course Level:
Advanced
Format:
Workshop
Software used:
R
Duration:
2 days
Language:
English
Fees:
Student: 220 €
Academic: 330 €
Commercial: 660 €
 
Keywords
Additional links
Lecturer(s): Anna-Carolina Haensch

About the lecturer - Anna-Carolina Haensch

Course description

Registration deadline: 17.05.2027  
Registration after this date might be possible, but is subject to sufficient demand and availability of places.
 
Open-ended questions let respondents answer in their own words. They are used to measure concepts that closed categories miss (e.g., "most important problem," occupation, reasons for nonresponse, survey motivation) and to find out how respondents understand survey questions. They are rarely analyzed to their full potential, however, because turning text into analyzable categories ("coding") is costly and error-prone. This course introduces you to state-of-the-art methods for analyzing open-ended responses, from traditional human coding to LLMs. We treat coding as a measurement problem: Every coding approach, whether human coders, supervised classifiers, or LLMs, produces codes that contain measurement error, and that error must be quantified and taken into account in downstream analyses.
On Day 1, you learn to develop and test codebooks, assess intercoder reliability, and build supervised classifiers. Day 2 covers zero- and few-shot coding with LLMs, validation against human-coded gold standards, and valid statistical inference with machine-generated codes. Practical issues such as multilingual data, cost, reproducibility, and data protection (proprietary APIs vs. open-weight models) are addressed throughout.
Teaching combines live lectures with hands-on exercises in R (Quarto notebooks) using real survey data. You are encouraged to bring your own open-ended data.
 
Organizational structure of the course
Mornings consist of live lectures with short breakout room exercises. Afternoons are hands-on lab sessions in which participants work through notebooks individually or in small groups. There is time to apply the methods to participants' own data. The lecturer is available throughout the lab sessions for individual consultation and troubleshooting, and each day ends with a joint discussion of results.
 


Target group

You will find the course useful if:
  • you have (or plan to collect) open-ended survey responses and need a scalable, defensible coding strategy;
  • you want to use LLMs for text annotation but are unsure how to validate them and report the results.


Learning objectives

By the end of the course, you will:
  • develop, pilot, and revise a codebook and evaluate intercoder reliability;
  • train and evaluate supervised classifiers for open-ended responses;
  • design, validate, and document LLM-based coding pipelines;
  • choose between human, supervised, and LLM coding based on accuracy, cost, reproducibility, and data protection.


Prerequisites

  • Basic R (tidyverse data handling, running scripts or Quarto documents);
  • Working knowledge of survey methodology and regression analysis;
  • Familiarity with classification metrics is helpful but not required.
 
Software requirements
R ≥ 4.3 and RStudio or Positron, with the following packages:
  • tidyverse for data handling
  • quanteda and tidytext for text preprocessing
  • irr for intercoder reliability
  • ellmer for LLM APIs
  • rollama for local open-weight models via Ollama


Schedule

Recommended readings