Schedule
Day 1 (10:00-17:30)
Survey processes in various interviewer-assisted and self-completion modes: Sampling strategies and coverage
The first day sets the scene of the course. First, we discuss the strengths and weaknesses of surveys as a research design in comparison to other research methods. We introduce the Total Survey Error (TSE) framework and discuss how the survey mode affects the potential for different survey errors. Dimensions of survey mode - computer- vs. paper-based, interviewer-assisted vs. self-completion, and aural vs. visual - are contrasted. During the second part of this day, we dive into the availability of sampling frames and their coverage of the population. We shortly contrast probability and non-probability samples.
On this first day, you receive the opportunity to develop a survey design (mode and sampling strategy) for your own research question and to be guided therein.
Suggested reading:
Lyberg, L. and Stukel, D. M. (2017). The roots and evolution of the Total Survey Error concept. In: Biemer, Paul P., Edith D. de Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars E. Lyberg, N. Clyde Tucker, and Brady T. West, eds. Total Survey Error in Practice. John Wiley & Sons. P. 1-22. https://www.wiley.com/en-us/Total+Survey+Error+in+Practice+-p-9781119041672Groves, R.M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau (2009) Survey Methodology, 2nd Edition. New York: Wiley and Sons. chapters 2, 3, 5 & 9. https://www.wiley.com/en-br/Survey+Methodology%2C+2nd+Edition-p-9780470465462Fowler, F. J. (2009), Survey Research Methods (4th Edition), London: Sage. Chapter 3 (Sampling). https://methods.sagepub.com/book/survey-research-methods/n3.xmlSalganik, M. J. (2018) Bit by Bit. Social Research in the Digital Age. Princeton, NJ: Princeton University Press. Chapter 3 (Asking Questions). https://press.princeton.edu/books/paperback/9780691196107/bit-by-bitCornesse, C., Blom, A., Dutwin, D., Krosnick, J., De Leeuw, E., Legleye, S., Pasek, J., Pennay, D., Philips, B., Sakshaug, J., Struminskaya, B., Wenz, A. (2020). A Review of Conceptual Approaches and Empirical Evidence on Probability and Nonprobability Sample Survey Research. Journal of Survey Statistics and Methodology 8(1), 4-36. https://doi.org/10.1093/jssam/smz041
Day 2 (09:00-16:00)
Nonresponse processes, prevention, and correction
On day two we cover the various types of nonresponse in survey data and how optimizing the data collection processes may minimize them. Specifically, we look at the effects of incentives on nonresponse rates and nonresponse bias. Surveys always contain some degree of nonresponse. The course will show you how you can correct for unit nonresponse by means of weighting. If time permits, we will talk about the role of interviewers and fieldwork procedures in general to monitor fieldwork. The effect of these treatments on analyzes is demonstrated.
On this second day, you design a fieldwork strategy for your survey that aims to minimize coverage and nonresponse errors.
Suggested reading:
Groves, R.M. et al. (2009), chapters 6, 10.5 & 10.6. https://www.wiley.com/en-br/Survey+Methodology%2C+2nd+Edition-p-9780470465462De Leeuw, E. D., J. J. Hox, and D. Dillman (2008). International Handbook of Survey Methodology. New York, chapters 17 & 19. http://joophox.net/papers/SurveyHandbookCRC.pdfLynn, P. (1996) Weighting for non-response. In Totman et al. Survey and statistical computing, available on: https://pdfs.semanticscholar.org/1e7d/d794cbaf774ecfe578b493859f29ea6c13f2.pdf
Day 3 (09:00-16:00)
Questionnaire design and data accuracy
The third day looks into the survey response process. We assess why operationalizing our research questions into survey questions can be intricate, which survey factors may affect responses and how different respondents differentially understand our questions. Finally, a bit of time is spent discussing effective lay-out of questionnaires, especially in light of designing web and designing or adapting surveys for mobile devices and using new possibilities of collecting smartphone sensor data.
On day three, you develop your own short questionnaire taking into account the design principles covered during class.
Suggested reading:
Groves et al. (2009), chapter 7. https://www.wiley.com/en-br/Survey+Methodology%2C+2nd+Edition-p-9780470465462Dillman, D.A., J.D. Smyth, and L.M. Christian (2009) Internet, Mail, and Mixed-Mode: The Tailored Design Method, 3rd Edition. Wiley and Sons, chapters 4 and 5. Fowler, F.J. (1996) Improving survey questions - design and evaluation. London, Sage, Chapters 1-4 Tourangeau, R. (2003). “Cognitive aspects of survey measurement and mismeasurement.” International Journal of Public Opinion Research, 15, pp. 3-7. https://doi.org/10.1093/ijpor/15.1.3
Day 4 (09:00-16:00)
Questionnaire testing, mode, interviewers, and cross-national comparisons
Day four continues with questionnaire development. Today, however, we will look into methods of pre-testing survey questions, including the qualitative technique of cognitive interviewing. We will focus on the effects of survey modes on measurement error. We will very briefly discuss eye tracking as a method to test questionnaires and detect errors and problems in questionnaire design.
Finally, if time permits, we discuss how to do surveys in different cultures, with the goal of comparing countries within the framework of Total Survey Error.
You will practice the pre-testing techniques learned in class with the questionnaires you developed on day three.
Suggested reading:
Groves et al. (2009), chapter 8. https://www.wiley.com/en-br/Survey+Methodology%2C+2nd+Edition-p-9780470465462Presser, S. , M.P. Couper, J.T. Lessler, E. Martin, J. Martin, J.M. Rothgeb, and E. Singer (2004) “Methods for Testing and Evaluating Survey Questions”, Public Opinion Quarterly, 68 (1): 109-130. https://doi.org/10.1093/poq/nfh008Fowler, F.J. (1996) Improving survey questions - design and evaluation. London, Sage, Chapters 5 and 6.De Leeuw, E. D., J. J. Hox, and D. Dillman (2008). International Handbook of Survey Methodology. New York, chapter 20. http://joophox.net/papers/SurveyHandbookCRC.pdfPennell, B., Cibelli Hibben, K. L., Lyberg, L., Mohler, P. P., & Worku, G. (2017). A total survey error perspective on surveys in multinational, multiregional, and multicultural contexts. In: Biemer, Paul P., Edith D. de Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars E. Lyberg, N. Clyde Tucker, and Brady T. West, eds. Total Survey Error in Practice. John Wiley & Sons. P. 179-202. https://www.wiley.com/en-us/Total+Survey+Error+in+Practice+-p-9781119041672
Day 5 (09:00-16:00)
Data preparation: assessing measurement quality, survey quality vs. costs
On the final day, we look at how to develop and assess measurement errors after we have collected data. This includes topics such as data preparation, coding, and editing. To conclude the course, we look back at the Total Survey Error framework and evaluate how different costs associated with survey design decisions might affect data quality. We discuss the methods that can be used to assess the different components of the Total Survey Error framework and discuss how we may trade-off survey quality against survey costs. Optionally, we can discuss questions of data protection and research ethics in survey research as well as emerging modes of data collection (combining surveys with apps, wearables, data donation).
On day 5, you will have an opportunity to present your current or future survey projects in the class meeting, and to receive feedback from all participants and the instructors on how to improve the survey design. For this, you are encouraged to send details of your future project to the instructors before Wednesday 12 pm of the week in which this Summer School course is taught. Upon the discussion with the instructors, you are encouraged to prepare at least two PowerPoint slides: (1) the description of the design/problem statement, (2) questions to the participants they would like to discuss.
Suggested reading:
Groves et al. (2009), chapters 2 (again) & 10. https://www.wiley.com/en-br/Survey+Methodology%2C+2nd+Edition-p-9780470465462Fowler, F. J. (2009), Survey Research Methods (4th Edition), London: Sage. Chapter 9.Struminskaya, B., Lugtig, P., Keusch, F., Höhne, J. (2020). Augmenting Surveys with Data from Sensors and Apps: Opportunities and Challenges. Social Science Computer Review Special Issue 'Using Mobile Apps and Sensors in Surveys' https://doi.org/10.1177/0894439320979951
