Scientific Coordination

The significant progress in computational models for text analysis offers immense potential for social science research (e.g., for large-scale analysis of opinions or harmful language). The rich statistical language representations captured in pre-trained modern language models (e.g., BERT, Llama-2/3, and GPT-3/4) provide an excellent starting point for use-case specific models. Most commonly, the pre-trained models are adapted to the task at hand through fine-tuning, a technique which involves further training of the models on labeled (also referred to as annotated or coded) data.

However, fine-tuning is costly given the enormous number of parameters that modern language models consist of (e.g., up to 340 million for BERT, up to 65 billion for Llama-2, up to trillions for GPT-4). Optimizing the full set of parameters for a new application task requires substantial computing resources, significant time, and enough training data. The energy-intensive process also raises environmental concerns. All these reasons can pose severe restrictions to social science scholars in leveraging the enormous potential of modern language models.

A more parameter-efficient solution for addressing these challenges is the use of adapter modules. Instead of fine-tuning the entire model, only a small number of newly introduced parameters are adapted for the specific application task. This approach not only significantly reduces complexity but also demonstrates performance that is typically on par with that of their fully-tuned counterparts.

In this course, we provide an introduction to adapters. After a brief recap of modern language model architectures, we go into detail about how adapters function, explore different types of adapters, and discuss their use cases. Alongside the theoretical foundations, participants will engage with practical Python-based implementation examples through hands-on tutorials using real digital behavior datasets. Participants will also have the possibility to discuss their own applications with the instructors.

Organizational Structure of the Course

The course will take place in a hybrid setting with participants on-site and online. In the morning sessions, we will cover the theoretical foundations and showcase the practical implementation using Python notebooks. In the afternoon sessions, participants will have the opportunity to apply the learned content autonomously during lab sessions with hands-on exercises. Participants can reach out for support to the lecturers at any time during the practical sessions. While we provide tasks and data for the lab sessions on day 1 and day 2, on day 3, participants have the option to transfer the gained knowledge onto their own research questions. This will be accompanied by individual consultation slots with the lecturers.

Target group

You will find the course useful if:

You work on or aim to apply state-of-the-art computational methods for text analysis

You have a background in Computational Social Science or related fields

You feel constrained by the computational costs of applying language models to your research / fine-tuning models for use with your data and target task

Learning objectives

By the end of the course you will:

Have a solid understanding of the problems and challenges of fine-tuning pre-trained large language models for your own analyses

Gain both theoretical and practical understanding of the concept of adapters and how they enable lightweight and parameter-efficient machine learning with transformer models

Be able to apply adapters to language models in Python to address your own research question

Prerequisites

Solid knowledge of programming in Python (e.g., data processing with pandas, basic syntax) (required)

First experiences and conceptual understanding of machine learning (e.g., classification) (required)

Basic understanding of natural language processing (e.g., preprocessing with tokenization) and language models (e.g., BERT) (required)

Familiarity with the Hugging Face ecosystem (recommended)

Software Requirements

We are planning to use Google Colab (free version) for the practical parts of the course for GPU access and to avoid system-specific troubleshooting. For this, the participants need a Google account. If participants choose not to use Google Colab (not recommended), they need to install the Transformers library on Python 3.8+ as described in https://huggingface.co/docs/transformers/installation (CPU-support only option if no GPU is locally available) and the Adapters library as described in https://github.com/adapter-hub/adapters.

Schedule

Monday, 02.06.
10:00-13:00	Transformer models I - Motivation - Overview of transformer architecture - Introduction into Hugging Face Transformers (Python library the course will work with) and important modules Transformer models II Step-by-step walkthrough of the fine-tuning pipeline using the example of hate speech detection: Data Model setup Training Evaluation Further considerations like model optimization and initialization for reproducibility
13:00-14:00	Lunch break
14:00-17:00	Practical exercise on fine-tuning transformer models in Python I - Basic application of the pipeline Practical exercise on fine-tuning transformer models in Python II - Advanced modifications of the pipeline
Tuesday, 03.06.
09:00-12:00	Adapters I - Short summary of day 1 - Motivation - Conceptual introduction of the adapter module Adapters II - Step-by-step walkthrough of how to integrate adapters with Hugging Face Transformers using the example of hate speech detection - Further considerations like types of adapters and combination of different adapters
12:00-13:00	Lunch break
13:00-16:00	Practical exercise on integrating adapters with transformer models in Python I - Basic application of the pipeline Practical exercise on integration adapters with transformer models in Python II - Advanced modifications of the pipeline
Wednesday, 04.06.
09:00-12:00	Adapters III - Short summary of days 1 and 2 - Extended overview of different types of adapters and their use cases Adapters IV - Discussion of interesting application examples of adapters for social science research - General Q&A for the participants	Literature: Examples of adapters for social science research: Falk, N., & Lapesa, G. (2023). Bridging Argument Quality and Deliberative Quality Annotations with Adapters. In Findings of the Association for Computational Linguistics: EACL 2023. 2469-2488. https://aclanthology.org/2023.findings-eacl.187 Ahnert, G., Pellert, M., Garcia, D., & Strohmaier, M. (2024). Extracting Affect Aggregates from Longitudinal Social Media Data with Temporal Adapters for Large Language Models. ArXiv preprint. https://arxiv.org/abs/2409.17990
12:00-13:00	Lunch break
13:00-16:00	Practical application to individual projects - Participants work on their own projects, applying the concepts learned to research questions and data examples of their interest. They may use data they bring themselves or example datasets that we provide. Individual Consultations Participants may discuss individual questions 1-1 with the course tutors and receive further guidance on how to use adapters in their own projects.

Scientific Coordination

Administrative Coordination

Adapters: Lightweight Machine Learning for Social Science Research

About the lecturer - Julia Romberg

About the lecturer - Julia Romberg

About the lecturer - Vigneshwaran Shankaran

About the lecturer - Vigneshwaran Shankaran

About the lecturer - Maximilian Maurer

About the lecturer - Maximilian Maurer

Course description

Target group

Learning objectives

Prerequisites

Schedule

Schedule

Recommended readings

Recommended readings