A Modular Workflow for Multimodal Reading Experiments
Source: arXiv:2608.05966 · Published 2026-08-06 · By Thomas Krämer, Thomas Kosch, Dagmar Kern, Daniel Hienert
TL;DR
This paper presents a novel modular web-based workflow designed to conduct real-time multimodal experiments on naturalistic online reading. The workflow integrates eye tracking, EEG, and interaction data (mouse, keyboard) synchronously via the Lab Streaming Layer (LSL), enabling precise alignment of gaze data to browser-based text down to the word and sentence levels. The system supports flexible adaptation to various sensors, tasks, and analysis goals, allowing metrics derived online from gaze data and post-hoc from EEG segments to be used immediately within the same lab session for participant feedback on selected text passages. The authors demonstrate the workflow’s utility with an experiment studying selective exposure to online news, using fixation duration and theta-band EEG power as criteria to select sentences for participant rating. The results illustrate how behavioral, neural, and linguistic metrics can be combined to investigate cognitive biases in ecologically valid contexts, bridging laboratory paradigms with real-world reading scenarios.
The workflow fills a methodological gap by enabling co-registered EEG and eye-tracking in naturalistic, self-paced online reading, with real-time gaze-text mapping and multimodal synchronization. Its modular design allows easy extension and reuse for other cognitive and affective reading research beyond this initial selective exposure use case. The implementation builds on state-of-the-art components for gaze mapping (EyeLiveMetrics), EEG artifact correction (MNE-Python + ICA), and linguistic feature extraction, all coordinated via LSL. This work advances the capability to empirically validate cognitive constructs and biases derived from physiological signals under realistic reading conditions, which are traditionally assessed only via self-report.
Key findings
- The pipeline synchronizes eye tracking, EEG, mouse, and keyboard input streams via Lab Streaming Layer into unified XDF files for multimodal analysis.
- EyeLiveMetrics achieves strong fixation-to-word mapping accuracy with Pearson correlations ≤ 0.96 against Tobii Pro Lab ground-truth fixation metrics.
- In a selective exposure study using a German news corpus of 192,685 articles, sentences were selected for feedback based on longest normalized fixation duration and theta band power increase during 500–1500 ms post-fixation.
- EEG preprocessing employs notch filtering, band-pass filtering (0.1–30 Hz), and ICA artifact removal using the Picard algorithm with automated ICLabel component classification.
- Fixation-locked EEG epochs are segmented relative to the first fixation onset of each word and bad epochs rejected via thresholding or dynamic criteria.
- Participants rated up to 60 sentences per session, sampled from sentences with either highest fixation duration or theta power increases, enabling integrated behavioral and neural feedback collection within a single lab session.
- Theta band EEG oscillations were targeted as neural correlates of memory retrieval and selective exposure processing during online reading.
- The workflow operates fully in web environments, enabling naturalistic, self-selected reading with real-time multimodal data processing and dynamic stimulus annotation.
Methodology — deep read
The authors implemented a modular experimental workflow for real-time integration of multimodal data streams in naturalistic online reading settings.
Threat Model & Assumptions: The workflow focuses on studying cognitive processes and biases during reading, assuming participants freely select reading material; no adversarial threat model is presented.
Data: In the use case, 192,685 German news articles published between April 2024 and April 2025 formed the corpus for realistic news search and reading tasks. Participants completed self-selected reading tasks on two topics (climate change and migration policy) with topics randomized. Eye-tracking data was recorded using Tobii Pro Spectrum at 300 Hz; EEG was recorded with OpenBCI Cyton+Daisy at 125 Hz with 16 channels plus references. Pre-experiment questionnaires collected attitudes and personality data. Eye tracker calibration and EEG impedance checks were performed before tasks.
Architecture / Algorithm: Eye tracking raw gaze points, pupil diameters, and validity flags were streamed via a customized Tobii Pro Connector App to Lab Streaming Layer (LSL). EEG signals were streamed simultaneously using OpenBCI GUI to LSL. Mouse and keyboard inputs were also streamed. LSL LabRecorder unified all streams into XDF files with synchronized timestamps. EyeLiveMetrics web browser plugin computed fixation and saccade metrics aligned to textual elements in real time. EEG data was preprocessed in MNE-Python with notch and band-pass filters, ICA using the Picard algorithm for artifact removal, and ICLabel for automatic component classification. EEG signals were segmented into epochs time-locked to word fixation onsets derived from gaze data. Fixation-related potentials and theta-band power metrics were extracted.
Training Regime: Not applicable, as this is a data collection/processing workflow rather than a ML training model. However, EEG preprocessing and artifact correction parameters such as filter cutoff frequencies, ICA component rejection, and epoch rejection thresholds are configurable.
Evaluation Protocol: EyeLiveMetrics fixation detection was validated vs Tobii Pro Lab ground truth with Pearson correlation ≤ 0.96 across fixation metrics. For selective exposure study, sentences were selected based on (1) longest fixation duration normalized by sentence length, (2) strongest theta band power increase in EEG between 500-1000 ms post-fixation, and (3) between 1000-1500 ms post-fixation. Selected sentences were rated by participants in the post-experiment phase. This allowed cross-validation of neural and gaze-derived signals against self-reported agreement.
Reproducibility: The system builds on publicly available tools like EyeLiveMetrics, MNE-Python, LSL, OpenBCI GUI, and pyxdf, with references to all software packages. However, codebases for custom integrations or full dataset releases are not explicitly stated as publicly available. The EEG and eye tracking devices are off-the-shelf hardware. The news corpus used is based on publicly published articles but not described as released alongside the paper.
End-to-end example: A participant completed pre-experiment questionnaires and sensor calibrations, then performed two news search and reading tasks via a web app. During reading, eye tracker and EEG streamed data to LSL and EyeLiveMetrics processed fixation metrics in real time, mapping gaze onto browser text at the word and sentence level. After tasks, fixation duration and theta power metrics from EEG were used to select text passages for participant feedback rating agreement or familiarity. The post-experiment questionnaire collected personality and demographics. The resulting multimodal data streams with stimulus annotations enable linking neural responses to precisely localized reading behavior and subjective judgments.
Technical innovations
- A fully integrated, web-based experimental workflow synchronizing eye tracking, EEG, and interaction data in real time via Lab Streaming Layer for naturalistic reading research.
- Online gaze-to-text mapping at word, sentence, and AOI levels enabling precise behavioral-neural alignment during self-paced online reading.
- Fixation-triggered epoching of EEG data combined with immediate fixation and EEG-derived metric computation to dynamically select stimuli for participant feedback within the same lab session.
- Modular pipeline design allowing easy adaptation to different sensor setups, cognitive tasks, and analytic goals, supporting ecological validity in cognitive neuroscience experiments.
Datasets
- German News Corpus — 192,685 articles — collected from April 14, 2024 to April 14, 2025, source not publicly released
Limitations
- Potential noise and bias in EEG fixation-locked analysis due to variability in eye tracker calibration, sampling frequency, display resolution, and dynamic web content.
- Artifact correction via ICA is less reliable for low-density mobile EEG setups as used, limiting EEG data quality.
- The impact of different hardware and text layout parameters on gaze-to-word assignment accuracy remains unquantified.
- No adversarial evaluation or robustness testing against deliberate manipulation or sensor failure was conducted.
- Data and code for the full workflow are not fully publicly available, limiting reproducibility.
- Results from the selective exposure use case pending detailed publication, currently only described in overview.
Open questions / follow-ons
- How do hardware variables (e.g., eye tracker type, sampling rate, display settings) quantitatively affect the accuracy of gaze-to-text mapping and subsequent EEG segmentation?
- Can regression-based EEG artifact correction methods that explicitly model ocular events improve fixation-related potential analysis in low-density EEG?
- What is the generalizability of the workflow to other naturalistic reading contexts involving multimedia stimuli beyond text, such as images or video?
- How do different linguistic features extracted online from text influence the cognitive-neural metrics and classification of reading behavior or biases?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this work illustrates a sophisticated approach to precisely synchronize and analyze multimodal biometric signals—eye tracking and EEG—in naturalistic online behavior. Although not directly related to bot detection, the pipeline’s integration of low-latency gaze mapping to dynamic web content offers insight into building systems that closely monitor user interactions at high resolution in real time. Such capabilities could inspire advanced human-bot interaction analyses or biometric CAPTCHA designs that incorporate cognitive and behavioral signals beyond traditional interaction metrics. The modular, extensible design signals potential for adapting similar sensor fusion workflows for continuous user verification under ecological user conditions, but practical application would require significant hardware and signal-processing resources not typical in large-scale deployments.
Cite
@article{arxiv2608_05966,
title={ A Modular Workflow for Multimodal Reading Experiments },
author={ Thomas Krämer and Thomas Kosch and Dagmar Kern and Daniel Hienert },
journal={arXiv preprint arXiv:2608.05966},
year={ 2026 },
url={https://arxiv.org/abs/2608.05966}
}