MEEDAV: A Synchronous Web Viewer for EEG, Eye-Tracking and Speech Data

Source: arXiv:2605.18296 · Published 2026-05-18 · By Jan Pijálek, Karel Vlk, Ondřej Bojar

TL;DR

This paper introduces MEEDAV, an open-source, web-based tool designed to synchronously visualize and explore multimodal psycholinguistic data streams—specifically EEG, eye-tracking, and audio recordings. Initially created to analyze the Eyetracked Multi-Modal Translation (EMMT) dataset (four-channel Muse 2 EEG, monocular eye-tracking, and audio), MEEDAV supports higher-density EEG data through a channel-agnostic pipeline. The system aligns modalities in time and integrates independent component analysis (ICA) for EEG denoising, offering interactive visualizations of simultaneous signals, fixation/saccade heatmaps, and cross-modal correlations within a single unified interface. MEEDAV enables filtering by participant and stimuli and supports streaming data from both local files and GitHub repositories.

MEEDAV’s main advance is facilitating intuitive, real-time multimodal data exploration via a lightweight browser interface using open-source Python libraries (Streamlit, Plotly). The tool solves the complex task of synchronized visualization of heterogeneous data types from cognitive neuroscience and translation studies, with modular backend design enabling generalizability beyond the original dataset. The authors demonstrate MEEDAV’s effectiveness on EMMT data and outline future extensions to support more file formats and improve frontend performance. Overall, MEEDAV exemplifies a practical, code-verified prototype supporting exploratory analysis of neurophysiological and behavioral data with multimodal integrations rarely combined in a single web viewer.

Key findings

• MEEDAV performs multimodal time alignment by resampling EEG, eye-tracking, and audio data to a common 256 Hz grid.
• ICA-based EEG denoising using FastICA removes artifacts in under 150 ms for a 30-second segment with 4 EEG channels by zeroing components with kurtosis > 3σ or peak-to-peak amplitude above 95th percentile.
• Interactive Plotly visualizations display synchronized EEG traces, audio waveforms, gaze-intensity time series, and event markers (saccades, fixations) mapped to trial basenames.
• Gaze intensity visualization converts 2D gaze coordinates to a combined horizontal/vertical delta magnitude aggregated in fixed windows, capturing semantics of reading and image comparison.
• Spatial kernel density estimate (KDE) heatmaps computed on fixation/saccade landing points generate smooth 100×100 resolution attention maps for screen coordinates (1280×1024).
• Cross-modal correlation computations (Pearson, Kendall, Spearman) operate on 10-ms sliding windows enabling data-driven exploration of relationships between EEG, eye-tracking, and audio signals.
• The modular backend supports both local file loading and GitHub streaming with interchangeable loader classes, facilitating flexible data access.
• MEEDAV adapts dynamically to arbitrary EEG channel counts beyond the original 4-channel Muse 2 data due to its channel-agnostic processing and plotting pipeline.

Threat model

Not a security-focused work; the adversary model is not applicable. MEEDAV is designed for exploratory research use with trusted, preprocessed cognitive neuroscience datasets.

Methodology — deep read

The authors designed MEEDAV for post-hoc exploratory visualization of synchronized EEG, eye-tracking, and audio data collected under a controlled psycholinguistic experiment with 43 participants reading, speaking, and translating English sentences into Czech. The threat model is implicit: the adversary is not relevant since the goal is research data exploration rather than security. MEEDAV assumes complete, time-synchronized, preprocessed data streams in CSV formats plus audio WAV files.

Data originated from the EMMT dataset, which contains 4-channel EEG signals recorded using a Muse 2 headband (channels RAW_TP9, RAW_AF7, RAW_AF8, RAW_TP10), monocular eye-tracking data including fixation and saccade events from EyeLink 1000 Plus, and raw microphone audio recordings segmented per trial into WAV files (16 kHz, mono). The dataset includes raw and preprocessed versions, where preprocessing involved time alignment and cleaning. Each trial consists of stimulus presentation segments identified by a shared basename comprising participant, stimulus, and order details (e.g., P03_S084_01_Read).

MEEDAV’s architecture separates into three layers: data loading/manipulation, preprocessing, and browser-based visualization. The loader accepts data from either a local directory or via GitHub REST API, exposing a uniform interface. The DataProcessor converts absolute timestamps to relative seconds, resamples all modalities to 256 Hz using linear interpolation, normalizes audio to [-1,1], and maps gaze coordinates to pixel units. Heavy filtering such as bandpass or ICA denoising is deferred to frontend-triggered actions.

The ICA denoising implementation uses FastICA on input EEG data matrix X (channels × time) to decompose into source matrix S and mixing matrix A (X = AS). Components with kurtosis magnitude exceeding 3 standard deviations or peak-to-peak amplitude above the 95th percentile are automatically zeroed to suppress artifacts. The cleaned signal is reconstructed as A times the filtered source estimates.

The Streamlit frontend presents a multi-panel Plotly figure with four EEG channels, audio waveform, gaze-intensity bars (encoding horizontal versus vertical eye movement magnitude), and event markers aligned on a shared timeline. Additional tabs show KDE heatmaps of fixation/saccade spatial distributions and a participant dashboard. The system supports filtering by participant and stimulus and toggling raw versus denoised EEG views.

Cross-modal correlation analyses compute Pearson, Kendall, or Spearman coefficients using sliding windows with a 10 ms grid, enabling hypothesis-generating exploration of temporal relationships between EEG, gaze, and audio signals. The system was implemented in Python, leveraging Pandas for data management and Librosa for audio processing. The tool was tested on the full EMMT dataset (43 participants, 20 probes, 32 stimuli each).

No formal statistical validation or adversarial robustness evaluation was reported. The implementation code is open-source, but the dataset (EMMT) has restricted access. The paper elucidates concrete steps with specific examples, e.g., processing of a trial with basename P03_S084_01_Read involving loading EEG/eye/audio files, resampling to 256 Hz, and rendering synchronized visualization.

Technical innovations

• Channel-agnostic multimodal data pipeline enabling integration of arbitrary-density EEG alongside eye-tracking and audio streams synchronized to 256 Hz resolution.
• Automated ICA-based EEG denoising triggered via frontend UI with artifact rejection heuristics based on kurtosis and amplitude thresholds, completing under 150 ms per trial segment.
• Conversion of 2D gaze coordinates into signed intensity time-series capturing semantically meaningful horizontal versus vertical eye movements reflecting reading and image inspection.
• Unified browser-based Plotly visualization combining EEG waveforms, audio amplitudes, gaze intensity, event markers, and spatial KDE heatmaps, all synchronized with interactive filtering and navigation.

Datasets

• EMMT (Eyetracked Multi-Modal Translation) — 43 participants, 20 probes with 32 stimuli each — publicly documented dataset with restricted access via authors.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2605.18296.

Fig 2: MEEDAV interface: synchronised EEG, audio, gaze-intensity bars, and event

Limitations

• Current implementation tightly coupled to the EMMT dataset’s specific file structure and naming conventions, limiting out-of-the-box applicability.
• Supports only CSV-like formats for EEG and eye-tracking data, forcing manual data conversion for other common neurophysiological file standards (e.g., FIF, EDF).
• ICA artifact rejection is automated and heuristic-based, lacking interactive manual correction which might improve precision as in comparable tools like EYE-EEG.
• Frontend performance limited by Plotly’s computational and memory demands, affecting scalability to longer recordings or larger EEG channel counts.
• No formal evaluation of robustness to timing misalignment or noisy real-world datasets beyond the curated EMMT corpus.
• No end-to-end statistical validation or comparison with other multimodal visualization tools presented.

Open questions / follow-ons

• How effectively can MEEDAV be adapted to other multimodal datasets with different formats, channels counts, and experimental designs?
• Can incremental ICA denoising with interactive manual artifact rejection improve EEG cleaning quality versus the current automated heuristic approach?
• What visualization and computational improvements result from migrating from Plotly to alternatives like Altair-Vega Lite for large-scale, high-density data?
• How do cross-modal temporal correlation metrics computed by MEEDAV correlate with ground-truth or clinical cognitive state annotations?

Why it matters for bot defense

While MEEDAV is not directly focused on bot defense or CAPTCHA technology, its approach to synchronized, multimodal data visualization offers a valuable example framework for explorations requiring integration of heterogeneous data signals such as user behavior tracking (e.g., eye movements) alongside system telemetry (analogous to EEG). Bot-defense engineers can draw inspiration from MEEDAV’s modular backend and unified timeline interface to integrate diverse sensor streams in real time for detailed behavioral analyses. The ICA denoising method showcases practical artifact removal techniques that could analogously serve noise-reduction in biometric signals used for authentication systems or anti-bot verification. Finally, MEEDAV’s open-source, browser-based design exemplifies how complex multimodal data exploration tools can be made lightweight and accessible, a useful consideration for deploying user-side defense telemetry interfaces.

Cite

@article{arxiv2605_18296,
  title={ MEEDAV: A Synchronous Web Viewer for EEG, Eye-Tracking and Speech Data },
  author={ Jan Pijálek and Karel Vlk and Ondřej Bojar },
  journal={arXiv preprint arXiv:2605.18296},
  year={ 2026 },
  url={https://arxiv.org/abs/2605.18296}
}

Read the full paper

• arXiv:2605.18296 (abstract)
• arXiv:2605.18296 (PDF)