Multi-modal transformer for signal classification in nanopore blockade experiments

Source: arXiv:2607.20323 · Published 2026-07-22 · By Sandro Kuppel, Julian Hoßbach, Samuel Tovey, Christian Holm

TL;DR

This paper addresses the challenge of reliably classifying molecules from complex nanopore blockade signals, which are measurements of ionic current disruptions caused by analytes passing through nanometer-scale pores. Traditional approaches rely on static features like mean current and dwell time, which discard rich temporal and spectral correlations. The authors propose a novel multi-modal deep learning architecture based on transformers that jointly processes three complementary input representations: raw time-series current signals, wavelet-transformed images exposing spectral features, and a set of selected statistical descriptors (catch22 features). This fusion enables the model to capture a broader range of signal characteristics than previous single-modality models.

The multi-modal transformer significantly outperforms prior approaches on a large 42-peptide dataset, improving macro-averaged accuracy from around 81.7% (best prior wavelet model) to 92.6%, and improving the worst-performing class accuracy by nearly 20 percentage points. Attention analysis reveals that the time-series and wavelet modalities attend to distinct, complementary signal aspects, supporting the benefit of information fusion. Furthermore, the model transfers effectively to a smaller, 20-amino-acid dataset, achieving near-perfect accuracy when fine-tuned from the larger dataset. This suggests the learned features generalize across peptides, enabling robust and scalable molecular identification using nanopore sensors.

Key findings

• The multi-modal transformer achieves a macro-averaged accuracy of 92.6% and micro-averaged accuracy of 92.1% on the 42-peptide dataset, exceeding the best single-modality baseline (ResNet18 wavelet model) by over 10 percentage points (81.7% macro and 81.5% micro).
• Worst class accuracy improved from 58.7% (wavelet ResNet18) to 77.2% in the multi-modal model, critical for diagnostic reliability.
• Best-performing class reaches 100% accuracy with the multi-modal model, higher than 95.6% for ResNet18 wavelet.
• Attention rollout shows time-series input attends mainly to entry/exit phases and deep blockades, while the wavelet input attends to high-frequency bands throughout the event, indicating complementary information capture.
• Transfer learning from the 42-peptide dataset to the XR7 20-amino acid dataset leads to faster convergence and higher final accuracy (~99.0% macro accuracy), with the worst class >90%, versus 70% for training from scratch on XR7.
• Single-modality models show broad, uneven class accuracy distributions, while the multi-modal transformer concentrates accuracies near perfect, supporting improved robustness.
• The model uses cross-attention between modality-specific classification tokens to effectively fuse information without large computational overhead.

Threat model

n/a — The paper focuses on molecular classification accuracy from nanopore signals rather than defensive security mechanisms or adversarial threats.

Methodology — deep read

1. Threat model & assumptions: The work assumes nanopore blockade signals generated by peptides passing through an aerolysin biological nanopore. No explicit adversarial threat model is defined since the focus is signal classification accuracy. The system assumes the same pore type for training and transfer.

2. Data: Two main datasets are used. The 'peptide ladder' dataset contains 42 peptide classes from six amino acid sequences of varying lengths with 350,000 labeled blockade events exhibiting strong class imbalance and ~3-5% label noise. Splits are 70% train, 15% validation, 15% test. The 'XR7' dataset has 20 amino acid classes with 38,000 labeled events, also imbalanced, split similarly.

Each raw ionic current event is segmented and processed into three modalities: (a) raw time-series current signals segmented into equal length chunks, (b) wavelet transformed 2D images (using the hhhat mother wavelet with µ=5, converted to log-absolute grayscale images), and (c) static feature vectors derived from five selected catch22 time-series descriptors plus mean current, std deviation, and log dwell time.

3. Architecture: The multi-modal model is based on the Vision Transformer (ViT) framework extended with three input branches for each modality. The time-series and image inputs are tokenized into fixed-length segments and patches respectively, each with learnable positional embeddings and a classification token. The feature vector modality is projected directly into a classification token via an MLP. Cross-attention is applied once before entering the multi-branch transformer encoder to transfer information among classification tokens of different modalities, while the encoder alternates per-branch self-attention and cross-attention blocks N times. Finally, classification tokens are passed through separate MLP heads, whose outputs are combined for final peptide classification.

Novelty lies in the multi-branch transformer design enabling efficient fusion of time-series, image, and static features simultaneously, whereas prior methods used only single representations.

4. Training regime: Training involves two stages—(a) self-supervised masked autoencoding pretraining masking random tokens in time-series and wavelet inputs to encourage cross-modal correlation learning and (b) supervised fine-tuning on labeled peptide classes using cross-entropy loss with label smoothing (to account for label noise). Training hyperparameters such as epochs, batch size, optimizer type, learning rates, hardware specifics, and seeds are detailed in the appendix (not fully described here). Transfer learning experiments involve fine-tuning the model pretrained on the peptide ladder dataset to the smaller XR7 dataset.

5. Evaluation protocol: Metrics reported include macro-averaged accuracy (mean over per-class accuracies), micro-averaged accuracy (overall event accuracy), maximum and minimum per-class accuracies. Several baselines are evaluated on the same test splits: ResNet18 trained on wavelet images, MLP on catch22 features, and a transformer trained on raw time-series alone. Per-class accuracy distributions and attention rollout visualizations are analyzed to understand model behavior. Transfer learning performance compares models trained from scratch vs pretrained and fine-tuned on XR7.

6. Reproducibility: The paper mentions extensive methodological detail, but no explicit code or dataset release is discussed. The peptide ladder data comes from prior publications [30,17] and XR7 from [24], which may be publicly available. Exact frozen weights are not reported.

End-to-end example: For a single peptide blockade event, the raw current trace is segmented into tokens, a wavelet transform produces a corresponding image token sequence, and the feature vector is extracted. Each modality is processed in its branch, cross-attention fuses information across modalities, and the final classification token outputs a probability distribution across 42 peptide classes, effectively accounting for temporal, spectral, and static descriptors. Attention maps highlight event regions each modality uses for decision-making.

Technical innovations

• A multi-branch transformer architecture that jointly processes raw time-series, wavelet images, and static feature vectors with cross-attention to fuse complementary information efficiently.
• Use of cross-attention between modality-specific classification tokens prior to multi-branch transformer encoding to reduce computational cost while enabling inter-modality communication.
• Inclusion of classification tokens for each modality branch and combining their final MLP outputs to improve classification robustness and accuracy.
• Self-supervised masked autoencoder pretraining applied jointly across multiple modalities to encourage cross-modal feature learning from unlabeled nanopore signals, facilitating better downstream supervised training and transfer learning.

Datasets

• Peptide ladder dataset — 350,000 labeled events, 42 peptide classes — from prior nanopore peptide ladder experiments [30,17]
• XR7 dataset — 38,000 labeled events, 20 amino acid classes — from residual current events of amino acids linked to arginine heptapeptide carriers [24]

Baselines vs proposed

• ResNet18 on wavelet images: macro accuracy 81.7%, micro accuracy 81.5% vs multi-modal transformer macro 92.6%, micro 92.1%
• MLP on catch22 features: macro and micro accuracy 73.6% vs multi-modal 92.6% macro, 92.1% micro
• Time series transformer baseline: macro accuracy 77.0%, micro accuracy 79.6% vs multi-modal 92.6% macro, 92.1% micro

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.20323.

Fig 1: Experimental workflow and model architecture. (A) Illustration of the

Fig 7: Pre-training Framework. During pretraining, the model is trained to recon-

Fig 8: Extended attention analysis of the vision transformer and the time

Fig 4 (page 14).

Fig 5 (page 14).

Fig 6 (page 14).

Fig 7 (page 14).

Fig 8 (page 14).

Limitations

• All datasets were recorded with the same aerolysin nanopore; transfer across different nanopore types is untested.
• Label noise in the peptide ladder dataset (~3-5%) potentially caps the maximum achievable accuracy.
• 42-class peptide ladder dataset represents only a small fraction of clinical molecular diversity, limiting generalizability assessment.
• No explicit adversarial or robustness evaluation against noisy or manipulated signals.
• Transfer learning tested only on a related peptide/amino acid dataset; generalization to structurally different analytes or more diverse datasets remains unknown.

Open questions / follow-ons

• Can the multi-modal transformer architecture generalize to nanopore signals recorded with structurally different biological or solid-state pores?
• How well does the approach scale to much larger, more diverse molecular libraries with hundreds or thousands of analyte classes?
• What is the impact of adversarial perturbations or experimental noise on model robustness in real-world sensing environments?
• Could additional modalities, such as electro-optical measurements, further improve classification performance when fused with the current modalities?

Why it matters for bot defense

For bot-defense and CAPTCHA engineers, this work illustrates how multimodal data fusion using transformer architectures can significantly improve classification accuracy and robustness over single-representation models. Analogously, bot detection systems might benefit from jointly modeling multiple correlated data streams (e.g., behavioral time-series, interaction heatmaps, and static device fingerprints) within a unified, cross-attentive transformer framework. Attention analysis highlights that different representations can emphasize complementary features, suggesting that combining raw signals with transformed or engineered features could improve classifier resilience and accuracy. The transfer learning results could inspire pretrained baseline models for CAPTCHA or bot-detection domains, reducing labeled data needs when deploying across new attack variants.

However, direct application requires careful consideration of domain-specific differences. The nanopore domain benefits from known physical modalities (time, frequency, statistical descriptors) with established biological meaning. In contrast, bot defense signals and challenges may be more heterogeneous and adversarial. Still, the demonstrated multi-modal fusion, cross-attention mechanisms, and pretraining strategies represent valuable methodological ideas to explore. Critically, the improved worst-class accuracy underscores the importance of uniformly high accuracy across classes—a lesson applicable to CAPTCHA systems needing to minimize false negatives on rare attack vectors.

Cite

@article{arxiv2607_20323,
  title={ Multi-modal transformer for signal classification in nanopore blockade experiments },
  author={ Sandro Kuppel and Julian Hoßbach and Samuel Tovey and Christian Holm },
  journal={arXiv preprint arXiv:2607.20323},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.20323}
}

Read the full paper

• arXiv:2607.20323 (abstract)
• arXiv:2607.20323 (PDF)