Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction via Dynamic Urban Congestion Integration

Source: arXiv:2606.28274 · Published 2026-06-26 · By Abdolazim Rezaei, Mehdi Sookhak, Mahboobeh Haghparast

TL;DR

This paper addresses the challenging problem of accurate network traffic prediction in dynamic urban cellular networks by proposing the Parameter-Efficient Hybrid Transformer (PEHT). Traditional methods struggle to incorporate complex urban mobility and congestion dynamics alongside high-dimensional spatio-temporal traffic data. PEHT innovates by integrating external urban mobility and congestion features through a multimodal fusion strategy into a Transformer-based architecture, while using Low-Rank Adaptation (LoRA) within the Transformer encoder to drastically reduce trainable parameters by over 90%. This approach effectively regularizes noisy data and leverages multi-source contextual cues for improved forecasting accuracy.

Extensive experiments on the Telecom Italia Milan dataset and five synthetic congestion scenarios demonstrate PEHT's strong performance gains over state-of-the-art baselines such as ST2T and MVSTGN. It lowers RMSE by 14.6% on SMS traffic and 11.9% on Internet traffic compared to top competitors. Ablations show that combining LoRA with multimodal fusion leads to consistent improvements in RMSE, MAE, and R2 metrics across diverse traffic congestion and mobility conditions. Overall, PEHT offers a parameter-efficient, scalable, and context-aware solution for urban network traffic forecasting.

Key findings

• PEHT achieves RMSE of 18.42 on SMS traffic versus 21.58 for ST2T baseline on Telecom Italia Milan dataset, a 14.6% improvement (Table I).
• Internet traffic RMSE improves from 138.65 (ST2T) to 122.10 (PEHT), an 11.9% reduction (Table I).
• LoRA reduces trainable parameters in Transformer encoder by over 90%, from ~19.1M to 69,952 for rank r=8 with input/output dimension 4,372.
• Ablation on 5 synthetic datasets shows the Full PEHT model consistently outperforms variants with only encoder fusion or only LoRA, providing best combined RMSE, MAE, and R2 results (Table II).
• LoRA alone achieves R2=0.9286 on the cell-edge congestion synthetic dataset, close to Full model’s 0.9203, highlighting strong regularization effect.
• Multimodal fusion of external urban mobility and congestion features into decoder improves forecasting without violating temporal causality during inference.
• PEHT maintains high predictive accuracy while being parameter efficient and computationally feasible for high-dimensional spatio-temporal network data.
• Training uses sequence length of 48 time steps with four attention heads, three encoder layers, two decoder layers, and dropout rate of 0.1.

Threat model

n/a - This paper focuses on improving network traffic prediction accuracy by integrating heterogeneous urban data via Transformers. It does not define or address an adversarial threat model but assumes trustworthy data inputs with causal temporal ordering and no future data leakage during inference.

Methodology — deep read

1. Threat Model & Assumptions: The paper assumes a standard predictive modeling threat domain focused on forecasting accuracy in dynamic urban cellular networks. The adversary is not explicitly defined since the work targets traffic prediction enhancement rather than adversarial robustness. The model leverages historical and external data sources under assumptions of causal temporal flow without future data leakage.

2. Data: The core dataset is the Telecom Italia Milan dataset, which contains aggregated network traffic per grid square including SMS, call, and Internet usage over time. Five synthetic datasets simulating diverse congestion and mobility scenarios were generated via CARLA simulator for ablation studies. The synthetic data isolate conditions like base-station congestion, cell-edge congestion, pedestrian-only and vehicular-only mobility, and regular mobility with 70/20/10 train/val/test splits.

3. Architecture / Algorithm: PEHT consists of a customized Transformer Encoder-Decoder with LoRA applied to the encoder. Input features are split into primary network communication features and secondary urban mobility/congestion features. The encoder processes the primary features with LoRA low-rank adaptation to reduce trainable parameters. Multimodal fusion concatenates the encoder output with external mobility features before passing to the decoder. Positional encoding injects temporal order. LoRA approximates weight updates by two low-rank matrices (A and B) during initial training rather than just fine-tuning. The decoder predicts future traffic autoregressively with masked self-attention to maintain causality.

4. Training Regime: Training used AdamW optimizer with learning rate 5×10^-4, weight decay 1×10^-4, sequence length 48, hidden dimension 128, four attention heads, three encoder layers, two decoder layers. Dropout rate 0.1 was employed. LoRA rank set to r=8 with scaling factor α=16 balancing parameter efficiency and accuracy. Models trained for 100 epochs with early stopping on validation loss. Training executed on NVIDIA Jetson AGX Orin development kit.

5. Evaluation Protocol: Metrics include RMSE, MAE, and coefficient of determination R2. Baselines compared include state-of-the-art Transformer and GNN models such as HGCRN, ST-DenseNet, STCNet, StTran, MVSTGN, ST2T. Ablation studies compare encoder fusion only, LoRA only, and the full model on Telecom Italia and synthetic datasets.

6. Reproducibility: The implementation is available on GitHub. Exact code and dataset splits used are not detailed, but the Telecom Italia dataset is publicly referenced. Synthetic data specifics are described but not published. The use of a standard simulator (CARLA) suggests reproducibility with domain knowledge.

Example end-to-end: For a given 48-step historical sequence for a virtual base station, primary network traffic and mobility features are embedded and encoded by the Transformer encoder with LoRA to efficiently capture dependencies while reducing parameter count. The output contextual embedding is fused with supplementary urban mobility and congestion features representing real-time contextual data. The decoder autoregressively predicts future traffic considering encoded context and extra fused features, producing an accurate forecast evaluated on held-out test data via RMSE and R2 metrics.

Technical innovations

• Introduction of PEHT integrating external urban mobility and congestion features using a multimodal fusion strategy directly into Transformer decoder to enhance traffic prediction.
• Application of Low-Rank Adaptation (LoRA) within the Transformer encoder at training time, not just fine-tuning, to drastically reduce trainable parameters by over 90% in high-dimensional spatio-temporal network traffic data.
• Design of grid-to-virtual-cell greedy clustering algorithm balancing temporal correlation, spatial proximity, and load to aggregate sparse traffic grids into virtual base stations for more robust input signals.
• Strict temporal causality maintained during multimodal fusion and decoder prediction by only using historical or autoregressive forecasted external features, avoiding future data leakage.

Datasets

• Telecom Italia Milan — ~several months of urban cellular traffic aggregated per grid square — public dataset from Telecom Italia Big Data Challenge
• Synthetic congestion and mobility scenarios — gen. with CARLA simulator (5 datasets for ablation): C Congested, E Congested, w/o Ped, w/o V, Regular — private/generated

Baselines vs proposed

• HGCRN baseline RMSE on SMS: 75.20 vs PEHT 18.42
• ST-DenseNet RMSE on Internet: 196.37 vs PEHT 122.10
• MVSTGN RMSE on SMS: 30.94 vs PEHT 18.42
• ST2T RMSE on SMS: 21.58 vs PEHT 18.42 (14.6% improvement)
• Encoder Fusion variant RMSE in synthetic 'C Congested': 13.12 vs Full Model 9.24
• LoRA variant RMSE in synthetic 'E Congested': 10.01 vs Full Model 8.48

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28274.

Fig 1: PEHT: The proposed framework initially applies LoRA on Encoder

Fig 2: NVIDIA Jetson AGX Orin Development Kit and supporting equipment

Fig 3: Ablation comparison of the Full Model against variants excluding

Limitations

• The synthetic datasets used for ablation are generated via simulation and may not fully capture real-world urban complexities or rare events.
• No explicit adversarial evaluation or robustness tests against unexpected traffic pattern shifts or sensor failures are reported.
• The scalability and inference latency of PEHT in real-time large-scale deployments remain unquantified in the paper.
• The integration of external mobility data requires reliable and timely data sources, which might not be available in all urban settings.
• Exact reproducibility is limited by lack of full public release of synthetic datasets and trained weights.

Open questions / follow-ons

• How does PEHT perform under sudden urban incidents or anomalies causing non-recurring traffic changes?
• Can the LoRA-based parameter efficiency enable real-time edge deployment or on-device forecasting at scale in cellular networks?
• What are the impacts of noisy or missing external mobility/congestion data on PEHT’s prediction robustness and how can this be mitigated?
• Could the multimodal fusion strategy be generalized to integrate other relevant urban sensing data, such as environmental or event data, to further improve forecasts?

Why it matters for bot defense

For bot-defense and CAPTCHA systems relying on network traffic or user behavior modeling to detect anomalies or automate resource allocation, the PEHT paper offers insights on efficiently incorporating external urban context data into predictive models without exploding parameter counts. Its use of Low-Rank Adaptation within Transformer encoders can inspire parameter-efficient modeling where high-dimensional multimodal inputs are involved. The strict maintenance of temporal causality during data fusion is particularly relevant for systems requiring real-time inference without leakage of future ground truth. Furthermore, the virtual base station clustering approach to handle sparse spatial data smoothing can be adapted in bot defense scenarios involving spatial- or network-location-based features. Overall, PEHT presents an approach that balances accuracy, scalability, and efficient fusion of heterogeneous contextual signals which can inform design choices for CAPTCHA and automated defense algorithms handling dynamic network traffic or user activity patterns.

Cite

@article{arxiv2606_28274,
  title={ Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction via Dynamic Urban Congestion Integration },
  author={ Abdolazim Rezaei and Mehdi Sookhak and Mahboobeh Haghparast },
  journal={arXiv preprint arXiv:2606.28274},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28274}
}

Read the full paper

• arXiv:2606.28274 (abstract)
• arXiv:2606.28274 (PDF)