Building a User Foundation Model for the Open Web

Source: arXiv:2607.28019 · Published 2026-07-30 · By Solal Vernier, Ivan Can Arisoy, Merwan Barlier, Blaž Škrlj

TL;DR

This paper addresses the challenge of building effective user foundation models (UFMs) for open-web real-time bidding (RTB) where user identity is fragmented and non-persistent, and browsing history is sparse or absent due to privacy and device fragmentation. Unlike prior industrial recommender foundation models trained on stable logged-in user accounts with dense histories, open-web RTB operates on brief, disjointed session sequences, limiting the usefulness of conventional feature engineering approaches that aggregate counters and recency buckets without leveraging sequential structure. The authors propose a self-supervised pre-training approach applying a Transformer encoder trained with masked language modeling (MLM) and a novel sequence-level contrastive objective on temporally-disjoint slices of user browsing histories represented as triplets of (publisher, advertiser, interaction). They fine-tune this encoder for click-through rate (CTR) prediction and integrate its pooled user representation into production CTR rankers, demonstrating statistically significant offline and online improvements over strong baselines.

A key novelty is the deployment of an LLM-in-the-loop neural architecture search over a curated set of reviewable, literature-anchored, code-level training pipeline edits (“lifters”), which systematically optimizes the model architecture and training strategy for this challenging domain. The optimized model improves Relative Information Gain (RIG) by up to +1.354% on the production CTR ranker and +1.197% on a bid win-rate prediction model, showing cross-task generalization. A 7-day live A/B test confirms meaningful gains in CTR (+2.13%) and reductions in effective cost per click, demonstrating feasibility under strict sub-millisecond RTB latency budgets and robustness to users without history. The results establish sequential user modeling with self-supervision as a viable, ranker-agnostic representation approach for open-web advertising contexts with sparse and fragmented user signals.

Key findings

• The NAS-optimized user foundation model (UFM-NAS) achieves +1.354% RIG over a strong production GDCN CTR ranker baseline (Table 3).
• UFM-NAS yields +0.99% RIG lift for users with history length=1 (history-absent regime) and +1.62% RIG lift for sequences with ≥2 events (history-present regime) (Table 4, Fig 3).
• The learned user embeddings improve a structurally different bid win-rate model by +1.197% RIG, demonstrating cross-task generalization (Table 5).
• LLM-in-the-loop neural architecture search reduces total pre-training loss (MLM + contrastive) by 25.2% relative and converges in 6 epochs vs 10 for baseline, with modest +2.6% wall-clock cost increase (Table 1, Fig 2).
• Offline embedding gains generalize across multiple CTR ranker architectures: +1.354% on GDCN and +0.894% on DCN2 (Table 5).
• Live 7-day, 50/50 user-level A/B test shows +2.13% CTR uplift and −1.13% eCPC reduction (80% CI excludes zero) on click-optimized traffic (Table 6).
• The deployed encoder runs within a sub-millisecond latency budget by serving embeddings from a dedicated data management platform and batching inference with width 8 (Section 5.3).
• Sequence-level contrastive learning on temporally-disjoint subsequences of browsing histories improves representation quality over masked token modeling alone (Table 2 ablation).

Threat model

The modeled threat is the highly fragmented and non-persistent identity environment of the open web, where users appear across multiple disconnected browsing sessions and devices without stable login IDs. Traffic is partly anonymized by user privacy opt-outs, resulting in portions of inference requests with no historical data. The adversary is therefore essentially the challenge of sparse and discontinuous user signals rather than an active attacker. The model must remain robust and provide stable embeddings both when histories are available and when they are completely absent, under strict sub-millisecond latency constraints. It is not designed to detect or mitigate malicious behavior, nor to link fragmented identities.

Methodology — deep read

The paper’s methodology centers on developing and deploying a user foundation model for open-web real-time bidding scenarios characterized by fragmented, non-persistent user identities and sparse browsing histories.

1. Threat Model & Assumptions: The adversary is implicitly the fragmented nature of open-web user data — users appear under multiple disconnected IDs and opt-out choices mean large portions of traffic lack any history. The model must produce useful embeddings both when there is rich history and when there is none, under strict sub-millisecond latency constraints.

2. Data: The authors use production-scale open-web browsing logs from Teads, represented as sequences of event triplets (publisher, advertiser, interaction type) ordered by timestamp. Histories are fragmented and short, with a max of 16 events; 7-day TTL is applied in storage. Data is split into an unlabeled pre-training corpus (user histories) and labeled fine-tuning corpora drawn forward-in-time with train/val/test splits for impression-level binary click-through labels.

3. Architecture: The encoder is a bidirectional Transformer encoder akin to BERT with 8 attention heads and embedding dimension 48 (final NAS-lifted config). Token input combines learned embeddings for publisher, advertiser, and event-type tokens, positional embeddings, and a continuous time embedding encoding elapsed time relative to the most recent event in the sequence slice per [13]. Input sequences are tokenized into 1+[3×N] tokens per N events, truncated to length 49 tokens.

Pre-training uses a joint loss: masked language modeling (MLM) with 30% masking rate to recover tokens, and a sequence-level contrastive loss (NT-Xent) aligning pooled [CLS] embeddings from two temporally-disjoint, chronological subsequences of the same user history (constructed by a random split point). This novel contrastive objective differs from past work by using temporal partitions as positive views, rather than random cropping or dropout-based augmentations.

4. Training Regime: Pre-training is performed with JAX/Flax/Optax on mini-batches of 1024 users for 6 epochs (NAS-lifted config), optimized with AdamW variants, layer norm pre-attention, GELU nonlinearity, and gradient clipping. Fine-tuning attaches a single dense sigmoid prediction head on the pooled embedding for per-impression CTR binary classification. Fine-tuning is supervised on click labels with standard binary cross entropy loss using user histories strictly before impression timestamps. The click head is discarded at inference; only the encoder embedding is used.

5. Evaluation Protocol: Offline evaluation uses Relative Information Gain (RIG) versus a base-rate predictor, AUC, and training time metrics. Multiple upstream models (GDCN, DCN2) and downstream tasks (CTR prediction, bid win-rate prediction) are evaluated to test generalization. Offline gains in RIG are analyzed by user history length buckets. A live 7-day, 50/50 user-level A/B test with CTR and cost efficiency metrics validates online impact under production serving constraints. Embeddings are integrated via lightweight feed-forward adapters before the CTR ranker’s gated cross network.

6. Optimization Pipeline: The model’s training architecture and hyperparameters are optimized via a two-stage neural architecture search using an LLM-in-the-loop to propose curated “lifters” (reviewable code-level edits anchored in literature) across 10 categories (embedding, optimization, temporal encoding, etc.). Candidates passing static checks are fully pre-trained, fine-tuned, and proxy evaluated. Improvements are committed incrementally, with a total of ~150 lifters in the catalog.

7. Reproducibility: The lifter-based search process is inspectable and replayable, producing a deterministic training pipeline configuration. Model source code and weights are not publicly released; data is proprietary. Exact seeds or multiple run averages are not reported, so variability is unknown.

A concrete example end-to-end: a user browsing history of length 10 events is randomly split into two temporal partitions of length 6 and 4 respectively, tokenized and embedded with the Transformer encoder, which is trained simultaneously to reconstruct masked tokens from each view and to align the pooled [CLS] embeddings of the two views in contrastive loss space. After pre-training, the encoder is fine-tuned on impression click labels using the earlier timeline segment only, producing an embedding fed to a CTR ranker via adapter layers. Offline RIG over baseline improves by ~1.35%, and the embedding runs in production with sub-millisecond latency using a microbatch size of 8. The entire architecture and training hyperparameters were optimized iteratively with the LLM-in-the-loop NAS pipeline.

Technical innovations

• Temporally-disjoint chronological partitioning of user browsing histories to create contrastive positive pairs for sequence-level contrastive learning, differing from prior random cropping or augmentation strategies.
• Representation of user histories as event triplets (publisher, advertiser, interaction type) with continuous time embedding capturing elapsed time within each subsequence.
• LLM-in-the-loop neural architecture search over a curated catalog of 150 reviewable code-level training pipeline edits (“lifters”) with literature grounding, enabling systematic optimization and reproducibility in an industrial setting.
• Serving the user foundation model separately in a low-latency data management platform, embedding sequences up to 16 events with 7-day TTL, and micro-batching encoder inference (width 8, max 1ms wait) to meet sub-millisecond production RTB latency requirements.

Datasets

• Open-web browsing histories — proprietary, production-scale dataset from Teads — unlabeled for pre-training; labeled impression-level CTR datasets derived from the same stream with forward-in-time splits for fine-tuning and evaluation.

Baselines vs proposed

• GDCN CTR ranker baseline: RIG = reference; with UFM-Base embedding: +1.065% RIG; with UFM-NAS embedding: +1.354% RIG (Table 3).
• For users with history length=1: UFM-NAS vs GDCN +0.99% RIG; for length≥2: +1.62% RIG (Table 4).
• DCN2 CTR ranker baseline vs with UFM-NAS embedding: +0.894% RIG (Table 5).
• Bid win-rate prediction model baseline vs with UFM-NAS embedding (click fine-tuned): +1.197% RIG (Table 5).
• NAS pre-training total loss: Baseline 2.5566 vs NAS-lifted 1.9113 (−25.2%); training time +2.6% (Table 1).
• NAS fine-tuning test set RIG: Baseline 0.0510 vs NAS-lifted 0.0547 (+7.3%) (Table 2).
• Live A/B test CTR uplift: +2.13% (80% CI [1.55%, 3.17%]); eCPC reduction: −1.13% (CI excludes zero) (Table 6).

Limitations

• Single-seed training runs; results reported as point estimates without variability or statistical significance for offline metrics.
• No controlled comparison of LLM-in-the-loop NAS vs alternative search methods; unclear contribution of catalog curation vs LLM proposer.
• Pre-training and fine-tuning datasets are proprietary with undisclosed size and exact composition, limiting external reproducibility.
• No adversarial evaluation or robustness testing against manipulated user sequences or adversary tampering.
• No evaluation reported for fine-tuning the model on tasks beyond click prediction, e.g., win-rate, limiting understanding of downstream task adaptation.
• Limited token vocabulary and fixed truncation length (max 16 events) may constrain long-term sequence modeling capacity.

Open questions / follow-ons

• Can the proposed contrastive sequential modeling approach be generalized or improved by incorporating longer-range dependencies beyond the current 16-event limit?
• How would fine-tuning the UFM encoder on non-CTR downstream tasks (e.g., bid win-rate or conversion prediction) affect embedding transferability and final performance?
• What is the relative contribution of different categories of lifters in the LLM-in-the-loop NAS to final model improvements, and how does LLM-driven search compare to random or Bayesian search over the same catalog?
• How robust is the model to adversarial user behavior or noisy fragmented histories, and can adversarial training improve resilience?

Why it matters for bot defense

This study demonstrates a scalable approach to learning sequential user behavioral representations in environments characterized by sparse, fragmented, and non-persistent user identities—conditions common in open-web RTB systems. For practitioners in bot defense and CAPTCHA applications, the methodology highlights the feasibility of extracting meaningful user embeddings even when historical signals are limited or absent, by leveraging self-supervised masked token and temporal contrastive objectives. The use of temporally-disjoint partitions as positive pairs offers a novel way to exploit sequential coherence despite fragmented sessions. Furthermore, the LLM-in-the-loop architecture and training pipeline optimization illustrate an effective strategy for tuning complex, industrial models with large hyperparameter spaces in a reviewable, reproducible manner.

Bot-defense engineers could adopt similar user foundation models to generate robust user embeddings that enhance detection features or behavioral anomaly signals, particularly in contexts where user sessions are short or user identifiers are ephemeral. The operational deployment insights—separating embedding inference into a dedicated low-latency service and integrating embeddings into downstream models while meeting stringent latency budgets—can inform CAPTCHA latency and embedding integration constraints. Lastly, the demonstrated generalization of user embeddings across prediction tasks suggests potential utility in multi-task user reputation or behavior scoring frameworks foundational for adaptive bot defenses.

Cite

@article{arxiv2607_28019,
  title={ Building a User Foundation Model for the Open Web },
  author={ Solal Vernier and Ivan Can Arisoy and Merwan Barlier and Blaž Škrlj },
  journal={arXiv preprint arXiv:2607.28019},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28019}
}

Read the full paper

• arXiv:2607.28019 (abstract)
• arXiv:2607.28019 (PDF)