Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

Source: arXiv:2607.27203 · Published 2026-07-29 · By Perry Dong, Ron Polonsky, Dorsa Sadigh, Chelsea Fin

TL;DR

This paper challenges the dominant assumption in reinforcement learning (RL) that pretraining the Q-function alongside a pretrained policy improves sample efficiency and final performance during online RL fine-tuning. The authors systematically investigate whether Q-function pretraining on offline data helps when starting from a pretrained base policy. Surprisingly, results across six challenging continuous control tasks show that naive Q-function pretraining often provides little to no improvement, and in some cases even hinders downstream performance compared to randomly initializing the Q-function and learning it entirely online. The root cause is a fundamental mismatch between the Q-function learned from offline pretraining (which reflects the pretrained policy) and the Q-function that online RL ultimately learns for optimal fine-tuning. Offline value maximization during pretraining is insufficient to close this gap since the Q-function targets the wrong value landscape. To address this, the authors propose Initialization via Policy Ensemble (IPE), a simple yet effective approach that trains multiple diverse policies on the same offline dataset and uses their aggregated rollouts to pretrain the Q-function. IPE broadens Q-function coverage over alternative actions, enabling better learning of the optimal Q-function during RL fine-tuning. Empirically, IPE yields an average 1.26x (26%) improvement in fine-tuning success rates across the studied tasks compared to naive Q-function pretraining.

Key findings

• Naive Q-function pretraining on offline data provides little to no improvement over random initialization for online RL fine-tuning, and can sometimes degrade early performance (Fig 2).
• Despite pretraining to convergence with a near-optimal base policy, the offline-pretrained Q-function Qπbase differs substantially from the Q-function Qπ*RL learned during online RL fine-tuning (Fig 3).
• Offline value maximization techniques (including learning an edit policy or distilling max-Q policies) do not improve the mismatch between Qπbase and Qπ*RL or downstream fine-tuning performance (Fig 4 and 5).
• Actions sampled from a single pretrained policy display limited diversity and narrow coverage in action space, causing the Q-function to collapse towards the value function Vπbase and reducing its usefulness (Fig 6 and 8).
• IPE, which pretrains the Q-function on rollouts collected from an ensemble of diverse policies trained on the same offline data, significantly improves downstream online RL fine-tuning performance, achieving on average 26% higher success rates than naive Q-function pretraining (Fig 7).
• Increasing the number of policies in the ensemble N used for IPE consistently improves fine-tuning performance, with N=5 policies showing the best results within tested range (Fig 9).

Threat model

Implicit adversary is the challenge of efficiently adapting a pretrained policy through online RL fine-tuning where the Q-function guides action selection. The Q-function pretraining must support learning toward a different Q-function than the offline-distribution one. Adversaries cannot modify the offline data but the core difficulty arises from distributional mismatch and limited offline action diversity constraining Q-function estimates.

Methodology — deep read

The paper investigates online RL fine-tuning starting from a pretrained base policy πbase trained offline via supervised imitation learning on an offline dataset Doffline. The key question is whether pretraining the Q-function Qϕ on this offline data improves downstream online RL fine-tuning performance compared to randomly initializing Qϕ and learning it purely online.

Threat model & assumptions: The considered adversary is implicit - the challenge is learning optimal policies online given only offline-pretrained base policies and offline data. The Q-function guides policy improvement through TD learning and influences sample efficiency and final performance.

Data: Six continuous control manipulation tasks from Robomimic and OGBench benchmarks are used, with offline datasets comprising expert transitions {(s,a,r,s')} collected from pretrained policies. The offline dataset is used to pretrain both πbase and optionally Qϕ.

Architecture & algorithm: The authors use EXPO, an off-policy RL algorithm that maintains a base policy πbase and a lightweight 'edit' policy πedit that edits πbase actions within a bounded range. The Q-function Qϕ is trained with TD learning, using a target network Qϕ'. During inference, an on-the-fly (OTF) policy selects the highest Q-value among samples from πbase and πedit. This setup allows fine control of policy updates while analyzing Q-function pretraining effects.

Training regime: The pretrained base policy πbase is learned offline with supervised learning on Doffline. Q-function pretraining is done via offline TD learning on Doffline until convergence. Online fine-tuning proceeds by collecting new transitions in the environment, appending them to a replay buffer, and continuing to update πedit and Qϕ with EXPO using an update-to-data ratio of 1. Hyperparameters and seeds are consistent across experiments. Further hyperparameter details are in the paper's appendix.

Evaluation protocol: Performance is measured by success rates over environment steps during online RL fine-tuning across six tasks. Comparisons are made between (1) naive Q-function pretraining, (2) random Q-function initialization, and (3) the proposed IPE method. The authors analyze preference accuracy metrics to quantify alignment between Qπbase and the optimal Qπ*RL function by comparing action rankings between policies. Ablations include testing offline value maximization approaches and scaling the number of policies N in IPE. Statistical tests are not explicitly reported.

Reproducibility: Experimental protocols, datasets, and evaluation benchmarks are openly cited but full code and pretrained model releases are not specified explicitly. The methodology is sufficiently detailed for reproduction given access to Robomimic, OGBench, and the EXPO codebase.

Concrete example: Online fine-tuning a pretrained Robot arm policy for block manipulation begins with πbase and an offline-pretrained Q-function. The naive Q-function pretraining does not improve success rates over random Q-init. However, initializing Qϕ with rollouts from multiple distinct policies trained on the same dataset (IPE) leads to better Q-function coverage and improved success rates during online fine-tuning.

Overall, the empirical protocol systematically evaluates how Q-function pretraining interacts with pretrained policies for online RL fine-tuning, identifies failure modes, and proposes a practical method to boost fine-tuning performance.

Technical innovations

• Demonstration that naive Q-function pretraining does not improve and can degrade online RL fine-tuning performance starting from a pretrained policy, challenging conventional assumptions.
• Identification of a fundamental mismatch between the offline-pretrained Q-function Qπbase and the online fine-tuning optimal Q-function Qπ*RL, explaining the ineffectiveness of naive Q-pretraining.
• Proposal of Initialization via Policy Ensemble (IPE), which trains multiple diverse policies on the same offline data to collect richer rollouts for Q-function pretraining, thereby providing better action coverage and improving online fine-tuning.
• Empirical scaling analysis showing that increasing the number of policies in the ensemble enhances downstream RL fine-tuning success.

Datasets

• Robomimic manipulation benchmark — task-specific size, public
• OGBench manipulation benchmark — task-specific size, public
• Offline expert transitions Doffline used for pretraining — sourced from expert demonstrations collected through policies on Robomimic and OGBench tasks

Baselines vs proposed

• Naive Q-function pretraining: final success rate comparable to or worse than random Q-function initialization baseline across 6 tasks.
• Random Q-function initialization baseline: served as de facto standard for online RL fine-tuning, beating naive Q-pretraining in some tasks.
• IPE (Initialization via Policy Ensemble): average 26% improvement in final fine-tuning success rate over naive Q-function pretraining (Fig 7).
• Performance improves as number of policies N increases, with N=5 outperforming N=1 or 0 (Fig 9).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.27203.

Fig 1: IPE. Left: naively pretraining does not help RL fine-tuning performance as Qπbase ̸= Qπ∗

Fig 2: Pretraining the Q-function does not improve performance for downstream fine-tuning. Success

Fig 3: Preference of Qπbase and Qπ∗

Fig 4: Performance of online RL after offline value maximization. Performing offline value maximization

Fig 5: Preference score analysis of offline value maximization. The preference of actions after performing

Fig 6: Actions from a single policy vs multiple policies. Actions sampled from one pretrained policy

Fig 7: IPE. Through using multiple policies to collect diverse rollouts, IPE is able to better learn Qπ∗

Fig 9: Scaling the number of policies for IPE. Increasing the number of policies used to collect rollouts

Limitations

• The analysis and results focus exclusively on off-policy RL fine-tuning; generalization to on-policy methods remains untested.
• IPE requires additional data collection overhead since multiple policies must be trained and rolled out to populate the replay buffer, increasing pretraining cost compared to naive Q-pretraining.
• The study uses smaller scale imitation-trained policies rather than large-scale pretrained Vision-Language-Action models directly, which might affect transferability of insights.
• No adversarial or distribution shift evaluations are presented to test robustness of IPE or Q-pretraining methods.
• Statistical significance measures or confidence intervals are not reported, limiting certainty around some performance claims.

Open questions / follow-ons

• How do these findings regarding Q-function pretraining and the effectiveness of IPE generalize to on-policy RL methods or other classes of algorithms?
• Can the data collection overhead of IPE be reduced by more efficient ensemble methods, policy distillation, or synthetic data augmentation?
• How do these patterns extend to large-scale pretrained policies, such as vision-language-action models, in more complex real-world robotics environments?
• Can alternative approaches to bridging the Qπbase and Qπ*RL gap be designed that do not require multiple policy rollouts but still increase Q-function action-space coverage?

Why it matters for bot defense

For bot-defense or CAPTCHA systems relying on online fine-tuning or adaptive policy learning, these findings emphasize that pretraining value functions (Q-functions) naively from offline data may not meaningfully accelerate adaptation or performance improvements despite pretrained control policies. Instead, diversity in policy behaviors used to bootstrap value estimation is critical. Applying an approach akin to IPE could allow such systems to better shape value functions and policy behaviors during fine-tuning by leveraging multiple related policies or behavior patterns to cover a broader action space. This could improve learning stability and robustness against evolving attack strategies in environments where policies must continuously adapt based on limited supervised data and online experience. It also cautions against over-reliance on Q-function pretraining without considering how offline data distributions differ from the fine-tuning target distribution, which can cause performance plateaus or degradation.

Cite

@article{arxiv2607_27203,
  title={ Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning? },
  author={ Perry Dong and Ron Polonsky and Dorsa Sadigh and Chelsea Fin },
  journal={arXiv preprint arXiv:2607.27203},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.27203}
}

Read the full paper

• arXiv:2607.27203 (abstract)
• arXiv:2607.27203 (PDF)