Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Source: arXiv:2607.01232 · Published 2026-07-01 · By Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin et al.
TL;DR
This paper investigates how reinforcement learning (RL) post-training adapts pre-trained large language models (LLMs) at the granularity of individual transformer layers. Contrary to the common practice of updating all parameters uniformly during RL fine-tuning, the authors demonstrate that training only a single carefully chosen transformer layer can recover nearly all performance gains achieved by full-parameter RL training and sometimes even surpass it. This phenomenon holds consistently across multiple model sizes (1.5B to 8B parameters), model families (Qwen3, Qwen2.5), RL algorithms (GRPO, GiGPO, Dr. GRPO), and diverse task domains including mathematical reasoning, code generation, and agentic decision-making.
The paper introduces a metric called layer contribution, quantifying the fraction of full-parameter RL improvement attributed to training a single layer in isolation. Their experiments reveal that the bulk of RL adaptation is concentrated in a small subset of layers clustered in the middle of the transformer stack, with input and output layers consistently contributing less. This concentration is stable across datasets, tasks, models, and algorithms. Furthermore, leveraging this insight, layer-aware RL training strategies that prioritize these high-contribution middle layers outperform conventional full-parameter RL, and ensembles combining layer-specialized models yield further gains. This provides a new structural understanding of how RL reshapes LLMs and suggests more efficient fine-tuning approaches.
Key findings
- Training a single transformer layer can recover from 87% to 114% of the performance gains of full-parameter RL training, e.g., Layer 10 on Qwen3-1.7B achieves 114% C(k) (Table 2), surpassing full training gains.
- Layer contributions follow a consistent structure across models, with the highest contributions centralized in middle transformer layers (roughly 40%-60% depth), and low contributions near input/output layers (Figure 1a, Figure 2).
- Layer contribution rankings correlate strongly across math and code datasets (Spearman ρ=0.76 between NuminaMath-CoT and DeepScaleR; ρ=0.59 between math and code tasks, Figure 3).
- This structural pattern holds across model families (Qwen3 vs Qwen2.5), RL algorithms (GRPO, Dr. GRPO, GiGPO), model scales (1.5B to 8B), and task domains including agentic decision-making (Tables 3 and 4, Figures 4 and 5).
- Selective training of only high-contribution layers outperforms full training; e.g., on Qwen3-8B, training the top 10 layers yields 69.1% math accuracy vs 66.4% for full RL (+4.1% absolute)(Figure 1b).
- Ensembling models trained on complementary layers yields further accuracy boosts via majority voting beyond single-model full RL.
- Learning rate ablations show that layer contribution rankings remain stable across hyperparameter changes (Appendix A.7).
- Some layers exhibit negative contribution when trained alone (e.g., Layer 0 on Qwen3-8B, C(k) = -0.51), indicating isolated layer training may degrade performance.
Threat model
n/a (The paper studies RL adaptation patterns in large language models post-training rather than adversarial or security threats.)
Methodology — deep read
The study assumes a threat model of analyzing where reinforcement learning (RL) improvements are localized within transformer-based large language models (LLMs) during RL post-training, with no adversarial context involved.
Data comes from multiple domain-specific datasets: primarily NuminaMath-CoT (mathematics), with additional math datasets DeepScaleR and Skywork, a coding dataset DeepCoder, and the ALFWorld environment for agentic tasks. Model families include Qwen3 (1.7B, 4B, 8B parameters) and Qwen2.5 (1.5B, 3B, 7B parameters). Dataset splits and preprocessing follow established benchmarks for each domain.
The architecture under study is the standard transformer encoder-decoder with L layers (28 to 36 depending on model). RL post-training uses Group Relative Policy Optimization (GRPO) or its variants (Dr. GRPO, GiGPO), optimizing expected verifiable rewards on task-specific prompts. The novelty lies in the 'single-layer RL training framework': for each layer k, all parameters except that layer’s weights are frozen, and RL gradient updates are applied only to that layer. Gradients still backprop through the entire network but only a single layer’s parameters are updated per run.
Layer contribution C(k) is defined as the fraction of the full-parameter RL performance gain recovered by training only layer k: C(k) = (performance with only layer k trained - base performance) / (full-parameter RL performance - base performance). Values >1 indicate surpassing full RL gains.
Training details standardize hyperparameters across all runs: learning rates are tuned on full-parameter baseline and identical rates used for single-layer runs to ensure fair comparison. Batch sizes, KL penalties, clipping ranges, and epochs are consistent. Experiments run to convergence on matched training steps. Hardware specifics and random seed protocols are not detailed but presumably standard for large RL training.
For evaluation, in-domain and out-of-domain benchmarks measure accuracy or success rate across math, code, reasoning, and language tasks, averaged to get overall scores. Layer-wise contribution is measured on these held-out sets. Statistical correlations (Spearman rho) quantify ranking consistency across tasks and datasets. The study also performs learning rate ablations and partial layer scans under computational constraints.
A concrete example: On Qwen3-1.7B, each of the 28 layers is individually RL-updated (GRPO) using NuminaMath-CoT data with all other layers frozen. Resulting models are evaluated on multiple math benchmarks. Layer 10 alone achieves 114% of the full-parameter RL gains, outperforming the joint training baseline, while layers near input/output show much lower gains. This process is repeated across scale, model, algorithm, and task variations.
Overall, the method rigorously isolates each layer’s ability to absorb RL improvements by constraining parameter updates, quantifies its relative importance via layer contribution metric, and validates stability across multiple axes.
Technical innovations
- Introduction of the 'layer contribution' metric quantifying how much RL improvement a single transformer layer can capture in isolation during RL post-training.
- Systematic layer-wise RL training framework that independently fine-tunes each layer while freezing others, isolating layers’ capacity to absorb RL gains.
- Empirical demonstration that middle transformer layers consistently concentrate the majority of RL adaptation across models, algorithms, tasks, and datasets.
- Development of layer-aware RL training strategies guided by layer contribution to selectively train high-impact layers, resulting in better performance than full-parameter training.
Datasets
- NuminaMath-CoT — math reasoning tasks — public benchmark
- DeepScaleR — mathematics — public benchmark
- DeepCoder — code generation tasks — public benchmark
- ALFWorld — agentic multi-step decision tasks — public benchmark
- Skywork — mathematics — internal/public benchmark (cited Liu et al. 2025)
Baselines vs proposed
- Full RL training (GRPO) on Qwen3-1.7B: overall accuracy = 50.8%; best single layer (Layer 10) = 51.8% (114% contribution)
- Full RL training on Qwen3-4B: math accuracy = 63.7%; Layer 16 single-layer training = 64.3% (106% contribution)
- Full RL training on Qwen3-8B: math accuracy = 66.5%; Layer 16 single-layer = 67.1% (107% contribution)
- Full RL on Qwen2.5-Math-1.5B with Dr. GRPO: avg = 36.1%; Layer 14 single-layer = 36.3% (101% contribution)
- Full RL on Qwen2.5-1.5B-Instruct with GiGPO: overall success = 87.8%; Layer 14 single-layer = 89.1% (102% contribution)
- Layer-aware training on Qwen3-8B (top 10 layers): math accuracy = 69.1% vs full RL 66.4% (+4.1% absolute gain)
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.01232.
Fig 9: Per-layer weight change magnitude ∥∆θk∥2 on Qwen3-1.7B-Base. Blue: full-parameter training (all layers
Limitations
- The study focuses on post-training RL with verifiable rewards; it does not explore other RL formulations or unsupervised RL setups.
- Layer-wise training is computationally demanding, limiting some experiments to partial layer scans rather than exhaustive scanning.
- Potential interactions between layers during joint full-parameter training are not explored—some layers may learn less effectively due to joint interference.
- Learning rate for single-layer training is inherited from full-parameter tuning without extensive tuning per layer, which could affect layer contribution estimates.
- The impact of RL adaptation on downstream real-world deployment or robustness to adversarial inputs was not addressed.
- The study mainly tests models sized 1.5B to 8B parameters; extrapolation to larger scale LLMs with hundreds of billions of parameters remains uncertain.
Open questions / follow-ons
- What causes the middle transformer layers to concentrate the majority of RL adaptation in terms of representational or architectural properties?
- Can layer-aware RL training reduce computational and memory costs significantly in large-scale deployment scenarios?
- How do joint interactions among multiple layers during full-parameter RL affect adaptation dynamics compared to isolated single-layer training?
- Would the discovered layer contribution pattern hold for even larger models (e.g., 100B+ parameters) or different transformer variants?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, the paper’s findings suggest that reinforcement learning signals and fine-tuning may disproportionately affect certain internal transformer layers rather than uniformly throughout the network. This structural insight could inspire more efficient layer-wise defenses or model adaptation strategies when deploying RL-trained language models in CAPTCHA or bot-detection systems. Specifically, knowing that RL improvements concentrate in middle layers could inform targeted monitoring or robustness testing of those layers to detection evasion attempts or adversarial manipulations.
Moreover, layer-aware training may help optimize fine-tuning resources and improve model interpretability by isolating which layers carry task-specific RL adaptations — useful when deploying dual-use language models for bot defense. However, as the findings focus on RL post-training for general capabilities, direct application to CAPTCHA-specific adversarial RL scenarios would require further validation.
Cite
@article{arxiv2607_01232,
title={ Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training },
author={ Zijian Zhang and Rizhen Hu and Athanasios Glentis and Dawei Li and Chung-Yiu Yau and Hongzhou Lin and Mingyi Hong },
journal={arXiv preprint arXiv:2607.01232},
year={ 2026 },
url={https://arxiv.org/abs/2607.01232}
}