SLORR: Simple and Efficient In-Training Low-Rank Regularization

Source: arXiv:2607.08754 · Published 2026-07-09 · By David González-Martínez, Shiwei Liu

TL;DR

This paper addresses the challenge of inducing low-rank structure in neural network weight matrices during training to improve compressibility without compromising model performance. Prior low-rank regularization methods require costly singular value decompositions (SVDs), modify model architectures, or rely on stateful cached quantities, limiting their practicality for modern large-scale models. In response, the authors propose SLORR (Simple and Efficient In-Training Low-Rank Regularization), an architecture-preserving, stateless framework that directly regularizes original weight matrices using GPU-efficient approximations of spectral quantities via polar factor estimation. Two variants are developed based on the Hoyer sparsity metric and the nuclear norm, with proven approximation guarantees.

Empirical results demonstrate that SLORR induces better compressibility than unregularized training while incurring low training overhead (under 8% for ImageNet-scale models and under 1% for large language model pretraining). Experiments on ImageNet-1K with ResNet and ViT architectures show SLORR matches or surpasses competitive baselines like Q3R and LoRITa in accuracy-compression tradeoffs. Large language model pretraining at 135M and 560M parameter scales further confirms that SLORR yields compressed models preserving perplexity and downstream task performance more robustly compared to unregularized baselines. SLORR thus provides a practical, scalable, and effective low-rank regularization method for compressible neural networks.

Key findings

• SLORR introduces less than 8% training overhead in ImageNet-1K classification training and under 1% overhead in LLM pretraining at 135M and 560M scales.
• SLORR achieves comparable or better post-training compression than recent baselines Q3R and LoRITa across ResNet-50, ViT-B/16, ViT-L/16, and ResNet-18 architectures.
• SLORR-Hoyer regularization significantly improves compressibility of LLaMA-like LLMs, preserving validation perplexity and zero-shot accuracy after SVD and SVD-LLM compression at multiple regularization strengths.
• Polar Express iterative approximation of the polar factor provides provable guarantees on the nuclear norm and Hoyer regularizer value and gradient approximations, enabling SVD-free efficient computation during training.
• SLORR operates directly on original weight matrices without architectural changes or cached states, simplifying integration and hyperparameter tuning compared to prior methods.
• In ViT models from ViT-T to ViT-H scale, SLORR’s training time and memory overhead scale modestly and remain well below the overhead of Q3R with frequent SVD refresh intervals.
• Under uniform and energy-based compression schemes, SLORR maintains higher accuracy than unregularized models at equivalent parameter/FLOPs retention ratios.
• SLORR-Hoyer promotes redistribution of singular values by shrinking small ones and enlarging large ones, effectively concentrating spectral energy and inducing low-rank compressibility as theoretically supported.

Methodology — deep read

1. Threat Model & Assumptions: The adversary context is not directly addressed as this work focuses on low-rank regularization rather than security. The assumption is that models trained without regularization have weight matrices not naturally amenable to strong low-rank compression without loss.

2. Data: For vision experiments, ImageNet-1K is used, including pretraining ResNet-18 for 110 epochs and continued training of ResNet-50, ViT-B/16, and ViT-L/16 for 30 epochs from pretrained checkpoints. Language model pretraining uses LLaMA-like transformers at 135M and 560M parameter scales trained on the FineWeb-Edu corpus with approximately 20 training tokens per parameter.

3. Architecture / Algorithm: SLORR directly regularizes original weight matrices without modifying architectures. Two variants, SLORR-Hoyer and SLORR-Nuc, are based on the squared Hoyer sparsity metric and nuclear norm, respectively. The regularizers involve singular values, but direct SVD is avoided by approximating the polar factor (UVᵀ from the thin SVD W=UΣVᵀ) using the iterative Polar Express method which is GPU-friendly and efficient. The approximated polar factor is used to compute forward regularizer values and gradients substituting exact SVD terms (Equation 1). Backpropagation is implemented as a custom PyTorch autograd function (Listing 1), with gradients corresponding to the Clarke generalized gradient.

4. Training Regime: Vision models use AdamW optimizer, batch sizes up to 1024, cosine learning rate decay, and standard augmentation. Regularization strength λ is swept over. For continued training, learning rates are low to maintain pretrained baseline performance. LLMs are pretrained from scratch with distributed data parallelism and AdamW, tuning λ for SLORR-Hoyer. Regularization is applied to all linear and convolutional layers except the first and last.

5. Evaluation Protocol: After training, models are compressed post-training through either uniform or energy-based rank truncation of weight matrices via SVD, and then evaluated on ImageNet-1K top-1 accuracy or LM perplexity and zero-shot downstream tasks. Baselines include unregularized training, Q3R, and LoRITa regularizers. Training overhead in wall-time and memory is measured. Multiple hyperparameter configurations are tested to establish accuracy-compression Pareto frontiers (Fig. 1). Ablations explore Polar Express iteration counts, refresh intervals (Q3R), and decoupled variants.

6. Reproducibility: The authors release code publicly on GitHub. Datasets like ImageNet-1K and FineWeb-Edu, and standard LLaMA-size architectures are public. Frozen weights or exact seeds are not specifically reported. Some hyperparameter details and ablations are provided in appendices. The method uses standard PyTorch and is easy to integrate into existing pipelines.

Concrete Example: For ResNet-50 continued training on ImageNet-1K, SLORR randomly initializes regularization strength λ and applies SLORR-Hoyer with 6 Polar Express iterations to approximate the polar factor UVᵀ at every training step. The weighted regularizer loss is added to task cross-entropy and optimized with AdamW for 30 epochs maintaining the pretrained baseline accuracy (via low learning rate). Post-training, layers are compressed by truncated SVD to various ranks set by energy or uniform criteria, then Top-1 accuracy is measured on validation. Compared to unregularized training and Q3R baselines, SLORR-trained models achieve better accuracy-retention curves with less than 5% extra training time and minimal memory increase (Table 2 and Fig. 1).

Technical innovations

• Introduction of SLORR: an in-training low-rank regularization framework that approximates spectral regularizers via efficient polar factor iterations instead of costly SVDs.
• Use of GPU-friendly Polar Express iterative method to approximate the polar factor UVᵀ enabling stateless, architecture-preserving, and zero-cached low-rank regularization.
• Formulation of two SLORR variants based on the squared Hoyer sparsity metric and nuclear norm, with provable approximation guarantees on regularizer values and gradients.
• Demonstration that SLORR-Hoyer promotes spectral energy concentration by shrinking smaller singular values and expanding larger ones without changing model norm in expectation.
• Design of a decoupled SLORR-Hoyer variant inspired by decoupled weight decay to improve training stability.

Datasets

• ImageNet-1K — 1.28M images — public benchmark for image classification
• FineWeb-Edu — corpus size not explicitly stated — used for LLaMA-like LLM pretraining

Baselines vs proposed

• Q3R: SLORR-Hoyer achieves similar or better post-training compressibility and accuracy on ImageNet-1K with less than half the training overhead (Q3R 1.656x vs SLORR 1.048x runtime, Table 2).
• LoRITa: SLORR outperforms LoRITa consistently in accuracy vs retention of parameters/FLOPs across ViT and ResNet architectures (Fig. 1).
• Unregularized baseline: SLORR-trained models maintain significantly higher accuracy/perplexity after compression at matched parameter ratios compared to unregularized models (Fig. 1 and 3).
• At 135M and 560M LLM scales, SLORR-Hoyer adds less than 1% training overhead and preserves better downstream zero-shot accuracy post SVD compression than unregularized models across multiple benchmarks (Fig. 3).

Limitations

• Approximation guarantees rely on assumptions about normalized singular value ranges and number of Polar Express iterations; practical error bounds may vary with floating point and implementation details.
• The method does not explicitly set or require a target rank; tuning regularization strength λ to obtain desired compressibility may require manual search.
• Evaluation focuses primarily on classification and standard LLM pretraining; performance on other modalities or architectures is not reported.
• No explicit adversarial robustness evaluation or tests under distribution shifts are provided.
• Decoupled variant is empirically explored only for SLORR-Hoyer and not extensively evaluated across all tasks.
• Large-scale SVD refresh techniques (e.g. Q3R) may achieve better results with fine-tuned refresh intervals not fully explored here.

Open questions / follow-ons

• How does SLORR perform on larger scale models beyond 560M parameters, especially in very large LLM training regimes?
• Can SLORR be integrated with activation-aware post-training compression methods to further improve compression-accuracy tradeoffs?
• What is the effect of SLORR on model robustness and calibration, especially when aggressively compressed?
• Could the Polar Express approximation be further optimized or combined with other spectral approximation techniques to reduce overhead further?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, SLORR represents a practical method to train neural networks that are compressible via low-rank approximations without expensive per-step SVD computations or architectural changes. This can help engineering teams optimize deployment of large models in latency- and memory-sensitive settings, such as CAPTCHA inference pipelines where model size impacts serving cost and response times. Furthermore, the low training overhead and stateless design ensure ease of incorporation into existing training workflows without introducing stateful caching or architectural complexity that could complicate maintenance or scaling. However, given that SLORR focuses on inducing low-rank structure rather than directly targeting robustness to adversarial inputs or bot-like query patterns, its application is primarily to improve model efficiency rather than security properties. Nonetheless, improved compressibility enables lighter-weight CAPTCHA or bot-detection models which can be updated or retrained more frequently. Incorporating SLORR could be a useful step within a broader model lifecycle aimed at balancing performance and resource footprint in bot-defense systems.

Cite

@article{arxiv2607_08754,
  title={ SLORR: Simple and Efficient In-Training Low-Rank Regularization },
  author={ David González-Martínez and Shiwei Liu },
  journal={arXiv preprint arXiv:2607.08754},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08754}
}

Read the full paper

• arXiv:2607.08754 (abstract)
• arXiv:2607.08754 (PDF)