LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

Source: arXiv:2607.29559 · Published 2026-07-31 · By Manith Adikari, Bei Peng, Samuele Vinanzi, Angelo Cangelosi

TL;DR

LEMUR addresses a key limitation in Multi-Objective Reinforcement Learning (MORL): the reliance on predefined, well-specified reward functions for each competing objective. Real-world tasks often have multiple conflicting goals such as speed vs efficiency or safety vs performance, yet handcrafting reward functions for each objective is challenging and leads to suboptimal policies. Existing preference-based RL (PbRL) methods learn from human feedback but typically assume a single scalar reward and do not handle multiple conflicting objectives. LEMUR introduces a novel framework that jointly learns multiple objective-specific reward models from the preferences of multiple human teachers and optimizes policies in the MORL setting to find balanced, Pareto optimal compromises guided by human feedback without requiring ground-truth rewards.

LEMUR operates in three stages: unsupervised pre-training to encourage exploratory behaviors, interactive reward learning from multiple human preference signals (one per objective), and multi-objective policy optimization with state-of-the-art MORL algorithms using vector reward relabeling to address non-stationarity. Evaluated on complex continuous control and robotic manipulation benchmarks with two to four objectives, LEMUR outperforms several baselines that collapse preferences into single rewards or use static offline rewards. It closely approaches oracle policies trained with true reward functions, is robust to label noise and varying querying budgets, and scales naturally to more objectives by learning separate reward models. Overall, LEMUR represents a significant advance in preference-based MORL, enabling principled alignment with human trade-offs in settings where explicit reward functions are inaccessible.

Key findings

• LEMUR outperforms all baselines (Naive, Utilitarian, MORAL, PbMORL, FPbRL) on MO-Lunarlander, MO-Hopper, MO-Cheetah, and MO-MetaWorld, most closely tracking oracle ground-truth reward policies on both objectives simultaneously (Fig. 2).
• LEMUR achieves the highest or comparable Hypervolume (HV) across all four benchmark environments, indicating policies that are high-performing and well-distributed on the Pareto front, with markedly lower sparsity (SPS) than PbMORL (Table 1).
• LEMUR’s learned reward models correlate strongly with ground-truth teacher reward orderings, with Spearman rank correlations up to 0.945 ± 0.002 and Trajectory Alignment Coefficients (TAC) up to 0.898 ± 0.041, outperforming PbMORL and FPbRL (Table 2).
• In ablation studies on MO-Cheetah, removing the shared replay buffer caused the most severe performance degradation; disabling vector reward relabeling moderately reduced performance; removal of unsupervised pre-training reduced final returns (Fig. 5a).
• LEMUR is robust to noisy teacher labels: performance remains nearly unaffected up to 10% label noise and degrades gracefully at 15% noise (Fig. 5b).
• LEMUR can learn effective policies with as few as 260 total preference queries per teacher and improves with larger query budgets (up to 5,200 queries) (Fig. 5c).
• LEMUR scales to more objectives: on MO-Cheetah with three and four conflicting teachers, LEMUR policies closely track oracle ground-truth rewards across all objectives without modification (Fig. 4).
• Weight-conditioned reward models outperform unconditioned ensembles of independent models in reward learning, converging to substantially higher returns (6,812 ± 39 and 4,404 ± 22 vs 4,556 ± 369 and 2,902 ± 245 on two objectives).

Threat model

LEMUR assumes an environment where the agent cannot directly observe ground-truth reward functions for multiple conflicting objectives and must infer them from preference feedback provided by multiple human teachers. The adversary in this context is the absence or inaccessibility of explicit reward signals, and possibly noisy or imperfect human feedback. The system does not consider adversarial manipulations or attacks on the preference data, and assumes honest, though possibly noisy, human annotators. The capabilities of the adversary do not include direct intervention in the agent’s internal learning process or access to oracle rewards.

Methodology — deep read

1. Threat model & assumptions: LEMUR assumes an agent interacting with an environment modeled as a Multi-Objective Markov Decision Process (MOMDP) with m conflicting objectives. The agent does not have access to ground-truth reward functions for these objectives. Instead, multiple human teachers provide preference feedback over pairs of trajectory segments per objective. The adversary is modeled as absence of perfect reward functions—the system must infer objectives from incomplete, noisy feedback. The assumption is that teachers express preferences through pairwise comparisons, possibly with some noise.

2. Data: Training data consists of trajectories collected via exploration and policy rollout. The dataset includes pairs of trajectory segments sampled uniformly from a shared replay buffer containing diverse experiences obtained through unsupervised pre-training maximizing state entropy. Human teachers (scripted for experiments) label pairs with preference values {0, 0.5, 1}, indicating which segment is preferred for each objective. Data splits and standardization are not detailed; evaluation uses five random seeds.

3. Architecture and algorithm: LEMUR uses m lightweight Multi-Layer Perceptrons (MLPs) parameterizing reward models ˆr_{ψj}(s,a) for each teacher/objective j. Each reward model outputs a full objective vector conditioned on a shared objective space; the scalar teacher-specific utility is computed as the dot product between the teacher’s preference anchor and the reward vector. Preference likelihood is modeled using the Bradley-Terry model with cross-entropy loss to train the reward models. For policy learning, LEMUR uses Multi-Objective Soft Actor-Critic (MO-SAC) with an evolutionary strategy. Policies are conditioned on scalarization weight vectors that adapt online via Pareto Simulated Annealing (PSA), enabling exploration of trade-offs. A shared replay buffer across policies enables data reuse.

4. Training regime: First, LEMUR performs unsupervised pre-training maximizing intrinsic motivation (state entropy estimated via k-nearest neighbor distances) for broad exploration. Next, an interactive loop alternates between querying teachers for new preference labels on sampled trajectory segment pairs, updating reward models via gradient descent on cross-entropy, and optimizing MORL policies via MO-SAC on the learned reward vector with scalarization weights. Training uses off-policy RL to maximize sample efficiency under limited feedback budgets. Experiments are run over 3900 queries per teacher by default, with multiple random seeds. Specifics on optimizer or hyperparameters (batch size, learning rate, epochs) are in appendices and not exhaustively detailed.

5. Evaluation protocol: LEMUR is evaluated on multi-objective continuous control (MO-Lunarlander, MO-Hopper, MO-Cheetah) and robotic manipulation (MO-MetaWorld Drawer-Close extended to two objectives) from MORL-Generalization benchmark. Scripted teachers generate ground-truth feedback for evaluation; reward functions are withheld from agents. Primary metrics include true objective returns compared to oracle ground-truth policies, Hypervolume and Sparsity of Pareto fronts, Spearman rank correlation and Trajectory Alignment Coefficient (TAC) for reward model alignment, and success rates on manipulation tasks. Baselines use matched query budgets and environment steps; ablations cover buffer sharing, vector reward relabeling, pre-training, noisy labels, query budget, and reward model architecture.

6. Reproducibility: Code, hyperparameters, and pseudocode are provided in appendices. The scripted teachers and benchmark environments are publicly available. The reward learning architecture and training procedures are detailed, enabling reproduction, though no frozen weights or exact seeds are stated.

Concrete example end-to-end (MO-Cheetah domain):

• The agent first performs unsupervised pre-training maximizing state entropy via intrinsic rewards to collect diverse behaviors.
• The replay buffer B accumulates transitions; pairs of trajectory segments are uniformly sampled from B.
• Two teachers provide pairwise preferences on the segments, reflecting conflicting objective preferences.
• Two reward models ˆr_{ψ1} and ˆr_{ψ2} are learned by minimizing cross-entropy loss from Bradley-Terry model on teacher labels.
• The learned reward models define vector rewards ˆrψ(s,a) used to relabel transitions on-the-fly during policy training.
• MO-SAC policies conditioned on scalarization weight vectors w optimize scalarized expected rewards w^T ˆrψ(s,a).
• Weights w adapt online via Pareto Simulated Annealing to explore trade-offs.
• The agent iteratively collects new trajectories, queries teachers for feedback, refines rewards, and updates policies.
• Performance is tracked via true objective returns and standard multi-objective metrics, showing LEMUR approaches oracle performance and outperforms baselines.

Technical innovations

• Joint learning framework that simultaneously learns multiple objective-specific reward models from separate human preference feedback streams and multi-objective policies via reinforcement learning.
• Weight-conditioned reward models where each teacher’s reward model predicts a full vector reward conditioned on shared objective space, enabling coherent scalarization per teacher and scalability to many objectives.
• Interactive online learning loop employing vector reward relabeling to address non-stationarity, recomputing rewards on-the-fly during training for sample-efficient off-policy multi-objective RL.
• Unsupervised pre-training via intrinsic motivation maximizing state entropy to generate diverse, informative trajectory data for efficient preference querying and robust reward learning.

Datasets

• MORL-Generalization Benchmark — high-dimensional continuous control and robotic manipulation tasks — public

Baselines vs proposed

• Naive (monolithic reward model on pooled feedback): underperforms, plateaus flat with suboptimal returns
• Utilitarian (single SAC optimizing mean of learned rewards): underperforms, flat suboptimal returns
• MORAL (AIRL per-teacher rewards, offline fixed): early improvement but mid-training declines, fails due to out-of-distribution issues
• PbMORL (weight-conditioned vector reward from pooled feedback): better than aggregation baselines but consistently trails LEMUR in returns and hypervolume
• FPbRL (fixed Generalized Gini welfare scalarization): fair but converges to single policy, fails on success rate and many objectives
• LEMUR: tracks oracle ground-truth returns closely, achieves highest hypervolume and lowest sparsity (Table 1), robust to noise and query budgets

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.29559.

Fig 1: Illustration of our framework LEMUR: (1) Unsupervised Pre-training for the MORL agent to explore and collect

Limitations

• Experiments rely on scripted teacher feedback rather than real human annotators; human study needed to validate usability and noise robustness in practice.
• Linear scalarization used for weighted combination of objectives may limit ability to recover policies in non-convex regions of the Pareto frontier.
• Evaluation environments are simulated benchmarks; real-world tasks with more complex or unknown objectives remain untested.
• No adversarial or strategic manipulation of preference feedback examined; system robustness to malicious or biased annotators unstudied.
• Relies on multiple distinct teachers providing preference feedback per objective; scalability and logistics for many objectives/humans in practice uncertain.

Open questions / follow-ons

• How does LEMUR perform with real human annotators, especially considering annotator disagreement, fatigue, and biases?
• Can non-linear scalarization or more expressive utility models enable learning policies in non-convex or more complex trade-off spaces?
• What active learning or query strategies could minimize human labeling effort while maximizing learning efficiency?
• How robust is LEMUR against adversarial, inconsistent, or malicious human feedback?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, LEMUR provides a robust framework to learn multi-objective policies from heterogeneous human feedback without requiring explicit, difficult-to-design scalar reward functions. In bot detection and CAPTCHA design where multiple competing objectives (e.g., security, usability, accessibility) must be balanced and reward design is challenging, LEMUR's interactive, multi-teacher preference learning could enable systems that better capture nuanced human priorities and trade-offs. The joint and online learning of multiple objective-specific rewards and policies ensures adaptability and alignment with evolving user and security needs, potentially reducing reliance on brittle heuristics. However, practitioners should note the current evaluation is on simulated benchmarks with scripted teachers; deploying in real-world bot-defense will require further validation with actual human annotators and exploration of non-linear trade-offs relevant to security tasks.

Cite

@article{arxiv2607_29559,
  title={ LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback },
  author={ Manith Adikari and Bei Peng and Samuele Vinanzi and Angelo Cangelosi },
  journal={arXiv preprint arXiv:2607.29559},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.29559}
}

Read the full paper

• arXiv:2607.29559 (abstract)
• arXiv:2607.29559 (PDF)