Reward Design With Language Model Guidance

via “reward model training for reinforcement learning from human feedback (rlhf)”

Shanghai AI Lab's multilingual foundation model.

Unique: InternLM provides pre-trained reward models that can be fine-tuned on domain-specific preferences, reducing training time compared to training from scratch; integrates with XTuner for efficient fine-tuning

vs others: More accessible than building custom reward models from scratch; comparable to OpenAI's reward modeling approach but with full transparency and ability to customize for specific domains

via “reward model training with configurable loss functions”

Reinforcement learning from human feedback — SFT, DPO, PPO trainers for LLM alignment.

Unique: Supports multiple loss variants (Bradley-Terry, Elo, margin-based) with automatic hyperparameter suggestions based on dataset statistics, and includes built-in reward calibration utilities to estimate preference probabilities from scores

vs others: More flexible than monolithic reward models because it supports both regression and ranking objectives; better integrated with TRL's ecosystem than standalone reward modeling libraries because it shares data pipeline and chat template handling

via “reward-guided video generation steering”

text-to-video model by undefined. 40,686 downloads.

Unique: Embeds reward optimization directly into LoRA adapter weights rather than using explicit reward scoring during generation — this is a training-time optimization approach where the adapters learn to implicitly maximize entertainment value, contrasting with inference-time reward guidance methods that compute rewards during generation

vs others: Eliminates inference-time reward computation overhead (which would add 50-100% latency) by baking optimization into adapter weights, enabling fast generation while maintaining entertainment-focused steering that generic models lack

via “implicit reward model extraction from language model log-probabilities”

* ⏫ 06/2023: [Faster sorting algorithms discovered using deep reinforcement learning (AlphaDev)](https://www.nature.com/articles/s41586-023-06004-9)

Unique: Mathematically proves that language model log-probability ratios encode reward information, eliminating the need for a separate reward model while maintaining theoretical grounding in reward-based RL frameworks

vs others: More interpretable than black-box RLHF reward models because the reward function is directly derived from model probabilities; more efficient than training separate reward models because no additional training is required

via “reward model training from pairwise human preference comparisons”

* ⭐ 03/2022: [Multitask Prompted Training Enables Zero-Shot Task Generalization (T0)](https://arxiv.org/abs/2110.08207)

Unique: Uses a language model itself as the reward model rather than a separate scoring function, enabling the reward model to understand semantic nuances in instructions and outputs. The pairwise comparison approach is more data-efficient than absolute scoring and better captures relative preferences.

vs others: More semantically sophisticated than hand-crafted reward functions or simple metrics, and more data-efficient than absolute rating scales because pairwise comparisons provide stronger training signals for preference learning.

* ⏫ 03/2023: [Reward Design with Language Models](https://arxiv.org/abs/2303.00001)

Unique: RLPD integrates LM-based reward design as a first-class component with automatic validation against offline data, whereas prior work treats reward engineering as a separate manual step. This enables end-to-end specification of RL tasks from natural language to learned policies.

vs others: More flexible than hand-crafted rewards because LMs can express complex multi-objective specifications, and more reliable than pure inverse RL because rewards are validated against ground-truth offline trajectories before deployment