name: fine-tuning-with-trl

description: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.

version: 1.0.0

author: Orchestra Research

license: MIT

dependencies: [trl, transformers, datasets, peft, accelerate, torch]

metadata:

hermes:

tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace]

TRL - Transformer Reinforcement Learning

Quick start

TRL provides post-training methods for aligning language models with human preferences.

Installation:

`bash

pip install trl transformers datasets peft accelerate

`

Supervised Fine-Tuning (instruction tuning):

`python

from trl import SFTTrainer

trainer = SFTTrainer(

model="Qwen/Qwen2.5-0.5B",

train_dataset=dataset, # Prompt-completion pairs

)

trainer.train()

`

DPO (align with preferences):

`python

from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)

trainer = DPOTrainer(

model=model,

args=config,

train_dataset=preference_dataset, # chosen/rejected pairs

processing_class=tokenizer

)

trainer.train()

`

Common workflows

Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)

Complete pipeline from base model to human-aligned model.

Copy this checklist:

`

RLHF Training:

• - [ ] Step 1: Supervised fine-tuning (SFT)
• - [ ] Step 2: Train reward model
• - [ ] Step 3: PPO reinforcement learning
• - [ ] Step 4: Evaluate aligned model

`

Step 1: Supervised fine-tuning

Train base model on instruction-following data:

`python

from transformers import AutoModelForCausalLM, AutoTokenizer

from trl import SFTTrainer, SFTConfig

from datasets import load_dataset

Load model

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

Load instruction dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

Configure training

training_args = SFTConfig(

output_dir="Qwen2.5-0.5B-SFT",

per_device_train_batch_size=4,

num_train_epochs=1,

learning_rate=2e-5,

logging_steps=10,

save_strategy="epoch"

)

Train

trainer = SFTTrainer(

model=model,

args=training_args,

train_dataset=dataset,

tokenizer=tokenizer

)

trainer.train()

trainer.save_model()

`

Step 2: Train reward model

Train model to predict human preferences:

`python

from transformers import AutoModelForSequenceClassification

from trl import RewardTrainer, RewardConfig

Load SFT model as base

model = AutoModelForSequenceClassification.from_pretrained(

"Qwen2.5-0.5B-SFT",

num_labels=1 # Single reward score

)

tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")

Load preference data (chosen/rejected pairs)

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

Configure training

training_args = RewardConfig(

output_dir="Qwen2.5-0.5B-Reward",

per_device_train_batch_size=2,

num_train_epochs=1,

learning_rate=1e-5

)

Train reward model

trainer = RewardTrainer(

model=model,

args=training_args,

processing_class=tokenizer,

train_dataset=dataset

)

trainer.train()

trainer.save_model()

`

Step 3: PPO reinforcement learning

Optimize policy using reward model:

`bash

python -m trl.scripts.ppo \

--model_name_or_path Qwen2.5-0.5B-SFT \

--reward_model_path Qwen2.5-0.5B-Reward \

--dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \

--output_dir Qwen2.5-0.5B-PPO \

--learning_rate 3e-6 \

--per_device_train_batch_size 64 \

--total_episodes 10000

`

Step 4: Evaluate

`python

from transformers import pipeline

Load aligned model

generator = pipeline("text-generation", model="Qwen2.5-0.5B-PPO")

Test

prompt = "Explain quantum computing to a 10-year-old"

output = generator(prompt, max_length=200)[0]["generated_text"]

print(output)

`

Workflow 2: Simple preference alignment with DPO

Align model with preferences without reward model.

Copy this checklist:

`

DPO Training:

• - [ ] Step 1: Prepare preference dataset
• - [ ] Step 2: Configure DPO
• - [ ] Step 3: Train with DPOTrainer
• - [ ] Step 4: Evaluate alignment

`

Step 1: Prepare preference dataset

Dataset format:

`json

{

"prompt": "What is the capital of France?",

"chosen": "The capital of France is Paris.",

"rejected": "I don't know."

}

`

Load dataset:

`python

from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

Or load your own

dataset = load_dataset("json", data_files="preferences.json")

`

Step 2: Configure DPO

`python

from trl import DPOConfig

config = DPOConfig(

output_dir="Qwen2.5-0.5B-DPO",

per_device_train_batch_size=4,

num_train_epochs=1,

learning_rate=5e-7,

beta=0.1, # KL penalty strength

max_prompt_length=512,

max_length=1024,

logging_steps=10

)

`

Step 3: Train with DPOTrainer

`python

from transformers import AutoModelForCausalLM, AutoTokenizer

from trl import DPOTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

trainer = DPOTrainer(

model=model,

args=config,

train_dataset=dataset,

processing_class=tokenizer

)

trainer.train()

trainer.save_model()

`

CLI alternative:

`bash

trl dpo \

--model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \

--dataset_name argilla/Capybara-Preferences \

--output_dir Qwen2.5-0.5B-DPO \

--per_device_train_batch_size 4 \

--learning_rate 5e-7 \

--beta 0.1

`

Workflow 3: Memory-efficient online RL with GRPO

Train with reinforcement learning using minimal memory.

Copy this checklist:

`

GRPO Training:

• - [ ] Step 1: Define reward function
• - [ ] Step 2: Configure GRPO
• - [ ] Step 3: Train with GRPOTrainer

`

Step 1: Define reward function

`python

def reward_function(completions, **kwargs):

"""

Compute rewards for completions.

Args:

completions: List of generated texts

Returns:

List of reward scores (floats)

"""

rewards = []

for completion in completions:

# Example: reward based on length and unique words

score = len(completion.split()) # Favor longer responses

score += len(set(completion.lower().split())) # Reward unique words

rewards.append(score)

return rewards

`

Or use a reward model:

`python

from transformers import pipeline

reward_model = pipeline("text-classification", model="reward-model-path")

def reward_from_model(completions, prompts, **kwargs):

# Combine prompt + completion

full_texts = [p + c for p, c in zip(prompts, completions)]

# Get reward scores

results = reward_model(full_texts)

return [r["score"] for r in results]

`

Step 2: Configure GRPO

`python

from trl import GRPOConfig

config = GRPOConfig(

output_dir="Qwen2-GRPO",

per_device_train_batch_size=4,

num_train_epochs=1,

learning_rate=1e-5,

num_generations=4, # Generate 4 completions per prompt

max_new_tokens=128

)

`

Step 3: Train with GRPOTrainer

`python

from datasets import load_dataset

from trl import GRPOTrainer

Load prompt-only dataset

dataset = load_dataset("trl-lib/tldr", split="train")

trainer = GRPOTrainer(

model="Qwen/Qwen2-0.5B-Instruct",

reward_funcs=reward_function, # Your reward function

args=config,

train_dataset=dataset

)

trainer.train()

`

CLI:

`bash

trl grpo \

--model_name_or_path Qwen/Qwen2-0.5B-Instruct \

--dataset_name trl-lib/tldr \

--output_dir Qwen2-GRPO \

--num_generations 4

`

When to use vs alternatives

Use TRL when:

• - Need to align model with human preferences
• - Have preference data (chosen/rejected pairs)
• - Want to use reinforcement learning (PPO, GRPO)
• - Need reward model training
• - Doing RLHF (full pipeline)

Method selection:

• - SFT: Have prompt-completion pairs, want basic instruction following
• - DPO: Have preferences, want simple alignment (no reward model needed)
• - PPO: Have reward model, need maximum control over RL
• - GRPO: Memory-constrained, want online RL
• - Reward Model: Building RLHF pipeline, need to score generations

Use alternatives instead:

• - HuggingFace Trainer: Basic fine-tuning without RL
• - Axolotl: YAML-based training configuration
• - LitGPT: Educational, minimal fine-tuning
• - Unsloth: Fast LoRA training

Common issues

Issue: OOM during DPO training

Reduce batch size and sequence length:

`python

config = DPOConfig(

per_device_train_batch_size=1, # Reduce from 4

max_length=512, # Reduce from 1024

gradient_accumulation_steps=8 # Maintain effective batch

)

`

Or use gradient checkpointing:

`python

model.gradient_checkpointing_enable()

`

Issue: Poor alignment quality

Tune beta parameter:

`python

Higher beta = more conservative (stays closer to reference)

config = DPOConfig(beta=0.5) # Default 0.1

Lower beta = more aggressive alignment

config = DPOConfig(beta=0.01)

`

Issue: Reward model not learning

Check loss type and learning rate:

`python

config = RewardConfig(

learning_rate=1e-5, # Try different LR

num_train_epochs=3 # Train longer

)

`

Ensure preference dataset has clear winners:

`python

Verify dataset

print(dataset[0])

Should have clear chosen > rejected

`

Issue: PPO training unstable

Adjust KL coefficient:

`python

config = PPOConfig(

kl_coef=0.1, # Increase from 0.05

cliprange=0.1 # Reduce from 0.2

)

`

Advanced topics

SFT training guide: See references/sft-training.md for dataset formats, chat templates, packing strategies, and multi-GPU training.

DPO variants: See references/dpo-variants.md for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.

Reward modeling: See references/reward-modeling.md for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.

Online RL methods: See references/online-rl.md for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.

Hardware requirements

• - GPU: NVIDIA (CUDA required)
• - VRAM: Depends on model and method

- SFT 7B: 16GB (with LoRA)

- DPO 7B: 24GB (stores reference model)

- PPO 7B: 40GB (policy + reward model)

- GRPO 7B: 24GB (more memory efficient)

• - Multi-GPU: Supported via accelerate
• - Mixed precision: BF16 recommended (A100/H100)

Memory optimization:

• - Use LoRA/QLoRA for all methods
• - Enable gradient checkpointing
• - Use smaller batch sizes with gradient accumulation

Resources

• - Docs: https://huggingface.co/docs/trl/
• - GitHub: https://github.com/huggingface/trl
• - Papers:

- "Training language models to follow instructions with human feedback" (InstructGPT, 2022)

- "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)

- "Group Relative Policy Optimization" (GRPO, 2024)

• - Examples: https://github.com/huggingface/trl/tree/main/examples/scripts