Open curriculum · verified editionContribute on GitHub
LLM EngineerClub
SearchESClub Quiz
Lesson6 min1,079 words

09 — Alignment and Fine-Tuning: SFT, RLHF, RLAIF, and Constitutional AI

Conceptual topic. The goal is to understand where model behavior originates and when to adapt a model; we will not train a foundation model in this program.

SourceImprove this page

Conceptual topic. The goal is to understand where model behavior originates and when to adapt a model; we will not train a foundation model in this program.

A pre-trained model predicts the next token. To convert it into an assistant, its behavior is adapted: following instructions, using tools, rejecting certain requests, and producing useful formats. Alignment names the broad problem of orienting the system toward human intentions and values; fine-tuning is a family of techniques, not a guarantee of alignment.

1. The Mental Pipeline#

pretraining masivo
      ↓
instruction tuning / SFT
      ↓
preferencias (RLHF, RLAIF, DPO y variantes)
      ↓
evaluación de capacidad + seguridad
      ↓
controles del sistema desplegado

Each layer retains the failures of the previous one and may introduce others. An aligned model on average does not know your application's permissions nor does it replace runtime guardrails.

2. Supervised Fine-Tuning (SFT)#

SFT trains on instruction-response pairs considered desirable. It is useful for:

  • repeatable style and format;
  • domain terminology;
  • tool calling patterns;
  • narrow tasks with high-quality examples;
  • specialized small models.

The typical objective minimizes cross-entropy over the response tokens. Dataset quality dominates: contradictory examples or unnecessarily long responses teach exactly those vices.

A serious dataset includes positives, abstentions, limits, tool errors, and difficult cases. Not just happy paths.

3. Reward Model and RLHF#

Classic pipeline:

  1. For a given prompt, multiple responses are generated.
  2. Human annotators rank or compare them.
  3. A reward model is trained to predict preference.
  4. A policy is optimized against that reward, historically using PPO with a KL penalty to prevent it from drifting too far from the reference model.

Simplified intuition:

reward_total = reward_preferencia - beta * divergencia_KL(politica, referencia)

Risks:

  • the reward model inherits annotators' preferences and biases;
  • reward hacking: the policy exploits shortcuts that increase the score without fulfilling the intent;
  • sycophantic or overly cautious responses;
  • reduced diversity;
  • high operational cost and complexity.

RLHF aligns with a distribution of feedback, not with a universal definition of truth.

4. RLAIF and Constitutional AI#

RLAIF replaces or supplements part of the human feedback with evaluations generated by another model. It scales better and allows for explicit criteria, but it inherits the judge's errors and may amplify preferences from its own family.

Constitutional AI proposes written principles —a "constitution"— to critique and revise responses, generate preferences, and train a policy that is more helpful and harmless. Its conceptual value lies in making part of the criteria explicit and auditable.

It is not enough to draft a constitution in the system prompt: the method includes the generation of critiques/revisions and preference learning. In an app, a list of policies remains useful, but it is prompting/runtime control, not the training described in the paper.

5. DPO and direct preference optimization#

Direct Preference Optimization (DPO) learns directly from triplets (prompt, respuesta_preferida, respuesta_rechazada) without training and optimizing a reward model separately. It simplifies the pipeline and has become the reference for preference-based adaptation.

Subsequent variants adjust robustness, length, label noise, or online/offline nature. The engineering question is not to memorize acronyms: it is what data you have, what behavior you seek, what evaluation capability you possess, and how much drift you tolerate.

6. PEFT and LoRA#

Full fine-tuning updates all weights and requires significant memory. Parameter-Efficient Fine-Tuning trains only a fraction. LoRA freezes the base model and introduces low-rank matrices:

W_adaptada = W_base + escala * (A · B)

Advantages:

  • less memory and storage;
  • task-specific adapters on the same model;
  • accessible training with less hardware;
  • simple rollback by removing the adapter.

Limitations: it does not create reliable knowledge from changing data, may degrade capabilities, and still requires compatible licensing, evaluation, and serving of the correct combination.

7. Fine-tuning vs. prompting, RAG, and tools#

Need First option
private or changing information RAG / source query
fixed format structured outputs
exact business rule code / tool
repeated style and behavior prompt; SFT if scale and evaluation justify it
reduce cost with small model distillation/SFT with benchmark
teach stable vocabulary RAG or SFT depending on task and volume
correct specific facts source of truth, not weights

Fine-tuning is not a database. Updating a policy tomorrow should not require training.

8. Tool use and alignment#

Models learn tool formats and decision patterns during SFT/preference tuning. But the final contract is defined by your application. You must test:

  • when to call and when to respond directly;
  • valid arguments and correction after error;
  • obedience to negative results;
  • resistance to malicious tools or documents;
  • confirmation before effects;
  • stop and budget.

A better-trained model reduces errors; it does not make a tool execute_any_sql safe.

9. Adaptation Dataset#

Best practices:

  1. deduplicate and separate train/validation/test by source or entity to avoid leakage;
  2. keep a frozen test set that never enters training;
  3. document license, consent, provenance, and PII;
  4. balance task types and abstention cases;
  5. include negatives and corrected errors;
  6. measure annotation quality and disagreement;
  7. version the dataset, code, base model, tokenizer, and configuration.

Synthetic data helps expand coverage, but should not be the only truth: a model that trains and evaluates its own patterns can create a self-confirming bubble.

10. Evaluation Before and After#

Compare the adapted model with the base model on:

  • target task by segments;
  • general capabilities you do not want to lose;
  • safety and prompt injection;
  • calibration/abstention;
  • format and tool calling;
  • latency, throughput, and cost;
  • memorization and data leakage.

Perform ablation: improved prompt with base model, RAG without fine-tuning, adapter, and combination. If the prompt already solves the problem, training adds maintenance without value.

11. System Alignment, Not Just Model Alignment#

The deployed unit includes model, prompt, retrieval, tools, permissions, UI, monitoring, and humans. Strong controls:

  • authorization and least privilege outside the model;
  • confirmation of effects and idempotent transactions;
  • sources with provenance;
  • input/output validation;
  • observability and incident response;
  • appeal/correction channels;
  • continuous evaluation after every change.

A model cannot guarantee objectives it does not observe. If it does not receive identity, permissions, or real state, it cannot reliably respect them.

Common Errors#

  1. Fine-tuning to add documentation that changes every week.
  2. Training before building a baseline and an eval.
  3. Using test cases within synthetic training data.
  4. Confusing frequent rejection with complete safety.
  5. Choosing a technique based on popularity without considering data and serving.
  6. Failing to measure regressions outside the target task.
  7. Thinking that model alignment replaces system permissions.

For Further Reading#