Open curriculum · verified editionContribute on GitHub
LLM EngineerClub
SearchESClub Quiz

LEARNING PATH

Professional LLM Engineering path

You do not need to finish everything to get value. Start with the assessment, repair weak competencies, and retain evidence from each milestone.

Take the assessmentView academic plan
01

MODULE 1 · 0.5 ECTS

LLM Foundations and APIs

15–18 hours

Transformers, tokenization, generation, current models, and multi-provider clients.

Exit evidence

A multi-provider client with observable tokens, latency, errors, and metadata.

Prerequisites

None

Contents12 resources
  1. Module I — LLMs and APIs Fundamentals (0.5 ECTS)Module map · 3 min
  2. Transformer Architecture: Encoder, Decoder, and Multi-Head AttentionLesson · 10 min
  3. Tokenization: BPE, WordPiece, and Model ComparisonLesson · 7 min
  4. Generation parameters: temperature, top_p, top_k, and penaltiesLesson · 7 min
  5. Current Landscape: GPT-5.6, Claude 5, Gemini 3.x, Llama 4, and MistralLesson · 8 min
  6. Generative AI Services on AWS: Bedrock, SageMaker JumpStart, and Amazon QLesson · 6 min
  7. Lab 01 — First OpenAI Call with the Responses APILab · 20 min
  8. Lab 02 — First Call to the Anthropic API (Claude)Lab · 20 min
  9. Lab 03 — Comparative Tokenization with tiktokenLab · 20 min
  10. Lab 04 — Sweeping temperature and top_p over the same promptLab · 20 min
  11. Lab 05 — Multi-model inference with the Amazon Bedrock Converse APILab · 20 min
  12. Exercises — Module IExercises · 3 min
02

MODULE 2 · 1.5 ECTS

Advanced Prompt Engineering

38–45 hours

Prompts as software: tools, structured outputs, evaluation, versioning, and mitigation.

Exit evidence

A prompt-evaluation pipeline with a dataset, A/B tests, and regression gates in CI.

Prerequisites

Foundations

Contents16 resources
  1. Module II — Advanced Prompt Engineering (1.5 ECTS)Module map · 3 min
  2. 01 — Base techniques: zero-shot, few-shot, chain-of-thought, and self-consistencyLesson · 8 min
  3. 02 — System Prompts, Roles, and Context DelimitationLesson · 6 min
  4. 03 — Function calling / tool use in OpenAI and AnthropicLesson · 6 min
  5. 04 — Structured outputs: JSON mode, response_format, Pydantic and InstructorLesson · 6 min
  6. 05 — Prompt Evaluation: Test Datasets, LLM-as-a-Judge, and A/B TestingLesson · 7 min
  7. 06 — Prompt Versioning and ManagementLesson · 5 min
  8. 07 — Biases and hallucinations: detection, measurement, and mitigationLesson · 6 min
  9. Lab 01 — Compare zero-shot and few-shot on the same casesLab · 20 min
  10. Lab 02 — Private reasoning and self-consistency with majority votingLab · 20 min
  11. Lab 03 — Complete function calling loop with the OpenAI Responses APILab · 20 min
  12. Lab 04 — Complete tool use loop with the Anthropic Messages APILab · 20 min
  13. Lab 05 — Native Structured Outputs and Instructor with PydanticLab · 20 min
  14. Lab 06 — Reproducible prompt evaluation pipeline on a datasetLab · 20 min
  15. Lab 07 — Paired statistical comparison of two promptsLab · 20 min
  16. Exercises — Module IIExercises · 3 min
03

MODULE 3 · 2 ECTS

RAG Systems and Evaluation

50–60 hours

Ingestion, embeddings, vector stores, chunking, reranking, grounding, and RAGAS.

Exit evidence

A RAG application with citations, abstention, and automated evaluation.

Prerequisites

Foundations · Prompt systems

Contents23 resources
  1. Module 3 — RAG Systems and Evaluation (2 ECTS · 55 h)Module map · 3 min
  2. 01 — RAG Architecture: Ingestion, Embedding, Indexing, and RetrievalLesson · 6 min
  3. 02 — Embeddings and Vector DatabasesLesson · 6 min
  4. 03 — Chunking StrategiesLesson · 6 min
  5. 04 — Reranking: cross-encoders, Cohere Rerank, and ColBERTLesson · 5 min
  6. 05 — Advanced RAG: HyDE, Multi-Query, Self-RAG, and CRAGLesson · 6 min
  7. 06 — Amazon Bedrock Knowledge Bases: Managed RAG on AWSLesson · 4 min
  8. 07 — Evaluating RAG with RAGASLesson · 6 min
  9. 08 — Hallucinations in RAG: Diagnosis and MitigationLesson · 6 min
  10. Lab 01 — Semantic similarity with local embeddingsLab · 20 min
  11. Lab 02 — Minimal RAG with retrieval, context with IDs, and optional generationLab · 20 min
  12. Lab 03 — Compare chunking with a retrieval metricLab · 20 min
  13. Lab 04 — Two-stage retrieval: cheap candidates and precise rerankingLab · 20 min
  14. Lab 05 — Advanced RAG with multi-query, HyDE, and Reciprocal Rank FusionLab · 20 min
  15. Lab 06 — Layered evaluation: deterministic retrieval and RAGAS 0.4.xLab · 20 min
  16. Exercises — Module IIIExercises · 3 min
  17. Module Project — NebulaOps Knowledge AssistantProject · 2 min
  18. Account SecurityResource · 2 min
  19. Billing and PaymentsResource · 3 min
  20. Frequently Asked QuestionsResource · 2 min
  21. Incident RunbookResource · 3 min
  22. Operations HandbookResource · 3 min
  23. Product, Plans, and LimitsResource · 3 min
04

MODULE 4 · 2 ECTS

AI Agents and Orchestration

50–60 hours

Agentic patterns, LangGraph, MCP, tool use, multi-agent systems, memory, and evaluation.

Exit evidence

A multi-agent system with an MCP server, traces, and failure recovery.

Prerequisites

Foundations · Prompt systems

Contents18 resources
  1. Module IV — AI Agents and Orchestration (2 ECTS)Module map · 3 min
  2. 01 — Agent Patterns: ReAct, Plan-Execute, Reflection, Self-CritiqueLesson · 7 min
  3. 02 — LangGraph: state graphs, cycles, conditionals, and checkpointsLesson · 7 min
  4. 03 — Model Context Protocol (MCP): Architecture and Server DevelopmentLesson · 6 min
  5. 04 — Advanced tool use: tool design, error handling, idempotencyLesson · 6 min
  6. 05 — Multi-agent systems: coordination, supervisors, communicationLesson · 6 min
  7. 06 — Agent Memory: Working, Episodic, and SemanticLesson · 5 min
  8. 07 — Agents for Amazon BedrockLesson · 5 min
  9. 08 — Agent Evaluation: Outcome, Trajectory, Cost, and SafetyLesson · 5 min
  10. 09 — Alignment and Fine-Tuning: SFT, RLHF, RLAIF, and Constitutional AILesson · 6 min
  11. Lab 01 — ReAct loop from scratch, with typed tools and step limitLab · 20 min
  12. Lab 02 — First state graph with LangGraph 1.2.x, fully offlineLab · 20 min
  13. Lab 03 — Bounded loop and conditional edges in LangGraphLab · 20 min
  14. Lab 04 — Threads, checkpoints, and history with InMemorySaverLab · 20 min
  15. Lab 05 — Small but complete 2.x MCP Notes ServerLab · 20 min
  16. Lab 06 — Multi-agent supervisor with explicit handoffs in LangGraphLab · 20 min
  17. Lab 07 — Working, episodic, and semantic memory with isolation and forgettingLab · 20 min
  18. Exercises — Module IVExercises · 3 min
05

MODULE 5 · 2 ECTS

LLMOps, Production, and Responsible AI

50–60 hours

Observability, continuous evaluation, cost, deployment, security, safety, and governance.

Exit evidence

A deployed system with SLOs, costs, evaluations, and an operational runbook.

Prerequisites

RAG · Agents

Contents19 resources
  1. Module V — LLMOps, Production, and Responsible AI (2 ECTS)Module map · 3 min
  2. LLM Observability: Traces, Spans, Metrics, and AlertsLesson · 7 min
  3. Continuous evaluation: eval pipelines and regression testing in CILesson · 7 min
  4. Cost optimization: semantic caching, model routing, and batchingLesson · 7 min
  5. Docker for LLMs: Containers, Multi-Stage Builds, and Image OptimizationLesson · 6 min
  6. Open-source LLM Deployment: vLLM, Ollama, and Triton Inference ServerLesson · 6 min
  7. 06 — Deployment and Monitoring on AWSLesson · 6 min
  8. 07 — Responsible AI: From Principles to Measurable ControlsLesson · 5 min
  9. 08 — LLM System Security and GovernanceLesson · 5 min
  10. 09 — Safety in Production: Policies, Guardrails, and ResponseLesson · 5 min
  11. Lab 01 — Manual JSONL traces without framework or sensitive dataLab · 20 min
  12. Lab 02 — The same instrumented chain with LangSmithLab · 20 min
  13. Lab 03 — Semantic cache with isolation, versioning, TTL, and a calibrable thresholdLab · 20 min
  14. Lab 04 — Auditable model routing between GPT-5.6 Luna, Terra, and SolLab · 20 min
  15. Lab 05 — Deterministic regression gate for CILab · 20 min
  16. Lab 06 — Local discovery and inference with the native Ollama APILab · 20 min
  17. Lab 07 — Prompt injection: measure attacks and apply layered controlsLab · 20 min
  18. Exercises — Module VExercises · 3 min
  19. Docker Lab — Hardened and Observable LLMOps APIResource · 2 min
06

MODULE 6 · 2 ECTS

Production Capstone

47–57 hours

End-to-end project, technical portfolio, deployment, defense, and final preparation.

Exit evidence

An interview-ready system with architecture, metrics, costs, and a live demo.

Prerequisites

RAG · Agents · LLMOps

Contents11 resources
  1. Module VI — Final Project / Capstone (2 ECTS)Module map · 6 min
  2. Project ProposalsProject · 7 min
  3. Capstone RAG Pipeline GuideProject · 6 min
  4. Autonomous Agent GuideProject · 6 min
  5. Cloud Deployment GuideProject · 7 min
  6. LLMOps Capstone GuideProject · 5 min
  7. Cost Analysis TemplateProject · 4 min
  8. Demo and Technical Defense GuideProject · 6 min
  9. ADR (Architecture Decision Record) TemplateProject · 3 min
  10. Architecture Document — [Project Name]Project · 5 min
  11. Final Capstone ChecklistProject · 5 min

COMPETENCY MAP

What you must be able to prove

A competency counts only when a reviewable artifact exists. These eight signals make up the professional passport.

01

Foundations and models

Explain transformers, tokenization, inference, and model selection through explicit trade-offs.

Evidence

An executable comparison and a model decision backed by evaluation.

02

Prompt systems

Treat prompts, schemas, and tools as versioned, testable contracts.

Evidence

A regression dataset and A/B pipeline with explicit criteria.

03

RAG and retrieval

Design ingestion, chunking, search, reranking, grounding, and RAG evaluation.

Evidence

A measured RAG system with answerable and unanswerable cases, citations, and retrieval metrics.

04

Agents and MCP

Build graphs, tools, memory, and failure recovery without losing control.

Evidence

An agent with three tools, traces, idempotency, and negative-path tests.

05

Evaluation

Separate model, retrieval, and application quality through reproducible evaluations.

Evidence

A regression suite with a baseline, rubric, failed samples, and acceptance gates.

06

LLMOps

Observe latency, cost, quality, safety, and distribution shifts in production.

Evidence

An operational dashboard, alerts, and an incident runbook.

07

Safety and governance

Model threats, constrain tools, and apply privacy, guardrails, and auditability.

Evidence

A threat model, prompt-injection test, and verifiable controls.

08

Product architecture

Make build-vs-buy, cost, reliability, and user-experience decisions.

Evidence

ADRs, cost analysis, and a technical capstone defense.

How progress works

Marks are stored in localStorage and shared across ES and EN through each resource’s canonical path. You can export them at any time. The site never needs an account.

View privacy and export