Open LLM Systems Engineering courseReviewed Aug 23, 2026
LLM EngineerClub
SearchESClub Quiz

FIELD PATH

Open LLM Systems Engineering path

LLM Systems Engineering designs the environment that turns a model into a reliable product. This path lives and improves in a public repository; the web edition turns it into a navigable experience with tools, progress, and assessment.

Check my levelView the complete mapView and improve on GitHub
01

MODULE 1

Model Interfaces and Foundations

15–18 h

Transformers, tokenization, generation, and multi-provider clients treated as replaceable components.

Proof of work

A multi-provider client with observable tokens, latency, errors, and metadata.

Prerequisites

None

Contents12 resources
  1. Module 01 — Model interfaces and foundationsModule map · 3 min
  2. Transformer Architecture: Encoder, Decoder, and Multi-Head AttentionLesson · 10 min
  3. Tokenization: BPE, WordPiece, and Model ComparisonLesson · 7 min
  4. Generation parameters: temperature, top_p, top_k, and penaltiesLesson · 7 min
  5. Current Landscape: GPT-5.6, Claude 5, Gemini 3.x, Llama 4, and MistralLesson · 8 min
  6. Generative AI Services on AWS: Bedrock, SageMaker JumpStart, and Amazon QLesson · 6 min
  7. Lab 01 — First OpenAI Call with the Responses APILab · 20 min
  8. Lab 02 — First Call to the Anthropic API (Claude)Lab · 20 min
  9. Lab 03 — Comparative Tokenization with tiktokenLab · 20 min
  10. Lab 04 — Sweeping temperature and top_p over the same promptLab · 20 min
  11. Lab 05 — Multi-model inference with the Amazon Bedrock Converse APILab · 20 min
  12. Exercises — Module IExercises · 3 min
02

MODULE 2

Context and Output Contracts

38–45 h

Context, instructions, tools, and structured outputs as versioned, evaluable interfaces.

Proof of work

A contract-evaluation pipeline with a dataset, A/B comparison, and regression gates in CI.

Prerequisites

Models

Contents16 resources
  1. Module 02 — Context and output contractsModule map · 3 min
  2. 01 — Base techniques: zero-shot, few-shot, chain-of-thought, and self-consistencyLesson · 8 min
  3. 02 — System Prompts, Roles, and Context DelimitationLesson · 6 min
  4. 03 — Function calling / tool use in OpenAI and AnthropicLesson · 6 min
  5. 04 — Structured outputs: JSON mode, response_format, Pydantic and InstructorLesson · 6 min
  6. 05 — Prompt Evaluation: Test Datasets, LLM-as-a-Judge, and A/B TestingLesson · 7 min
  7. 06 — Prompt Versioning and ManagementLesson · 5 min
  8. 07 — Biases and hallucinations: detection, measurement, and mitigationLesson · 6 min
  9. Lab 01 — Compare zero-shot and few-shot on the same casesLab · 20 min
  10. Lab 02 — Private reasoning and self-consistency with majority votingLab · 20 min
  11. Lab 03 — Complete function calling loop with the OpenAI Responses APILab · 20 min
  12. Lab 04 — Complete tool use loop with the Anthropic Messages APILab · 20 min
  13. Lab 05 — Native Structured Outputs and Instructor with PydanticLab · 20 min
  14. Lab 06 — Reproducible prompt evaluation pipeline on a datasetLab · 20 min
  15. Lab 07 — Paired statistical comparison of two promptsLab · 20 min
  16. Exercises — Module IIExercises · 3 min
03

MODULE 3

Retrieval Engineering

50–60 h

Ingestion, embeddings, vector stores, chunking, reranking, grounding, and retrieval evaluation.

Proof of work

A retrieval system with citations, abstention, traceability, and automated evaluation.

Prerequisites

Models · Contracts

Contents23 resources
  1. Module 03 — Retrieval EngineeringModule map · 3 min
  2. 01 — RAG Architecture: Ingestion, Embedding, Indexing, and RetrievalLesson · 6 min
  3. 02 — Embeddings and Vector DatabasesLesson · 6 min
  4. 03 — Chunking StrategiesLesson · 6 min
  5. 04 — Reranking: cross-encoders, Cohere Rerank, and ColBERTLesson · 5 min
  6. 05 — Advanced RAG: HyDE, Multi-Query, Self-RAG, and CRAGLesson · 6 min
  7. 06 — Amazon Bedrock Knowledge Bases: Managed RAG on AWSLesson · 4 min
  8. 07 — Evaluating RAG with RAGASLesson · 6 min
  9. 08 — Hallucinations in RAG: Diagnosis and MitigationLesson · 6 min
  10. Lab 01 — Semantic similarity with local embeddingsLab · 20 min
  11. Lab 02 — Minimal RAG with retrieval, context with IDs, and optional generationLab · 20 min
  12. Lab 03 — Compare chunking with a retrieval metricLab · 20 min
  13. Lab 04 — Two-stage retrieval: cheap candidates and precise rerankingLab · 20 min
  14. Lab 05 — Advanced RAG with multi-query, HyDE, and Reciprocal Rank FusionLab · 20 min
  15. Lab 06 — Layered evaluation: deterministic retrieval and RAGAS 0.4.xLab · 20 min
  16. Exercises — Module IIIExercises · 3 min
  17. Module Project — NebulaOps Knowledge AssistantProject · 2 min
  18. Account SecurityResource · 2 min
  19. Billing and PaymentsResource · 3 min
  20. Frequently Asked QuestionsResource · 2 min
  21. Incident RunbookResource · 3 min
  22. Operations HandbookResource · 3 min
  23. Product, Plans, and LimitsResource · 3 min
04

MODULE 4

Agent and Tool Interfaces

50–60 h

Agent patterns, tool use, MCP, memory, and agent collaboration with explicit authority.

Proof of work

An agent with an MCP server, typed tools, authority boundaries, and complete traces.

Prerequisites

Models · Contracts

Contents18 resources
  1. Module 04 — Agent and tool interfacesModule map · 3 min
  2. 01 — Agent Patterns: ReAct, Plan-Execute, Reflection, Self-CritiqueLesson · 7 min
  3. 02 — LangGraph: state graphs, cycles, conditionals, and checkpointsLesson · 7 min
  4. 03 — Model Context Protocol (MCP): Architecture and Server DevelopmentLesson · 6 min
  5. 04 — Advanced tool use: tool design, error handling, idempotencyLesson · 6 min
  6. 05 — Multi-agent systems: coordination, supervisors, communicationLesson · 6 min
  7. 06 — Agent Memory: Working, Episodic, and SemanticLesson · 5 min
  8. 07 — Agents for Amazon BedrockLesson · 5 min
  9. 08 — Agent Evaluation: Outcome, Trajectory, Cost, and SafetyLesson · 5 min
  10. 09 — Alignment and Fine-Tuning: SFT, RLHF, RLAIF, and Constitutional AILesson · 6 min
  11. Lab 01 — ReAct loop from scratch, with typed tools and step limitLab · 20 min
  12. Lab 02 — First state graph with LangGraph 1.2.x, fully offlineLab · 20 min
  13. Lab 03 — Bounded loop and conditional edges in LangGraphLab · 20 min
  14. Lab 04 — Threads, checkpoints, and history with InMemorySaverLab · 20 min
  15. Lab 05 — Small but complete 2.x MCP Notes ServerLab · 20 min
  16. Lab 06 — Multi-agent supervisor with explicit handoffs in LangGraphLab · 20 min
  17. Lab 07 — Working, episodic, and semantic memory with isolation and forgettingLab · 20 min
  18. Exercises — Module IVExercises · 3 min
05

MODULE 5

Harness Engineering

25–35 h

Architecture and causal evaluation of harnesses: context, loops, capabilities, sandboxes, the current ecosystem, and reproducible benchmarks.

Proof of work

An executable harness and causal benchmark of current runtimes, with isolation, traces, injected failures, and reproducible evidence.

Prerequisites

Contracts · Agents

Contents11 resources
  1. Module 05 — Harness EngineeringModule map · 3 min
  2. 01 — Harness anatomy and contractsLesson · 3 min
  3. 02 — Context, capabilities, and isolationLesson · 3 min
  4. 03 — Traces and the evaluation harnessLesson · 3 min
  5. 04 — Current landscape: compare architectures, not brandsLesson · 8 min
  6. 05 — Causal benchmarking of harnessesLesson · 8 min
  7. Lab 01 — Harness with a capability manifest, approval, sandbox, and tracesLab · 20 min
  8. Lab 02 — Evaluation harness with isolated trials and separate gradersLab · 20 min
  9. Lab 03 — Design valid comparisons between current harnessesLab · 20 min
  10. Exercises — Harness EngineeringExercises · 4 min
  11. Project — Verifiable repair harnessProject · 4 min
06

MODULE 6

Loop Engineering

25–35 h

Governable execution loops: state, progress, budgets, termination, retries, and resumption.

Proof of work

A durable loop with multidimensional budgets, verifiable stopping, idempotency, and recovery.

Prerequisites

Agents · Harnesses

Contents8 resources
  1. Module 06 — Loop EngineeringModule map · 3 min
  2. 01 — State, budgets, and terminationLesson · 3 min
  3. 02 — Errors, idempotency, and resumptionLesson · 3 min
  4. 03 — Progress, concurrency, and loop evaluationLesson · 3 min
  5. Lab 01 — Bounded loop with terminal states and progress detectionLab · 20 min
  6. Lab 02 — Checkpoint, resumption, and idempotency for one effectLab · 20 min
  7. Exercises — Loop EngineeringExercises · 3 min
  8. Project — Durable research loopProject · 2 min
07

MODULE 7

Graph Engineering

25–35 h

Execution, knowledge, and provenance graphs for LLM systems that must explain what they know and what they did.

Proof of work

A hybrid system with a state graph, knowledge graph, provenance, and query evaluation.

Prerequisites

Retrieval · Agents · Loops

Contents8 resources
  1. Module 07 — Graph EngineeringModule map · 3 min
  2. 01 — Three graphs and their invariantsLesson · 3 min
  3. 02 — Knowledge, identity, and provenanceLesson · 3 min
  4. 03 — Hybrid retrieval and evaluationLesson · 3 min
  5. Lab 01 — Typed graph that rejects relationships without provenanceLab · 20 min
  6. Lab 02 — Lexical retrieval plus a graph path with verifiable provenanceLab · 20 min
  7. Exercises — Graph EngineeringExercises · 3 min
  8. Project — Operational map with knowledge and provenanceProject · 2 min
08

MODULE 8

Production Engineering for LLM Systems

50–60 h

Observability, continuous evaluation, cost, deployment, security, safety, and governance.

Proof of work

A deployed system with SLOs, costs, evaluations, and an operational runbook.

Prerequisites

Retrieval · Harnesses · Loops · Graphs

Contents19 resources
  1. Module 08 — Production Engineering for LLM systemsModule map · 3 min
  2. LLM Observability: Traces, Spans, Metrics, and AlertsLesson · 5 min
  3. Continuous evaluation: eval pipelines and regression testing in CILesson · 7 min
  4. Cost optimization: semantic caching, model routing, and batchingLesson · 7 min
  5. Docker for LLMs: Containers, Multi-Stage Builds, and Image OptimizationLesson · 6 min
  6. Open-source LLM Deployment: vLLM, Ollama, and Triton Inference ServerLesson · 6 min
  7. 06 — Deployment and Monitoring on AWSLesson · 6 min
  8. 07 — Responsible AI: From Principles to Measurable ControlsLesson · 5 min
  9. 08 — LLM System Security and GovernanceLesson · 5 min
  10. 09 — Safety in Production: Policies, Guardrails, and ResponseLesson · 5 min
  11. Lab 01 — Manual JSONL traces without framework or sensitive dataLab · 20 min
  12. Lab 02 — The same instrumented chain with LangSmithLab · 20 min
  13. Lab 03 — Semantic cache with isolation, versioning, TTL, and a calibrable thresholdLab · 20 min
  14. Lab 04 — Auditable model routing between GPT-5.6 Luna, Terra, and SolLab · 20 min
  15. Lab 05 — Deterministic regression gate for CILab · 20 min
  16. Lab 06 — Local discovery and inference with the native Ollama APILab · 20 min
  17. Lab 07 — Prompt injection: measure attacks and apply layered controlsLab · 20 min
  18. Exercises — Module VIIIExercises · 3 min
  19. Docker Lab — Hardened and Observable LLMOps APIResource · 2 min
09

MODULE 9

Field Project

47–57 h

A real, bounded, verifiable system integrating decisions, failures, metrics, operations, and public review.

Proof of work

A defensible system with architecture, evals, traces, costs, a runbook, and a reproducible demo.

Prerequisites

Retrieval · Harnesses · Loops · Graphs · Production

Contents11 resources
  1. Module 09 — Field projectModule map · 5 min
  2. Project ProposalsProject · 7 min
  3. Field project RAG Pipeline GuideProject · 6 min
  4. Agentic System GuideProject · 6 min
  5. Cloud Deployment GuideProject · 3 min
  6. LLMOps Field project GuideProject · 5 min
  7. Cost Analysis TemplateProject · 4 min
  8. Demo and Technical Review GuideProject · 6 min
  9. ADR (Architecture Decision Record) TemplateProject · 3 min
  10. Architecture Document — [Project Name]Project · 4 min
  11. Final Field project ChecklistProject · 4 min

Open, verifiable curriculum

This web edition is generated from the public repository that preserves the lessons, labs, exercises, and their contribution history.

View and improve on GitHub

COMPETENCY MAP

What you must be able to prove

A competency counts only when a reviewable artifact exists. These 11 signals connect product decisions, system control, and operational evidence.

01

Foundations and models

Explain transformers, tokenization, inference, and model selection through explicit trade-offs.

Evidence

An executable comparison and a model decision backed by evaluation.

02

Prompt systems

Treat prompts, schemas, and tools as versioned, testable contracts.

Evidence

A regression dataset and A/B pipeline with explicit criteria.

03

RAG and retrieval

Design ingestion, chunking, search, reranking, grounding, and RAG evaluation.

Evidence

A measured RAG system with answerable and unanswerable cases, citations, and retrieval metrics.

04

Agent interfaces

Design tools, MCP, memory, and authority boundaries without delegating control to the model.

Evidence

An agent with three typed tools, explicit permissions, and negative-path tests.

05

Harness Engineering

Build the environment that supplies context, capabilities, isolation, signals, and evaluations to an agent.

Evidence

A reproducible harness with a capability manifest, sandbox, traces, and isolated evals.

06

Loop Engineering

Control iterations, budgets, progress, retries, concurrency, stopping, and resumption.

Evidence

A bounded, resumable loop that proves termination, idempotency, and failure recovery.

07

Graph Engineering

Model execution, knowledge, and provenance graphs with explicit invariants and measurable queries.

Evidence

A hybrid system that connects state, evidence, and decisions without losing provenance.

08

Evaluation

Separate model, retrieval, and application quality through reproducible evaluations.

Evidence

A regression suite with a baseline, rubric, failed samples, and acceptance gates.

09

LLMOps

Observe latency, cost, quality, safety, and distribution shifts in production.

Evidence

An operational dashboard, alerts, and an incident runbook.

10

Safety and governance

Model threats, constrain tools, and apply privacy, guardrails, and auditability.

Evidence

A threat model, prompt-injection test, and verifiable controls.

11

Product architecture

Make build-vs-buy, cost, reliability, and user-experience decisions.

Evidence

ADRs, cost analysis, and a technical field-project review.

How progress works

Your progress is stored under a pseudonym so you can resume it. You can export or erase it at any time without creating an account.

View privacy and export