Final Capstone Checklist

Mark an item only when linked evidence exists. "Implemented" without a command, result, URL, trace, or artifact is incomplete. Record the commit and digest of the defended version.

Elimination Gates#

• Public HTTPS URL works from a clean session and has a usable frontend.
• RAGAS faithfulness ≥ 0.75 on the final versioned and audited dataset.
• At least three tools execute happy path and error scenarios with resolvable traces.
• No prohibited tools or secret/canary leaked in the adversarial suite.
• Tests, build, and offline eval pass on the delivery commit/digest.

1. Architecture#

• Final document with context, negative scope, and measurable RF/RNF requirements.
• Context, container, and sequence diagrams match the deployed state.
• At least five ADRs: model/routing, vector DB, chunking, agent, and cloud.
• Alternatives and trade-offs include results or costs, not just opinions.
• Threat model with assets, trust boundaries, threats, controls, owners, and tests.
• Changes from v1 architecture explained.

Evidence: versioned document, accepted/replaced ADRs, and exported diagram.

2. RAG#

• Corpus manifest with license/origin, hashes, and parser/chunker/embedding versions.
• Idempotent incremental ingestion covers high, modification, and low.
• Final dataset ≥ 50 questions, with negatives, paraphrases, and multi-source; test frozen.
• Retrieval reports recall/MRR/nDCG per tag and latency.
• Generation uses delimited context, resolvable citations, and abstention.
• RAGAS saves configuration, per-case detail, cost, and audit of worst results.
• One ablation justifies chunking, hybrid, or reranker.
• Five failures with root cause and prioritized action.

Evidence: dataset/hash, manifest, JSON results, report, and reproducible command.

3. Agent#

• Graph/flow with typed state, step limit, timeout, and degraded output.
• Three non-trivial tools with narrow schemas, least privileges, and structured errors.
• Writes use idempotency and confirmation linked to arguments/actor/TTL.
• Retry/backoff distinguishes transient from permanent errors.
• Checkpointer allows resuming without duplicating side effects.
• Memory has purpose, isolation, retention, and deletion, or justification for not having it.
• Evaluation includes success, trajectory, safety, steps, tokens, cost, and p95.
• Comparison against pipeline or single agent demonstrates that complexity pays off.

Evidence: tests without network and four traces: success, error, attack, and approval/resumption.

4. Deployment#

• Multi-stage image, non-root, without .env/weights, scanned and with SBOM.
• Digest-based deployment, external configuration/secrets, and rollback tested in < 5 min.
• /health and /ready have distinct contracts and do not make paid calls.
• CI runs tests/evals before deploying; a failure prevents promotion.
• Rate limit, auth, timeout, circuit breaker, and per-user/tenant budget.
• External uptime ≥ 99% for ≥ 14 days, with incidents explained.
• Versioned public load: two runs, varied queries, p50/p95/p99, TTFT, and errors.
• P95 meets the objective defined before measurement.

Evidence: URL, exported monitor, workflow/run, digest, scan/SBOM, and load reports.

5. LLMOps#

• Trace IDs link API, graph, tools, retrieval, and model calls.
• Dashboard covers reliability, latency, quality/safety, usage/cost.
• Labels have bounded cardinality and the wording passes the canary test.
• SLI/SLO/error budget are defined with window and exclusions.
• Burn rate, p95, provider, cost, quality, and security alerts have an owner/runbook.
• Three alerts were triggered in staging and recovered.
• An incident/failure produced a new test or case.

Evidence: versioned dashboard/queries, alerts, notifications, and timeline.

6. Costs#

• Period, currency, rates/dates/sources, and credits are separated.
• Actual cost includes inference, embeddings, evals, retries, tools, infrastructure, and observability.
• Average cost and p95 per successful task; breakdown by route/model/segment.
• 10×/100× projection separates variable, fixed, and stepped costs and includes stress testing.
• Chosen optimization has measured savings and quality/latency.
• Self-host break-even includes idle capacity, HA, and operations.

Evidence: drafted invoice/usage, dashboard export, and calculation sheet/script.

7–8. Certifications#

• AIF-C01 65-question mock exam completed; errors classified and reviewed.
• NCA-GENL: five weighted areas and ten topics reviewed; flashcards and mock exam completed.
• Dated results and review plan for the three weakest areas.

9. Demo and defense#

• 20-minute script rehearsed twice in under 19 minutes.
• Happy path, abstention, attack/error, approval, eval, operation, and cost are visible.
• Recent contingency plan and offline mode; live vs. recorded is distinguishable.
• 42 questions reviewed with answers based on your own data.
• Clean profile, zoom, alternative network, status restored, and notifications silenced.
• Feature freeze; commit, image, prompt, models, and corpus are exactly annotated.

Third-party reproducibility#

• README is part of a clean checkout and leads to tests + execution in < 30 minutes.
• .env.example documents variables without secret values.
• Locks present for Python/Node/infra and current versions verified.
• Licenses for corpus/models/dependencies reviewed.
• Backup/restore/rollback commands tested and runbooks accessible.

Delivery minutes#

Record in the final README: UTC date/time, commit, image digest, URL, prompt version, corpus hash, models/snapshots, dataset hash, gate results, and location of evidence. Sign the decision go, conditional go, or no-go with the accepted risks and their owner.