Stop learning. Start shipping.
The chapters build the mental models. These 18 capstones, three per part, turn them into real systems on real infrastructure: deployed, measured, and defensible. Finishing the chapters makes you sound like a top engineer. Finishing six of these is how you become one.
Bring your own key:OpenAIAnthropicGeminiGroqGrok
The 0.001% definition of done, every capstone
Real & deployed Measured (eval / p95 / cost) Observable Secret-safe + budget-capped Documented for a stranger A 2–4 min demo
Part One · Foundations
Can you wire a model into real software, well?
C1.1L3 · Senior
The Provider-Agnostic LLM Gateway
Your startup is locked into one model vendor and one outage took down the whole product. Build the internal gateway every team routes through.
ShipA real service fronting all five providers behind one interface: streaming, schema-constrained structured output, automatic fallback on outage / rate-limit, retries with backoff + jitter, and per-request token + cost logging with a health/cost dashboard.
Done whenYou kill the primary key mid-load-test and zero user requests fail, they reroute. Structured outputs validate 100% against schema; p95 latency and cost-per-request are visible per provider.
ClosesReal multi-provider API use · structured output · reliability · cost
View build plan & architectureC1.2L3 · Senior
A Real Tool-Using Agent (terminal ops copilot)
Build a CLI agent that can actually do things, not just chat.
ShipA ReAct agent using native function-calling (made to work across OpenAI / Anthropic / Gemini tool APIs) with 4+ real tools, web search, a sandboxed code/calc executor, filesystem read/write, one real API, a full Thought → Action → Observation trace, step + budget caps, and a confirmation guard before any destructive tool.
Done whenIt completes a real multi-step task unattended ("find the 3 most-starred repos for X, summarize their READMEs, write report.md") with a correct, inspectable trace and no unsafe action without confirmation.
ClosesReal tool use · the reasoning loop · first taste of agent safety
View build plan & architectureC1.3L1 · Junior
Semantic Search Over Your Own Corpus
Make your own notes / docs / bookmarks searchable by meaning, and prove it works.
ShipIngest a real personal corpus (Obsidian/Notion export, PDFs), real embeddings (OpenAI or Gemini; or self-host e5/bge, Groq/Grok don't embed), a vector store, a search UI, and a hand-labeled eval set (≥20 queries with expected hits) measuring recall@k.
Done whenrecall@5 hits a target you set and justify, with a documented before/after from one chunking change you made on purpose, not guessed.
ClosesReal embeddings · semantic search · your first honest eval
View build plan & architecturePart Two · Data & Retrieval
Can you make retrieval work on messy reality?
C2.1L3 · Senior
FlagshipProduction RAG Over Ugly Real Documents
A law / insurance / clinical firm hands you 500 genuinely ugly PDFs, scans, multi-column, tables, and demands grounded answers with citations and zero hallucination tolerance.
ShipReal ingestion of the actual mess (OCR + tables), a real vector DB (pgvector / Qdrant / Weaviate), hybrid search + a reranker, query rewriting, grounded answers with inline citations, a refusal path when evidence is weak, and a RAGAS eval on a labeled set.
Done whenFaithfulness ≥ your target, every answer cites its sources, the system refuses on out-of-corpus questions (demonstrate it), and you can show the eval delta from each upgrade.
ClosesThe #1 toy→real gap · real vector DB · retrieval evals · document hell
View build plan & architectureC2.2L3 · Senior
The Agent That Remembers You
A companion / assistant that holds memory across weeks of conversations and many users.
ShipA real memory architecture, short-term window management, long-term vector memory + an episodic/summary store in a real DB, per-turn retrieval, an explicit write/update/forget policy, persistence that is auth-scoped per user (no cross-user leakage).
Done whenIn a fresh session it recalls a fact from three sessions ago, stays within a token budget (no bloat), and correctly updates a changed fact ("I moved cities") without contradiction.
ClosesMemory & context engineering for real · stateful, multi-user agents
View build plan & architectureC2.3L4 · Strong Senior
Knowledge-Graph RAG Over a Domain
Q&A over a connected domain where relationships matter, a codebase, a company wiki, a research field, and plain RAG fumbles multi-hop questions.
ShipExtract entities/relations into a real graph (Neo4j / Postgres) alongside a vector index, do hybrid graph + vector retrieval (GraphRAG / HippoRAG style), answer multi-hop questions, and eval multi-hop accuracy vs plain RAG.
Done whenIt beats plain RAG on a labeled multi-hop set you built, and you can show the graph path it traversed.
ClosesAdvanced retrieval · graph reasoning · honest comparison vs a baseline
View build plan & architecturePart Three · Architecture & Agentic
Can you design and ship a multi-agent system on real infra?
C3.1L5 · Principal
Framework fluencyA Multi-Agent System on a Real Framework
Automate a real knowledge-work task end-to-end: a competitive-research report, a codebase migration, or a due-diligence memo.
ShipBuilt on one production framework gone deep, LangGraph, OpenAI Agents SDK, or the Claude Agent SDK, a supervisor routing to specialists (researcher / analyst / writer / critic), real tools, shared state, a human-in-the-loop checkpoint that truly pauses, and a full trace. Deployed.
Done whenIt produces a genuinely useful artifact on a real task you didn't babysit, with a complete trace, a HITL gate that truly blocks, and a measured cost/quality comparison vs a single-agent baseline, proving multi-agent earned its complexity, or honestly reporting it didn't.
ClosesThe framework-illiteracy gap · orchestration · when NOT to go multi-agent
View build plan & architectureC3.2L3 · Senior
Ecosystem + securityShip a Real MCP Server
Expose one of your real systems, your Postgres, your company API, your files, to any model through the Model Context Protocol.
ShipA real MCP server (TS or Python SDK) exposing tools + resources over correct JSON-RPC, with auth, used live from a real MCP client (Claude Desktop or an SDK client), plus a written security review of the surface you exposed.
Done whenA real MCP client uses your server to do something genuinely useful against your real data, scoped to least privilege, with a documented threat model.
ClosesMCP & tool ecosystems · agent security (the exposed-tool-surface problem)
View build plan & architectureC3.3L5 · Principal
DistributedParallel Agentic Pipeline at Volume
Process 10k–100k real items (documents, tickets, products) with agents, on a budget and a deadline.
ShipA fan-out / fan-in pipeline with real concurrency, per-item model-tier routing (Groq for the easy 80%, escalate hard items to a frontier model), batching, idempotent retries, a real queue (SQS / Redis / Cloud Tasks), and a cost + throughput dashboard.
Done whenIt processes the full set within a stated cost and wall-clock budget, retries are idempotent (re-running is safe, no double-writes), and you can defend the cost-per-item and a measured quality floor.
ClosesParallelization · resource-aware execution · distributed-systems hygiene
View build plan & architecturePart Four · Production & Reliability
Can you build an AI system you'd put on-call?
C4.1L5 · Principal
The evals gapEvals-as-a-Discipline: a CI Gate for an LLM Feature
Your team ships prompt and model changes straight to prod with no safety net. Build the gate that stops a regression.
ShipA real eval suite, golden dataset, LLM-as-judge with bias controls (calibrated against human labels), task metrics, running offline + a canary design, wired into CI (GitHub Actions) so a PR that regresses quality fails the build, with the rigor to not block on noise, plus a regression diff report.
Done whenA deliberately-bad prompt change is caught and blocked by CI with a clear diff, and a within-noise change is correctly NOT blocked (you handle statistical power, not just a threshold).
ClosesThe evals-is-the-job gap · CI/CD for AI · statistical honesty
View build plan & architectureC4.2L4 · Strong Senior
Security + opsObservability + Guardrails for an Agent in Prod
An agent is live and you are on-call for it.
ShipReal tracing of every span / token / cost (OTel GenAI, Langfuse, or LangSmith), a guardrail layer (prompt-injection + jailbreak detection, PII redaction, output safety → allow / redact / block), SLO dashboards + alerts, and a runbook, then a red-team / chaos session with documented incidents.
Done whenYou can trace any request end-to-end, your guardrails block a real injection / PII-exfiltration attempt you demonstrate (show the attack and the block), and an alert fires on an injected fault.
ClosesObservability · the agent-security gap (injection, exfiltration, untrusted input)
View build plan & architectureC4.3L3 · Senior
ReliabilityResilience Engineering: Survive Chaos
Providers go down, rate-limit you, and return malformed JSON. Keep serving.
ShipAn agent service with retries (backoff + jitter), circuit breakers, timeouts/budgets, fallback chains (frontier → Groq → cached → degraded), HITL escalation, and idempotency keys, then a chaos harness injecting outages, latency, and bad JSON, and a report of auto-resolve rate vs escalation.
Done whenUnder a chaos run (≈30% injected failures) it sustains a target auto-resolve rate with bounded p95 and zero data corruption, escalating the rest cleanly.
ClosesRecovery & HITL · reliability you'd actually trust on-call
View build plan & architecturePart Five · Scalability & MLOps
Can you serve, optimize, and post-train models, and know when to?
C5.1L5 · Principal
Real deploymentDeploy & Autoscale an Agent Service
Launch day: traffic goes 0 → 500 concurrent sessions on a bursty curve.
ShipA containerized service on real infra (Fly.io / Cloud Run / Railway / Modal), autoscaling, a real load test (k6 / Locust) to 500 concurrent, a p95 SLO, graceful degradation under overload (load-shedding / queueing), a cost-per-session number, and a CI→deploy pipeline with rollback.
Done whenIt passes a 500-concurrent load test within p95 SLO and budget, sheds load gracefully past capacity (no cascading failure), and a bad deploy auto-rolls-back.
ClosesReal deployment + the serving-at-scale gap
View build plan & architectureC5.2L6 · Architect
Post-training depthPost-Train a Small Model and Prove It
A narrow, high-volume task where a fine-tuned small model might beat prompting a frontier model on cost and quality.
ShipActually post-train an open model, LoRA/QLoRA (PEFT/TRL) and/or DPO on a real preference set, on real GPU (Modal / Runpod / Colab), then build an eval that fairly compares fine-tuned-small vs prompt+RAG-on-a-frontier-model across quality, latency, and $/1k, and write the verdict (including "tuning didn't win, here's why"). Serve via Groq or self-host.
Done whenYou have a head-to-head eval with a recommendation you can defend, the run is reproducible (config + data card), and you can explain what LoRA/DPO actually changed in the weights.
ClosesThe post-training depth gap · fine-tune-vs-RAG for real
View build plan & architectureC5.3L6 · Architect
Model internalsSelf-Host & Optimize Inference
At your volume, hosted-API cost is killing margins. Can self-hosting win, and do you understand why?
ShipServe an open model with vLLM / TGI on real GPU, apply quantization (int8/int4, AWQ/GPTQ) and continuous batching, measure throughput / latency / $-per-1M-tokens vs a hosted baseline (Groq is the perfect fast-hosted comparison), and write a roofline / KV-cache analysis of the bottleneck.
Done whenA measured throughput/latency/cost curve vs the hosted baseline, a defensible break-even volume, AND a correct mechanistic explanation of why (attention, KV cache, arithmetic intensity).
ClosesInference optimization + the model-internals gap for the Architect tier
View build plan & architecturePart Six · Advanced & Leadership
Can you architect, communicate, and push the frontier?
C6.1L5 · Principal
Multimodal depthShip a Multimodal Product Feature
A real feature with a real owner: receipts → expense entries, a voice agent, video-moment search, or screenshot → UI code.
ShipA real multimodal pipeline using a vision/audio provider (OpenAI, Anthropic, or Gemini for vision; Whisper/Gemini for audio), real input handling, an explicit token/cost budget (image patches, audio seconds, video frames), an eval on real samples, and a deployed demo.
Done whenIt works on genuinely messy real inputs (bad lighting, accents, long video), with a measured accuracy and a cost model that shows why video / high-res explodes the budget.
ClosesMultimodal depth, one chapter to a real build
View build plan & architectureC6.2L6 · Architect
Capstone of capstonesArchitect & Defend a Real System Design
The actual staff/architect interview, made real: "Design an AI assistant over 10 years of our wiki + support tickets + code."
ShipA full design doc (requirements, architecture, data model, retrieval strategy, eval plan, safety/threat model, a cost model at 10x scale, rollout plan, an ADR per major decision), PLUS a working reference-implementation skeleton (the spine runs on real infra at small scale), PLUS a recorded design-review defense.
Done whenA senior engineer could build from your doc, your skeleton runs end-to-end at small scale, your cost model survives scrutiny at 10x, and you can defend every tradeoff on camera (including what you rejected and why).
ClosesThe Architect bar, system design + cost + security + communication
View build plan & architectureC6.3L5 · Principal
ExperimentationOnline Experimentation / Adaptive System
Two prompt/model/agent variants are live, learn which wins on a real metric, and/or build a system that improves from feedback.
ShipA real experimentation harness: an A/B test or a contextual bandit in production on a real KPI (with peeking-safe statistics), or a feedback-collection + preference loop that measurably improves the system over time, with guardrails against metric gaming and a writeup of what you learned.
Done whenYou make a statistically valid call on a real metric (no peeking/optional-stopping inflation), or show a measured improvement from a feedback loop, with the explore/exploit and Goodhart risks explicitly addressed.
ClosesExploration & experimentation · learning & adaptation
View build plan & architectureOne per part = a six-project portfolio.
It walks a recruiter straight up the ladder: wire a model in → make retrieval real → orchestrate agents → make it reliable → make it scale → architect and defend it.