AI Systems Architecture — Mastery5 / 9
Evaluation Pipelines as Infrastructure
In AI systems, evaluation is not QA you do at the end — it's infrastructure you build first. Without it, every change is a prayer.
In normal software, tests are pass/fail and you write them as you go. In AI systems, "correct" is fuzzy and outputs vary — so evaluation stops being QA and becomes infrastructure you stand up before optimizing anything.
Offline: the eval set
A curated set of representative inputs with reference answers or rubrics. Run it on every prompt change, model swap, or retrieval tweak and you get a number — did this help or hurt? Include hard and out-of-scope cases, not just the happy path.
Online: production metrics
Offline can't catch everything. Track online signals — thumbs up/down, task completion, escalation rate, regeneration rate — and feed surprising production cases back into the offline set. The eval set is a living asset.
LLM-as-judge, with guardrails
A strong model can grade quality at scale, but:
- Give it a strict rubric, not "is this good?"
- Calibrate against human labels on a sample.
- Use a different model/lens than the one being graded where bias matters.
Gate changes in CI
You can now measure. Next: making the system affordable — cost engineering.
Series — AI Systems Architecture — Mastery
- Part 01Architecting AI Products — First PrinciplesAI systems fail differently from normal software: they're non-deterministic, costly per call, and hard to test. The architecture has to account for all three.
- Part 02Single Agent vs. Multi-Agent — Choosing a TopologyMulti-agent is fashionable and usually premature. Here is how to decide honestly — and why most products should start with one well-equipped agent.
- Part 03Orchestration Patterns — Pipelines, Routers, SwarmsOnce you have multiple steps or agents, how they're wired together decides cost, latency and reliability. Four patterns cover almost everything.
- Part 04Context & Memory ArchitectureThe context window is your most expensive, most contested resource. What you put in it — and what you remember between calls — is an architectural decision.
- Part 05Evaluation Pipelines as Infrastructure — you are hereIn AI systems, evaluation is not QA you do at the end — it's infrastructure you build first. Without it, every change is a prayer.
- Part 06Cost Engineering — Token Budgets That HoldAn AI feature that delights at 100 users can bankrupt you at 100,000. Cost is an architectural constraint, designed in — not discovered on the invoice.
- Part 07Latency & Throughput at ScaleInference is slow and bursty. Streaming, parallelism, and the async boundary are what keep an AI product feeling fast under real load.
- Part 08Reliability — Retries, Fallbacks, GuardrailsModels return malformed output, providers go down, and outputs drift. A reliable AI system expects all three and keeps working anyway.
- Part 09The Reference Architecture in ProductionTopology, orchestration, memory, eval, cost, latency and reliability — composed into one blueprint for an AI system that survives real users.