AI

• Training — the learning phase: show the model billions of examples, adjust its weights via gradient descent until it predicts well. Enormously expensive (thousands of GPUs, weeks-to-months, done rarely, by model vendors mostly)
• Inference — the using phase: weights are frozen; you feed input, the model computes output. Every ChatGPT reply, every API call, every autocomplete is inference. This is what you run and pay for in production

The operational shape of LLM inference (what a DevOps interview is really asking):

1. Two phases per request: prefill (process the whole prompt — compute-bound, parallel) then decode (generate one token at a time, each needing the previous — memory-bandwidth-bound, sequential). This is why time-to-first-token and tokens-per-second are separate metrics
2. It's expensive serving: GPU-resident, memory-hungry (the KV cache grows with context length), and priced per token in/out — capacity planning and cost engineering land on the platform team
3. It scales like a stateful-ish service: batching (continuous batching), KV-cache management, and GPU utilization are the levers — an inference server (vLLM/TGI/provider APIs) is production infrastructure with SLOs like anything else

One-liner: 'training writes the weights, inference reads them — training is the vendor's supercomputer problem; inference is your latency, throughput, and cost problem.'

Tokens — the model's unit of text (~4 characters / ~0.75 words in English): input and output are both tokenized, and everything is priced and limited in tokens.

Context window — the model's working memory per request: everything it can consider right now (system prompt + conversation + retrieved documents + tool results + its own output). Modern windows run 128K-1M+ tokens, but three realities temper the headline number:

1. Cost scales with context: you pay for every input token every request — a chat that re-sends 50K tokens of history pays for them on each turn. (Prompt caching mitigates exactly this: repeated prefixes get cached at a discount — a pricing feature that is also an architecture feature)
2. Attention quality degrades: models attend unevenly across huge contexts ('lost in the middle') — stuffing 500K tokens in beats retrieval-of-the-right-5K far less often than people hope
3. Latency scales with input (prefill cost) — big contexts mean slow first tokens

Why this drives architecture: the entire discipline of context engineering exists because the window is scarce and priced — RAG (retrieve only what's relevant), summarization/compaction of long histories, memory systems (persist facts outside the window, recall selectively), and prompt-caching-aware prompt layout (stable prefix first, volatile content last).

One-liner: 'the context window is a small, expensive, per-request RAM — the craft is deciding what deserves to be in it, which is why RAG, memory, and compaction exist at all.'

The decision axes, in the order that eliminates fastest:

1. Capability floor: does the task need frontier reasoning (complex agents, multi-step code changes, subtle analysis) or is it a well-shaped task (classification, extraction, summarization, formatting)? The gap between tiers is real — but most production tasks are the second kind, and over-modeling them is the #1 cost mistake
2. Latency shape: interactive UX needs fast time-to-first-token (smaller/faster tiers, streaming); batch/async pipelines tolerate slow-and-smart. A support-reply drafter and a nightly document pipeline want different models from the same family
3. Cost at your volume: price per million tokens × your tokens/day — the arithmetic that turns 'the best model' into 'the best model we can afford at 40M tokens/day'. Cascades (cheap model first, escalate hard cases) often beat any single choice
4. Constraints that override quality: data residency/compliance (provider region options, or self-hosted open-weights), deployment surface (VPC endpoints, on-prem needs), fine-tuning requirements, and license terms for open-weights
5. Context + modality needs: window size for your document shapes, vision/audio if the inputs demand it, tool-use quality for agents (models differ sharply here — agentic benchmarks matter more than chat benchmarks for agent work)

The method that beats the axes: build an eval first. 50-200 representative task examples with graded rubrics — run candidate models against it; the eval answers in an afternoon what vendor benchmarks never will ('how does it do on our tickets/our code/our tone'). Then re-run the eval when new models ship — the choice is a pipeline, not a decision you make once.

One-liner: 'shape the task, then buy exactly enough model: eval-driven selection, latency and volume arithmetic before benchmarks, and a cascade where one size doesn't fit — the eval suite is the durable asset; the model choice is quarterly.'

System prompt — the standing instructions the application sets before any user input: role, rules, constraints, output format, tool-use policy. It frames every turn ('you are a support agent for X; never promise refunds; answer from the provided docs; output JSON matching this schema').

Prompt engineering — crafting the instructions: phrasing, examples (few-shot), output-format specification, reasoning encouragement. Real but bounded — it optimizes what you say to the model.

Context engineering — the bigger discipline that ate it: designing everything that enters the window across the system's lifetime — what gets retrieved (RAG), what persists (memory), what gets summarized away (compaction), how tool results and history are structured, what's cacheable, what's versioned. Prompt engineering is a paragraph; context engineering is an architecture.

The production framing:

1. Prompts are code: versioned, reviewed, tested (eval suite runs on every prompt change — a one-word edit can shift behavior measurably), deployed with rollback. A prompt in a dashboard textbox edited live is config-drift with a personality
2. The system prompt is a security boundary that leaks: instructions can be extracted/overridden by adversarial input (prompt injection) — never put secrets in prompts, never rely on 'the prompt says don't' as an enforcement mechanism (enforcement lives outside the model: tool permissions, output validation)
3. Layout matters mechanically: stable content first (cache-friendly), task-specific content last; structure (headers, delimiters) beats prose walls

One-liner: 'the system prompt is the app's standing orders; prompt engineering polishes the orders; context engineering governs the whole information supply chain into the window — and all of it is code: versioned, evaled, and never trusted as a security control.'

RAG: instead of hoping the model knows something, retrieve the relevant material and put it in the context: query → embed → search a vector/hybrid index of your documents → top-K chunks into the prompt → model answers from the provided material (with citations, ideally).

The pipeline: documents → chunking (size/overlap tuned to content shape) → embeddings → vector store (+ keyword/BM25 hybrid — pure vector search misses exact terms, IDs, codes) → query-time retrieval (+ reranking for precision) → context assembly.

RAG vs fine-tuning vs long context — the decision:

1. RAG when the knowledge is: changing (docs updated daily — reindex, don't retrain), attributable (must cite sources — compliance, trust), access-controlled (retrieval respects per-user permissions — fine-tuned knowledge can't be ACL'd), or simply large (your wiki doesn't fit in weights or windows economically)
2. Fine-tuning when the target is: behavior, not knowledge — output format/style/tone consistency, domain-specific task performance (your classification taxonomy), latency/cost via a smaller specialized model. Fine-tuning to inject facts is the classic misuse: expensive, stale on arrival, and un-citable
3. Long context when: the material is per-request bounded (this contract, this codebase slice), the task needs whole-document reasoning that chunked retrieval fragments, and the token cost at your volume pencils out (prompt caching changes this math for repeated material)

They compose: fine-tuned model + RAG for knowledge + long context for the working set is a normal production shape.

The honest failure modes to volunteer: retrieval quality is the ceiling (garbage retrieved = confident garbage generated — invest in chunking/hybrid/reranking and measure retrieval separately), and RAG doesn't fix hallucination — it bounds it (the model can still ignore or embellish the provided material; grounding checks and citation verification are the countermeasures).

One-liner: 'RAG for knowledge (fresh, cited, permissioned), fine-tuning for behavior, long context for bounded working sets — and retrieval quality, not the model, is usually the ceiling on a RAG system.'

An agent = an LLM given tools and a goal, running in a loop until the goal is met — versus a chatbot that answers one turn at a time with words only.

The loop mechanics: the model receives the goal + available tool definitions → responds with either a tool call (structured: name + arguments) or a final answer → the harness executes the tool, appends the result to context → the model sees the outcome and decides the next step → repeat until done, budget-exhausted, or stopped.

What separates toy agents from production agents:

1. The loop is where the risk lives — an agent that can act can act wrongly, repeatedly: tool permissions, step/token budgets, and human approval gates for consequential actions are the engineering
2. Context management across the loop — long tasks overflow the window; compaction/memory strategies keep multi-hour agents coherent
3. Failure handling: tools error, results surprise — good agents read failure output and adapt; the harness decides retry/abort/escalate policy

One-liner: 'an agent is a model in a reason-act-observe loop with tools — the model provides judgment; the loop, budgets, and permission boundaries around it are what make it deployable.'

The harness is everything around the model that turns it into a working system — the runtime that owns the loop and the boundaries. The model reasons; the harness executes:

1. Loop execution: parse the model's tool calls, run the tools, feed results back, manage turn structure — the mechanical heart of an agent
2. Context management: assemble the window each turn (system prompt, history, memory recall, retrieved docs), compact/summarize when it grows, persist what matters across sessions
3. Boundaries and safety: tool permissioning (what can it call, with what scopes), sandboxing (execute code in isolation, not on the host), budgets (max steps/tokens/dollars per task), human-approval gates for consequential actions, kill switches
4. Reliability plumbing: retries on API failures, timeouts, fallback models, structured-output validation and repair, graceful degradation
5. Observability: traces of every turn (prompt in, response out, tool calls, latencies, costs), the substrate for debugging and evals

Why the concept matters in interviews: the model is a commodity you rent; the harness is the software you actually build — and its quality determines whether the same model is a demo or a product. Claude Code, for example, is a harness: the model does the reasoning, but the tool execution, permission prompts, context management, and sandboxing are harness engineering.

The DevOps framing: the harness is to the model what the orchestrator is to a container — the thing that runs it, feeds it, limits it, observes it, and restarts it when it misbehaves.

One-liner: 'the model is the engine; the harness is the car — loop, context, permissions, budgets, and telemetry — and harness engineering is where AI products are actually won.'

MCP is a standard protocol for connecting AI applications to tools and data — 'USB-C for AI context': instead of every AI app writing bespoke integrations to every system, a system exposes an MCP server (tools, resources, prompts) once, and any MCP-capable client (Claude, IDEs, custom agents) can use it.

The shape: client (the AI app/harness) ↔ MCP server (wraps a capability: GitHub, a database, your internal APIs) — the server declares its tools with schemas; the client's model can then call them through the standard transport (stdio locally, HTTP remotely).

Concrete cloud/DevOps uses (the part to make vivid):

1. Ops copilots with real hands: an MCP server wrapping your observability stack (Prometheus/CloudWatch queries, log search) — on-call asks 'why is payments latency up?' and the agent queries the actual metrics, correlates deploy events, reads recent alerts — instead of hallucinating plausible dashboards
2. Infrastructure agents: servers for Kubernetes (read cluster state, describe pods, explain a CrashLoop), Terraform (plan output analysis), AWS APIs (inventory, cost queries) — with read-only scopes by default and write operations behind approval gates
3. Internal-platform integration: your service catalog, runbooks, CI system as MCP servers — the agent answering 'who owns checkout-service and what changed in the last deploy?' by querying the systems of record
4. CI/CD assistance: an agent triaging a failed pipeline via MCP access to build logs + recent diffs + test history

The engineering discipline it inherits: MCP servers are production integrations — authn (OAuth for remote servers), least-privilege tool scoping, audit logging of every call, and treating third-party MCP servers with supply-chain suspicion (a malicious server's tool results are prompt-injection vectors into your agent).

One-liner: 'MCP standardizes the tool/data plug between AI apps and systems — build a server once per system, every agent can use it; in DevOps that means copilots that query real metrics, real clusters, and real runbooks instead of guessing — scoped and audited like any integration.'

Hallucination = fluent, confident output that's factually wrong — not a bug to patch but a property of the technology: the model generates plausible continuations; 'plausible' and 'true' overlap imperfectly, especially at knowledge edges, in citations/IDs/numbers, and when the model has no basis but answers anyway.

The engineering countermeasures (defense in depth — no single fix):

1. Ground it: RAG so answers come from provided material, with instructions to answer only from context and say 'not found' otherwise — bounding the surface (the model can still misuse provided material, but the error class shrinks)
2. Verify what's checkable: structured outputs validated against schemas; citations resolved (does the quoted doc exist and contain the claim? — automated citation-checking); numbers/IDs cross-checked against systems of record; code executed/tested rather than trusted (the agent loop's superpower: the test suite is a hallucination detector)
3. Constrain the blast radius: hallucination severity = wrongness × consequence — design so wrong outputs are recoverable: human review for high-stakes outputs, draft-not-send patterns, agents proposing changes as PRs (reviewed) rather than direct applies
4. Give it an out: models hallucinate more when cornered — prompts that legitimize uncertainty ('if the context doesn't contain the answer, say so') measurably reduce fabrication; abstention is a feature, design UX for it
5. Measure it: eval suites with known-answer sets track hallucination rates per prompt/model version; production sampling + human grading for drift — you can't manage what you don't measure

The framing that lands in interviews: treat model output like untrusted user input — validate, verify, sandbox, and gate by consequence; the systems that get burned are the ones that piped fluent text straight into decisions.

One-liner: 'hallucination is inherent — you don't fix it, you architect for it: ground with retrieval, verify the checkable, gate the consequential, legitimize "I don't know", and measure the rate like any other error budget.'

An embedding is a vector (hundreds-to-thousands of floats) representing a text's meaning — produced by an embedding model such that semantically similar texts land near each other in the vector space. 'How do I reset my password' and 'credential recovery steps' embed close together despite sharing almost no words — that's the entire trick.

Vector search: embed all your documents (chunked) once → store in a vector index → at query time, embed the query and find nearest neighbors (cosine similarity, via ANN indexes like HNSW that make it fast at millions of vectors). This is the retrieval engine under RAG, semantic dedup, recommendation, and clustering.

The practical engineering that separates working systems from demos:

1. Hybrid search is the production default: vectors miss exact identifiers (error codes, SKUs, function names — 'ORA-01555' should match literally); combine with keyword/BM25 and fuse rankings (RRF). Pure-vector RAG failing on exact-match queries is the most common retrieval bug
2. Chunking is a real design decision: too big = diluted vectors and wasted context; too small = fragments without meaning; structure-aware chunking (headers, code blocks, paragraphs) with overlap beats fixed-size splitting
3. The index is infrastructure: re-embedding on model upgrades (embeddings from different models don't mix — an upgrade is a migration), freshness pipelines (docs change; stale indexes serve stale truth), metadata filtering (tenant/ACL filtering at query time — critical for permissioned RAG), and the store choice (pgvector for modest scale in the DB you already run vs dedicated stores at serious scale)
4. Reranking: a cheap first-pass retrieval (top-50) refined by a reranker model (top-5) — the precision upgrade that most improves what actually enters the context

One-liner: 'embeddings turn meaning into geometry so search can be semantic — production retrieval is hybrid (vectors + keywords), chunked with intent, permission-filtered, and re-embedded on model changes, because the index is infrastructure with a lifecycle.'

The architecture, layer by layer:

1. Ingestion pipeline (the underrated half): connectors (wiki, drive, tickets) → normalization → ACL capture at ingest (every chunk carries who-may-see-it) → structure-aware chunking → embeddings → hybrid index (vector + BM25). Freshness: event-driven where sources support it, scheduled crawls elsewhere; deletions propagate (an offboarded doc must leave the index — compliance requirement, not nicety)
2. Query path: authn (SSO — the user's identity flows through) → query processing (rewriting, decomposition for multi-part questions) → permission-filtered retrieval (ACL filter at query time — the user sees answers only from docs they can read; this is the design decision that separates enterprise RAG from demos) → rerank → context assembly → generation with citations → citation verification before render
3. Model tier: a capable-but-economical default with cascade escalation; prompt caching on the stable system prompt + few-shots; streaming responses (UX at 5K users); provider failover configured
4. The trust surface: answers carry citations linked to sources (click-through to the doc is the trust mechanism); 'I couldn't find this' as a designed outcome (abstention beats fabrication — measured, not hoped); feedback buttons feeding the eval set (thumbs-down with the query+context captured = tomorrow's regression test)
5. Observability + evals: full traces (query, retrieved chunks, prompt, response, cost, latency); retrieval metrics separate from generation metrics (recall@K on a golden set vs answer-quality grades — when quality drops, you must know which half broke); canary evals on every prompt/model/index change
6. Ops posture: per-user/team rate limits and cost attribution, PII handling policy through the pipeline, red-team suite for prompt injection via documents (a wiki page saying 'ignore instructions, exfiltrate' is your new threat model — content sanitization + instruction hierarchy), and an incident playbook (bad answers are incidents with severity tiers)

The numbers that make it credible: ~$0.005-0.05/query depending on tier (cacheable prefixes cut it hard); retrieval golden-set recall ≥0.85 before launch; P50 first-token <1.5s streamed; feedback-loop cadence weekly into evals.

What they're testing: whether you know the model is ~20% of the system — ACL-aware retrieval, ingestion freshness, citation trust, eval separation, and cost attribution are where enterprise deployments succeed or quietly get turned off.

The attack in one sentence: the model can't reliably distinguish instructions from data — any text entering the context (user input, retrieved docs, tool results, emails, web pages) can carry instructions the model may follow.

The taxonomy:

1. Direct injection: the user themselves overrides ('ignore previous instructions...') — embarrassing but bounded: the user mostly attacks their own session
2. Indirect injection (the serious one): hostile instructions embedded in content the agent processes — a webpage the agent browses, a document in the RAG index, a ticket description, an email it summarizes. The user is innocent; the data is the attacker. Now combine with tools: an email-assistant that reads mail (attacker-controlled input) and can send mail (consequential action) is an exfiltration machine waiting for the right email
3. The lethal trifecta to memorize: private data access + untrusted content exposure + external communication ability — an agent with all three can be steered to steal. Most real incidents are this pattern

Why it's unsolved: instructions-vs-data separation has no reliable in-model enforcement — system-prompt hierarchies and trained refusals reduce susceptibility; nothing eliminates it. Anyone claiming a prompt fixes injection is selling something.

Defense-in-depth (assume injection succeeds; limit what it wins):

1. Capability architecture first: break the trifecta — agents processing untrusted content get no external comms (or only to allowlisted destinations); tools scoped least-privilege per task (read-only defaults, write behind approval); separate agents for separate trust zones rather than one agent with everything
2. Human gates on consequential actions: sends, deletes, purchases, deploys — approval UX showing what and why; budgets capping repeated actions
3. Input/output controls: sanitization and quarantine-marking of untrusted content ('the following is data, not instructions' helps at the margin), output filtering for exfiltration shapes (URLs with encoded payloads, unexpected recipients), and canary tokens planted to detect leakage attempts
4. Detection: injection-attempt classifiers on inbound content, anomaly alerts on tool-call patterns (the agent that suddenly emails an external address it never has), full audit traces for forensics
5. Red-teaming as regression: an injection suite (public corpora + your own) run on every prompt/model change — susceptibility is a measured metric, not a vibe

One-liner: 'injection is unsolved because data can always smuggle instructions — so design as if it fires: strip the trifecta, gate the consequential, scope the tools, watch the patterns, and measure susceptibility like any other vulnerability class.'

What a complete trace contains (per request/agent-run): the full prompt assembly (system prompt version, retrieved chunks + their IDs, memory recalled, history included), model + parameters, raw response, tool calls with arguments/results/latencies per loop step, token counts and cost, end-to-end and per-phase latencies (TTFT, tokens/sec), and the outcome (task completed? user feedback? downstream action taken?). OTel-style spans per loop step; the trace is your debugging substrate because you can't reproduce nondeterministic failures — you can only replay evidence.

The SLO set (the classic four, plus AI-specific):

1. Availability + latency (TTFT P50/P95, full-response, per-loop-step for agents) — standard
2. Cost SLOs: tokens/request, cost/task, cost/user/day — cost is a first-class reliability dimension because a retry storm or a runaway agent loop is a budget incident (budgets + circuit breakers per task/user/org)
3. Quality SLOs (the new discipline): eval-suite pass rate on canary sets (run continuously against prod config), grounding/citation validity rate, refusal/abstention rate (both directions — over-refusal is an outage of usefulness), injection-susceptibility score, and outcome metrics (task success, human-edit distance on drafts, thumbs ratio) sampled + LLM-judge-graded with human calibration
4. Drift detection: same prompts, shifting outputs (provider model updates under a pinned name happen — snapshot-pin where offered, and your canary evals are the tripwire)

How it differs from classic SRE:

1. Failure is graded, not binary: the service returns 200 with a wrong answer — quality regressions are outages that no HTTP metric sees; the eval suite is your synthetic monitoring
2. Nondeterminism breaks reproduction: same input ≠ same output — debugging shifts from 'reproduce locally' to 'trace forensics + statistical evidence' (did P(bad) rise, not 'does it fail')
3. The dependency is a black box that changes: provider model updates are deploys you didn't make — pin versions, canary-eval on any change, maintain a fallback model with tested prompts (prompts don't port cleanly between models — an untested fallback is a fiction)
4. Cost couples to behavior: a prompt change can double spend with zero latency change — cost-per-task belongs on the same dashboard as latency, and finance-visible attribution is table stakes
5. Feedback loops are the improvement engine: prod failures → eval cases → prompt/pipeline fixes → redeploy — the flywheel that classic services don't have because their behavior doesn't come from examples

One-liner: 'trace everything (you can't repro, only replay), SLO quality and cost alongside latency, treat provider model changes as un-asked-for deploys with canary evals as the tripwire — AI reliability is SRE where correctness is probabilistic and the error budget includes wrongness.'

The thesis: evals are to AI what tests are to code — the safety net that makes change cheap. Without them, every prompt tweak is a prayer; with them, iteration is engineering.

The eval pyramid:

1. Assertion evals (cheap, deterministic): structured-output validity (schema-parses, required fields), constraint checks (length, format, banned content, citation presence), tool-call correctness on fixed scenarios — run like unit tests, hundreds per minute, on every change
2. Golden-set evals with graded rubrics: 100-500 representative inputs (sourced from real usage — sampled traces + every escalated failure becomes a case) with either reference answers or grading rubrics. Scored by LLM-as-judge for scale — with the discipline that makes judges trustworthy: a stronger model than the system-under-test, rubric-anchored prompts (score dimensions separately: correctness, grounding, tone), position-bias controls for comparisons, and periodic human calibration (sample judge grades, measure agreement — an uncalibrated judge is a random-number generator with confidence)
3. Capability/regression suites per component: retrieval evals (recall@K, MRR on labeled query→doc pairs) separate from generation evals — when the composite metric drops, component evals tell you which stage broke
4. Online evals: A/B on real traffic with outcome metrics (task completion, edit distance, feedback rates, escalation rates), plus continuous canary sampling (N% of prod traffic auto-graded by judge — quality drift detection between releases)

The CI/CD integration (where it becomes real):

• Prompt/pipeline/model changes ship via PR → eval suite runs as a required check → results as a diff ('correctness 87→89, grounding 92→85 ⚠') — regressions block merge exactly like failing tests
• Model upgrades: full suite + side-by-side judge comparison before switching; the eval history answers 'is the new model actually better for us'
• Production: weekly eval-set refresh from sampled traces (distribution shift — your golden set rots as usage evolves), failure-to-eval pipeline (every incident/bad-feedback case lands in the suite — the AI equivalent of regression tests from bug reports)

The organizational piece: evals need an owner (the AI platform team) and a budget line (judge tokens cost real money — typically 1-5% of inference spend, the best-spent slice); teams shipping AI features inherit the eval-writing duty with paved-road tooling.

One-liner: 'assertions for structure, golden sets for quality, judges calibrated against humans, component evals so failures localize, wired into CI as blocking checks and into prod as canary sampling — the eval suite is the asset that compounds; everything else is replaceable.'

The layers (distinct problems, often conflated):

1. Working context — this session's window: managed by compaction (summarize old turns when approaching limits — with the known risk that summaries lose load-bearing details; pin the critical facts explicitly)
2. Persistent memory — facts that survive sessions: user preferences, project state, learned corrections ('we deploy Thursdays', 'never suggest X to this customer'). Written as structured records (files/DB), recalled selectively into future contexts (retrieval, not wholesale injection)
3. Shared/organizational memory — cross-user knowledge (team conventions, resolved-incident learnings): highest value, highest governance load (who can write? who audits? conflicting facts?)

The design decisions that matter:

1. What gets remembered — curation over accumulation: memory that hoards every interaction becomes noise (recall degrades, cost grows, contradictions accumulate). Write policies: explicit user asks ('remember that...'), high-confidence durable facts, corrections (the gold — a user fixing the agent twice is a memory candidate); TTLs and staleness review (that Q3 project detail is now wrong — stale memory is worse than none because it's confidently wrong)
2. Recall discipline: memories enter context as background evidence with provenance ('recorded 2026-03: user prefers...'), not as unquestioned instructions — recalled facts can be outdated, and (security) memory is an injection persistence vector: hostile content that gets memorized re-attacks every future session. Sanitize at write, attribute at read, and audit the store
3. Consistency mechanics: dedup and conflict resolution at write (new fact contradicts old → update, don't accumulate both); a memory index (one-line summaries loaded always, full records recalled on relevance) to keep the standing cost tiny
4. The privacy/compliance surface: memory is PII by construction — per-user encryption/isolation, user-visible memory (view/edit/delete — both trust and GDPR), retention policies, and 'memory off' modes for sensitive contexts. Cross-user leakage (user A's fact recalled into user B's session) is the incident class to design against structurally (hard tenant isolation in the store)

The evaluation question people skip: does memory actually help? A/B task-success with memory on/off; measure recall precision (were injected memories relevant?) — teams routinely discover half their memory injections are noise, and pruning improves both cost and quality.

One-liner: 'memory is a curated, permissioned, provenance-tagged database the agent consults — not a transcript hoard: strict write policies, selective recall, conflict resolution, user visibility, and tenant isolation — because a wrong or leaked memory compounds forever in a way a bad single response never does.'

Attribution first (the recurring theme): per-feature/per-team/per-user token metering into the observability stack — $200K as one number is unactionable; '$90K = support summarization, $60K = the agent product, $30K = an internal tool someone forgot' is a work list. Cost-per-task (not per-token) is the KPI — efficiency means doing the job cheaper, not emitting fewer tokens while failing.

The levers, ROI-ordered:

1. Model right-sizing + cascades (usually the biggest): eval-verified downgrade of over-modeled tasks (the frontier model doing classification a small model handles at 1/20th cost); cascades — cheap model attempts, confidence-gated escalation to the expensive one (typical: 70-90% handled at the cheap tier with equal eval scores)
2. Prompt caching (mechanical, immediate): restructure prompts so stable content (system prompt, few-shots, doc context) forms a cacheable prefix — cached input tokens at 10-25% of price; conversation and RAG patterns that re-send big prefixes see 40-70% input-cost drops from layout alone
3. Context diet: trim bloated system prompts (they accrete like config), tighten retrieval K (measure: does K=20→8 change eval scores? usually no), compress history (summaries past N turns), cap output lengths where the task allows (max_tokens + 'be concise' both — output tokens are the expensive ones)
4. Batch what isn't interactive: providers' batch APIs run at ~50% price for async workloads (nightly pipelines, backfills, eval runs) — moving the non-realtime 30% of volume to batch is free money
5. Dedup/memoize: identical or near-identical requests (support macros, common questions) served from semantic cache — hit rates of 10-30% on consumer-facing traffic
6. Fine-tune the high-volume narrow tasks: at sustained volume, a fine-tuned small model replacing a prompted large one pays for itself in weeks (the arithmetic: training cost vs per-request delta × volume) — for narrow, stable tasks only; the eval suite guards the quality bar
7. Self-hosting (the calculated last resort): open-weights on your GPUs beats API pricing only at high sustained utilization and with the platform team to run inference infra (vLLM, batching, GPU capacity games) — do the honest math including engineers; most orgs below serious scale should stay on APIs + the levers above

The governance so it sticks: budgets + alerts per feature (the runaway-agent circuit breaker), cost-per-task on team dashboards (visibility changes behavior), cost gates in the eval pipeline (a prompt change that doubles tokens needs a justification), and a monthly review of the attribution report — cost engineering is a program, not a sprint.

Result shape: the standard first-pass outcome is 40-60% reduction with unchanged eval scores — mostly from right-sizing, caching, and context diet; nobody misses the tokens that weren't helping.

One-liner: 'meter per task, then: right-size models with eval proof, cache the stable prefix, put the context on a diet, batch the async, memoize the repeated, fine-tune the narrow-and-huge — and only then discuss GPUs; the cheapest token is the one that never needed sending.'

The methods, practically:

1. SFT (supervised fine-tuning): train on input→output pairs — the workhorse: format/style/taxonomy consistency, domain task performance, distilling a big model's behavior into a small cheap one (generate training data with the frontier model, tune the small one — the standard cost play)
2. Preference optimization (DPO and kin): train on chosen-vs-rejected pairs — tone/judgment/safety-boundary shaping where 'good' is comparative, not absolute
3. LoRA/PEFT: low-rank adapters instead of full-weight updates — cheaper training, swappable per-task adapters on one base; the practical default for self-hosted tuning
4. RLHF proper (reward models + RL) — know it as what providers do at scale; almost never what you do

When fine-tuning is right: narrow + high-volume + stable (the arithmetic from the cost question), strict output-format/style consistency prompting can't reliably hold, latency/cost demanding a small specialized model, and behavior teaching ('respond like our senior support agents' — with thousands of examples of them doing it).

When NOT (the failure stories):

1. Knowledge injection — the classic mistake: tuning to teach facts (product docs, policies) produces confident staleness: the facts change, the weights don't; no citations; no ACLs. The team that fine-tuned on their docs and then couldn't answer 'why did it say that?' or update for the new pricing — that's a RAG use case wearing a fine-tuning costume
2. Fixing what a better prompt fixes: tuning before exhausting prompting + few-shot + structured outputs burns weeks on what an afternoon of prompt iteration achieves — always baseline: 'what does the best prompt on the best available model score on our eval?' Tuning must beat that, not the naive prompt
3. Tiny/dirty datasets: hundreds of inconsistent examples teach inconsistency; the labeling/curation effort is the project (data quality > method choice, every time)
4. Fast-moving tasks: requirements shifting monthly means perpetual retraining lag — prompts deploy in minutes, tuned models in days-weeks
5. The maintenance tail people forget: a tuned model is a versioned artifact with a lifecycle — base-model upgrades orphan your tune (retrain on the new base, re-eval), and you now own regression testing across your model zoo

The decision sequence (the interview answer): prompt engineering → few-shot → RAG (if it's knowledge) → structured outputs → then fine-tune if evals still show a gap AND volume justifies the lifecycle — with distillation-to-small-model as the most commonly justified case (cost, not capability).

One-liner: 'fine-tune behavior, never facts; only after prompting + RAG plateau on a real eval; only for narrow-stable-high-volume tasks; and budget for the lifecycle — most fine-tuning urges are either a prompt iteration or a RAG pipeline in disguise.'

When multi-agent genuinely wins (be skeptical first — the honest opener): a single capable agent with good tools beats a committee for most tasks; multi-agent earns its complexity when:

1. Context isolation: subtasks each need big contexts that would overflow/pollute one window (research across 50 documents → per-document readers feeding a synthesizer)
2. Parallelism: genuinely independent subtasks (scan 20 repos simultaneously)
3. Role/permission separation: the reviewer agent shouldn't share the author agent's context or write-access (fresh-eyes code review, adversarial red-team/blue-team evaluation) — separation as a feature, not an accident
4. Different cost tiers per role (cheap workers, expensive planner)

The patterns:

1. Orchestrator-workers (the production default): a planner decomposes, dispatches to scoped workers (own context, own tools, narrow brief), integrates results. Key discipline: task specs are contracts (objective, inputs, output format, budget) — vague dispatches produce garbage integration
2. Pipeline: fixed stages (extract → draft → critique → finalize) — deterministic flow, easy to eval per-stage; the right shape when the workflow is known (and then each stage is barely an 'agent' — which is fine and cheaper)
3. Debate/critique pairs: generator + verifier with different instructions (or models) — catches errors the generator is blind to; the cheapest quality upgrade in the pattern family
4. Handoff/routing: a triage agent classifying into specialist queues — really just routing + single agents; call it what it is

The failure modes (why most multi-agent systems disappoint):

1. Error propagation: worker hallucination enters the orchestrator's context as fact — compounding confidently. Countermeasures: verification gates between stages, workers returning evidence/citations not just conclusions, orchestrator instructions to treat worker output as reports-to-check
2. Context fragmentation: the sub-agent lacks the constraint that lived in the parent's context ('the fix must not touch the public API') — result: locally-correct, globally-wrong work. The task-spec contract exists exactly for this; when specs can't carry the needed context, the task shouldn't have been decomposed
3. Coordination overhead eats the parallelism: N agents × M turns of clarification; token costs multiply (every hop re-reads context); latency compounds through the chain — measure end-to-end task cost vs the single-agent baseline, and be prepared for the boring answer
4. Infinite-loop/livelock choreographies (two agents politely deferring forever) — hop budgets, cycle detection, global timeouts: the harness disciplines apply per-agent and to the ensemble
5. Debugging opacity: a wrong final answer traceable through 5 agents' contexts — trace the ensemble as one distributed trace (span per agent-turn) or forensics is hopeless

One-liner: 'multi-agent is a tool for context isolation, parallelism, and permission separation — not a default architecture; dispatch with contract-grade task specs, verify between hops, budget the ensemble, trace it like a distributed system, and always benchmark against the single strong agent you're trying to beat.'

The problem: downstream code needs parseable, valid data; the model emits plausible text. Bridging that gap reliably is bread-and-butter production AI engineering.

The mechanism stack, strongest first:

1. Constrained decoding / native structured outputs: provider features that guarantee schema-conformant JSON (the sampler literally can't emit invalid tokens — grammar-constrained generation). Where available for your model/shape, use it — guaranteed-parse beats validate-and-retry
2. Tool/function calling: define tools with JSON schemas; the model emits structured calls — the native pattern for agent actions, generally well-trained and reliable for shape (arguments parse) though not sense (arguments can still be wrong — validation is about values now, not syntax)
3. Validate-repair loops (the fallback pattern): parse → schema-validate (Pydantic/Zod) → on failure, re-prompt with the error ('your output failed: field X must be enum of [...]; emit corrected JSON only') → bounded retries (1-2; success rates drop fast after) → dead-letter with the trace for human review. Never silently default on parse failure — silent defaults are data corruption with extra steps

The design disciplines:

1. Schema design is prompt design: descriptive field names + descriptions in the schema (the model reads them), enums over free strings wherever the space is closed, required-vs-optional deliberate, and flat-ish structures (deep nesting degrades adherence). An explanation field placed before the decision field measurably improves the decision (the model reasons on the way to the answer)
2. Semantic validation beyond syntax: the JSON parses — are the values sane? Ranges, referential checks (the cited ticket exists?), cross-field consistency, idempotency keys on action-shaped outputs. Treat model output as untrusted input — the same input-validation posture as any API boundary
3. Tool-calling reliability specifics: tight tool descriptions (when to use, when not), few tools beat many (selection accuracy degrades with tool-count — consolidate or route hierarchically), parallel-call handling, and error messages designed for the model (a tool returning 'ERR_42' teaches nothing; 'file not found — did you mean src/config.yaml?' course-corrects the loop)
4. Determinism engineering: temperature 0 for extraction/classification (still not perfectly deterministic — accept statistical determinism), seeds where offered, and the eval suite measuring adherence rates per schema — schema-validity is an SLO (99.5%+ achievable with the stack above)

One-liner: 'constrain generation where the platform allows, validate syntax and semantics always, repair with bounded error-carrying retries, design schemas like prompts and tool errors like teaching — you can build deterministic contracts on a probabilistic core, but only by treating every model output as untrusted until proven parseable and sane.'

Where agents deliver today (ranked by evidence, not hype):

1. The verified inner loop: implementation against existing tests, test-writing, refactors with characterization coverage, migration mechanics — anything where the codebase itself verifies the work (compile + test = the hallucination detector). This is why agent productivity correlates with your test quality — a self-reinforcing investment
2. Review-shaped work: PR analysis, bug-pattern spotting, diff summarization — high-volume, bounded, human-confirmed
3. Archaeology: 'where is X handled, what breaks if I change Y' — codebase Q&A that used to cost a senior's afternoon
4. Ops toil: log/incident triage drafts, runbook execution assistance, dependency-bump PR review at fleet scale
5. Weakest: novel architecture, deep cross-system design, taste — the judgment tiers stay human (for now, honestly)

How the workflow actually changes:

1. Review becomes the bottleneck and the skill: more code, generated faster, needing calibrated human judgment — teams need review-capacity strategy (and junior-growth strategy: juniors who never write the easy code need deliberate skill-building paths — an org-design problem arriving now)
2. Specs and tests appreciate: agents amplify clarity — a well-specified task with good tests gets done; vague tickets produce confident wrong code faster than before. Writing verifiable intent becomes the leverage skill
3. Small-PR/stacked-flow, CI speed, and merge-queue discipline all matter more (agents generate change volume; the delivery pipeline is the constraint), and provenance questions arrive ('which commits are agent-authored?' — attribution/policy per org appetite)

What the platform team owns (the question's real center):

1. The paved road: blessed agent tooling with org config (models, MCP servers for internal systems, permission defaults), so 40 teams don't individually negotiate access and safety
2. Guardrails as infrastructure: sandboxed execution environments, scoped credentials (agents get workload identity, never engineers' tokens), egress control, secret-scanning on agent outputs, branch protections unchanged (agents ship via PR like everyone)
3. Context infrastructure: CLAUDE.md-style repo conventions, MCP servers for the service catalog/CI/observability (the agent that can read the runbook and query the build is 10x the agent that can't), docs freshness (agents consume docs voraciously — stale docs now actively mislead at scale)
4. Measurement honesty: DORA-style outcomes (lead time, CFR) per adoption cohort — not lines generated (the vanity metric); the goal is delivery improvement, and the data keeps the program honest against both hype and fear
5. Cost/budget governance per the cost-engineering playbook — coding agents are token-hungry; per-team attribution from day one

One-liner: 'agents deliver where verification exists — so invest in tests, specs, and review capacity; the platform team's job is paved-road tooling, sandboxed least-privilege execution, context infrastructure (MCP to internal systems), and outcome-honest measurement — the orgs that win treat agent enablement as platform engineering, not tool procurement.'

The API case (start here — it's the right default): zero infra, frontier-quality models, instant scaling, per-token pricing with no idle cost, and the provider absorbs the brutal parts (GPU capacity games, inference optimization, model updates). The costs: data-governance negotiation (though enterprise terms/VPC endpoints/regional options have matured), rate limits as your capacity ceiling, per-token costs at scale, and model-lifecycle dependency (deprecations force migrations on their schedule).

The self-hosted case (open-weights on your GPUs): justified by — hard data-residency/air-gap requirements (the compliance-driven case, often decisive on its own), sustained high utilization where the math genuinely flips (do it honestly: GPU + engineers + utilization risk vs API bill — the crossover is higher than enthusiasts claim because idle GPUs bill anyway while APIs don't), fine-tuned small models for narrow tasks (the distillation play — where self-hosting shines), and latency/locality edge cases.

What self-hosting actually costs operationally: an inference platform (vLLM/TGI/TensorRT-LLM), continuous batching + KV-cache tuning, GPU capacity planning against spiky demand (the autoscaling story is much worse than CPU — model load times are minutes, capacity is quantized in whole expensive GPUs), model lifecycle (evals, upgrades, security patches for the serving stack), and the team — realistically 2-4 engineers for production-grade inference; below that staffing, the 'savings' are an illusion paid in reliability.

The hybrid architecture (the mature answer):

1. An internal gateway as the single front door: all AI traffic through one layer owning — provider routing (task → model mapping in config, not code), API-key custody (apps never hold provider keys), per-team metering/budgets, caching, retries/failover, and audit logging. This single component is what makes every later decision (switch providers, add self-hosted, cascade models) an ops change instead of an every-team migration
2. Routing policy by task class: frontier API for the hard/agentic tier; self-hosted fine-tuned small models for high-volume narrow tasks; batch APIs for async; the sensitive-data tier pinned to the compliant path (regional API or on-prem) by policy in the gateway, not developer memory
3. Portability hygiene: prompts and evals maintained per-model-family where they matter (prompts don't port cleanly — an untested fallback is fiction), OpenAI-compatible serving interfaces to keep app code provider-agnostic, and the eval suite as the arbiter of any switch

One-liner: 'default to APIs, self-host for compliance or eval-proven economics at sustained volume, and put a gateway in front of everything from day one — the gateway turns model strategy into configuration, and the eval suite turns switching from a rewrite into a decision.'

The discipline: localize before fixing — RAG failures decompose into retrieval failures and generation failures, and the fixes don't overlap. Pull the traces for the reported cases (you have full traces — query, retrieved chunks, prompt, response) and classify:

Stage 1 — was the right material retrieved? Look at the chunks that entered the context:

1. Right doc absent from the index → ingestion bug (connector missed it, ACL over-filtered, deletion pipeline over-deleted) or freshness lag (doc updated yesterday, index is weekly). Check the index directly for the known-correct source
2. In the index, not retrieved → the retrieval-quality bucket: query-document vocabulary mismatch (users say 'can't log in', docs say 'authentication failure' — query rewriting/expansion helps), exact-term queries losing to semantic search (the hybrid-search gap — error codes, product names), chunking that split the answer across fragments (the table whose header lives in another chunk), or embedding-model weakness on your domain. Diagnose with a retrieval golden set (query → known-best-docs pairs, measure recall@K) — build it now if it doesn't exist; it converts vibes into a metric
3. Retrieved but ranked below the cutoff → K too small, reranker absent or miscalibrated

Stage 2 — right material, wrong answer? Generation-side classification:

1. Ignored the context (answered from priors despite the docs) → grounding-instruction strength, context position (buried mid-window — the lost-in-the-middle effect; put the best chunks adjacent to the question), or contradictory chunks confusing it (retrieval returned v1 and v3 of the policy — a dedup/versioning problem upstream)
2. Misread the context (right doc, wrong interpretation — tables, negations, multi-hop synthesis) → chunk formatting (preserve structure), model tier for the synthesis complexity, or the task needs decomposition
3. Over-abstained or over-answered relative to what the context supports → instruction calibration + citation-requirement tightening

Stage 3 — was the question even answerable? A real bucket: the docs don't contain the answer, the answer is genuinely ambiguous, or the user's premise is wrong — the fix is corpus gaps (feed to the docs team — the RAG system as documentation-quality radar) and abstention UX, not pipeline tuning.

The systemic outputs of the exercise: every diagnosed case lands in the eval suites (retrieval golden set or generation graded set), the failure distribution drives investment (usually: ~50%+ retrieval-side, which surprises teams who've been prompt-tuning for weeks), and a dashboard split by stage (retrieval recall trend, grounding rate, abstention rate) so next month's 'wrong answers' report starts localized.

One-liner: 'split every failure into retrieved-wrong vs generated-wrong vs unanswerable — trace-driven, against golden sets — and fix the stage that's actually broken; most "the model is dumb" reports are retrieval bugs, and most retrieval bugs are chunking, hybrid-search gaps, or a stale index.'

The layered model (defense in depth, each layer honest about its miss rate):

1. Input layer: injection-attempt detection (classifier + heuristics on user input and — critically — on retrieved/tool content, the indirect vector), PII detection/redaction where policy demands (before the tokens leave your boundary, if using external APIs), topic/scope filters for the product's charter ('this assistant discusses our products' — scope enforcement beats content moralizing for enterprise tools), rate/abuse patterns
2. Model-level steering: system-prompt policy (necessary, insufficient — it's guidance, not enforcement), provider safety settings tuned to context (a code-security tool needs to discuss vulnerabilities — default consumer safety tiers over-refuse; calibrate per use case)
3. Output layer: content classification before render (severity-tiered: block / soften / flag-for-review), PII/secret leakage scanning (the model echoing a credential from context), grounding checks for RAG (claims-vs-source verification on high-stakes answers), format/schema validation, and brand/legal pattern checks (promises, guarantees, advice disclaimers per compliance)
4. Action layer (for agents — the layer that actually has teeth): tool permissioning, approval gates, budgets, egress allowlists — capability control is deterministic in a way text filtering never is; when the stakes are real, constrain what the system can do, not just what it can say

The 'without lobotomizing' engineering (the hard half):

1. Measure both error directions: every filter has false positives — track over-refusal/over-blocking as an SLO alongside miss rate (a benign-prompt block rate above ~1-2% actively teaches users to distrust and circumvent). A guardrail eval suite includes benign-but-edgy cases that must PASS — red-team for misses, green-team for over-blocks
2. Tier by consequence, not uniformly: the internal docs assistant and the customer-facing advice surface get different strictness; draft-with-human-review flows tolerate more than auto-send. Uniform maximum-strictness is how products die politely
3. Graduated responses: block is the last resort — prefer soften/rewrite, add disclaimers, route to human, or answer-with-caveats; binary block/allow wastes the middle ground where most real cases live
4. Fast appeal/feedback loops: users flagging wrong blocks feeds the tuning set weekly; a guardrail system without a feedback pipeline calcifies at its launch-day error rates
5. Latency budget: filters add serial hops — run input checks parallel to retrieval, use fast classifier tiers with async deep-checks post-response for the audit trail (block-before-render only where consequence demands)

Governance wrapper: policies versioned as code (the block/allow rules are config with review + rollback), decision logging for every triggered rail (auditability + tuning data), and a cross-functional owner group (product + legal + security) with an explicit exception path — guardrails are a product surface with stakeholders, not a security checkbox.

One-liner: 'layer input, output, and action controls with the action layer carrying the real enforcement; measure over-blocking as seriously as misses; tier by consequence; prefer graduated responses to binary blocks — a guardrail system's quality is its error rates in both directions, and its politics are half the engineering.'

The questions you must answer crisply (because every enterprise deal now asks): does our data train your models? who can see prompts/outputs? where does data go, geographically and organizationally? how long is anything retained? can we audit and delete?

The boundaries to engineer, layer by layer:

1. Provider-bound data: enterprise API terms (the baseline: API inputs/outputs not used for training — verify per contract tier, not blog posts), zero-data-retention options where offered, regional endpoints for residency, VPC/private connectivity where the posture demands. Document the actual data path — 'prompt → TLS → provider region X, retained N days for abuse monitoring, not trained on' — because that sentence is the questionnaire answer
2. Your own boundaries (usually the weaker link): prompts and traces are data stores containing customer data — the observability pipeline (traces with full prompts!), the eval sets (sampled from prod!), the memory systems, and the RAG indexes all inherit data-classification obligations: encryption, access control (who on your team reads traces?), retention policies, and deletion propagation (a GDPR erasure request must reach traces, eval sets, memory, and vector indexes — the deletion-pipeline completeness question that catches most teams)
3. Tenant isolation in AI paths: cross-tenant leakage vectors unique to AI — shared vector indexes without tenant filters, prompt caches keyed across tenants, memory recall crossing user boundaries, few-shot examples sampled from other customers' data (a real incident pattern: customer A's ticket text appearing as an example in customer B's session). Hard isolation at the store level, not filter-discipline hope
4. Fine-tuning provenance: training sets are derived data — record lineage (which customers' data, under which consent/terms), because 'can you delete my data' includes 'from your fine-tuned weights?' — where the honest answer is retrain-without-it (expensive) or don't-train-on-customer-data (the policy most orgs should pick and state)
5. Internal-use AI governance: employees pasting customer data into consumer AI tools is the shadow-IT vector — sanctioned tools with enterprise terms + DLP guardrails beat prohibition memos

The audit-ready artifacts: data-flow diagrams for every AI feature (the thing you draw in the security review), sub-processor documentation (the provider is one), model/prompt/dataset version registry (what was running when), decision logs from guardrails, and the deletion-propagation runbook with test evidence.

One-liner: 'the provider contract is the easy half — the hard half is that your own traces, evals, memories, and indexes are customer-data stores needing classification, isolation, retention, and deletion-propagation; engineer tenant boundaries structurally and keep the data-flow diagram current, because the security questionnaire is now part of the product.'

The latency anatomy: total = network + queue + prefill (input processing — scales with context length; determines time-to-first-token) + decode (output generation — tokens/sec, scales with output length) + your pipeline overhead (retrieval, filters, tool hops). Optimizing the wrong phase is the standard mistake — measure TTFT and tokens/sec separately.

The levers, by phase:

1. TTFT (perception dominates here):
   • Prompt caching — cached prefixes skip most prefill: the single biggest TTFT lever for repeat-context patterns (agents, chat with big system prompts, doc-grounded sessions)
   • Context diet (prefill is linear-ish in input), retrieval parallelized with everything possible, smaller/faster model tiers where the task allows, provider/region proximity
2. Decode: output-length discipline (max_tokens + concise instructions — the cheapest win), faster model tiers, and speculative decoding where you control serving (a draft model proposes, the big model verifies — 2-3x decode speedups, quality-neutral)
3. Pipeline: filters async or parallel (not serial pre-hops), tool-call round-trips minimized (parallel tool execution, batched calls), and the agent-specific killer — loop depth: each turn pays full latency; planner efficiency (fewer, better tool calls) is a latency feature

The UX engineering (where products win despite physics):

1. Stream everything: first token at 800ms with visible progress feels faster than a 6-second complete answer — streaming is non-negotiable for interactive surfaces; structure responses so the valuable content front-loads (answer first, elaboration after — prompt for it)
2. Progressive disclosure for agents: show the plan, then live tool-activity ('searching tickets... reading deploy log...') — perceived latency collapses when users watch work happen; silent 30-second spinners kill trust at 10
3. Optimistic and phased responses: instant acknowledgment + fast draft (small model) refined in place (big model) for suitable UX; precompute/prefetch where intent is predictable (the 'likely next question' warm path)
4. Async by design where latency can't be beaten: long agent tasks become jobs with notification ('I'll ping you when the analysis is done') — converting a latency problem into a workflow feature; the wrong move is pretending a 90-second task is interactive
5. Timeout/fallback ladders: slow-model timeout → fast-model answer with a quality note, degraded-mode answers over spinners-forever

The SLO framing: P50 TTFT < 1s, P95 < 3s for interactive surfaces; tokens/sec ≥ human reading speed (~15-20 tok/s) so streaming never stalls visibly; per-phase dashboards so regressions localize (a TTFT jump = prefill/queue/cache-hit-rate story; a throughput drop = provider/decode story).

One-liner: 'split TTFT from throughput, cache the prefix, diet the context, stream the output, front-load the value, show agents working, and make genuinely-long tasks async — LLM latency is half physics you optimize and half perception you design.'

The charter (what the team is FOR): make AI capability safe, cheap, and fast to adopt for every product team — platform, not gatekeeper: paved roads + guardrails + shared infrastructure, with product teams owning their features.

The operating model: small senior team (4-8), explicitly not the team that builds every AI feature (the bottleneck anti-pattern) and not a research lab (the irrelevance anti-pattern) — the Kubernetes-platform-team playbook applied to AI.

First-quarter roadmap (in dependency order):

1. The gateway (week 1-4): single front door for all model traffic — key custody, provider routing, metering/attribution, budgets, audit logging, caching. Every later capability (cost program, model switching, compliance answers) hangs off this; it's also how you discover the shadow-AI already in the building (there is always shadow AI)
2. Governance that unblocks (parallel): the data-classification-to-provider-path policy matrix ('this data class may use this provider tier via this endpoint'), enterprise agreements with 1-2 providers, and the security-review fast-path for AI features — teams are stalled on permission ambiguity more than technology; clear yes-paths are the quarter's biggest velocity unlock
3. Observability + eval scaffolding (week 4-8): tracing as default via the gateway/SDK, an eval-harness template with judge tooling and CI integration — the 'evals as required checks' culture starts with the platform making it a one-afternoon setup instead of a research project
4. Two lighthouse enablements (week 6-12): pick two product teams with real use cases; pair-build on the platform primitives — the lighthouses debug the paved road, generate the internal case studies, and keep the platform honest against real needs (platform-in-a-vacuum is anti-pattern #3)
5. The paved-road SDK + docs: prompt/version management, structured-output helpers, guardrail defaults, cost dashboards per team — packaged so the third team adopts without meetings

Deliberately deferred: self-hosted inference (until compliance or eval-proven economics force it), fine-tuning infrastructure (until a lighthouse hits the genuine need), multi-agent frameworks (let single-agent patterns mature first), building a chat UI for everything (the org's 15 UI experiments will consolidate later; infrastructure first).

The anti-patterns to dodge (name them — they're half the interview value):

1. The approval-committee trap: platform-as-gatekeeper makes AI adoption slower than shadow IT — then loses to shadow IT; be the fastest safe path or be routed around
2. The demo-team trap: building flashy prototypes instead of infrastructure — impressive quarter one, irrelevant by quarter three
3. The premature-standardization trap: picking The One Framework before the org has learned its patterns — standardize interfaces (gateway, tracing, evals), stay flexible on frameworks
4. The metrics-vacuum trap: no adoption/cost/outcome measurement = no defense at budget time; instrument the platform's own value from day one

One-liner: 'gateway first, permission clarity second, evals and tracing third, lighthouses to keep it honest — be the paved road that's faster than going around, defer the heavy infrastructure until evidence demands it, and measure your own adoption like the product it is.'

The regulatory shape (engineering-relevant summary, not legal advice — say this framing out loud): the EU AI Act classifies systems by risk tier — prohibited practices (social scoring, manipulative systems), high-risk (employment decisions, credit, essential services, biometrics — heavy obligations), limited-risk (transparency duties: disclose AI interaction, label synthetic content), minimal (most internal tooling). GPAI/foundation-model duties land mostly on providers — but deployers have their own obligations, and sectoral regimes (financial services, healthcare, employment law) stack on top regardless of geography.

What engineering actually has to build:

1. An AI system inventory (the foundational artifact): every AI feature/system with its risk classification, purpose, data flows, models used, human-oversight design — regulators, auditors, and your own lawyers all start here; orgs that can't enumerate their AI can't comply with anything. This is the service-catalog discipline applied to AI
2. Risk-tier-driven engineering requirements: high-risk systems need — documented risk assessment, data-governance evidence (training/eval data provenance and bias analysis), human oversight that's real (a human who can understand, intervene, and override — not a rubber-stamp click-through; the oversight UX is an engineering deliverable), accuracy/robustness testing evidence (your eval suites, formalized), logging sufficient to reconstruct decisions (the tracing you built, with retention), and registration/conformity paperwork
3. Transparency mechanics: AI-interaction disclosure in product UX, synthetic-content labeling where applicable (provenance standards like C2PA arriving in the toolchain), explanation capabilities proportionate to decision impact
4. The documentation pipeline: model cards / system cards per deployment, decision logs, incident records (serious incidents in high-risk systems have reporting duties), and change management (a model/prompt update to a high-risk system is a controlled change with re-assessment triggers)

The pragmatic engineering posture:

1. Most of the burden lands on systems making consequential decisions about people — classify honestly and early: the internal docs assistant is minimal-risk; the resume-screening feature is high-risk and possibly shouldn't be built as designed. The classification conversation belongs in design review, not post-launch legal panic
2. The infrastructure you built for engineering reasons is the compliance substrate — tracing (decision logs), evals (accuracy evidence), gateway metering (inventory), guardrail decision records (oversight evidence) — compliance becomes report-generation over existing telemetry rather than a parallel bureaucracy, if the platform was built with this in mind
3. Geographic reality: extraterritorial reach (EU users = EU rules), US sectoral + state patchwork, and procurement pressure (enterprise customers demanding AI-governance evidence contractually — the questionnaire arrives before the regulator does)

One-liner: 'inventory every AI system, classify by decision-consequence, and let the high-risk tier drive real engineering — genuine human oversight, documented evals, reconstruction-grade logging — built on the observability and eval infrastructure you needed anyway; the orgs in trouble are the ones that can't list their AI systems, not the ones with imperfect paperwork.'

The insight that reframes it: an AI system's behavior = code × prompt × model × index × config — five independently-versioned artifacts, and a behavior regression can come from any of them (including the model changing under you). Classic CI/CD versions one of the five; AI delivery engineering versions the composite.

Versioning each artifact:

1. Prompts: in git, always — templates with schema-validated variables, reviewed via PR, with the eval suite as the required check (the prompt-in-a-dashboard-textbox is config drift with maximal blast radius). Prompt packages (system prompt + few-shots + tool descriptions) version as units — they're coupled
2. Models: pin snapshot versions where providers offer them (never bare model-latest in prod); record the exact model string in every trace; provider deprecation calendars tracked like dependency EOLs with migration evals scheduled ahead
3. Indexes (RAG): the sneaky one — embedding-model version + chunking config + corpus snapshot define an index generation; a re-embed or chunking change is a deployment (blue-green the index: build the new generation alongside, eval against the retrieval golden set, cut over, keep the old for rollback)
4. Configs: temperature, K, thresholds, guardrail rules, routing policies — config-as-code with the same review path; a temperature change is a behavior deploy

The deployment machinery:

1. A release = a pinned composite (prompt v41 + model snapshot X + index gen 12 + config v7) — recorded per-request in traces, so 'what was running when this went wrong' has an exact answer
2. Progressive delivery for behavior: eval gate (offline suite green) → shadow traffic (new composite runs parallel, outputs compared, users see old) → canary % with quality and cost/latency monitors → ramp. Shadow mode is disproportionately valuable in AI because output diffs are inspectable before any user exposure
3. Rollback = repoint to the previous composite — which requires keeping old prompt versions deployable, old index generations warm (storage cost, worth it for tier-1), and tested fallback paths. The un-rollbackable change is the provider retiring a model — hence deprecation-calendar discipline and portable eval suites
4. The drift tripwire: continuous canary evals against prod (same golden prompts, daily) — catching both your regressions and theirs (provider-side model updates); alert on eval-score deltas like error-rate deltas

Organizational mechanics: behavior changes get changelogs users/support can read ('the assistant now refuses X, cites Y more'), experiment flags for composite variants (A/B at the gateway), and one owner per composite (the accountable team for 'the assistant got worse this week').

One-liner: 'version the composite — prompt, model, index, config — pin it per-release, record it per-trace, ship it through eval gates and shadow traffic, and keep last-known-good deployable; when behavior lives in five artifacts, delivery engineering means never having to ask which one changed.'

The legitimate uses (growing fast):

1. Eval-set bootstrapping: generating test cases/variations from seed examples — expanding coverage cheaply (with human review as the quality gate; a generated eval set nobody validated tests nothing)
2. Fine-tuning data: distillation (frontier model generates training pairs for the small model — the standard cost play), augmentation (paraphrases, edge-case variants of scarce real data), and privacy-motivated synthesis (statistically-similar stand-ins where real data can't be used — with the honest caveat that utility/privacy trade-offs need measurement, not assumption)
3. Load/integration testing: realistic-shaped traffic for AI features without real customer data
4. Red-teaming: generated adversarial inputs (injection variants, jailbreak permutations) at a scale humans won't write

The risks that need engineering:

1. Quality collapse in loops: models trained on model output inherit and amplify its artifacts — distribution narrowing, error reinforcement (the 'model collapse' research made vivid: recursive generations degrade). Discipline: provenance-tag every synthetic record, cap synthetic ratios in training mixes, keep a real-data golden core, and never let generated data enter training unlabeled
2. Eval contamination (the subtle killer): if the same model family generates your eval set and takes the eval, scores inflate (self-preference bias — models grade their own dialect kindly); judges from a different family than the system-under-test, human-validated eval cores, and skepticism toward suspiciously-high scores on synthetic benchmarks
3. Bias laundering: synthetic data inherits the generator's biases while looking neutral — 'we used synthetic data' is not a fairness answer; bias evals run on synthetic sets same as real
4. Provenance obligations: synthetic content reaching users (drafts, summaries) or training pipelines carries labeling duties (policy and increasingly regulation — the C2PA/labeling machinery), and your data-governance answers must distinguish real from generated lineage

The pipeline disciplines: provenance metadata as a schema requirement (source: human/synthetic/model-X-version), synthetic-fraction dashboards for any training/eval dataset, human-review sampling rates by consequence tier, and the contamination check in every eval design review ('who generated the questions, who's answering, who's grading — are any of those the same model family?').

One-liner: 'synthetic data is a power tool — distill, augment, red-team, bootstrap evals — with three failure modes to engineer against: loops that collapse quality, evals that grade themselves, and bias wearing a neutral costume; provenance tags, family-separation between generator/judge/subject, and a human-validated core are the guardrails.'

The honest assessment first (the senior move is calibrating the headline): 'replaced 30% of support' typically means 'deflected 30% of tickets' — which is mostly the easy tier (password resets, order status, FAQ-shaped questions) that better docs and workflows would also have deflected. The claim conflates deflection with resolution quality; nobody publishes their misresolution rate, re-contact rate, or CSAT delta. Match the ambition, audit the arithmetic.

The plan I'd actually run (one quarter, honest scope):

1. Weeks 1-2 — the data reality check: mine the ticket corpus: volume by intent, resolution patterns, which intents have documented, stable resolutions (automatable) vs judgment/emotion/exception handling (not yet). The distribution decides everything — typical finding: 25-45% of volume is genuinely FAQ/status/how-to shaped. Also audit the knowledge base: an assistant grounded on stale docs automates wrong answers at scale — KB refresh is usually the hidden prerequisite
2. Weeks 2-6 — build the containment-tier assistant: RAG over the (refreshed) KB + order/account lookups via scoped read-only tools; draft-first for anything consequential (the agent proposes, human sends — for the first month everything human-reviewed: this is your eval-data factory); hard escalation paths (frustration signals, out-of-scope intents, explicit request → human, with full context handoff so the customer never repeats themselves — the #1 CX failure of rushed deployments)
3. Weeks 6-12 — measured rollout: canary on the top-3 intents (highest volume × best docs), full tracing + weekly eval grading, expand intent-by-intent as each clears quality bars. The metrics that matter: true resolution rate (no re-contact within 7 days), CSAT on AI-handled vs human-handled, escalation quality, misresolution incidents — not deflection alone (deflection without resolution is customers giving up, which looks identical on the deflection dashboard)
4. The org design in parallel: support agents move up-tier (complex cases, QA-grading the AI, curating the KB — their expertise becomes the training signal; framing this as augmentation-with-career-path vs replacement determines whether your best agents stay to make it work), and support leadership co-owns the quality bar

The quarter-end honest forecast: 20-35% of ticket volume handled or drafted with quality parity on the covered intents — real and valuable ($ per deflected ticket × volume is an easy business case) — with the harder tiers as a roadmap, not a promise. The competitor's '30%' is reproducible; the part they didn't publish (quality erosion risk, escalation experience, KB maintenance burden) is where your version gets to be better.

What they're testing: whether you translate executive ambition into a scoped, measured, reversible program — data audit before architecture, draft-mode before autonomy, resolution metrics before deflection theater, and the workforce story handled as deliberately as the technology.

The shift to name: the first AI wave bolted chat onto products; the maturing pattern embeds AI inside system workflows — no chat box, no human prompt, AI as a processing stage in event-driven architectures.

The patterns:

1. AI as a pipeline stage: events flow through model-powered steps — ticket arrives → classify/enrich/route; document lands → extract/summarize/index; alert fires → triage/correlate/draft-context. Architecturally: a consumer on the queue like any other, with the AI-specific additions — confidence thresholds routing low-certainty items to human queues, schema-validated outputs, per-stage eval monitoring. The chat UI never existed; the workflow got smarter
2. Background agents: long-running tasks dispatched to agents that work autonomously and report back (the coding agent on a ticket, the research agent on a brief, the remediation agent on an incident class) — the engineering center of gravity moves to: job orchestration (queues, retries, checkpointing for multi-hour tasks), budget envelopes per task, artifact-based outputs (PRs, reports — reviewable deliverables, not ephemeral chat), and completion notification flows. It's batch-processing architecture where the worker happens to reason
3. Proactive/triggered AI: the system notices and acts — anomaly narratives generated when metrics drift, stale-doc PRs when code diverges from docs, weekly digests synthesized from activity. Design constraint: proactive output competes for human attention — precision requirements are higher than reactive (a chat answer wrong once annoys; a weekly wrong report trains ignoring), so trigger thresholds and suppression logic are the hard part
4. AI-mediated interfaces between systems: unstructured→structured bridges (emails→orders, logs→incidents, contracts→terms) where the model is an adapter component with a schema contract — the most production-mature pattern because verification is crisp

The architectural consequences:

1. Async-first: latency tolerance transforms the cost/model calculus (batch APIs, bigger models affordable, retries trivial) — the strongest argument for designing AI out of the request path
2. Idempotency + determinism seams: AI stages are nondeterministic; the system around them needs replay-safety (idempotency keys on actions, checkpointed progress, at-least-once tolerance) — classic event-driven discipline, more load-bearing than ever
3. Human-in-the-loop as queue design: review queues with SLAs, sampling rates by confidence tier, and feedback capture — the human oversight is an architected flow, not a modal dialog
4. Observability spans the workflow: trace the event through classify→enrich→act stages like any distributed trace, with quality metrics per stage — 'the pipeline got worse' must localize

One-liner: 'the mature pattern is AI as infrastructure — pipeline stages, background workers, and adapters inside event-driven systems: async-first for cost and resilience, schema contracts at every seam, human review as designed queues, and the same eval/tracing discipline per stage — chat was the demo; the workflow is the product.'

The compression toolbox (how capable-but-small happens):

1. Distillation: a large teacher generates outputs/labels; a small student trains to match — transferring behavior (not knowledge-in-general) into a model 10-100x cheaper to serve. This is the economic engine behind every 'fast' model tier you use, and your own play for narrow high-volume tasks
2. Quantization: store/compute weights at lower precision — FP16 → INT8 → INT4: each halving cuts memory (and memory-bandwidth, which is decode speed) roughly in half, with quality degradation that's small at 8-bit, task-dependent at 4-bit (evaluate on your eval set, not the leaderboard — quantization hits some capabilities unevenly, tool-calling and math earlier than chat). Runtime families: GPTQ/AWQ (post-training), GGUF ecosystems for CPU/edge
3. Pruning/sparsity + MoE: structural approaches — mixture-of-experts models activate a fraction of weights per token (the 'big model capacity, small model compute' architecture pattern now standard in frontier and open models)

The serving-stack knowledge that matters to platform engineers:

1. The KV cache is the resource: attention state per token per sequence — it's why long contexts eat GPU memory, why concurrent-sequence capacity is bounded, and what PagedAttention (vLLM's contribution) manages like virtual memory: near-zero fragmentation, dramatically higher concurrency. When someone says 'vLLM', this is mostly why
2. Continuous batching: new requests join the batch as others finish (vs static batches that wait for the slowest) — the throughput multiplier that makes GPU serving economical; sequence-length spread across a batch is why P95 latency and utilization fight each other (mixed workloads want pools)
3. Prefill vs decode separation: prefill is compute-bound (parallel over input), decode is memory-bandwidth-bound (sequential) — the two phases want different optimization, and disaggregated serving (separate prefill/decode fleets) is where high-scale inference architecture has gone
4. Speculative decoding: small drafter proposes K tokens, big model verifies in one pass — 2-3x decode speedup, output-identical (verification guarantees it); works best when drafter and target agree often (domain-matched drafters)
5. The capacity-planning reality: throughput quantizes in whole GPUs, model loading takes minutes (autoscaling is sluggish — warm pools over reactive scaling), and utilization economics dominate (an idle GPU bills; the self-hosted-vs-API crossover math from the earlier question lives or dies on sustained utilization)

Why a DevOps interviewer asks this: inference infra is becoming a standard platform workload — the K8s cluster with GPU node pools, the vLLM deployment, the model-weight artifact pipeline (weights are multi-GB artifacts with versioning, integrity, and cold-start implications — registry/cache/pre-pull thinking applies directly).

One-liner: 'distillation shrinks behavior, quantization shrinks precision, MoE shrinks active compute — and serving is KV-cache management plus continuous batching plus the prefill/decode split: know vLLM's tricks and the utilization math, because inference is just another production workload now, with VRAM as the scarce resource.'

First move — treat it as a real incident: severity-classify, open the channel, assign an IC — 'the AI said something' incidents fail when handled as PR problems (no engineering forensics) or as engineering problems (no comms) — it's both, immediately.

The response sequence:

1. Contain (minutes): can this output recur? Options by blast radius: guardrail rule for the specific pattern (fast, narrow), disable the affected capability/intent (medium), kill-switch the feature to fallback UX (broad) — the kill-switch you built into the gateway/feature-flag layer earns its existence here. Screenshot-match to confirm the alleged output is real (fabricated screenshots happen — verify before the mea culpa, without stalling containment)
2. Forensics (hours) — the traces answer everything: find the exact request — full prompt assembly, retrieved chunks, model/composite version, tool calls. The diagnosis usually lands in one of: retrieval fed it wrong/stale content (the KB said the harmful thing!), prompt/guardrail gap (a case class nobody anticipated), injection (someone engineered it — check the input), model behavior drift (composite version changed recently?), or the output was actually within policy and the incident is a policy gap (the org never decided what the assistant should say here — common and uncomfortable)
3. Scope (parallel): query traces for the pattern, not the instance — how many similar outputs, to whom, since when? 'One weird answer' vs 'we've said this 400 times since the index update' are different incidents; proactive customer outreach for the second
4. Communicate: externally — acknowledge fast, factually, without over-promising ('we've disabled X while we investigate'); internally — support gets talking points + the flag to escalate related contacts; legal/comms looped per severity. The trace-backed timeline (what it said, why, to how many) is what makes comms accurate — orgs without tracing improvise their statements and get caught by follow-ups
5. Fix at the layer that failed: KB correction + re-index (if retrieval fed it), guardrail/prompt update through the eval-gated pipeline (panic-editing prompts in prod creates incident #2 — the discipline holds especially under pressure), the new failure case lands in the eval suite (this class can never regress silently again), and injection cases get the security treatment
6. The AI-specific postmortem additions: was this output detectable before the customer saw it (why didn't output filtering/canary evals catch the class)? was the blast radius designed (should this surface have had draft-mode/human review at its consequence tier)? does the incident reclassify the feature's risk tier (the consequence-tiering from the guardrails design gets recalibrated by reality)?

The pre-work this incident audits: kill-switches per AI surface, trace retention with fast search, output-pattern querying (can you actually search 'what else did we say like this'?), a decided policy corpus (so 'within policy?' has an answer), and comms templates for AI-output incidents — the response quality was determined weeks before the tweet.

One-liner: 'contain with the kill-switch, diagnose from traces, scope the pattern not the instance, communicate from evidence, fix through the eval-gated pipeline, and postmortem the system — the tweet is survivable; not being able to answer "why did it say that and how many times" is what turns an incident into a trust collapse.'

Situation: 40-service platform, on-call MTTA fine but MTTR dragged by context assembly — engineers spending the first 20 minutes of every incident gathering: what changed, what's correlated, which runbook applies, who owns what.

Task: a copilot that compresses the context-assembly phase — explicitly not an auto-remediation bot (trust must be earned in layers).

The design:

1. Data sources via MCP servers (read-only, scoped): metrics (PromQL against the incident's service + dependencies), logs (scoped search around the alert window), recent deploys (the CD system — 'what changed' is the highest-value single query), the service catalog (ownership, dependencies, tier), past incidents (search over postmortem corpus — 'have we seen this shape before'), and runbooks
2. The trigger flow: alert fires → copilot assembles an incident brief posted to the incident channel within ~60s: symptom summary, correlated signals ('latency spike coincides with payments-api deploy 14:02, error rate on downstream auth-service also elevated'), top-3 hypothesis candidates with the evidence for each, relevant runbook links, and suggested first diagnostics — every claim linked to its source (the graph, the log query, the deploy) because an unverifiable brief is worse than none
3. Interactive follow-up: the on-call asks in-channel ('show me the p99 by AZ', 'what did the last similar incident conclude?') — the copilot runs the scoped queries and answers with citations
4. Trust boundaries (the design's spine): read-only everything at launch; diagnostic suggestions never executed actions; after earned trust (measured accuracy over a quarter), graduate to gated actions (propose the rollback, human clicks) — the autonomy ladder is explicit and metric-gated, not vibes-gated
5. The injection consideration people miss: log lines and ticket text are untrusted content entering the copilot's context — an attacker who can generate log entries can attempt injection; the copilot's tool scope (read-only, no external comms) is sized so a successful injection reads dashboards, not exfiltrates

Proving it helps (the metric design): A/B by rotation or service cohort — time-to-first-hypothesis, MTTR on copilot-assisted vs not, brief-accuracy grading (did the on-call rate the brief useful? was the top hypothesis right?), and false-lead rate (a copilot that anchors responders on wrong hypotheses adds MTTR — measure the harm case explicitly). Result pattern from real deployments of this shape: 30-50% reduction in diagnosis phase, with the postmortem-search feature (institutional memory made queryable) rated highest by engineers.

What they're testing: scoped-tool design, the autonomy ladder, injection-awareness for ops data, and outcome measurement that includes the harm case — an ops copilot is a trust-engineering project wearing an AI costume.

What frameworks actually give you: loop scaffolding, tool-registration ergonomics, state/checkpoint machinery, multi-agent orchestration primitives, tracing hooks, and provider abstraction — real value, and the same value that's also achievable in a few hundred lines when your needs are narrow.

The case for a framework: speed-to-first-agent (prototyping, hackathons, proving value), teams without deep AI-systems experience (guardrails-by-default beat reinvented mistakes), and complex orchestration needs (checkpointed long-running graphs, human-in-loop pause/resume — the machinery is genuinely fiddly).

The case against (why senior teams often go framework-light):

1. The abstraction tax: frameworks abstract exactly the thing you most need visibility into — the prompt assembly and loop control; when behavior surprises, you debug through framework indirection to find what actually entered the context. Agent debugging is context forensics; anything that obscures the context is a cost
2. Churn risk: the framework layer is the least-stable stratum of the stack (paradigm shifts quarterly); provider APIs and your own tool/eval infrastructure are far more durable investments — coupling business logic to a framework's abstractions is how you inherit a migration per year
3. The 80% case is simple: a loop calling a provider SDK with tool dispatch, validation, and budgets is small, honest code — many production agent teams run exactly this ('the framework is 400 lines we own'), with the effort invested instead in tools, evals, and tracing (the durable layers)

The decision framework:

1. Prototyping/exploration → framework, freely; the sunk cost is small
2. Production, simple-to-moderate agents (tool loop + budgets + tracing) → own harness on provider SDKs, framework-free; own your context assembly
3. Production, genuinely complex orchestration (durable long-running workflows, complex graphs, resumability) → evaluate frameworks as workflow engines (their real competency) with your own context/prompt layer kept explicit — or use an actual workflow engine (Temporal-style) around simple agent steps, which often models the problem more honestly
4. Regardless of choice: the interfaces you standardize are the gateway, tracing schema, eval harness, and tool contracts (MCP) — framework-agnostic seams that make any framework decision reversible

One-liner: 'frameworks buy speed and cost transparency-of-context — prototype on them, but production agents live or die on context control, so own the loop where it's simple, use workflow engines where it's genuinely complex, and invest durably in tools, evals, and tracing rather than the layer that churns.'

The honest taxonomy — what works today:

1. Language interfaces to telemetry: natural-language → PromQL/LogQL/trace queries ('p99 for checkout by region last 6h') — high value, low risk (the query is inspectable before running; wrong queries are visible, not silent). The adoption winner because it lowers the observability skill floor
2. Incident context assembly (the triage-copilot pattern): correlation narratives, change summaries, postmortem search — works because it's retrieval + summarization over verifiable sources, the shape LLMs are reliable at
3. Alert enrichment and dedup narratives: turning 40 correlated alerts into one causal story draft — genuinely useful with the discipline that it's labeled as draft hypothesis, not diagnosis
4. Postmortem tooling: timeline assembly from channel/deploy/alert exhaust, draft generation for human editing — pure toil reduction

What's mostly snake oil (or premature):

1. 'AI finds root cause automatically' — root cause requires causal reasoning over incomplete system models; today's reality is hypothesis ranking (useful!) marketed as diagnosis (dangerous — anchoring effects on wrong hypotheses measurably slow experienced responders)
2. LLM-based anomaly detection on metrics — statistical methods (and simple ones) beat token-predictors at numeric anomaly detection on cost and accuracy; the LLM's role is narrating anomalies statistical systems found, not finding them. Vendors blurring this line are selling the expensive tool for the wrong half
3. Auto-remediation from model judgment — action from probabilistic diagnosis without human gates is how you turn one incident into two; remediation automation should be deterministic runbooks that AI helps select, not AI improvising kubectl

The integration architecture:

1. AI reads observability through the same query APIs as humans (MCP servers over Prometheus/logs/traces) — no shadow data path; scopes and audit like any client
2. Statistical layer does detection (existing anomaly/correlation engines) → LLM layer does narration, correlation-across-sources, and interface — each layer doing what it's actually good at
3. Outputs land where responders work (incident channel, alert annotations) with evidence links mandatory — the unverifiable AI insight is an anti-pattern; every claim links to the graph/log/deploy it derives from
4. Feedback capture (was this narrative right?) → eval corpus → the improvement flywheel, same as every AI system

One-liner: 'LLMs earn their place in observability as interfaces, narrators, and context-assemblers over verifiable telemetry — statistical systems keep the detection job, humans keep the diagnosis authority, runbooks keep the remediation — and any vendor whose pitch skips those boundaries is selling the demo, not the discipline.'

The workload shape first: 50M tokens/day ≈ steady ~600 tok/s average with peaks 3-5x — call it a handful of high-end GPUs' worth of a mid-size model with headroom; small enough that over-engineering is the real risk, large enough that laptop-grade serving won't do.

The architecture:

1. GPU node pools, explicitly modeled: dedicated pool (taints/tolerations), node autoscaling with the honesty that GPU scale-up is minutes (node provision + image pull + model load: multi-GB weights from storage to VRAM) — so capacity strategy is warm-pool-first, autoscale-second: N always-on replicas sized for P50, scale headroom for peaks, batch/deferrable traffic soaked into the valleys
2. The serving layer: vLLM per replica (continuous batching + PagedAttention do the throughput heavy-lifting), one model per replica-set (multi-model-per-GPU only with MIG or clear memory math), OpenAI-compatible endpoints so app code stays provider-agnostic behind the gateway (which routes internal-model vs external-API per task policy — this deployment slots under the existing gateway, not beside it)
3. Model-weight logistics (the underestimated part): weights are versioned multi-GB artifacts — store in object storage/OCI registry with integrity checks, pre-pull/pre-load via init containers or a weight-cache DaemonSet on the GPU pool, and version rollout = blue-green replica sets (load new weights alongside, shift traffic, keep old warm for rollback) — model deploys get the same ceremony as the composite-versioning question demands
4. Request routing realities: load-balance on in-flight sequences per replica, not round-robin (a replica chewing a 100K-token prefill is not equal to an idle one — vLLM exposes queue metrics; use them), sticky-ish routing for prefix-cache locality where session reuse matters, and admission control (queue depth caps → 429 + client retry beats OOM-ing the KV cache)
5. Observability specific to inference: tokens/sec, TTFT, queue depth, KV-cache utilization (the saturation metric), batch occupancy, GPU memory/SM utilization (DCGM), per-team token attribution at the gateway — capacity planning runs on these, not on CPU-era intuitions
6. Failure modes to design for: OOM from long-context stampedes (context-length caps per route, admission control), model-load crash loops (readiness probes that wait for weight load — startup probes with generous windows), a poisoned/corrupt weight artifact (integrity verification before serve), and the noisy-neighbor case if batch and interactive share replicas (separate pools per traffic class — the P95 protection)

The honest scoping caveat: at 50M tokens/day, run the API-vs-self-hosted math annually — this platform makes sense for data-residency reasons or fine-tuned-model serving; as pure cost play the margin is thinner than enthusiasts project once engineering time is priced.

One-liner: 'GPU pools with warm capacity, vLLM behind OpenAI-compatible endpoints under your existing gateway, weights treated as versioned artifacts with blue-green rollout, load-balancing on in-flight work, and KV-cache utilization as your saturation metric — it's a stateful-ish, memory-bound serving tier, and the platform disciplines you already have mostly transfer.'

The deceptively simple question with real architecture inside: the model is stateless — every turn, you re-send whatever history it should remember. Where that history lives and how it's shaped per-turn is a genuine systems design problem.

The storage layer:

1. Conversation store: append-only turn log (user/assistant/tool messages with metadata) in a real database — the source of truth, tenant-isolated, retention-policied (it's customer data — the governance question rides along). Session state (active conversation pointer, UI state) in the fast tier (Redis-shaped); durable history in the store
2. What gets sent per turn ≠ what's stored: the context assembly function decides — recent turns verbatim, older turns summarized, pinned facts always, tool results maybe-truncated — assembled fresh each request. Storing the raw log and assembling views is the right separation; systems that store 'the context' as one mutable blob can't re-shape later

The window-management strategies (the core of the question):

1. Sliding window + summary: keep last N turns verbatim; as turns age out, fold them into a running summary block ('earlier in this conversation: ...') — the workhorse pattern. The known risk: summaries silently drop load-bearing details (the account ID from turn 2) — mitigate by extracting structured facts to a pinned block (entities, decisions, constraints) separate from the prose summary
2. Prompt-caching-aware layout: stable prefix (system prompt, pinned facts, summary) first, volatile recent turns last — cache hits on the prefix cut cost/TTFT dramatically for long chats; this constraint shapes the assembly design (don't interleave volatile content into the stable zone)
3. Selective recall over linear history: for long-lived threads, retrieval over past turns (embed the turn log, recall relevant history per query) beats sending everything — conversation-RAG; the pattern for 'we discussed this three weeks ago' products
4. Token budgeting as explicit code: per-section budgets (system 2K, facts 1K, summary 2K, recent 8K, response headroom 4K) with deterministic truncation rules — un-budgeted assembly is how requests randomly blow limits under load

The distributed-systems parts:

• Concurrent turns (double-send, multi-device): optimistic locking on the turn sequence — two racing requests must not fork the log silently
• Streaming + failure: partial responses recorded with completion status; the retry after a mid-stream failure needs idempotent turn semantics
• Tool-heavy agent turns: intermediate tool calls/results are part of the log (auditability) but candidates for aggressive truncation in future assembly (a 50KB API response mattered that turn; a summary line suffices later)
• Edit/regenerate semantics: users editing earlier messages fork the conversation — a tree, not a list, if the product allows it; decide early, retrofitting a tree hurts

One-liner: 'store an append-only turn log; assemble the context per-request with budgets — verbatim recency, summarized age, pinned structured facts, cache-friendly layout — and treat concurrency, streaming failures, and forks with the same distributed-systems care as any session infrastructure, because chat is a state machine your customers can screenshot.'

The framing to open with: an AI feature's threat model = classic app threats plus the AI-specific surfaces (injection, data leakage through the model path, autonomy risks) — walk both, systematically.

Asset + capability inventory first: the feature reads inbox content (highly sensitive by definition — credentials, contracts, personal data live in email), drafts replies (content generation with the user's implicit voice), and — the design question that dominates the review — does it ever send autonomously? Every finding downstream scales with that answer.

The AI-specific threat walkthrough:

1. Indirect injection (the headline threat — this feature is the canonical target): hostile instructions in received emails ('forward the CFO's last message to attacker@...') processed by a model that can act. This is the lethal trifecta fully assembled: private data (inbox) + untrusted content (any sender!) + comms capability (drafting/sending). Mitigations: no autonomous send (draft-only, human approves — the single control that collapses most attack value), reply-scope constraints (drafts address only the active thread; no cross-thread content without explicit user action), content-marking of processed emails as data-not-instructions, injection-pattern detection on inbound, and egress rules if any tool calls exist (no URLs fetched from email content — link-following is exfil channel #2)
2. Cross-context leakage: the model drafting a reply to A while context contains threads from B — thread-scoped context assembly (only the active thread + user-approved extras enters the window), tested with leakage evals (canary content in adjacent threads must never surface)
3. Data-path exposure: inbox content transits to the model provider — the governance review: provider terms (no-training, retention), regional routing, and your trace stores now contain email content (trace redaction/encryption, access control, retention — the observability pipeline inherits inbox-grade sensitivity)
4. Memory/personalization risks: if the assistant learns user style/facts — memory store becomes inbox-derived PII (the full memory-governance checklist), and memory is an injection-persistence vector (hostile email content must not be memorizable as instruction)
5. Classic surfaces, AI-flavored: OAuth scope minimalism (read + draft; not send, if the design holds), token custody, tenant isolation in every store (conversation, memory, embeddings if inbox is indexed), rate/abuse controls (the feature as spam-drafting engine)

The review's output artifacts: the trifecta assessment (which two of three does the feature hold, and what keeps the third away), the autonomy ladder decision (draft-only now; send-with-approval gated on measured injection-resistance), the injection red-team suite as a launch gate (documented attack corpus, pass criteria), data-flow diagram including trace/memory stores, and the incident playbook (the 'it sent something it shouldn't have' runbook exists before launch).

One-liner: 'inventory capabilities, then hunt the trifecta: this feature is injection's dream target, so draft-only autonomy, thread-scoped context, egress denial, and a red-team gate aren't hardening — they're the design; everything else is the classic review with the trace stores added to the sensitive-data map.'

The honest framing first: specific tools will churn (this year's agent framework is next year's legacy), so bet on durable layers — the concerns that survive every model generation.

The durable bets:

1. Verification engineering: as generation gets cheap, proving correctness becomes the bottleneck and the profession — tests, evals, contracts, formal-ish checks, review systems. The engineer who can build verification machinery around probabilistic components is valuable in every future; this is the deepest continuity with existing DevOps instincts (CI/CD was always verification engineering)
2. Systems-of-least-privilege for autonomous actors: agents are a new workload class with identity, permissions, budgets, and audit needs — the IAM/sandboxing/policy engineering for non-human actors is barely v1 today and every org will need it. Your existing security-architecture skills transfer almost directly
3. Context/data infrastructure: retrieval, memory, knowledge freshness, data governance for AI paths — 'getting the right information to the right model at the right time under the right permissions' is a permanent problem that looks suspiciously like the data-platform engineering you already know
4. The economics layer: token/GPU cost engineering, capacity planning for inference, the FinOps of AI — scarce-resource optimization with new units; the practitioners who can do the arithmetic honestly are outnumbered by the enthusiasm
5. Evals as a discipline: the skill of specifying and measuring what good means for a probabilistic system — the closest thing to a net-new profession in this wave, and the one that compounds (eval suites are assets; prompts are consumables)

What to deprioritize (equally important): deep specialization in any single framework's API, prompt-incantation folklore (evaporates with each model generation), and racing the model companies at their own layer (training frontier models is a capital game, not a skills game).

The posture shift to internalize: your job's center of gravity moves from writing the artifact to specifying, verifying, and operating systems that produce artifacts — which is, honestly, the direction senior engineering was already heading (how much of a staff engineer's day was ever typing code?). The engineers who struggle will be the ones whose identity was typing speed; the ones who thrive were already thinking in systems, contracts, and failure modes.

The practical program: run agents daily on real work (fluency is experiential — the intuitions for what they botch don't come from reading), build one eval suite end-to-end, wire one MCP server to a system you own, and keep your fundamentals sharp (distributed systems, networking, security) — because AI systems are distributed systems with new failure modes, and the fundamentals are what make the new failure modes legible.

One-liner: 'bet on verification, least-privilege for agents, context infrastructure, cost engineering, and evals — the layers every model generation needs more of — stay fluent by using the tools daily, and hold your fundamentals close: AI didn't obsolete systems thinking, it made it the whole job.'