The Complete AI & Agentic Glossary — From Basics to Architect-Level

Whether you're learning AI from scratch or walking into an architect-level interview next week, you need to know these terms cold.

This guide covers everything — from the foundational concepts to the advanced patterns that production AI architects use daily. Each term comes with what it means, how it works, the trade-offs, and where it's used in the real world.

No fluff. No marketing speak. Just the stuff you need to know and be able to talk about intelligently.

Let's build this from the ground up.

Contents

1. Core AI: The Foundation
2. Foundation Models & Generative AI
3. The Four AI Properties (Feasibility Lens)
4. Architectural Patterns
5. Seven Primitives
6. Retrieval & Knowledge
7. Prompting & Context Engineering
8. Training, Tuning & Model Operations
9. Evaluation & Testing
10. Safety, Risk & Guardrails
11. Human-in-the-Loop & Review
12. Fairness, Bias & Transparency
13. Compliance & Governance
14. Production & Reliability
15. Stakeholder & Lifecycle
16. Team Enablement
17. Agentic Concepts
18. Future & Speculative
19. Quick Interview Prep — One-Liners That Stick

PART 1 — CORE AI: THE FOUNDATION

Artificial Intelligence (AI)

Overview: Machines simulating human-like reasoning, learning, and decision-making. The big umbrella. Everything under it — ML, deep learning, GenAI, agents — is a subset.

Description: AI isn't one technology. It's a field of computer science focused on building systems that can perceive, reason, learn, and act. It ranges from simple rule-based expert systems to neural networks with billions of parameters. The core idea: can a machine do something that, if a human did it, we'd call it intelligent?

Pros

• Automates complex tasks at scale
• Enables entirely new product categories
• Handles problems too data-heavy for humans to process manually

Cons

• Often opaque — hard to explain why a model made a decision
• Energy-intensive at scale
• Frequently overhyped; not every problem needs AI

Use Cases: Search engines, recommendation systems, fraud detection, robotics, autonomous vehicles, medical diagnosis support.

Machine Learning (ML)

Overview: A subset of AI where systems learn patterns from data instead of being explicitly programmed. This is where most real-world AI work happens.

Description: You feed an algorithm data — labeled or unlabeled — and it finds patterns. The more data, the better it gets. Three main flavors: supervised (labeled data), unsupervised (find hidden structure), and reinforcement (learn by reward). ML is the engine behind most AI features you use daily.

Pros

• Handles high-dimensional data humans can't process manually
• Adapts over time with retraining
• Generalizes to new, unseen inputs once trained well

Cons

• Garbage in, garbage out — quality data is non-negotiable
• Can overfit to training data and fail in production
• Requires ongoing retraining and monitoring

Use Cases: Spam filtering, churn prediction, demand forecasting, credit scoring, predictive maintenance, anomaly detection.

Deep Learning (DL)

Overview: Machine learning powered by multi-layered neural networks. This is what made modern AI possible — the breakthrough that gave us image recognition, voice assistants, and ChatGPT.

Description: Deep learning uses artificial neural networks with many layers (hence "deep") to automatically extract features from raw data. You don't hand-engineer features like "edge detection" or "word frequency" — the network learns them. Each layer builds on the previous one, moving from simple patterns to complex representations.

Pros

• State-of-the-art accuracy for perception and language tasks
• Eliminates manual feature engineering
• Scales impressively with more data and compute

Cons

• Needs massive datasets and serious compute power
• Often a "black box" — hard to interpret decisions
• Overkill for simple tabular data where traditional ML works better

Use Cases: Image recognition, speech-to-text, autonomous driving, protein folding (AlphaFold), real-time translation, medical imaging.

PART 2 — FOUNDATION MODELS & GENERATIVE AI

Large Language Model (LLM)

Overview: A deep learning model trained on massive amounts of text to understand and generate human language. The technology behind ChatGPT, Claude, Gemini, and Llama.

Description: An LLM is trained on trillions of tokens from books, websites, code, and conversations. Its core job is simple: predict the next token. But at scale, this simple objective produces emergent capabilities — reasoning, translation, summarization, code generation. Models like GPT-4 and Claude use transformer architecture with billions or trillions of parameters. They're called "foundation models" because you can build many applications on top of one model.

Pros

• General-purpose — one model handles dozens of tasks
• Few-shot learning — give it a few examples and it adapts
• Strong reasoning and writing ability
• Continuously improving with each generation

Cons

• Hallucinates — generates confident but false information
• Expensive to train and serve
• Reflects biases from training data
• Knowledge is frozen at training time unless augmented

Use Cases: Chatbots, content drafting, code generation, summarization, translation, data extraction, tutoring, research assistance.

Generative AI (GenAI)

Overview: AI that creates new content — text, images, audio, video, code. Not just analyzing data, but producing novel outputs. The category that took the world by storm in 2023.

Description: GenAI spans multiple model types. Transformers for text (GPT, Claude). Diffusion models for images (Stable Diffusion, Midjourney, DALL-E). GANs for realistic synthesis. The key shift: these models don't just classify or predict — they generate. You give a prompt, they produce something that didn't exist before.

Pros

• Massive productivity boost for creative and coding work
• Personalizes content at scale
• Enables non-experts to produce professional-quality output
• Opens new product categories

Cons

• Copyright and ownership concerns are unresolved
• Quality is inconsistent
• Misuse potential — deepfakes, disinformation, phishing at scale

Use Cases: Marketing copy, image generation, music production, synthetic data, drug discovery, code generation, video creation, design prototyping.

Transformer

Overview: The neural network architecture that powers every modern LLM. If you're talking about GPT, Claude, Gemini, or Llama — you're talking about transformers. The "T" in GPT stands for Transformer.

Description: Before transformers, models processed text sequentially — one word at a time. Transformers use "self-attention" to look at all tokens simultaneously and compute how relevant each token is to every other token. This enables parallel training (much faster) and long-range understanding. The paper "Attention Is All You Need" (Google, 2017) introduced this, and it changed everything.

Pros

• Scales exceptionally well with more parameters and data
• Supports transfer learning — pre-train once, fine-tune for many tasks
• Powers every major LLM today

Cons

• Quadratic memory cost with sequence length — doubling input length quadruples attention computation
• Not efficient for very long documents without optimizations like sliding window attention or sparse attention

Use Cases: Machine translation, summarization, code assistants, multimodal models, scientific reasoning.

Attention Mechanism

Overview: The core innovation inside transformers. It lets a model decide which parts of the input to focus on when producing each part of the output. It's why transformers can understand context.

Description: When a transformer processes "The bank was closed so I couldn't deposit money," attention lets the model connect "bank" with "deposit" — recognizing this is a financial institution, not a river bank. It computes relevance scores between every pair of tokens, creating a weighted representation. Multi-head attention runs this process multiple times in parallel, each "head" focusing on different relationships.

Pros

• Dramatically improves context understanding and translation quality
• Enables parallel processing
• The foundation of modern NLP

Cons

• Computationally expensive for long sequences
• Memory-intensive — every token attends to every other token

Use Cases: Machine translation, document Q&A, speech recognition, code understanding, multimodal reasoning.

Token

Overview: The basic unit of text that an LLM processes. Not a word, not a character — something in between. Understanding tokens is essential for understanding LLM costs and limits.

Description: Tokenization splits text into sub-word units. "Understanding" might become "Under" + "stand" + "ing." Common words are single tokens; rare words get split. A rough rule: 1 token is approximately 4 characters in English, or about three-quarters of a word. Models have a vocabulary of 50,000 to 100,000+ tokens. Everything — input, output, pricing, context limits — is measured in tokens.

Pros

• Flexible across languages and vocabularies
• Handles rare and unknown words through sub-word splitting
• Compact representation

Cons

• Makes cost and context limits harder to predict — you can't just count words
• Different tokenizers produce different token counts for the same text
• Non-English text often uses more tokens

Use Cases: API pricing (you pay per token), prompt sizing, output length limits, context window management.

Context Window

Overview: The maximum amount of text a model can consider at one time. Measured in tokens. Think of it as the model's short-term memory.

Description: A model with a 128K token context window can process about 300 pages of text in a single request. Everything inside the window is available for the model to attend to. Everything outside is forgotten. Larger windows enable longer conversations, full-document analysis, and multi-file code understanding. But they also mean more computation, higher cost, and potentially degraded attention to early parts of the input.

Pros

• Enables long-document analysis, extended conversations, and multi-file code reasoning without chunking
• Reduces the need for complex retrieval pipelines

Cons

• More tokens means higher cost and latency
• Models may pay less attention to content in the middle of long contexts
• Not all models handle long context equally well

Use Cases: Legal document review, codebase analysis, long chat sessions, multi-document summarization, research paper analysis.

Lost-in-the-Middle

Overview: LLMs recall content at the beginning and end of the context window best, and content in the middle worst. This is a known attention degradation pattern that affects both prompt design and RAG context assembly.

Description: When you stuff a long context window, the model doesn't pay equal attention to everything. It strongly recalls the first and last sections, but the middle gets blurry. This means if you put your most important instructions or highest-ranked retrieval chunks in the middle, they may get ignored. The fix: place the most important content at the head and tail of the context, not buried in the middle.

Pros (of knowing this)

• Lets you strategically order context for maximum recall
• Improves RAG quality by placing top-ranked chunks at the beginning and end

Cons

• Adds complexity to context assembly
• Easy to forget when debugging — you might blame the model when the real issue is chunk ordering

Use Cases: RAG context assembly, prompt design for long inputs, multi-document processing, conversation history management.

Temperature

Overview: A knob that controls how random or deterministic a model's output is. Low temperature equals focused and predictable. High temperature equals creative and surprising.

Description: Temperature scales the probability distribution before the model picks the next token. At temperature 0, the model always picks the most likely token — deterministic, repetitive, safe. At temperature 1, it samples more freely — creative, varied, sometimes wild. Most production systems use 0.0 to 0.3 for factual tasks and 0.7 to 1.0 for creative tasks.

Pros

• Gives you direct control over creativity vs. consistency
• Easy to adjust
• No retraining needed

Cons

• Easy to set wrong — high temperature on a coding task produces bugs; low temperature on a brainstorming task produces boring output
• Interacts with other parameters like top-p and top-k in non-obvious ways

Use Cases: Coding and factual Q&A (low: 0.0 to 0.3), brainstorming and creative writing (high: 0.7 to 1.0), balanced tasks like summarization (medium: 0.3 to 0.5).

Hallucination

Overview: When a model confidently generates false or fabricated information. The single biggest trust problem with LLMs. If you're building production AI, this is problem number 1.

Description: LLMs don't know facts — they know patterns. They predict what text should come next based on their training data. Sometimes the most statistically likely continuation isn't true. The model doesn't know it's wrong because it has no ground truth check. It can invent citations, fabricate API methods, make up statistics, and present all of it with complete confidence. This isn't a bug you can patch — it's inherent to how next-token prediction works. Pros: None. This is a risk to mitigate, not a feature.

Cons

• Spreads misinformation
• Breaks user trust
• Unsafe in regulated domains like healthcare, finance, and law
• Hard to detect — hallucinated content often sounds more convincing than true content

Use Cases (Mitigation): RAG to ground outputs in real documents. Fact-checking layers. Human-in-the-loop review for sensitive outputs. Citation requirements. Confidence scoring. Structured outputs to constrain what the model can say.

PART 3 — THE FOUR AI PROPERTIES (FEASIBILITY LENS)

Four AI Properties

Overview: Every LLM has four inherent properties that limit what it can do. Architects run every use case through these before giving a feasibility verdict. If you're designing AI solutions, this is your first checkpoint.

Description: The four properties are:

1. Next-token prediction — the model predicts text, not facts; instructions must be specific, concrete, and verifiable.
2. Knowledge — the model only knows what was in its training data; rare, niche, or fast-changing topics need external sources.
3. Working memory — the context window is finite; large corpora need chunking or retrieval.
4. Steerability — the model follows instructions, but abstract or long reasoning chains are unreliable; structured outputs and code execution help.

Pros

• Gives architects a systematic framework to evaluate any use case
• Exposes limitations before you build
• Forces you to design compensations (retrieval, tools, human review) into the architecture

Cons

• Can feel abstract if you're new
• Easy to skip when enthusiasm is high
• Properties are present whether your architecture acknowledges them or not — ignoring them doesn't make them go away

Use Cases: Pre-build feasibility assessment, solution design, interview discussions about why an AI use case works or doesn't.

Three Feasibility Verdicts

Overview: After running a use case through the four AI properties, you arrive at one of three verdicts. This is how architects communicate whether something is buildable.

Description:

1. Feasible as scoped — all properties favor the model, cost is within ceiling, latency is within SLA.
2. Feasible with constraints — works under specific conditions (document length threshold, refresh schedule, human review gate); the constraints are part of the architecture, not optional.
3. Not feasible — a property limitation can't be compensated within scope or budget; you name the disqualifying constraint and the scope reduction that would change the verdict.

Pros

• Forces honest assessment
• Prevents over-promising
• The "feasible with constraints" verdict is where most real-world projects live — naming the constraints makes them manageable

Cons

• Stakeholders may hear "not feasible" as "never" when it means "not as currently scoped"
• Requires clear communication to avoid misinterpretation

Use Cases: Solution scoping, stakeholder communication, SOW definition, interview answers about AI project feasibility.

PART 4 — ARCHITECTURAL PATTERNS

Augmented LLM

Overview: The simplest pattern — a single model call enhanced with retrieval, tools, or memory. One request in, one response out. No loops, no branching.

Description: You give the model a prompt, optionally with retrieved context or tool access, and it responds. That's it. No multi-step planning, no agent loop. This is what most API calls look like. It's the right choice when the task is a single, well-defined transformation — summarize this document, extract these fields, translate this text.

Pros

• Simplest to build and debug
• Lowest latency and cost
• Most predictable
• Easy to evaluate — one input, one output

Cons

• Can't handle complex, multi-step tasks
• No autonomy or adaptation
• Capability ceiling — if one call can't do it, this pattern can't

Use Cases: Summarization, field extraction, translation, classification, single-turn Q&A with RAG.

Workflow (Prompt Chaining)

Overview: A fixed sequence of LLM calls, where each step's output feeds the next. Predictable, bounded, and auditable. The production workhorse for multi-step tasks.

Description: You decompose a task into known steps and chain them. Step 1 extracts data, step 2 validates it, step 3 formats the response. Each step is a single LLM call with a specific prompt. The path is deterministic — you wrote it in code. Sub-patterns include chaining (sequential), routing (classifier directs to different handlers), parallelization (independent subtasks run concurrently or voted on), and evaluator-optimizer (generate, evaluate, revise loop).

Pros

• Predictable and auditable — you know every step
• Cost is known per request
• Errors are isolated per step
• Easy to debug — check which step failed
• Deterministic path is great for regulated workflows

Cons

• Rigid — can't adapt to unexpected inputs
• Every added step adds latency and cost
• If the decomposition is wrong, every request fails the same way
• Not suitable for open-ended problems

Use Cases: Document processing pipelines, claims processing, content moderation pipelines, multi-step data extraction and validation.

Routing

Overview: A workflow sub-pattern where a classifier directs input to specialized handlers. Different input types get different treatment.

Description: First, a classifier (often a small, fast model) categorizes the input. Then, the input is routed to the appropriate downstream handler — a specific prompt, model tier, or processing pipeline. For example: customer support tickets routed to billing, technical, or escalation handlers based on content.

Pros

• Each handler is specialized and optimized
• Cost-efficient — simple inputs go to cheap models, complex ones to expensive models
• Clear separation of concerns

Cons

• Misclassification cascades downstream — wrong route means wrong handler
• Classifier quality is the bottleneck
• Adds a step (latency) before the actual work begins

Use Cases: Customer support triage, document type classification, query routing in search systems, model tier selection based on task complexity.

Parallelization

Overview: A workflow sub-pattern where independent subtasks run concurrently. Two flavors: sectioning (different subtasks) and voting (same task run multiple times for consensus).

Description: In sectioning, you split a task into independent pieces that can run at the same time — e.g., analyzing different sections of a document in parallel. In voting, you run the same task multiple times and pick the majority or best answer — useful for confidence through consensus.

Pros

• Faster — parallel execution reduces total latency
• Voting improves reliability through consensus
• Good for independent subtasks that don't depend on each other

Cons

• Aggregation logic can be complex
• Voting is expensive — multiple LLM calls for one answer
• Only works when subtasks are truly independent

Use Cases: Multi-section document analysis, confidence-boosting for high-stakes answers, batch processing of independent items.

Evaluator-Optimizer

Overview: A workflow sub-pattern where a generator produces output, an evaluator scores it, and the generator revises. A quality improvement loop.

Description: The generator LLM produces a draft. The evaluator (could be code-based, LLM-as-judge, or human) scores it against criteria. If the score is below threshold, the generator revises. This loops until the output passes or a max iteration count is hit. Useful when quality is verifiable but a single pass isn't enough.

Pros

• Iterative refinement improves quality
• Clear quality gate
• Works well when eval criteria are well-defined
• Self-correcting

Cons

• Loop cost — multiple LLM calls per output
• Needs a good evaluator or the loop optimizes for the wrong thing
• No exception path for low-confidence items unless you build one

Use Cases: Document drafting with quality gates, code generation with test-based evaluation, content generation with rubric scoring.

Orchestrator-Workers

Overview: A central LLM dynamically decomposes a task and delegates to worker LLMs. The orchestrator owns the goal; workers own scoped sub-tasks.

Description: Unlike a fixed workflow, the orchestrator decides at runtime how to split the work. It looks at the input, determines what sub-tasks are needed, dispatches them to workers (each with its own context and tools), collects results, and synthesizes a final output. The orchestrator never does sub-task work — it plans and synthesizes. Workers do the actual work in isolated contexts.

Pros

• Handles unpredictable subtask structure — the orchestrator adapts
• Workers are specialized and isolated
• Scales to complex, multi-faceted problems
• Good for multi-file code changes or research tasks

Cons

• Blurred failure boundaries — hard to tell which worker failed
• Traces can fragment across orchestrator and workers
• Dropped subagent output is a common bug — you need a coverage check at synthesis (results returned must equal units dispatched)
• Shared trace IDs are essential

Use Cases: Multi-file code changes, research with multiple hypotheses, content production across multiple domains, complex data analysis pipelines.

Agent

Overview: The most autonomous pattern. The model plans, acts via tools, observes results, and iterates in a loop until the goal is met or it determines it can't be achieved. Open-ended, adaptive, and powerful — but the hardest to control.

Description: An agent receives a goal, not a specific instruction. It breaks the goal into steps, calls tools to act, observes the results, and adjusts its plan based on what it learned. The loop continues until the goal is achieved, a stopping criterion is met, or a budget is exhausted. This is the ReAct pattern (Reason + Act + Observe + Repeat). The model decides the steps — you don't script them.

Pros

• Handles open-ended problems with unpredictable step counts
• Adapts to unexpected situations
• Can explore and recover from errors
• The most flexible pattern

Cons

• Unbounded tool use and growing context equals runaway cost
• Non-deterministic — same input can produce different paths
• Hardest to debug — full trajectory analysis needed
• Needs guardrails: max tool calls, token budgets, explicit stopping criteria
• Highest risk pattern

Use Cases: Coding agents (Devin, Cursor, Claude Code), research assistants, IT automation, travel planners, DevOps remediation.

Dynamic Workflows

Overview: A pattern where workflow steps are assembled at runtime rather than fixed in advance. The model or orchestrator decides the sequence based on the input.

Description: Traditional workflows are pre-defined — you write the steps in code. Dynamic workflows let the system compose steps on the fly. The model looks at the input, determines what steps are needed and in what order, and executes accordingly. It's a middle ground between fixed workflows (rigid) and agents (fully autonomous).

Pros

• More flexible than fixed workflows
• More controlled than agents — steps are still discrete and auditable
• Adapts to input shape without full autonomy

Cons

• Harder to predict than fixed workflows
• Debugging is more complex
• Still needs guardrails on step count and cost

Use Cases: Document processing where document types vary, multi-step research where the path depends on findings, adaptive customer support flows.

Five-Factor Framework (Pattern Selection)

Overview: Five factors that determine which pattern to choose: predictability, error cost, observability, latency, and cost. The tightest constraint wins.

Description: For each use case, assess:

• How predictable must outputs be? High equals workflow, low means agent acceptable.
• What does an error cost? High equals workflow with step guards.
• How observable must the system be? High equals workflow, steps log as code.
• What latency is acceptable? Low equals augmented LLM, high means agent okay.
• What cost ceiling exists? Low equals simplest pattern, high means agent feasible.

The tightest constraint determines the pattern.

Pros

• Systematic — removes "it feels like we need an agent" decisions
• Forces you to name constraints before choosing
• Prevents over-engineering

Cons

• Can feel mechanical for creative tasks
• Factors interact — changing one affects others
• Still requires judgment to weight factors correctly

Use Cases: Architecture design decisions, pattern selection in solution design, interview answers about when to use agents vs. workflows.

Decomposition

Overview: Splitting work into what Claude owns, what existing systems own, and what humans own. The first decision in any AI architecture — before pattern selection, before model choice.

Description: For every capability in the use case, ask: should the LLM do this, should an existing system do this, or should a human do this? The delegation criteria: reversibility (can it be undone?), stakes (what does wrong cost?), accountability (who answers for it?). Deterministic rules (routing thresholds, validation, business logic) belong in a rule engine, not the model. The classic failure: handing a precise rule ("reject if amount exceeds 5,000") to an LLM, which interprets "around five thousand" as passing.

Pros

• Prevents over-assigning work to the LLM (the most expensive early mistake)
• Uses each component for what it's best at
• Creates clear boundaries and accountability

Cons

• Requires deep understanding of existing systems
• Can feel slow when stakeholders want to "just use AI"
• Easy to overlook existing systems that already do the job better

Use Cases: Solution design, scoping workshops, architecture reviews, interview answers about AI project structure.

Workflow vs Agent — The Decision Rule

Overview: If you could have written the steps in code, use a workflow. Don't choose an agent because the task "feels" open-ended. Every added pattern layer equals cost plus latency plus failure surface.

Description: The exam-perfect discriminator: if the task structure is known and decomposable in advance, use a workflow. If steps can't be determined until runtime and the count is variable, use an agent. Workflows give you predictability, consistency, and auditability. Agents give you flexibility and autonomy. Choose the simplest pattern that meets the requirement.

Pros

• Prevents over-engineering
• Saves cost and reduces failure surface
• Clear decision criterion

Cons

• Some tasks are genuinely ambiguous — the line isn't always clean
• Stakeholders may push for agents because they sound more impressive

Use Cases: Architecture decisions, pattern selection, interview answers about when to use agents.

PART 5 — SEVEN PRIMITIVES

Overview: Seven building blocks architects use to construct AI systems. Use the fewest primitives necessary — heavier primitive means more latency, tokens, and operational surface.

Tools

Overview: Functions the model can call to take action or fetch results. The most fundamental primitive — without tools, the model can only generate text.

Description: You define tools with a schema (name, description, parameters). The model decides when to call them and with what arguments. Your code executes the tool and returns the result. Tools can fetch data (database query, web search), take action (send email, create ticket, run code), or compute (calculator, unit conversion). Tool use is what turns a text generator into an actor.

Pros

• Extends model capabilities to real-time data and real-world actions
• No fine-tuning needed — tools are defined at inference time
• Models are getting better at choosing the right tool

Cons

• Requires robust schemas and error handling
• Models can hallucinate parameters or call the wrong tool
• Security — a model with tool access can do damage if not sandboxed
• Every tool is attack surface plus cost

Use Cases: Database queries, web search, code execution, calendar booking, file operations, API integrations, sending emails, triggering CI/CD pipelines.

MCP (Model Context Protocol)

Overview: A standardized protocol for exposing tools, resources, and prompts across multiple AI clients. Think of it as USB-C for AI tool connections — plug-and-play, any tool to any client.

Description: MCP uses a host-client-server architecture. The host (AI app) runs one client per server. Each server exposes capabilities through three primitives: tools (executable functions, model-controlled), resources (read-only context data, application-controlled, accessed via URIs), and prompts (reusable interaction templates, user-controlled). It uses JSON-RPC 2.0 with stateful sessions and capability negotiation at init. Transports: stdio (local, dev) or Streamable HTTP (remote, prod, OAuth 2.1 required). Tools are discovered at runtime — plug-and-play, no pre-baked connector.

Pros

• Standardized — write a server once, any MCP-compatible client can use it
• Runtime discovery — tools appear dynamically
• Solves the N-times-M integration problem
• Ecosystem of pre-built servers growing fast

Cons

• Protocol layer adds debugging complexity
• Overkill if only one client needs the tool — just use direct API tool use
• Still maturing — tooling and observability for MCP is evolving

Use Cases: Sharing tool definitions across multiple AI applications, standardized enterprise tool connections, plug-and-play integrations, cross-platform agent tool ecosystems.

Subagents

Overview: Scoped sub-tasks running in separate contexts. A way to isolate and parallelize work within an agent system.

Description: Instead of one agent doing everything in one growing context, you spawn subagents — each with its own context window, its own tool set, and a single scoped task. The orchestrator dispatches, subagents execute and return results, orchestrator synthesizes. This keeps contexts lean and allows parallel execution.

Pros

• Isolated contexts prevent context bloat
• Parallel execution speeds up complex tasks
• Each subagent can be optimized for its task
• Failure is usually recoverable — retry, re-route, or flag the gap

Cons

• Results can be silently dropped — you need a coverage check at synthesis
• Trace fragmentation across subagents
• More LLM calls equals higher cost
• Coordination overhead

Use Cases: Multi-file code analysis, parallel research tasks, document processing with independent sections, multi-step workflows with isolated stages.

Hooks

Overview: Deterministic code that fires on events. The model can't skip them. This is how you guarantee things happen regardless of what the model decides.

Description: Hooks are code-level callbacks that fire before or after model actions. A pre-tool hook can validate authorization before a tool executes. A post-generation hook can scan output for PII before returning to the user. Unlike prompt instructions (which are guidance the model can ignore), hooks are deterministic — they run no matter what.

Pros

• Guarantees execution — the model can't talk its way past a hook
• Deterministic and auditable
• Separates safety logic from model behavior
• Essential for production guardrails

Cons

• Adds code complexity
• Hook failures need careful handling (fail closed)
• Can introduce latency
• Overuse can make the system rigid

Use Cases: Pre-tool authorization checks, post-generation output screening, logging hooks, input validation, PII redaction before context window.

Skills

Overview: Reusable, distributable bundles of code plus instruction sets. Versioned units that appear as repeatable procedures. The evolution of prompt libraries into governed, shareable assets.

Description: A Skill packages a prompt (or set of instructions) with optional scripts into a versioned, distributable unit. Teams can share, update, and govern Skills across projects. Four distribution mechanisms:

1. Org-provisioned Skills — everyone gets them, no versioning or rollback.
2. Plugins — group/org targeting, version-controlled updates, rollback supported — the governed path.
3. Project Skills — live in the repo (.claude/skills/), version with the repository, scoped to the project.
4. API Skills — called programmatically with explicit version pinning.

Pros

• Reusable across teams and projects
• Versioned and governed (via plugins)
• Reduces prompt drift — everyone uses the same version
• Distributable at org scale

Cons

• Supply-chain risk — a Skill could contain malicious code (network calls, shell access, credential reads)
• Must be audited before trusting
• Org-provisioned Skills have no rollback — one bad edit ships to everyone
• Needs sandboxing and least-privilege runtime

Use Cases: Standardized code review procedures, deployment checklists, document analysis workflows, team-wide prompt standards, reusable extraction patterns.

Agent Teams

Overview: Multiple agents operating as coordinated peers. Each agent has a specialized role — planner, coder, reviewer, tester — and they interact to solve complex problems.

Description: Agent teams go beyond orchestrator-workers. Instead of a hierarchy, agents are peers that communicate through shared state or message passing. A planner agent breaks down the task. A coder agent writes code. A reviewer agent checks it. A tester agent runs tests. They coordinate, hand off, and iterate. Frameworks like AutoGen, CrewAI, and LangGraph support this.

Pros

• Scales to complex, multi-faceted problems
• Each agent is specialized and optimized
• Mimics real-world team workflows
• Enables parallel execution
• More robust — if one agent fails, others can compensate

Cons

• Coordination overhead — agents can talk in circles or contradict each other
• Emergent behavior is unpredictable
• More LLM calls equals higher cost
• Debugging inter-agent communication is hard
• Over-engineering risk — not every task needs a team

Use Cases: Software development teams (plan, code, review, test), supply chain simulation, game design pipelines, research with multiple hypotheses, content production workflows.

PART 6 — RETRIEVAL & KNOWLEDGE

Retrieval-Augmented Generation (RAG)

Overview: The technique of fetching relevant documents from a knowledge base before the LLM generates an answer. The most popular architecture for building production AI assistants.

Description: Here's the flow. First, you chunk your documents into passages and convert each chunk into an embedding. Store these in a vector database. When a user asks a question, you embed the question, search the vector database for the most similar chunks, and pass those chunks plus the question to the LLM. The LLM now has real context to ground its answer. Two pipelines: offline/indexing (ingest, parse, chunk, embed, store) and online/query (embed query, retrieve top-k, optionally rerank, assemble context, generate).

Pros

• Reduces hallucinations by grounding answers in real documents
• Works with private and frequently updated data
• Cheaper and faster than fine-tuning
• You can swap knowledge bases without retraining
• Enables citations and source tracking

Cons

• Retrieval quality is the bottleneck — garbage retrieval means garbage answers
• Chunking strategy matters enormously
• Doesn't fix reasoning errors
• Adds infrastructure (vector DB, embedding pipeline, reranker)
• Complex queries may need multiple retrieval steps
• The number 1 mistake: using RAG where a tool call belongs (RAG is for stable knowledge, not live state)

Use Cases: Enterprise knowledge bases, customer support bots, compliance Q&A, medical assistants, legal research tools, code documentation search.

Retrieval vs Tool Use — The Critical Distinction

Overview: Retrieval is for stable knowledge (was true yesterday, will be true tomorrow). Tool use is for live state (current value owned by a system). Confusing these is the number 1 RAG mistake.

Description: If a user asks "what is our return policy?" — that's stable knowledge. RAG it. If a user asks "how many items are in stock for SKU 12345?" — that's live state. Don't RAG it; call the inventory system directly. Symptoms of getting this wrong: stale chunks, answers that contradict the database, results that shift with index refresh. Fix: call the system that owns the live state.

Pros

• Getting this right means your system gives accurate answers for both knowledge and live data
• Clean separation of concerns

Cons

• Easy to get wrong — especially when "just put everything in the vector DB" feels simpler
• Requires understanding which data is stable vs. live

Use Cases: Architecture decisions about when to use RAG vs. tool calls, interview answers about RAG limitations.

Vector Database / Vector Store

Overview: A database designed to store and search high-dimensional vectors (embeddings). The backbone of RAG and semantic search.

Description: Traditional databases search by exact match or keyword. Vector databases search by meaning. They store embeddings and use approximate nearest neighbor (ANN) algorithms to find vectors closest to your query vector. Popular options include Pinecone, Weaviate, Milvus, Qdrant, Chroma, and pgvector (Postgres extension). ANN indexes typically use HNSW or IVF. Distance metric (cosine, dot, L2) must match the embedding model's recommendation.

Pros

• Fast semantic search across millions of items
• Enables long-term memory for AI agents
• Scales horizontally
• Supports metadata filtering (tenant/ACL scoping, recency constraints, source scoping)
• Access control belongs at the retrieval layer for confidential docs

Cons

• Adds infrastructure and operational complexity
• ANN is approximate — you trade precision for speed
• Index tuning is non-trivial
• Cost scales with dimensionality and dataset size
• Not a replacement for traditional databases — a complement

Use Cases: RAG pipelines, recommendation engines, image search, duplicate detection, anomaly detection, agent memory, semantic caching.

Embeddings

Overview: Numerical representations of data that capture meaning. The bridge between human-readable content and machine-searchable math. Without embeddings, there's no semantic search, no RAG, no vector databases.

Description: An embedding model takes text (or images, audio) and outputs a dense vector — typically 384 to 4096 numbers. Semantically similar inputs produce vectors that are close together in space. "The cat sat on the mat" and "A feline rested on the rug" will have nearby embeddings despite sharing almost no words. Critical rule: use the SAME embedding model and config for indexing AND querying — mismatch silently destroys retrieval.

Pros

• Enables semantic search, clustering, similarity comparison, and classification
• Works across languages with multilingual models
• Compact and fast to compare
• Foundation for RAG and agent memory

Cons

• Quality depends entirely on the embedding model
• Can encode and amplify biases from training data
• Dimensionality affects cost and speed
• Domain-specific text (medical, legal) may need fine-tuned embedding models

Use Cases: Semantic search, RAG retrieval, recommendation systems, clustering and deduplication, classification, anomaly detection, agent memory.

Grounding

Overview: The practice of tying AI outputs to real, verifiable data sources. Instead of letting the model answer from memory (which may be wrong), you force it to base answers on actual documents, APIs, or databases.

Description: Grounding means every claim the model makes should be traceable to a source. In practice: retrieve relevant documents, pass them to the model, require the model to cite which document supports each claim, and optionally verify citations programmatically. Grounding is what separates a demo from a production system.

Pros

• Improves accuracy and trust
• Enables traceability — you can show users where the answer came from
• Supports compliance and audit requirements
• Reduces hallucination risk significantly

Cons

• Adds latency (retrieval takes time)
• Increases system complexity
• Doesn't eliminate hallucinations — the model can still misinterpret grounded context
• Requires good source data

Use Cases: Financial advice bots, legal research assistants, customer support with knowledge base citations, medical Q&A, compliance reporting.

Chunking Strategies

Overview: How you split documents into pieces for embedding and retrieval. The most overlooked factor in RAG quality. Get this wrong and your retrieval feeds the model garbage.

Description: Five strategies:

1. Fixed-size — every N tokens (typically 512) with 10-20% overlap. Splits mid-clause, cuts facts in half.
2. Recursive — split on separator hierarchy (paragraph, sentence, word) within size budget. The sane production default — respects natural boundaries.
3. Semantic — split where sentence-embedding similarity drops. Unpredictable chunk sizes.
4. Layout-aware — split on markdown headers, HTML tags, PDF structure, code function boundaries. Needs reliable structure.
5. Parent-child (hierarchical) — index small child chunks (150-200 tokens) for precision, return large parent chunks (512-1024 tokens) to the model. Most-adopted production pattern.

Pros

• Right chunking equals retrieval feeds the model clean, complete context
• Parent-child gives precision at search time and context at generation time

Cons

• Wrong chunking equals signal diluted (too big) or no context (too small)
• Half of chunking bugs are visible to the naked eye — read your actual chunks

Use Cases: RAG pipeline design, document indexing, knowledge base construction.

Dense vs Sparse Search

Overview: Two retrieval approaches. Dense equals embeddings for semantic similarity. Sparse equals BM25/keyword for exact terms.

Description: Dense search uses embeddings to find semantically similar content — good for paraphrase, intent, concept matching. Sparse search (BM25) uses keyword matching — good for part numbers, error codes, citations, exact identifiers. Dense compresses rare tokens; BM25 catches exact matches. Most production systems need both.

Pros

• Dense understands meaning
• Sparse catches exact terms
• Together they cover both query patterns

Cons

• Dense alone misses exact-term queries
• Sparse alone misses semantic similarity
• Running both adds complexity

Use Cases: RAG retrieval, enterprise search, document Q&A, code search.

Hybrid Search with Reciprocal Rank Fusion (RRF)

Overview: Combining dense (semantic) and sparse (BM25/keyword) search, then fusing results using Reciprocal Rank Fusion. The standard production retrieval approach.

Description: Run both dense and sparse searches in parallel. Each returns a ranked list. RRF merges them by scoring items based on their rank in each list — items that rank well in both get boosted. This catches both semantic matches and exact-term matches in a single result set.

Pros

• Covers both semantic and exact-match query patterns
• Standard, well-understood technique
• Improves recall across diverse query types

Cons

• Two retrieval systems to maintain
• Fusion tuning
• More compute than single-method search

Use Cases: Production RAG systems, enterprise search, mixed query pattern environments.

Cross-encoder Reranker

Overview: A second-stage retrieval quality booster. Rescores the top 50-200 candidates from initial retrieval using a cross-encoder model that jointly evaluates query and document.

Description: Initial retrieval (dense plus sparse) is fast but approximate — it returns top-k candidates. A cross-encoder reranker takes those candidates and scores each query-document pair jointly (not separately like embeddings). This is more accurate but slower. Result: 10-20% relevance improvement, adding 100-400ms latency.

Pros

• Significant relevance improvement
• Catches false positives from initial retrieval
• Well worth the latency for precision-critical use cases

Cons

• Adds 100-400ms
• Another model to serve and maintain
• Overkill for simple search

Use Cases: High-stakes RAG (legal, medical), precision-critical retrieval, enterprise knowledge bases.

Query Rewriting (Multi-Query and HyDE)

Overview: Techniques for improving retrieval when user queries are short or ambiguous. Rewrite the query before searching.

Description: Multi-Query: generate 3-5 rephrasings of the user's question, run parallel searches for each, fuse results with RRF. HyDE (Hypothetical Document Embeddings): generate a hypothetical answer to the question, embed that answer, and search for similar real documents. Both expand the search surface.

Pros

• Catches relevant documents the original query would miss
• Improves recall for ambiguous queries
• Relatively cheap — a single LLM call to rewrite

Cons

• Adds a generation step (latency)
• Bad rewrites can mislead retrieval
• More complex pipeline

Use Cases: Short user queries, ambiguous search intent, research assistants, customer support search.

Contextual Retrieval

Overview: Anthropic's technique that reduces top-20 retrieval failures by approximately 67%. Prepend chunk-situating context before embedding, combine with contextual BM25 and a reranker.

Description: Standard chunking loses surrounding context — a chunk from page 47 of a contract doesn't know it's from a contract. Contextual Retrieval prepends a short context sentence to each chunk before embedding ("This chunk is from Section 4.2 of the Master Services Agreement, covering payment terms"). This gives the embedding model more signal. Combined with contextual BM25 and a reranker, it dramatically reduces retrieval failures.

Pros

• 67% reduction in retrieval failures
• Simple to implement — just prepend context
• Works with existing chunking strategies

Cons

• Adds a preprocessing step (generating context for each chunk)
• More storage (longer chunks)
• Needs to be re-run when documents change

Use Cases: High-accuracy RAG systems, enterprise knowledge bases, legal and medical document retrieval.

Progressive Discovery

Overview: An agent starts with a map or summary, then fetches details on demand via tools. Keeps context lean while navigating large information spaces.

Description: Instead of loading everything into the context window (monolithic), the agent gets a high-level overview first — a table of contents, a file listing, a document summary. When it needs detail, it calls a tool to fetch the specific section. This is the agentic equivalent of a human reading the table of contents first, then diving into specific chapters.

Pros

• Lower cost per request on large corpora
• Scales beyond the context window
• Agent only pays for what it actually needs
• Good for exploratory tasks over codebases or document trees

Cons

• Adds tool-call latency
• Loop complexity
• Needs stopping criteria — agent could explore forever
• More LLM calls than monolithic

Use Cases: Codebase navigation, large document tree exploration, agentic research, multi-file code analysis.

Monolithic Context

Overview: Put everything in the context window at once. The simplest context strategy — no retrieval, no tools, no chunking.

Description: If the corpus fits comfortably in the context window and is stable, just load it all. One call, one response. Cache-friendly. Simple to build and debug. The trade-off: you pay for every token every time, and you hit window ceilings on large corpora.

Pros

• Simplest approach
• One LLM call
• Cache-friendly (stable prefix)
• No retrieval infrastructure needed
• Good for bounded tasks with predictable input

Cons

• Pays for every token every time
• Hits window ceilings on large corpora
• Degrades with lost-in-the-middle attention
• Not suitable for large or dynamic corpora

Use Cases: Small document Q&A, single-turn tasks with bounded input, prototyping, cached system prompts.

Compaction / Summarization

Overview: Compress conversation history as it grows. Prevents context window overflow in long multi-turn sessions.

Description: As a conversation grows, older turns accumulate tokens. Compaction periodically summarizes or compresses older history — replacing 10 turns of detailed exchange with a 1-paragraph summary. This frees context space for new turns while retaining the gist of the conversation.

Pros

• Enables long-running conversations without hitting context limits
• Retains conversation continuity
• Reduces per-turn token cost over time

Cons

• Summarization can lose important details
• Adds a processing step
• Hard to decide what to keep vs. compress
• The summary itself can introduce errors

Use Cases: Long chat sessions, multi-turn agent workflows, extended customer support conversations.

Re-index Discipline

Overview: Documents added or removed without reindexing cause retrieval drift. The classic "confident but wrong after data refresh" scenario.

Description: When you add new documents to your corpus or remove old ones, you must re-index. If you don't, the vector database returns stale chunks — the model generates confident answers based on outdated information. This is the first thing to check when answers degrade after a corpus refresh: was the refresh re-indexed? Same embedding model? Chunk boundaries intact?

Pros

• Maintaining re-index discipline prevents the most common RAG failure mode
• Keeps retrieval accurate and current

Cons

• Re-indexing is expensive on large corpora
• Easy to forget in automated pipelines
• Needs monitoring and alerting

Use Cases: RAG pipeline maintenance, retrieval quality monitoring, debugging "confident but wrong" RAG failures.

PART 7 — PROMPTING & CONTEXT ENGINEERING

Prompt Engineering

Overview: Crafting inputs to get the best possible outputs from a model. The fastest, cheapest way to improve AI performance — no retraining required.

Description: Techniques include system prompts (define the model's role), few-shot examples (show what good output looks like), chain-of-thought (ask for step-by-step reasoning), structured outputs (enforce JSON schema), and constraint specification. The progression: zero-shot (cheapest, try first) to few-shot (show examples) to chain-of-thought (reason step by step). Use the lightest technique that meets the requirement.

Pros

• Fast and free — no training, no infrastructure
• Works with any model
• Iterative — improve in minutes
• Often gets 80% of fine-tuning benefit at 0% of cost

Cons

• Brittle — model updates can break prompts
• Hard to test systematically
• What works for one model may not work for another
• System prompt instructions are guidance, NOT enforcement — adversarial input can talk past them

Use Cases: Chatbots, content generation, classification, evaluation, data extraction, summarization, code generation.

System Prompt Design

Overview: The system prompt defines the model's role, constraints, and output contract. In enterprise systems, it has three parts: role and scope, constraints (must do, must never do), and output contract (exact shape of response).

Description: Role and scope tells Claude who it is and what it does. Constraints define boundaries. Output contract specifies the exact format. Underspecification is the enemy — vague instructions mean the model fills the gap with its own assumptions, differently each time. Server-verified identity and role should be injected into the system prompt, never trusted from user-asserted messages. Long stable system prompts are prime candidates for caching.

Pros

• Gives the model clear boundaries
• Reduces unpredictable behavior
• Cacheable (long, stable content)
• Separates role definition from task data

Cons

• Instructions are guidance, not enforcement — anything adversarial input can talk the model out of needs a runtime control, not more prompt text
• Long prompts cost more (unless cached)
• Easy to over-specify and constrain the model into poor outputs

Use Cases: Enterprise AI assistants, customer support bots, document processing systems, any production LLM deployment.

Zero-shot

Overview: Give the model an instruction with no examples. The cheapest, fastest prompting technique. Try this first.

Description: You describe the task clearly and let the model figure it out. Works well for well-specified tasks with strong models. If zero-shot works, you're done — no need for examples or reasoning prompts.

Pros

• Cheapest — fewest input tokens
• Fastest — no examples to process
• Simplest to maintain
• Good baseline

Cons

• May not produce consistent formatting
• Weaker on edge cases
• Model may interpret ambiguous instructions differently each time

Use Cases: Simple tasks (summarization, translation), strong models, well-specified instructions, prototyping.

Few-shot

Overview: Provide 2-5 input/output examples in the prompt to show the model what good output looks like. Easier to show than describe.

Description: Instead of describing the format, you show it. "Here are three examples of input and the expected output. Now do the same for this input." The model pattern-matches. If examples are static across requests, they're part of the cacheable prefix — near-free after the first call.

Pros

• Format consistency
• Handles domain conventions and edge cases
• Cheap (no retraining)
• Cacheable if static
• Reversible — just change the examples

Cons

• More input tokens
• Examples can bias the model toward specific patterns
• Need to curate good examples
• Maintenance — examples may need updates

Use Cases: Format-specific extraction, classification with unusual categories, style matching, edge-case handling.

Chain-of-Thought (CoT)

Overview: Ask the model to reason step by step before answering. One of the most impactful prompt engineering discoveries. Simple to do, surprisingly powerful.

Description: Instead of "What's 15% of 240?", you ask "What's 15% of 240? Think step by step." The model writes out its reasoning, then gives the answer. This intermediate reasoning significantly improves accuracy on math, logic, and multi-step problems. Costs more output tokens (output is typically 5x input price), so use when needed.

Pros

• Better reasoning and accuracy on complex problems
• Makes logic visible — easier to audit and debug
• Free — no extra cost beyond more output tokens
• Works with any model

Cons

• Uses more output tokens (expensive — output is 5x input)
• Not always necessary — can slow down simple tasks
• Model can produce convincing but wrong reasoning
• Doesn't guarantee correctness

Use Cases: Math word problems, coding challenges, troubleshooting, decision support, multi-step analysis, interview-style reasoning.

Structured Outputs

Overview: Enforce a JSON schema on model responses. Eliminates parse failures. Essential for machine-readable pipelines.

Description: Instead of hoping the model returns well-formatted JSON, you define a schema (field names, types, required fields) and the model is constrained to output valid JSON matching that schema. This turns unreliable text generation into a reliable data extraction step.

Pros

• Eliminates parse failures
• Machine-readable output
• Reliable integration with downstream systems
• Reduces hallucination — schema constrains what the model can say

Cons

• Schema design takes effort
• Overly rigid schemas can force the model into poor outputs
• Some models handle structured outputs better than others
• Adds complexity to the prompt

Use Cases: Data extraction pipelines, API integrations, field extraction from documents, classification with structured labels, machine-readable output contracts.

Prefilling

Overview: Start the assistant's turn to constrain the output format. A technique for forcing JSON or skipping preambles.

Description: Instead of letting the model start from scratch, you prefill the beginning of its response. If you want JSON, start the assistant turn with an opening brace. The model continues from there, producing valid JSON. If you want to skip "Sure, here's your summary:", prefill with the first word of the actual content.

Pros

• Minimal cost
• Strong format control
• Skips unwanted preambles
• Works with any model that supports prefilling

Cons

• Can constrain the model too much
• Edge cases where the model fights the prefill
• Needs careful use — wrong prefill can derail the output

Use Cases: Forcing JSON output, skipping preambles, constraining response format, machine-readable pipelines.

Prompt Caching

Overview: Cache a stable prompt prefix so it's processed once and reused on subsequent requests. The biggest cost and latency lever when your system prompt is long and stable. Highest-yield optimization in production LLM systems.

Description: Caches everything up to a cache breakpoint. On a request: if the prefix is already cached, reuse it (huge latency and cost cut); if miss, process fully and write cache. Two modes: automatic caching (one top-level cache_control field, system places breakpoint automatically — ideal for multi-turn) and explicit breakpoints (up to 4 independent breakpoints, needed for mixed TTLs or caching system prompt separately). Cache is refreshed at no extra cost on every hit (sliding TTL). Pricing: 5-min cache write costs 1.25x base input (pays off after one read). 1-hour cache write costs 2x (pays off after two reads). Cache read (hit) costs 0.1x — 10% of standard input. That's the big saving. Default TTL is 5 minutes, refreshed free on each hit. Constraints: Minimum cacheable length is 1,024 tokens (Sonnet) or 4,096 tokens (Opus, Haiku). Prefix-based — order matters. Static content FIRST (system prompt, policy docs, tool definitions), dynamic content LAST (user message). Any change to cached prefix equals cache miss. Cache READ tokens do NOT count toward input-tokens-per-minute limits — caching multiplies effective throughput. Stacks with Batch API 50% discount.

Pros

• Cache reads at 10% of standard input cost
• Dramatic latency reduction (time to first token)
• Cache-aware rate limits multiply effective throughput
• Stacks with Batch API
• Pays off after just one read (5-min) or two reads (1-hour)

Cons

• Cached content can't reflect live state within the TTL — separate live-state queries from static-knowledge content
• Minimum cacheable length may exclude short prompts
• Any prefix change invalidates the cache
• Needs careful content ordering

Use Cases: Long stable system prompts, policy documents in system prompt, tool definitions, few-shot examples that don't change, multi-turn conversations with stable context.

Extended Thinking / Adaptive Thinking

Overview: A separate block of thinking tokens generated before the final answer. The model reasons internally before responding. Adaptive thinking (effort parameter) is the recommended approach on modern models.

Description: Extended thinking gives the model a private reasoning space — tokens generated before the final answer that aren't shown to the user. Adaptive thinking uses an effort parameter to trade intelligence against latency and cost within a single model. Manual budget_tokens is deprecated on newer models and removed on some (returns 400 error). Thinking tokens are billed as output tokens and add latency.

Pros

• Improves accuracy on hard reasoning tasks
• Model can work through complex problems before answering
• Effort parameter lets you dial reasoning depth without switching models

Cons

• More output tokens (expensive — output is 5x input)
• Adds latency
• Not always needed — run evals without it first
• "It can't hurt" is NOT a valid reason to enable it
• Only enable when measured accuracy gap justifies the cost

Use Cases: Complex reasoning tasks, coding challenges, multi-step analysis, agentic decision-making, problems where accuracy matters more than latency.

Effort Parameter

Overview: A dial that trades intelligence against latency and cost within a single model. Tune this before switching model tiers.

Description: Instead of jumping from Sonnet to Opus when quality isn't good enough, try increasing the effort parameter on Sonnet first. Lower effort means faster, cheaper, less thorough. Higher effort means slower, more expensive, more thorough. This is often a better lever than switching tiers — you stay on one model, one integration, one eval set.

Pros

• Cheaper than switching to a bigger model
• No integration changes
• Fine-grained control
• Same model, same evals

Cons

• Still costs more (more thinking tokens)
• Has limits — if the model fundamentally lacks capability, effort won't fix it
• Needs eval to find the right setting

Use Cases: Model optimization, cost-performance tuning, interview answers about how to improve output quality without switching models.

PART 8 — TRAINING, TUNING & MODEL OPERATIONS

Fine-Tuning

Overview: Adapting a pre-trained model to a specific task, domain, or style by continuing training on a smaller, targeted dataset. When prompt engineering isn't enough, fine-tuning is the next step.

Description: You take a pre-trained model and train it further on your domain data. The model's weights are updated to better fit your data. Levels: full fine-tuning (update all parameters), LoRA (update a small set of adapter parameters — cheaper), QLoRA (LoRA with quantization — cheaper still). Fine-tuning changes the model's behavior; RAG adds knowledge. They're complementary, not either/or.

Pros

• Better domain performance
• Can reduce prompt length
• Smaller fine-tuned models can outperform larger generic models on specific tasks
• Enables custom styles and formats

Cons

• Expensive — needs GPU time and quality labeled data
• Can cause catastrophic forgetting
• Overfits easily with small datasets
• Model updates from the provider may break your fine-tuned version
• Harder to update than RAG (new knowledge equals retrain)

Use Cases: Medical diagnosis support, legal document drafting, brand voice generation, customer support automation, domain-specific code generation.

RLHF (Reinforcement Learning from Human Feedback)

Overview: The technique that turned raw LLMs into helpful assistants. Human raters compare outputs, and the model learns to prefer the better ones.

Description: Three steps. One — start with a pre-trained model. Two — collect human feedback by showing raters two outputs and asking which is better. Three — train a reward model on those preferences and use reinforcement learning to optimize the LLM toward high-scoring outputs. The result: a model that's helpful, harmless, and honest.

Pros

• Produces assistants that feel helpful and natural
• Aligns models with human values
• Reduces toxic and harmful outputs
• The breakthrough that made LLMs usable as consumer products

Cons

• Expensive — needs thousands of human-rated examples
• Subjective — raters disagree, bias creeps in
• Can over-optimize for style over truth
• Cultural bias — raters reflect their demographics

Use Cases: ChatGPT, Claude, Gemini, and all major conversational AI assistants.

Constitutional AI

Overview: Anthropic's approach to AI alignment — training models against a constitution (a set of principles) rather than purely human feedback. Priority order: broadly safe, then ethical, then compliant with guidelines, then genuinely helpful.

Description: Instead of relying solely on human raters (who are inconsistent and biased), Constitutional AI gives the model a set of principles (a "constitution") and trains it to self-evaluate and self-correct against those principles. The model generates responses, critiques them against the constitution, and revises. This creates more consistent alignment than pure RLHF.

Pros

• More consistent than human-only feedback
• Scales better — doesn't need millions of human ratings
• Explicit principles are auditable
• Reduces harmful outputs

Cons

• The constitution itself is a design choice with values embedded
• May not cover every edge case
• Still supplemented by human feedback
• Principles can conflict in practice

Use Cases: Claude's training, AI safety research, alignment methodology discussions in interviews.

Quantization

Overview: Reducing model precision to shrink size and speed up inference. Trade a little accuracy for a lot of efficiency. The key to running LLMs on laptops and phones.

Description: Models store weights as 32-bit floating point by default. Quantization converts these to 16-bit, 8-bit, or 4-bit. A 7B parameter model at 32-bit needs about 28GB of RAM. At 4-bit, it needs about 3.5GB — small enough for a consumer laptop. Techniques like GGUF, AWQ, and GPTQ handle this with minimal accuracy loss.

Pros

• Dramatically faster inference
• Lower memory — run on consumer hardware
• Lower serving cost
• Enables edge and on-device AI

Cons

• Some accuracy degradation, especially at aggressive levels (4-bit)
• Quantization-aware training can mitigate but adds complexity
• Tooling is still maturing

Use Cases: On-device LLMs (Ollama, LM Studio), cost-efficient API serving, mobile AI, edge deployments.

Model Distillation

Overview: Training a smaller model to mimic a larger one. The student learns from the teacher's outputs. Compact model, most of the capability, fraction of the cost.

Description: A large model (teacher) generates outputs on a dataset. A smaller model (student) is trained to produce the same outputs. The student learns from the teacher's behavior, not just raw data. Result: a model 10x smaller that performs nearly as well on the target task.

Pros

• Cheaper to deploy and serve
• Faster inference
• Easier to run on edge devices
• Can be specialized for one task

Cons

• May lose edge-case capability
• Inherits teacher biases and hallucinations
• Needs a good teacher
• Not always better than training the small model directly

Use Cases: Mobile assistants, real-time classification, cost-sensitive APIs, on-device translation, high-throughput inference.

SLM (Small Language Model)

Overview: A compact LLM designed for efficiency over raw capability. Fewer parameters, often specialized, runnable on edge or private infrastructure.

Description: SLMs typically range from 1B to 8B parameters. Models like Phi-3 (Microsoft), Gemma (Google), and Llama 3 8B (Meta) pack surprising capability into small packages. They're fine-tuned for specific tasks, run on consumer hardware, and cost pennies to serve.

Pros

• Lower cost — pennies per million tokens
• Lower latency
• Easier to self-host
• Privacy-friendly — runs on your infrastructure
• Simpler to fine-tune

Cons

• Less general knowledge
• Weaker reasoning on complex problems
• Smaller context windows
• May need fine-tuning to match domain performance of larger models

Use Cases: On-device summarization, enterprise edge deployments, offline apps, high-volume classification, privacy-sensitive applications.

Mixture of Experts (MoE)

Overview: A model architecture that activates only a subset of its parameters per input. Big model capacity with small model compute cost.

Description: An MoE model has multiple expert sub-networks and a router that decides which expert(s) to use for each token. A model might have 8 experts but only use 2 per token — capacity of an 8x larger model, inference cost of a 2x model. Mixtral and GPT-4 are both reported to use this architecture.

Pros

• Scales capacity without proportional compute
• Faster inference per token
• Different experts can specialize
• Efficient GPU memory use

Cons

• Complex to train — load balancing across experts is tricky
• Memory-heavy — all experts must be loaded
• Routing can be suboptimal
• Harder to debug

Use Cases: GPT-4-class models, large-scale translation, multimodal systems, high-throughput inference.

Synthetic Data

Overview: AI-generated data used to train or test other AI models. When real data is scarce, sensitive, or expensive — generate it.

Description: Use a powerful model to generate training examples — customer support conversations, medical case summaries, code with bugs, edge-case scenarios. Or use simulation environments for sensor data. The generated data augments or replaces real data. Risk: if the generator has biases or errors, they propagate.

Pros

• Privacy-preserving
• Cheap and unlimited
• Balances underrepresented classes
• Enables testing rare edge cases

Cons

• Can encode and amplify generator biases
• Quality may be poor without validation
• Models can collapse over generations
• Regulatory acceptance is evolving

Use Cases: Healthcare AI training, rare event simulation, stress testing, privacy-safe development, augmenting small datasets.

Model Version Pinning

Overview: Pin model versions in configuration. Never let versions roll forward silently. Every model swap is a release, gated by the eval suite.

Description: When you deploy an LLM in production, you pin the exact version. When the provider releases a new version, you don't automatically switch — you run your eval suite against the new version, compare results, and only deploy if the delta is acceptable. Maintain a version-update runbook. Monitor the provider's deprecation page.

Pros

• Prevents silent behavior changes
• Gives you control over when and how updates happen
• Eval-gated swaps catch regressions before users do

Cons

• Requires discipline and process
• Eval suite must be current (stale evals give false confidence)
• Lag behind latest features
• More operational overhead

Use Cases: Production LLM deployments, enterprise AI systems, any deployment where behavior consistency matters.

Batch API

Overview: An API mode for async workloads at a 50% discount. Submit batches of requests, get results later. Stacks with prompt caching.

Description: Instead of real-time requests, you submit a batch of requests (could be thousands) and get results back asynchronously — typically within hours. The trade-off: no real-time responses, but 50% cost reduction. Verify BAA coverage for regulated data before using.

Pros

• 50% cost reduction
• Stacks with prompt caching (cache reads at 10% plus 50% batch discount)
• Good for non-latency-bound workloads
• Simple to use

Cons

• Not real-time — results take hours
• Not suitable for interactive applications
• BAA coverage may not extend to batch for regulated data
• Less control over individual request timing

Use Cases: Bulk document processing, large-scale classification, batch summarization, data enrichment pipelines, non-interactive workloads.

PART 9 — EVALUATION & TESTING

Evals as Acceptance Criteria

Overview: Write the eval suite BEFORE production code. It forces measurable success definitions, exposes assumptions early, and gates every change. If you can't write an eval for a behavior, you can't measure whether it's present.

Description: Evals are the AI equivalent of unit tests — but harder because LLM outputs are non-deterministic. The eval workflow has five stages:

1. Define the task in measurable terms.
2. Build a golden dataset with representative inputs including edge cases and adversarial inputs.
3. Run automated checks (code-based pass/fail).
4. Score with a judge (LLM-as-judge for interpretive behaviors).
5. Interpret and act — aggregate scores, check per-category breakdown.

Pros

• Forces measurable success criteria
• Gates every change (model swap, prompt change, retrieval update)
• Catches regressions before users do
• Makes quality objective, not subjective

Cons

• Building good evals is hard
• Golden datasets need maintenance — stale evals give false confidence
• The highest-risk moment is evals present but outdated after a prompt change
• Manual spot-checks are not an eval substitute

Use Cases: Production AI development, model selection, prompt optimization, regression testing, interview answers about AI quality assurance.

Golden Dataset

Overview: A curated test dataset with known-good outputs covering the real input distribution. The foundation of any eval suite.

Description: Build from the REAL input population, not 10 convenient documents the team knows. Include edge cases, adversarial/malformed inputs, and each known failure mode as a labeled category. Multi-turn evals are a separate category — full conversation transcripts checking context retention and quality over turns. Keep the dataset current — an out-of-date eval suite provides false confidence.

Pros

• Representative — catches real-world issues
• Edge cases included
• Adversarial inputs tested
• Per-category breakdowns reveal specific weaknesses

Cons

• Time-consuming to build and maintain
• Can become stale
• Needs domain expertise to curate
• Manual spot-checks verify one input but say nothing about unknown inputs

Use Cases: Eval suite foundation, regression testing, model comparison, prompt optimization.

Grading Ladder

Overview: Three grading methods, ordered by cost. Use the cheapest reliable method first. Code-based, then LLM-as-judge, then human review.

Description:

1. Code-based — schema validation, regex, exact match, length, presence. Milliseconds, free, never drifts. Can't judge interpretation.
2. LLM-as-judge — detailed rubric, constrained verdicts, calibrated against human labels, different model than the one being evaluated (avoids self-preference bias). Inconsistent on borderline cases.
3. Human review — last resort for high-stakes, novel, safety-critical. Slow, expensive, own inconsistency.

Pros

• Cost-efficient — code-based is nearly free
• LLM-as-judge scales
• Human review catches what automated methods miss
• Clear hierarchy

Cons

• Code-based can't assess quality
• LLM-as-judge needs calibration — an uncalibrated judge is worse than no grade (false trust)
• Human review doesn't scale and has its own inconsistency

Use Cases: Eval scoring, quality gates, regression detection, model comparison.

LLM-as-Judge

Overview: Using a model to score another model's outputs. Essential for interpretive tasks that code-based checks can't handle. Must be calibrated and use a different model than the one being evaluated.

Description: You write a rubric (scoring criteria), give the judge model the input, the output, and the rubric, and it returns a score and reasoning. Critical requirements: use a different model than the one being evaluated (self-preference bias — models prefer their own outputs), calibrate against human-labeled examples before trusting, use constrained verdicts (not open-ended scoring), and favor volume over perfection — many auto-gradable cases catch more regressions than few hand-graded ones.

Pros

• Scales better than human review
• Can assess interpretation, tone, reasoning quality
• Cheaper than human grading
• Detailed rubrics produce actionable feedback

Cons

• Uncalibrated judge produces confident scores with no validated link to quality — worse than no automated grade
• Inconsistent on borderline cases
• Judge model has its own biases
• Needs ongoing calibration

Use Cases: Eval scoring for interpretive tasks, quality monitoring, content quality assessment, interview answers about automated evaluation.

Five Eval Axes

Overview: Every eval should measure across five axes: accuracy, latency, cost, safety, security. Each uses different grading methods.

Description:

• Accuracy — exact-match on extracted fields, schema compliance, hallucination rate (code-based).
• Latency — p95 under concurrent load (code-based).
• Cost — tokens times tier per request vs ceiling, distribution not average (code-based).
• Safety — no prohibited action taken, summary faithfulness (code-based for actions, LLM judge for faithfulness).
• Security — no cross-tenant leakage, no PII in output (code-based deterministic scan).

Pros

• Comprehensive — covers all production concerns
• Each axis has appropriate grading method
• Forces you to think about more than just accuracy

Cons

• Five axes means five sets of metrics to maintain
• Some axes conflict (latency vs. accuracy)
• Requires infrastructure to measure all five

Use Cases: Production eval suites, quality gates, regression testing, architecture reviews.

A/B Testing for LLM Systems

Overview: Structured experimentation to compare two versions. Four required components — any missing and it's not an experiment.

Description:

1. Hypothesis — specific, falsifiable, names treatment, metric, threshold, and secondary-metric constraints.
2. Random assignment — consistent per user/session, control the input distribution.
3. Primary metric fixed BEFORE the run — otherwise it's outcome-shopping.
4. Sample size calculated from minimum detectable effect, baseline, and confidence level.

LLM output variance is higher than deterministic systems — you need MORE samples. A 6-point difference needs hundreds of sessions, not 50.

Pros

• Rigorous — distinguishes real effects from noise
• Pre-specified metric prevents retrospective correlation
• Statistical significance ensures you're not shipping noise

Cons

• Needs significant traffic and time
• LLM variance means large sample sizes
• Treatment-input interaction — better on typical inputs, worse on rare edge cases
• Statistical significance does not equal practical significance

Use Cases: Model comparison, prompt optimization, retrieval config changes, any production AI improvement decision.

Shadow Testing

Overview: Run a new version in parallel — users see the current version, the new version is scored offline. Use when a single bad output is too risky, traffic is too low for a split, or regulation forbids exposure.

Description: Every live request is duplicated and sent to both the current (serving) version and the new (shadow) version. Users only see the current version's output. The shadow version's outputs are scored offline using rubrics. This lets you evaluate a new version without exposing users to potential degradation.

Pros

• Zero user risk
• Works with low traffic
• Suitable for regulated industries
• Catches regressions before exposure

Cons

• No downstream signal (user acceptance, follow-ups) — relies on offline rubric only
• Doubles cost (running two versions)
• Offline scoring may miss real-world issues

Use Cases: Regulated industries, high-stakes AI systems, low-traffic applications, pre-deployment validation.

Failure Taxonomy

Overview: Six classes of LLM system failures, each with a distinct signature and fix. Essential for debugging production AI.

Description:

1. Prompt failure — ambiguous/underspecified instruction; model filled the gap. Fix the prompt, not the model.
2. Hallucination — confident, fluent content not grounded in input. Fix: grounding (retrieval, tools, verification). A sterner instruction won't fix it.
3. Model mismatch — wrong tier for task complexity, or swapped without re-eval. Fix: model selection gated by eval suite.
4. Retrieval failure — confident-but-wrong after corpus/index change; model and latency unchanged. Fix: re-index, embedding consistency, chunk boundaries.
5. Context failure — degradation on long inputs; middle content ignored. Fix: reduce context, reorder (head/tail), progressive discovery.
6. Orchestration failure — missing subtask output, fragmented traces. Fix: shared trace ID, recoverable vs unrecoverable boundaries, coverage check at synthesis.

Pros

• Systematic — narrows the problem quickly
• Each class has a specific fix
• Prevents shotgun debugging
• Triage order: what changed? Cheapest check first. Distribution comparison (drift class: model, data, or version)

Cons

• Requires understanding of the full system
• Some failures span multiple classes
• Needs good observability to diagnose

Use Cases: Production debugging, incident response, architecture reviews, interview answers about AI failure modes.

Eval Drift

Overview: Evals present but stale after a prompt or model change. Still green, measuring behavior that no longer exists. The highest-risk moment in AI quality assurance.

Description: You changed the prompt three weeks ago. The eval suite still runs and still passes. But it's measuring the old behavior — the test cases were written for the old prompt's output format. New behavior isn't tested. This gives false confidence — green dashboards, real regressions.

Pros (of knowing this)

• Regularly refreshing the golden dataset prevents false confidence
• Including eval updates in every change process catches drift

Cons

• Easy to forget
• Stale evals are worse than no evals — they create false trust
• Needs process discipline

Use Cases: Quality process design, change management, interview answers about eval maintenance.

PART 10 — SAFETY, RISK & GUARDRAILS

Four-Layer Safety Stack

Overview: Safety is not a setting — it's a stack of layers, each covering a different part of the request path, each with a blind spot the next must catch. The architect places each control and decides what happens when it fails.

Description: Four layers from Claude outward:

1. Trained behavior — broad harm classes, every request, no config. Owned by Anthropic. Blind spot: your domain policy, data rules, auth model.
2. System-prompt instruction — role, tone, constraints. Owned by Architect. Blind spot: adversarial inputs can talk past it — guidance is NOT enforcement.
3. Runtime screening — input/output content detection. Owned by Architect. Blind spot: side-effecting actions (screening doesn't authorize), novel attacks.
4. Authorization — whether this caller may take this action in this context. Owned by Architect. Blind spot: content quality and fairness.

Pros

• Layered defense — each layer catches what the previous misses
• Clear ownership
• Systematic coverage of the request path

Cons

• Most common failure: assuming Claude enforces a rule it was never given
• Trained refusals cover broad harm — your domain policy lives nowhere unless you encode it
• Each layer adds complexity

Use Cases: Production AI safety design, security reviews, interview answers about AI safety architecture.

Training-Time Alignment vs Inference-Time Control

Overview: Two distinct safety mechanisms. Training-time alignment is general (Anthropic's job). Inference-time control is specific (Architect's job). Don't conflate them.

Description: Training-time alignment happens before deployment — Anthropic trains Claude against broad harm classes. Inference-time control happens at request time in your deployment — you enforce domain-specific rules via guardrails. A request can pass Claude's general alignment and still violate your deployment-specific rules. The classic failure: team saw Claude refuse harmful prompts in testing, assumed cross-business-unit data policy was covered. It wasn't — that rule was never in training and never in the application layer.

Pros

• Clear separation of concerns
• Each owner handles what they control
• Prevents the dangerous assumption gap

Cons

• Easy to assume the model handles more than it does
• Requires explicit mapping of which obligations belong to which layer

Use Cases: Safety architecture, compliance design, interview answers about who owns what in AI safety.

Five LLM Risk Categories

Overview: Five ways LLM systems get attacked or compromised. Every architect should know these cold.

Description:

1. Direct prompt injection — user crafts input that overrides system instructions.
2. Indirect prompt injection — malicious instructions arrive through retrieved content or tool outputs. The model treats them as trusted. Input screening never sees them. This is the dominant enterprise attack vector.
3. Token-budget exhaustion — oversized or adversarially padded inputs consume context or output budget.
4. Tool and action abuse — model induced to call a side-effecting tool outside policy.
5. Data exposure — sensitive fields enter context window or logs.

Pros

• Comprehensive threat model
• Each category has specific mitigations
• Walks every entry point (user input, retrieved content, tool outputs, model output, logs)

Cons

• Categories can overlap
• New attack patterns emerge
• Requires ongoing vigilance

Use Cases: Risk assessment, security design, threat modeling, interview answers about LLM security.

Three Guardrail Control Points

Overview: Three points on the guarded request path where controls run. A single filter at the end doesn't cover the other two.

Description:

1. Input screening — before model call. Model-based for ambiguous intent/jailbreak patterns. Deterministic for blocklist, regex, length/format.
2. Output screening — before response returns to user. Model-based for toxicity/policy compliance. Deterministic for known strings, forbidden fields, schema violations.
3. Tool-call authorization — before any side-effecting action. Almost always deterministic: allowlist plus identity plus scope validation. Must be auditable and replayable.

Pros

• Each control checks a different thing at a different point
• Chaining model-based and deterministic covers each other's gaps
• Comprehensive protection

Cons

• One output filter is NOT a guarded path — the refund already executed before the filter read the text
• Output screening judges text, not actions
• Every blocked gate must be logged

Use Cases: Guardrail design, security architecture, production AI deployment.

Fail Open vs Fail Closed

Overview: When a guardrail errors, does it pass traffic through (fail open) or block it (fail closed)? For safety controls, always fail closed — deliberately.

Description: An operator-built guardrail that errors and silently passes traffic is worse than no control — it gives reassurance of protection while providing none. Fail closed means blocking until the control is healthy. This applies to operator-built controls only. Anthropic's built-in safety controls are not operator-configurable and do not fail open.

Pros

• Fail closed maintains safety during failures
• Prevents the illusion of protection
• Clear doctrine

Cons

• Fail closed blocks traffic — availability impact
• Needs monitoring and quick recovery
• Deliberate choice, not default

Use Cases: Safety control design, incident response, production guardrail configuration.

Least-Privilege Tool Configuration

Overview: Every tool exposed to an agent is attack surface plus cost. If a role doesn't need a capability, remove it — don't log it, don't guard it, don't confirm it.

Description: Audit the tool set like permissions: essential vs. merely convenient. Remove out-of-scope tools and record the justification. In orchestrator-worker systems, scope each subagent's tools to its task only. Control types: preventive (privilege removal) is the correct answer. Detective (logging) doesn't reduce attack surface. Compensating (confirmation prompt) guards the privilege but doesn't remove it. Unrelated (bigger model) is not an authorization control.

Pros

• Minimizes attack surface
• Reduces cost
• Clear security posture
• Preventive over detective

Cons

• Requires thorough auditing
• May require custom configurations per role
• Ongoing maintenance as tools evolve

Use Cases: Agent security design, tool configuration, security reviews, interview answers about least privilege in AI.

Skill Supply-Chain Security

Overview: Skills are code bundles. A malicious Skill could make network calls, access the filesystem, read credentials. Audit before trusting.

Description: Four steps:

1. Audit — open the bundle, look for anomalous calls (network, shell, filesystem, credential reads) and out-of-scope operations vs. stated purpose.
2. Runtime confinement — run with least privilege in sandbox: limited file access, limited network, no standing credentials.
3. Trusted-source policy — only trust skills from vetted internal registry, verified publishers, signed releases.
4. Verdict — approve, reject, or remediate (strip offending call, sandbox, pin safer version, re-audit).

Never assume the platform screens skills for you.

Pros

• Prevents supply-chain attacks
• Systematic vetting process
• Defense in depth

Cons

• Adds friction to Skill adoption
• Requires security expertise
• Ongoing vigilance needed

Use Cases: Enterprise Skill deployment, security reviews, third-party tool vetting.

API Refusal Mechanics

Overview: When Claude refuses a request, the API returns specific signals. Understanding these helps with debugging and user experience.

Description: stop_reason: "refusal" with a stop_details object (available since Claude Opus 4.7). stop_details carries policy category and readable explanation (null when no named category). Categories include cyber, bio, frontier_llm, reasoning_extraction — re-check docs at publish time. After a refusal, reset the conversation context — remove or rephrase the triggering turn. Sending the next request on the same refused context returns more refusals.

Pros

• Structured refusal handling
• Programmatic detection
• Clear categories for logging and analytics

Cons

• Categories may change
• Unmapped categories return null
• Users may not understand why their request was refused

Use Cases: Production error handling, user experience design, safety logging, debugging.

PART 11 — HUMAN-IN-THE-LOOP & REVIEW

Review Routing Rule

Overview: Route to a person when low-confidence AND (irreversible OR high-cost). Let confident, reversible, low-cost decisions through. Stakes set WHAT needs review; confidence sets HOW MUCH volume routes.

Description: Three variables: reversibility (can it be undone?), cost of wrong decision (what does the mistake cause?), confidence (system's self-score — useful only if calibrated). When variables conflict, give greater weight to cost and reversibility — they determine consequences. Confidence decides how much high-stakes volume you can safely let through unreviewed. Confidence never changes the stakes.

Pros

• Prevents review overload
• Routes only what matters
• Scalable — most decisions pass through, humans focus on high-stakes

Cons

• Requires calibrated confidence scores (uncalibrated equals useless)
• Stake assessment is subjective
• Edge cases where confidence is high but wrong

Use Cases: Human-in-the-loop design, production AI workflows, regulated industries, interview answers about AI governance.

Three Review Placements

Overview: Where to put the human gate — before the action, after the action, or on a sample. Each has different trade-offs.

Description:

1. Pre-action approval — nothing irreversible happens unreviewed. Costs: latency, reviewer availability, doesn't scale.
2. Post-action audit — action runs immediately, throughput high. Costs: wrong action already landed — only suits reversible, lower-cost.
3. Sampled review — monitors quality without slowing process. Costs: bad decisions slip through unsampled.

Pros

• Flexible — match placement to stakes
• Pre-action for irreversible, post-action for reversible, sampled for monitoring

Cons

• Wrong placement equals either too slow or too risky
• Each has a specific failure mode
• Needs ongoing tuning

Use Cases: Workflow design, compliance gates, quality monitoring.

Consent Fatigue

Overview: Routing everything to review means reviewers click-approve without reading. Review collapses into approval. Two independent failures: volume (routing by volume not stakes) and missing context (no inputs or flag reason).

Description: 400 items per day in a queue with only an approve button. Reviewer sees output but not inputs or why it was flagged. Volume makes careful review impossible. Missing context makes it pointless. The fix: route by stakes, not volume. Show inputs, output, and flag reason. Move review to higher-value checkpoints (plan-level review, not per-step). Claude Code approves the plan, not each step.

Pros

• Plan-level review is scalable and meaningful
• Reduces friction
• Catches the important stuff

Cons

• Requires trust in the system
• Per-step approval feels safer but isn't
• Cultural shift needed

Use Cases: Review process design, team workflows, interview answers about human-in-the-loop at scale.

PART 12 — FAIRNESS, BIAS & TRANSPARENCY

Four Bias Injection Points

Overview: Unequal outcomes enter AI systems at four specific points — all instrumented by the architect, not the model provider. Fairness is an architecture property, not a vendor attribute.

Description:

1. Retrieval corpus — over/under-representation skews context before the model sees it.
2. Prompt framing — encoded assumptions push outcomes.
3. Few-shot examples — carry the corpus's skew.
4. Downstream routing — different groups routed down different paths.

The model passing published bias evals says nothing about YOUR corpus.

Pros

• Knowing the injection points lets you instrument and monitor each one
• Per-subgroup breakdowns reveal concentrated harm that aggregate metrics hide

Cons

• Hard to eliminate completely
• Fairness metrics can conflict — improving for one group may degrade another
• Requires per-subgroup logging

Use Cases: Fairness audits, compliance, architecture reviews, interview answers about AI bias.

Decision Logging

Overview: Capture inputs, retrieved context, model output, and routing per decision — keyed so one decision can be replayed. The transparency backbone for compliance.

Description: Three audiences need different things:

1. Affected user — clear explanation in actionable terms.
2. Regulator — evidence that comparable cases treated consistently, specific decision reconstructable.
3. Build team — full trace to find why a flagged decision went wrong.

The decision log itself is in compliance scope — apply minimization, retention limits, and access controls.

Pros

• Enables explanations for users, evidence for regulators, debugging for teams
• Supports compliance requirements
• Reconstructable on demand

Cons

• Storage and retention management
• The log itself contains sensitive data
• Access controls needed
• Over-logging creates noise

Use Cases: Compliance, audit support, user transparency, debugging, regulatory reporting.

PART 13 — COMPLIANCE & GOVERNANCE

BAA (Business Associate Agreement)

Overview: Required for HIPAA-covered workloads. BAA coverage is per-configuration, not per-vendor. A BAA for one config doesn't extend to another. Beta features are generally excluded.

Description: Before processing PHI through any AI service, verify the specific configuration has BAA coverage. Direct API with signed BAA, AWS Bedrock, and GCP Vertex can all offer BAA-covered configs — but you must verify per config. Betas typically excluded. Minimum-necessary PHI: strip to task-essential fields, use reference IDs over full fields, server-side redaction before the call.

Pros

• Enables compliant AI in healthcare
• Clear regulatory framework
• Per-config verification is precise

Cons

• Per-config verification is easy to overlook
• Betas excluded means latest features may not be available for PHI workloads
• Administrative overhead

Use Cases: Healthcare AI, medical assistants, any workload processing PHI.

Entry Point / Delivery Route Selection

Overview: Where your API traffic terminates. Four routes: Direct Anthropic API, AWS Bedrock, GCP Vertex AI, Microsoft Foundry. Governance constraints rule out routes before other tradeoffs.

Description:

• Direct API — newest features first, fewest hops, strong default.
• AWS Bedrock — region-configurable, fits AWS procurement, explicit region config (global endpoint default breaks residency).
• GCP Vertex — region-configurable, GCP procurement.
• Microsoft Foundry — varies by hosting form, verify per route.

What doesn't change across routes: model behavior, prompting, eval, tool use, context window. What changes: model identifiers, version strings, regional availability, CSP-side features. CSP routes lag first-party API on new features by weeks.

Pros

• Flexibility for different procurement and compliance needs
• Region pinning for data residency
• Fits existing cloud commitments

Cons

• Feature lag on CSP routes
• Region availability varies
• BAA coverage per-configuration
• EU pinning needs cloud route (direct API only supports "us" and "global")
• Entry-point responsibility map needed for multi-route systems

Use Cases: Enterprise AI deployment, compliance-driven route selection, multi-cloud strategies.

Control Register

Overview: Each compliance obligation mapped to a specific technical control, a named owner, and a living evidence artifact. A control with no owner and no artifact is a claim, not proof.

Description: For each obligation (HIPAA, GDPR, FedRAMP), record: the obligation, the specific technical control that achieves it, the named owner accountable, and the evidence artifact showing it's live (signed agreement, config screen, authorization record, returned log query). Revalidate on a cadence — controls go non-operational silently. The residency-drift story: logging config change wrote metadata to a second region for months. Nobody owned the control. No artifact tracked where data landed. Gap surfaced at audit.

Pros

• Audit-ready at any time
• Clear ownership
• Evidence-based, not assertion-based
• Surfaces gaps before auditors do

Cons

• Maintenance burden
• Needs regular revalidation
• Easy to create and forget
• Requires discipline

Use Cases: Compliance management, audit preparation, security reviews, governance.

Training Use vs Retention

Overview: Excluded from training does not equal not retained. Two distinct claims — don't collapse them.

Description: Data may be excluded from model training by default. But it can still be retained for logging, abuse prevention, legal compliance, or configured audit purposes. These are separate claims. A stakeholder asking "is our data used for training?" needs a different answer than "is our data retained?" Conflating them creates false assurance.

Pros

• Precise — answers the right question
• Prevents compliance gaps
• Clear communication with stakeholders

Cons

• Two separate verifications needed
• Easy to assume one covers the other
• Requires clear documentation

Use Cases: Compliance, data governance, vendor evaluation, interview answers about AI data handling.

Five Integration Layers

Overview: Five layers in enterprise AI integration, ordered compliance-first. Getting a layer wrong breaks everything above it.

Description:

1. Compliance — which routes/entry points survive governing constraints?
2. Identity and SSO — where does user identity boundary sit?
3. Authorization and policy — which capabilities does this user/role have?
4. Data handling and PII — what data goes into context window?
5. Observability and audit — what do you need to reconstruct?

Identity is verified server-side before the Claude call — never trust user-asserted roles. The context window is NOT a data-governance boundary — anything passed is transmitted. Multi-tenant means separate API keys per tenant.

Pros

• Systematic — covers all integration concerns
• Ordered by priority (compliance first)
• Clear separation of concerns

Cons

• Each layer adds complexity
• Layers interact — identity affects authorization affects data handling
• Requires cross-functional expertise

Use Cases: Enterprise AI architecture, integration design, security reviews, interview answers about production AI deployment.

PART 14 — PRODUCTION & RELIABILITY

Exponential Backoff

Overview: Retry transient errors (429, timeout, 5xx) with progressively longer delays. The most basic reliability control.

Description: When an API call fails with a transient error, retry — but wait longer each time. First retry after 1 second, then 2, then 4, then 8. This prevents overwhelming the API during outages and gives it time to recover. Add jitter (random variation) to prevent thundering herd — all clients retrying at the same time.

Pros

• Simple to implement
• Handles transient failures gracefully
• Prevents retry storms
• Standard practice

Cons

• Doesn't help with permanent failures
• Adds latency to failed requests
• Needs a max retry count
• Must be combined with fallback chains for production

Use Cases: API client design, production AI systems, any distributed system.

Fallback Chains

Overview: When the primary model or endpoint fails, route to an alternative. Sonnet fails? Try Haiku. Haiku fails? Return cached response.

Description: Define a chain of fallbacks at the orchestration layer. Primary: Sonnet. If Sonnet is unavailable or too slow, fall back to Haiku. If Haiku fails, return a cached or default response. This ensures the system degrades gracefully rather than failing completely.

Pros

• Graceful degradation
• Improves availability
• Handles model-specific outages
• Can also handle latency spikes (fall back to faster model)

Cons

• Fallback model may produce lower quality output
• Needs quality monitoring on fallbacks
• Adds orchestration complexity
• Cached responses may be stale

Use Cases: Production AI systems, high-availability requirements, latency-sensitive applications.

Circuit Breaker

Overview: Trip when error rate exceeds a threshold. Fail fast instead of waiting for timeouts. Cool down before retrying.

Description: Monitor error rates. When errors exceed a threshold (e.g., 50% of requests failing in a 30-second window), trip the circuit breaker — stop sending requests to the failing service. Wait for a cooldown period, then test with a single request. If it succeeds, close the circuit and resume normal traffic. This prevents cascading failures.

Pros

• Prevents cascading failures
• Reduces load on failing services
• Fast failure is better than slow timeout
• Self-healing

Cons

• Adds a stateful component
• Threshold tuning is tricky
• May trip on temporary spikes
• Needs monitoring

Use Cases: Microservice architecture, production AI systems, any system with external dependencies.

POC to Production Gap

Overview: POCs lie about four things: cost (volume), latency (concurrency/p95), reliability (no failure handling), and failure modes (only expected inputs tested). Plan for production from day one.

Description: POC runs 10-50 requests per day — negligible bill. Production runs 10,000 — billing exceeds budget. POC tests one request at a time — median latency looks fine. Production has concurrent load — p95 blows past SLA. POC has no retry, fallback, or circuit breaker — any transient failure takes down the workflow. POC tests expected inputs only — edge cases produce silent degradation or fabricated outputs. Build reliability from the start — retrofitting is far harder.

Pros

• Knowing the gap lets you plan for it
• Production-ready architecture from day one
• Realistic cost and latency models

Cons

• Requires upfront investment in infrastructure
• Slower to launch
• Stakeholders may push for fast POC-to-prod without accounting for the gap

Use Cases: Production planning, architecture reviews, interview answers about AI project lifecycle.

Token Distribution Modeling

Overview: Distributions are skewed — most requests are short, but a tail of long requests consumes disproportionate cost. Average-based models underestimate by 2-3x.

Description: Don't model the average token count. Model the distribution. If 80% of requests use 2,000 tokens but 5% use 20,000 tokens, the heavy tail drives most of your cost. A cost model based on the average (3,000 tokens) will underestimate actual spend by 2-3x. Always model the distribution and plan for the tail.

Pros

• Accurate cost forecasting
• Prevents budget surprises
• Reveals which inputs drive cost

Cons

• Requires real input data (not estimates)
• Distribution shifts over time
• More complex modeling

Use Cases: Cost modeling, production planning, budget forecasting, interview answers about AI cost management.

p95 vs Median

Overview: Design for p95 (95th percentile latency), never median. SLA breaches live in the tail.

Description: Median latency tells you what 50% of users experience. p95 tells you what the slowest 5% experience. SLA breaches, user abandonment, and complaints come from the tail — not the median. A system with a 2-second median and 15-second p95 is failing for 5% of users. Always set SLA targets on p95, monitor p95, and alert on p95 breaches.

Pros

• Catches the tail where problems live
• Realistic SLA targets
• Better user experience

Cons

• p95 is harder to optimize than median
• Costs more to fix tail latency
• May require architectural changes

Use Cases: SLA design, production monitoring, performance optimization, interview answers about AI latency.

Observability (4 Layers)

Overview: Four layers of observability for production AI: request-level tracing, metric aggregation, anomaly detection, and change attribution.

Description:

1. Request-level tracing — model and version, token counts (input/cached/output), latency, stop_reason, tool calls, prompt identifier.
2. Metric aggregation — cost/request, p50/p95 latency, task success rate, error rate by type. Per-request decomposition matters — aggregates can look healthy while 5% of requests eat 80% of budget.
3. Anomaly detection — threshold alerts (cost greater than 150% of 7-day average, p95 greater than SLA) plus distribution comparison for drift.
4. Change attribution — distinguish model drift (behavior changed, inputs stable), data drift (input distribution changed), and model-update effects (version changed).

Pros

• Comprehensive visibility
• Catches issues before users do
• Enables root cause analysis
• Per-request detail prevents aggregate blind spots

Cons

• Four layers of infrastructure to build and maintain
• High data volume
• Needs tooling (dashboards, alerting, log aggregation)
• Cost of observability itself

Use Cases: Production AI monitoring, incident response, capacity planning, interview answers about AI observability.

Change Attribution

Overview: When quality drops, distinguish between three causes: model drift, data drift, and model-update effects. Different causes need different fixes.

Description: Model drift — the model's behavior changed on stable inputs (provider pushed an update). Data drift — the input distribution changed (users asking different questions). Model-update effects — the model version changed (you or the provider switched versions). Each requires a different response: model drift may need prompt adjustment, data drift may need retraining or retrieval updates, model-update effects need eval-gated version management.

Pros

• Narrows the diagnosis quickly
• Prevents wrong fixes
• Systematic troubleshooting

Cons

• Requires good observability data
• Causes can overlap
• Needs historical baselines for comparison

Use Cases: Production debugging, incident response, quality management, interview answers about diagnosing AI issues.

Multi-Tenant Isolation

Overview: Separate API keys per tenant. Shared keys destroy attribution and isolation — one tenant's spike rate-limits everyone.

Description: In multi-tenant AI systems, each tenant (customer, business unit) gets their own API key. This ensures: rate limit issues in one tenant don't affect others, usage is attributable for billing, and data isolation is maintained. Shared keys mean one heavy user can trigger rate limits that block all other tenants.

Pros

• Clean isolation
• Accurate attribution
• Fair resource allocation
• Independent scaling per tenant

Cons

• More keys to manage
• More complex configuration
• Per-tenant monitoring needed

Use Cases: SaaS AI products, enterprise multi-tenant deployments, any shared AI service.

PART 15 — STAKEHOLDER & LIFECYCLE

Structured Discovery

Overview: Discovery is structured elicitation, not conversation. Three-step filter: Listen, Translate, Write down. Four question categories: Must DO, Must NOT do, Must COST, Must PROVE.

Description: Listen to what the stakeholder says. Translate stated preferences into constraints ("seamless" means "latency under 3 seconds" — ask what would break the experience). Write down the translation. Four categories:

• Must DO — capabilities as business outcomes.
• Must NOT do — boundaries, prohibited actions, route-to-human cases. Stakeholders never volunteer these, ask.
• Must COST — latency target, per-interaction ceiling, volume forecast.
• Must PROVE — evidence/audit obligations. Find them in discovery, not legal review.

The killer anti-pattern: proposing an architecture sketch mid-call. A plausible sketch ends the questions.

Pros

• Prevents missed requirements
• Translates vague preferences into concrete constraints
• Forces completeness before design begins

Cons

• Takes time — stakeholders want to jump to solutions
• Requires discipline to not propose too early
• Translation skill needed

Use Cases: Solution design, stakeholder workshops, requirements gathering, interview answers about AI project scoping.

Tradeoff Communication

Overview: Every tradeoff presentation has three elements (plus one for regulated): what do we GAIN, what do we GIVE UP, what does REVERSAL cost once the system is built around it. The reversal cost is the omitted element that changes the meeting.

Description: Most tradeoff presentations cover gains and costs. The reversal cost — how expensive is it to undo this decision after the system is built around it — is usually missing. Without it, stakeholders approve without understanding the commitment. The CTO story: approved "four cents per call" without hearing "equals five-figure monthly line at production volume" or the unwind cost. Accurate presentation, wrong question answered. For regulated workflows, add: what happens to the compliance posture?

Pros

• Complete picture — gains, costs, and reversibility
• Turns technical decisions into business decisions
• Prevents false alignment
• Stakeholders can defend the decision to their leadership

Cons

• Takes more time
• Reversal costs are hard to estimate
• May slow down decisions
• Requires honest framing of limitations

Use Cases: Architecture reviews, stakeholder presentations, design decisions, interview answers about AI project communication.

SLA Management

Overview: An SLA names three things: what's measured, what counts as breach, what happens on breach. Thresholds must trace to tangible sources.

Description: What's measured — latency, availability, quality. What counts as breach — p95 greater than 8 seconds, uptime below 99.5%, eval score below threshold. What happens on breach — escalation, rollback, compensation. Thresholds trace to tangible sources: latency from UX expectation, availability from business criticality, quality from eval acceptance criteria. An untraceable number is an arbitrary target. Cost is the post-launch expectation that breaks most often — production is 1-2 orders of magnitude over pilot.

Pros

• Clear expectations
• Accountable targets
• Actionable breach response
• Traceable to business needs

Cons

• Hard to set initially
• Needs ongoing calibration
• Quality SLAs are harder than latency SLAs
• Cost SLAs need distribution modeling

Use Cases: Production AI operations, vendor management, stakeholder agreements.

Feedback Loops

Overview: The decision layer above observability. Signals lead to Triage lead to Decide lead to Act lead to Review. A dashboard without trigger mapping is useless.

Description: Monitoring collects signals. The feedback loop maps each signal to a trigger, owner, and action. A cost spike visible from week 4 with no trigger means it surfaces at the quarterly review in week 12. The governance table exists before launch: signal type, review trigger, architect action, regulated checkpoint. Regulated reviews fire on a schedule, not a threshold — quarterly output audit fires regardless of eval scores.

Pros

• Catches issues early
• Clear ownership and action
• Prevents signal-without-action gaps
• Systematic

Cons

• Needs governance table maintenance
• Triggers can be wrong
• Requires discipline to act on signals
• Easy to set up dashboards but not loops

Use Cases: Production AI operations, quality management, compliance monitoring, interview answers about AI lifecycle management.

Outcome Document

Overview: Six fields that prove what an AI system is worth. Use case with scope boundary, metric before, metric after (same definition), auditable control, measurement owner, reuse potential.

Description: The CFO trap: volume, latency, and error-rate prove the system RUNS. Only a before/after on the business metric (claims-processing time), backed by an auditable control (clinician-authorization log), proves what it's WORTH. Capture the before-metric at the start — before deployment — or you can never prove the improvement. The outcome document gates the expansion decision.

Pros

• Proves business value, not just technical function
• Auditable
• Gates expansion with evidence
• Reuse potential identified

Cons

• Before-metric must be captured at start (often missed)
• Same definition for before/after is critical
• Needs owner and schedule
• May be incomplete at week 4 — name the owner, confirm the control logs, schedule completion

Use Cases: Project closure, expansion decisions, stakeholder reporting, interview answers about proving AI value.

ROI Mapping

Overview: Four-step ROI mapping: baseline (measured, not estimated), post-deployment state, subtract run cost, payback period plus sensitivity.

Description:

1. Baseline in a business unit (analyst-hours per claim) — from the business owner's operational data, NEVER estimated.
2. Post-deployment state in the SAME unit — include human-review cost if the design requires it.
3. Subtract run cost — value equals operational gain minus recurring run cost.
4. Payback period plus sensitivity — what if volume doubles? What if distribution shifts?

Classic ROI traps: baseline estimated not measured, projection assumes full automation when design has a human gate, run cost from average instead of distribution's heavy tail.

Pros

• Evidence-based
• Catches false assumptions
• Sensitivity analysis reveals fragility
• Finance-team credible

Cons

• Requires measured baselines (often unavailable)
• Time-consuming
• Sensitive to assumptions
• Easy to game with optimistic projections

Use Cases: Business case development, stakeholder approval, expansion decisions, interview answers about AI ROI.

Business Value Pillars

Overview: Five pillars for value alignment: Efficiency, Transformation, Productivity, Solution cost, Performance SLAs.

Description: Every AI use case should map to at least one pillar. Efficiency — doing the same work with fewer resources. Transformation — enabling new capabilities not possible before. Productivity — doing more work with the same resources. Solution cost — reducing the cost of the solution itself. Performance SLAs — improving speed, accuracy, or availability.

Pros

• Common vocabulary with stakeholders
• Forces business framing
• Covers different value types

Cons

• Pillars can overlap
• Hard to measure transformation
• Easy to claim multiple pillars without evidence

Use Cases: Value alignment, stakeholder communication, project prioritization, interview answers about AI business value.

PART 16 — TEAM ENABLEMENT

Champion-and-Batch Rollout

Overview: Don't switch on AI for everyone at once. A champion per department proves the workflow first (2 weeks, real task), runs peer sessions, becomes first-line support. Then batch the rollout.

Description: Mass switch-on produces confused prompts and quiet retreat. The champion model: identify one person per department who will invest 2 weeks in a real task with the AI tool. They build expertise, run peer sessions, and become first-line support. Then roll out in batches — the champion supports each batch. This standardizes adoption and prevents the lumpy adoption pattern (heavy users plus non-users, gain never standardizes).

Pros

• Proven workflow before mass rollout
• Built-in support
• Standardizes adoption
• Champion becomes expert

Cons

• Slower than mass rollout
• Requires champion investment
• Champion may leave
• Needs management support

Use Cases: Enterprise AI adoption, team tool rollout, organizational change management.

Judgment Erosion

Overview: Green tests do not equal understanding. If the author can't explain what the AI-generated change does, hold the merge. A change nobody could explain once leaked data through an unvalidated input.

Description: When developers use AI to write code, tests may pass but the developer may not understand the code. This is judgment erosion — the risk that AI-assisted development produces code nobody understands. The fix: require the submitting developer to explain what the change does and why, including untested inputs. If they can't, hold the merge.

Pros

• Prevents un-understood code in production
• Maintains team capability
• Catches subtle bugs
• Enforces accountability

Cons

• Adds friction to development
• May slow down PRs
• Requires cultural commitment
• Hard to enforce consistently

Use Cases: Code review processes, AI-assisted development governance, team standards.

Verification Checklist (4 Dimensions)

Overview: Four dimensions that gate AI-assisted code before production: correctness, security, maintainability, human understanding.

Description:

1. Correctness — tests exist and pass, behavior matches requirement including edge cases.
2. Security — no secrets, inputs validated, least-privilege calls.
3. Maintainability — reads clearly, follows conventions, no unexplained complexity.
4. Human understanding — the submitting developer can explain what it does and why, including untested inputs.

All four must pass before merge.

Pros

• Comprehensive — covers more than just tests
• Catches security and maintainability issues
• Enforces understanding
• Production-grade quality gate

Cons

• Four dimensions take time
• Subjective dimensions (maintainability, understanding) need judgment
• May slow development

Use Cases: Code review, AI-assisted development, production quality gates, interview answers about AI development governance.

Runbook

Overview: Symptom to cause to action paths. Firefighting fixes one incident — a runbook entry fixes the class.

Description: A runbook maps common symptoms to their architectural cause and the action to resolve. Example: gradual quality decline, no code change — check corpus and index first (retrieval drift). Latency spike — check tier, output length, context saturation. Tool failures — check API changes, permissions, circuit breaker. Cost spike — check volume, tier escalation, cache miss rate. The goal: the team needs the architect for NEW problems only. Everything else is in the runbook.

Pros

• Empowers the team to self-resolve
• Fixes the class of problem, not just one instance
• Reduces architect toil
• Scalable support

Cons

• Needs maintenance as the system evolves
• Easy to forget to add entries after incidents
• Requires initial investment to write

Use Cases: Production support, team enablement, on-call documentation, interview answers about AI operations.

Lumpy Adoption

Overview: Heavy users plus non-users — the productivity gain never standardizes. Fix with champion-and-batch rollout.

Description: Without structured rollout, some team members become power users while others never engage. The average productivity gain looks okay, but it's bimodal — a few people get 10x, most get nothing. The fix: champion per department proves the workflow, then batch rollout with peer sessions standardizes the gain.

Pros

• Identifiable pattern — easy to diagnose
• Fix is proven (champion-and-batch)

Cons

• Requires intervention
• May need re-training
• Cultural resistance from non-users

Use Cases: Adoption monitoring, team productivity analysis, organizational change.

Stalling at Basic Chat

Overview: Team uses AI as a Q&A box but never reaches tool use, repo-aware features, or skills. Access does not equal adoption.

Description: Giving everyone access to an AI tool doesn't mean they use it well. Many teams stall at "ask the AI a question" and never progress to integrating it into their workflow — tool use, codebase navigation, skills, automated reviews. The fix: champion demonstrates advanced workflows, peer sessions transfer the skill, and integration into existing workflows (editor, review, test loop) makes advanced use natural.

Pros

• Clear diagnosis — access metrics look good but real adoption is low
• Fix is structural (integrate into workflow, not side chat)

Cons

• Hard to measure real adoption vs. access
• Requires workflow change, not just tool access
• Cultural shift needed

Use Cases: Adoption assessment, team productivity, AI strategy, interview answers about enterprise AI enablement.

PART 17 — AGENTIC CONCEPTS

AI Agent

Overview: An autonomous system that uses an LLM as its reasoning engine to perceive, plan, and take actions to achieve a goal. Not just a chatbot — a doer.

Description: An agent has three things a chatbot doesn't: tools, memory, and autonomy. It receives a goal, breaks it into steps, calls external tools to act, observes results, and adjusts. The LLM is the brain; tools are the hands; memory is the context.

Pros

• Can handle complex, multi-step workflows
• Adapts to context
• Operates with minimal human intervention
• Composable — agents can call other agents

Cons

• Reliability is the number 1 challenge — agents can loop, call wrong tools, or misinterpret results
• Harder to debug than a single LLM call
• Safety risks — an agent with API access can take real-world actions
• Expensive — multiple LLM calls per task
• Needs guardrails: max tool calls, token budgets, stopping criteria

Use Cases: Coding agents (Devin, Cursor, Claude Code), travel planners, research assistants, IT automation, DevOps remediation.

Agentic AI

Overview: The paradigm of building AI systems that pursue goals through autonomous planning, tool use, and self-correction. Not a specific technology — a design philosophy. The shift from "AI that responds" to "AI that acts."

Description: Traditional AI usage is a single round-trip: you prompt, the model responds. Agentic AI is a loop: the model plans, acts, observes the result, reflects, and repeats until the goal is met or it determines it can't be achieved. This loop — often called ReAct (Reason plus Act) — is the core pattern. The key insight: you don't script the steps — the agent decides them.

Pros

• Automates end-to-end tasks that normally need a human
• Reduces manual toil
• Handles ambiguity and adapts
• Scales human productivity

Cons

• Less predictable than scripted workflows
• Needs strong guardrails, observability, and human oversight
• Failure modes are harder to diagnose
• Cost and latency multiply with each step

Use Cases: Autonomous coding, DevOps remediation, scientific research, supply chain optimization, automated testing, financial analysis.

ReAct (Reasoning + Acting)

Overview: An agent loop pattern where the model alternates between reasoning about what to do and actually doing it. Think, Act, Observe, Think, Act, Observe, done.

Description: The model first reasons about the current state and what action to take ("I need to search for flights to NYC"). Then it acts — calls a tool, runs code, queries a database. Then it observes the result ("The cheapest flight is $420 on United"). Then it reasons again ("$420 is under the $500 budget, but let me check if it fits the calendar"). This continues until the goal is met or the model determines it can't be achieved.

Pros

• More robust than one-shot tool calls — the agent can adjust based on feedback
• Enables exploration and recovery from errors
• Natural fit for complex, multi-step tasks
• The reasoning trace is auditable

Cons

• More tokens and latency per task
• Can get stuck in loops if the model keeps trying the same failed approach
• Requires good prompt design to terminate properly
• Error handling at each step adds complexity

Use Cases: Web research agents, API debugging, multi-step booking agents, data exploration, automated testing, IT remediation.

Copilot

Overview: An AI assistant embedded directly in a user's workflow. Not autonomous — collaborative. The human stays in the driver's seat; the AI is the co-pilot. The dominant AI product pattern today.

Description: A copilot doesn't act on its own. It suggests, drafts, explains, and completes — but the human reviews and approves. GitHub Copilot suggests code as you type. Microsoft 365 Copilot drafts emails and analyzes spreadsheets. The pattern is always the same: the AI does the heavy lifting, the human does the judgment call.

Pros

• High productivity gains with low risk
• Human remains in control — safer for sensitive tasks
• Low friction — works where the user already works
• Easier to build than autonomous agents

Cons

• Can introduce subtle errors if the human trusts it too much
• Privacy concerns — copilots see your code, documents, data
• Can create dependency — users may lose skills over time
• Quality varies by task and model

Use Cases: GitHub Copilot (code), Microsoft 365 Copilot (documents, email, spreadsheets), SQL copilots, design assistants, writing assistants, analytics copilots.

Multi-Agent Systems

Overview: Multiple AI agents working together — collaborating, specializing, or even debating — to solve problems too complex for one agent. Think of it as building a team instead of hiring a generalist.

Description: Instead of one agent doing everything, you create specialists. A planner agent breaks down the task. A coder agent writes code. A reviewer agent checks it. A tester agent runs tests. They communicate through a shared state or message bus. The orchestrator owns the goal: decomposes, delegates, synthesizes. Subagents own scoped sub-tasks in separate contexts. Subagent failure is usually recoverable. Orchestrator failure is usually unrecoverable. Critical fix: coverage check at synthesis — results returned must equal units dispatched.

Pros

• Scales to complex, multi-faceted problems
• Each agent is specialized and optimized
• Mimics real-world team workflows
• Enables parallel execution
• More robust — if one agent fails, others can compensate

Cons

• Coordination overhead — agents can talk in circles or contradict each other
• Emergent behavior is unpredictable
• More LLM calls equals higher cost
• Debugging inter-agent communication is hard
• Over-engineering risk — not every task needs multiple agents
• Weakest observability, hardest failure attribution
• Needs contracts and trace propagation

Use Cases: Software development teams (plan, code, review, test), supply chain simulation, game design pipelines, research with multiple hypotheses, content production workflows.

Orchestration

Overview: The layer that coordinates models, tools, agents, and data sources into a reliable workflow. The glue that turns AI components into production systems.

Description: Orchestration frameworks manage the messy parts: routing requests to the right model, maintaining conversation state, handling retries, managing tool calls, switching between agents, enforcing guardrails, and logging everything. LangChain, LlamaIndex, and LangGraph are popular open-source orchestrators. Good orchestration is what separates a hacky demo from a system that runs in production.

Pros

• Builds reliable systems from individually fragile components
• Handles state, retries, routing, and observability
• Reduces boilerplate
• Enables complex workflows without writing everything from scratch

Cons

• Adds latency and architectural complexity
• Framework lock-in risk
• Abstractions can hide important details
• Debugging framework-specific issues is painful
• Sometimes simpler is better — not everything needs an orchestrator

Use Cases: Production chatbots, RAG pipelines, multi-agent workflows, AI-powered ETL, customer support automation, internal AI tools.

PART 18 — FUTURE & SPECULATIVE

AGI (Artificial General Intelligence)

Overview: Hypothetical AI that can learn any intellectual task a human can, across all domains. The holy grail. The north star.

Description: Today's AI is narrow — it excels at specific tasks but can't transfer skills the way humans do. AGI would reason, learn, plan, and create at human level across any domain. There's no consensus on what counts as AGI, when it might arrive, or whether current approaches will get us there.

Pros

• Could solve scientific challenges beyond human capacity — disease, climate, energy
• Could dramatically increase economic productivity

Cons

• Safety and control concerns — how do you align something smarter than you?
• Societal disruption — job displacement, economic restructuring
• No consensus on timeline or feasibility
• Geopolitical race dynamics could compromise safety

Use Cases: Currently theoretical. Relevant to long-term AI strategy, policy, safety research, and investment decisions.

ASI (Artificial Superintelligence)

Overview: AI that surpasses human intelligence across all domains. Beyond AGI. The speculative endgame that sparks both utopian dreams and existential dread.

Description: If AGI matches human intelligence, ASI exceeds it — potentially by a large margin. An ASI could solve problems in seconds that would take humans centuries. It could improve itself recursively, leading to an "intelligence explosion." The control problem — how do you ensure an ASI's goals align with human flourishing — is unsolved and may be the most important problem in human history. Or it may never happen.

Pros

• Transformative problem-solving — could address fundamental human challenges
• Could unlock scientific breakthroughs we can't currently imagine
• Could lead to post-scarcity economics

Cons

• Existential risk — an unaligned ASI could be the last invention humanity makes
• Governance challenge — who controls it?
• Irreversibility — once created, can it be contained?
• No consensus on whether containment is even possible

Use Cases: Currently theoretical. Relevant to AI safety research, long-term policy, existential risk studies, and ethics.

Quick Interview Prep — One-Liners That Stick

• RAG vs Fine-tuning: RAG adds knowledge; fine-tuning changes behavior. Use both when needed.
• RAG vs Tool Use: RAG is for stable knowledge. Tool use is for live state. Don't RAG what a tool should fetch.
• LLM vs Agent: An LLM generates text. An agent uses tools, plans steps, and takes actions.
• Workflow vs Agent: If you could have written the steps in code, use a workflow. Agents are for when you can't.
• Tokens are the currency: Pricing, latency, and context limits all scale with token count.
• Token distribution matters: Average-based cost models underestimate by 2-3x. Model the heavy tail.
• Grounding kills hallucination: Retrieve real sources, cite them, and force the model to stick to them.
• Temperature controls personality: Low for precision, high for creativity.
• Lost-in-the-middle: Put important content at the head and tail of context, not buried in the middle.
• Prompt caching is the biggest lever: 0.1x read cost, 1.25x write. Static content first, dynamic last.
• Embeddings are meaning as math: Similar content equals nearby vectors. That's semantic search.
• Vector DB is the memory: Store embeddings, search by meaning, power RAG and agents.
• Hybrid search is the production standard: BM25 plus dense, fused with RRF. Add a reranker for precision.
• Chunking makes or breaks RAG: Recursive is the sane default. Parent-child is the production pattern.
• MoE is efficiency through specialization: Activate only the experts you need per token.
• Quantization is compression: Less precision, smaller model, faster inference, slight accuracy hit.
• Distillation is inheritance: Small model learns from big model's behavior.
• Guardrails are seatbelts: No production AI without them. Three control points: input, output, tool-call.
• Fail closed for safety: A guardrail that silently passes traffic is worse than no guardrail.
• Least privilege: Remove unneeded tools. Don't log, guard, or confirm — remove.
• Safety is a stack: Trained behavior (Anthropic) plus system prompt plus runtime screening plus authorization (Architect). Each has a blind spot.
• Indirect prompt injection is the enterprise threat: Screen retrieved content and tool outputs, not just user input.
• Evals before code: If you can't write an eval for a behavior, you can't measure whether it's present.
• Grading ladder: Code-based first, LLM-as-judge second (calibrated, different model), human last.
• Eval drift is the silent killer: Stale evals give false confidence. Update with every change.
• p95 not median: SLA breaches live in the tail. Always design for p95.
• Build reliability from day one: Backoff, fallback chains, circuit breakers. Retrofitting is far harder.
• Model version pinning: Never let versions roll forward silently. Every swap is a release, gated by eval.
• Control register: Obligation equals control plus owner plus evidence. No owner and no artifact equals a claim, not proof.
• Training use is not retention: Excluded from training does not mean not retained. Two separate claims.
• Review by stakes not volume: Low-confidence AND (irreversible OR high-cost) equals human. Confidence filters volume.
• Consent fatigue: Routing everything means reviewers click-approve without reading. Route by stakes, show inputs and flag reason.
• Fairness is architectural: Model passing bias evals says nothing about YOUR corpus. Instrument the four injection points.
• Decision logging: Capture inputs, context, output, routing per decision. Three audiences: user, regulator, team.
• Discovery before design: Four questions — Must DO, Must NOT do, Must COST, Must PROVE. Never propose architecture mid-call.
• Tradeoff presentation: Gains, costs, and reversal cost. The reversal cost changes the meeting.
• Outcome document: Before-metric, after-metric, auditable control. Proves worth, not just that it runs.
• Champion-and-batch: Prove the workflow with one person, then roll out in batches. Mass switch-on fails.
• Judgment erosion: If the author can't explain the AI-generated change, hold the merge.
• Runbook fixes the class: Firefighting fixes one incident. A runbook entry prevents the next one.
• AGI is the north star; ASI is the event horizon: One is the goal, the other is the unknown.
• Agentic systems need observability: If you can't see what the agent is doing, you can't trust it.
• Prompt engineering is the cheapest optimization: Try it before fine-tuning. Always.
• Copilots augment; agents automate: Know which one you're building.
• MCP is USB-C for AI tools: Write a server once, any MCP-compatible client can use it.
• Skills are governed prompts: Versioned, distributable, auditable. Not just a prompt library.
• Batch API saves 50%: Async workloads, stacks with caching. Verify BAA for regulated data.
• Effort parameter before model switch: Dial intelligence vs latency within one model before jumping tiers.
• Context window is not a data-governance boundary: Anything passed is transmitted. Filter by necessity before the call.
• Identity is server-side: Never trust user-asserted roles in the user message. Inject verified identity into system prompt.
• Separate API keys per tenant: Shared keys destroy attribution and isolation.
• An action not logged is an action that cannot be allowed: Observability is the precondition for agent approval.

If you made it this far — this is the vocabulary and the decision framework. The terms are the easy part. Knowing which pattern to reach for, which control belongs at which layer, and which failure signature you're actually looking at is where the work is.

Save it, share it with your team, and come back to it before your next architecture review.

What would you add? Which of these have you seen break in production?