Model the token DISTRIBUTION, not the average — heavy-tailed inputs underestimate cost by 2-3×
Output tokens ≈ 5× input token price at every tier
p95 latency, never median — SLA breaches live in the tail
Batch API = 50% discount for async workloads
1.4 Lost-in-the-Middle
LLMs recall content at beginning and end of context best, middle worst. Place most important content at head and tail. Relevant to prompt design AND RAG context assembly.
2. RAG — Retrieval-Augmented Generation
2.1 What is RAG?
Analogy: Open-book exam. Don't memorize — look up the right sections, read what's relevant, write answer grounded in source material. RAG = LLM (student with reasoning) + Vector Database (textbook).
2.2 Two Myths Debunked
"RAG is dead" → Maturing, not dying. Corrective RAG, Self-RAG, Agentic RAG are responses to limitations
"Bigger context windows replace RAG" → Cost (1M tokens/query is astronomical), latency (massive context = slow), accuracy (signal buried in noise). Well-built RAG wins on all three
2.3 Chunking Strategies
Strategy
Description
When
Fixed-size
N-token pieces (e.g., 512) + 10-20% overlap
Quick prototyping — loses context at boundaries
Recursive
Split on separator hierarchy (paragraph → sentence → word)
Sane production default
Semantic
Embedding model detects topic shifts
Long-form prose, wandering topics
Layout-aware
Markdown headers, HTML tags, PDF structure
Structured docs, codebases
Hierarchical (parent-child)
Index small child chunks (150-200 tok); return large parent (512-1024 tok)
Most-adopted production pattern
Sizing: 512-1024 tokens/chunk. Too big = signal diluted. Too small = no context. Read your actual chunks.
Retrieval drift (corpus grew, index didn't keep pace)
Check corpus/index first
Exact-term queries failing
Pure vector search misses exact matches
Add BM25 hybrid with RRF
The #1 RAG mistake: Using retrieval where a tool call belongs. Symptoms: stale chunks, answers contradict database. Fix: Call the system that owns live state directly.
Batch vs real-time, caching embeddings, adaptive routing
3. Vector Databases
3.1 The Problem They Solve
Traditional DBs: exact/partial text matching — "Find documents where 'vacation' appears" works.
Vector DBs:meaning-based matching — "How many days off do I get per year?" — traditional DB fails because "vacation" doesn't appear in the query.
3.2 How Vectors Work
Convert words/sentences/images into list of numbers (vector / embedding)
Numbers positioned in mathematical space to capture meaning
Similar meanings → similar numbers → close together in vector space
Documents → Embedding Model → Vectors → Vector Database
User Query → Embedding Model → Query Vector → Similarity Search
→ Top Results + Query → LLM → Grounded, Accurate Response
3.4 Chunking — Critical Mistake
Too large → lose precision (embedding averages to mush)
Too small → lose context ("this applies to enterprise customers" — what is this?)
Starting point: 300-500 tokens/chunk, 50-100 tokens overlap, use semantic chunker
3.5 Decision Framework
Situation
Recommended
Learning / experimenting locally
Chroma DB
Performance + self-host control
Qdrant
Production, no infra management
Pinecone
Hybrid search (vector + keyword)
Weaviate
AWS-native ecosystem
OpenSearch / pgvector
3.6 Beyond RAG
Use Case
How
Music recommendation (Spotify)
Listening history → vector → match song vectors
Content recommendation (Netflix)
Viewing history → vector → match content vectors
Visual search (Pinterest)
Image → vision model → vector → find similar pins
Anomaly detection (Cybersecurity)
Normal traffic clusters; outliers = alerts
3.7 Indexing Concepts
ANN (Approximate Nearest Neighbor): HNSW or IVF indexes for fast similarity search at scale
Distance metric (cosine/dot/L2) must match embedding model's recommendation
Metadata filtering on index = security boundaries (tenant/ACL scoping)
Access control belongs AT THE RETRIEVAL LAYER for confidential docs
Store raw text separately from vectors so you can re-embed without re-parsing
3.8 Vector DB for DevOps/SRE
Concern
Infra Experience
Application
Self-hosting
K8s, Docker, EKS
Qdrant on EKS with Terraform, auto-scaling
Managed service
RDS, ElastiCache
Pinecone (fully managed)
Monitoring
Grafana, Prometheus
Query latency, recall@k, index size, memory
Multi-tenancy
IAM, namespace isolation
Per-tenant collections, RBAC on indexes
Backup & DR
EBS snapshots, cross-region
Vector index backups, point-in-time recovery
4. AI Agents
4.1 What is an AI Agent?
Analogy: LLM on its own = brilliant person locked in a room — can only talk. Agentic AI = giving that person a phone, computer, internet, calendar — can search, run calculations, send emails, write code, call APIs.
An AI agent is an LLM that can perceive its environment, make decisions, and take actions.
4.2 Four Core Components
Component
Role
Brain (LLM)
Reasoning engine — decides what to do next
Tools
Functions the agent can call — search, run code, call APIs
Memory
Short-term: conversation history. Long-term: store/retrieve across sessions
Planning & Reasoning
Break big goal into smaller steps, figure out order
Planning & reasoning is what separates a truly agentic system from one that just answers questions.
4.3 Autonomy Spectrum
Level
Description
Risk
L0: Smart Chatbot
Single Q&A
Minimal
L1: Tool-Use
Model uses tools, you approve every step
Low
L2: Multi-Step Agent
Agent chains steps, you supervise at end
Medium
L3: Fully Autonomous
Long sequences across systems, surfaces results when done
High
More autonomy = more capability BUT also more risk. Start Level 1-2 in production.
4.4 Four Agentic Patterns
Pattern
Description
ReAct
Thought → Action → Observation → Thought → ... — most foundational
What shape is the work? (augmented / workflow / agent)
3
Reference architecture
Can you name the reference architecture?
4
Entry point, model, context
Where does work interact with Claude?
Order: Decomposition → Pattern → Reference Architecture → Model/Context/Entry Point. Governance constraints can rule out entry points BEFORE any other tradeoff.
5.2 Three Layers (Don't Conflate!)
Layer
What It Is
Chosen By
Entry Points
What user/system interacts with
User & work
Build-time interfaces
How engineer programs against Claude
Engineering team
Delivery routes
Where API traffic terminates
Cloud commitments & compliance
5.3 Decomposition — Who Does What?
Owner
What Belongs Here
Claude
Language understanding, summarization, planning, drafting, tool-mediated action
Existing systems
Anything already reliable: order-status service, policy engine, rules table, DB of record
Delegation criteria: Reversibility, Stakes, Accountability.
Key framing: Ask "where do the four properties argue for Claude over the system that already does this right?" — NOT "where can Claude help?"
5.4 Pattern Selection
Pattern
Predictability
Autonomy
When
Augmented LLM
High
Low
Single bounded task, verifiable output
Workflow
Medium
Medium
Predictable shape, bounded judgment per step
Agent
Low
High
Steps can't be determined in advance
Workflow Sub-Patterns
Sub-pattern
When
Chaining
Clear stage handoffs
Routing
Different input types need different handling
Parallelization
Independent sub-tasks
Evaluator-optimizer
Quality verifiable but single pass insufficient
Five-Factor Framework
Factor
Augmented LLM
Workflow
Agent
Predictability
Low risk
Low risk
High risk
Error cost
Medium
Low
High
Observability
Medium
Low
High
Latency
Low
Medium
High
Cost
Low
Medium
High
Agent vs Workflow rule: If you could have written the steps in code, use a workflow.
Try prompting before fine-tuning: Prompt → tools/retrieval → stronger pattern → fine-tuning (last resort).
5.5 Reference Architectures
Architecture
Key Failure Mode
Agent
Non-deterministic failures in trajectory
RAG
Retrieval applied to live state (most common mistake!)
Document processing
No exception path for low-confidence extractions
Customer service / ticket triage
Missing escalation path, unguarded tools
Coding agent
—
Combining architectures: Do it when different parts break differently. Don't do it because you haven't decided what problem you're solving.
5.6 Model & Context Strategy
Model Selection Rules
Start small (Haiku/Sonnet), upgrade on evidence
Effort parameter before model switch
Route per step: Haiku for cheap steps, Opus/Sonnet for hard steps
Model version pinning: pin versions, monitor deprecation, keep runbook, every swap re-runs eval
Price Ladder (MEMORIZE)
Model
Input/Output per MTok
Context
Haiku
$1 / $5
200k
Sonnet
$3 / $15
1M
Opus
$5 / $25
1M
Fable
$10 / $50
1M
Output ≈ 5× input. Batch API = 50% off.
Context Strategies
Strategy
When
Monolithic
Bounded tasks, predictable input
Progressive
Most production workloads — only what next step needs
Retrieval (RAG)
Large corpus, can't preload
Compaction
Long-running, context filling
Budget rule: Don't budget the full context window. Budget for largest realistic conversation + retrieved context + system prompt + margin. Window is a ceiling, not a target.
Extended Thinking
Separate block of thinking tokens before final answer
Billed as output tokens, add latency
Decision rule: Run evals without it first. Enable only when measured accuracy gap justifies cost
Underspecification = gap the model fills with its own assumption, differently each time
System prompt instructions are guidance, NOT enforcement — anything adversarial input can talk past needs a runtime control
5.9 Entry Points & Delivery Routes
Route
When
Anthropic first-party
No cloud preference, want newest features
AWS Bedrock
Existing AWS enterprise agreement
GCP Vertex AI
GCP ML stack
Microsoft Foundry
Microsoft enterprise agreement, Azure footprint
What doesn't change: Model behavior, prompting, eval, tool use, context window.
What changes: Model identifiers, version strings, regional availability. CSP routes lag first-party API by weeks.
Regulated-Industry Constraints (Rule out BEFORE other tradeoffs!)
Constraint
What Survives
Attorney-client privilege
API/SDK behind firm's own gateway
HIPAA (PHI)
API/SDK on BAA-covered config (BAA is per-configuration, not per-vendor)
GDPR / data residency
CSP route with region pinned + DPA
FedRAMP
Claude for Government, Bedrock GovCloud, Vertex Assured Workloads
6. Enterprise Integration & Production
6.1 Evals as Acceptance Criteria
If you cannot write an eval for a behavior, you have no reliable way to measure whether that behavior is present.
Eval Workflow (5 Stages)
Define task — behavior in measurable terms + prompt
Build golden dataset — representative inputs INCLUDING edge cases, counterexamples, adversarial inputs
Run automated checks — code-based pass/fail on unambiguous behaviors
Score with judge — model-based rubric scoring for interpretive behaviors
Code-based → 2. LLM-as-judge (calibrated, different model than evaluated) → 3. Human grading
Judge calibration: Uncalibrated judge = confident scores with no validated link to quality = worse than no automated grade. Favor volume over perfection.
Evals as Gate for Every Change
Every change (model swap, prompt revision, retrieval config) runs through eval suite
Eval suite must be kept current — out-of-date eval = false confidence
Highest risk moment: evals present but out of date
Manual spot-checks ≠ eval substitute
6.2 POC to Production — Four Dimensions Where POC Misleads
Dimension
POC
Production Reality
Cost
10-50 req/day = negligible
Billing exceeds budget
Latency
One request at a time
p95 under concurrent load ≠ median
Reliability
No retry, no fallback
Any transient failure takes down workflow
Failure modes
Expected inputs only
Silent degradation on edge cases
Reliability Controls
Control
What
Where
Exponential backoff
Retry with growing delays on transient errors (429, 5xx)
Close to API call
Fallback chains
Route to alternative model/endpoint
Orchestration layer
Circuit breaker
Trip on error-rate threshold; fail fast
Service boundary
Build reliability from the start — retrofitting is much harder.
Failure Modes by Architecture
Architecture
What Breaks First
Mitigation
Agent
Unbounded tool use, growing context
Per-turn budgets, max tool calls, stopping criteria
RAG
Retrieval quality drift
Keep retrieval quality in eval loop, monitor precision/recall
Document pipeline
No exception path for low-confidence
Confidence scoring → human review queue
Orchestrator-workers
Blurred failure boundaries, dropped subagent
Shared trace ID, coverage check at synthesis
6.3 Use-Case Sizing & Feasibility
Scoping Sequence
Business requirement → capability list
Capability list → architecture sketch (Claude / systems / humans)
Architecture sketch → boundary conditions
Boundary conditions → scope in SOW
Three Feasibility Verdicts
Feasible as scoped — all properties favor, cost within ceiling, latency within SLA
Feasible with constraints — works under conditions (doc-length threshold, refresh schedule, human review gate)
Not feasible — property limitation can't be compensated; name disqualifying constraint + scope reduction that would change verdict
ROI Mapping (4 Steps)
Baseline from operational data, NOT intuition
Post-deployment state in same unit — include human review cost
Subtract run cost — value = operational gain minus recurring run cost
Payback period + sensitivity — what if volume doubles / distribution shifts?
Common ROI Errors
Baseline estimated rather than measured → finance team rejects
Projection assumes full automation when design requires human review → actual hours don't fall
Run cost from average not distribution → understates cost on heavy-tailed inputs
6.4 Enterprise Integration — Five Layers (Compliance First!)
Layer
What Breaks When Wrong
Compliance
Built on route that fails legal/security review
Identity & SSO
Claude can't scope responses to authorized data
Authorization & policy
Users access unauthorized data through unguarded path
Data handling & PII
PII in plaintext in request logs
Observability & audit
No evidence after incident
Identity & Authorization Rules
Identity verified server-side, BEFORE Claude call — inject role + authorized data into system prompt
Never trust user-asserted identity — "As a senior manager..." in user message is fakeable
Authorization is deterministic — allowlist + identity + scope before ANY side-effecting tool call
Context window is NOT a data-governance boundary
Multi-tenant = separate API keys per tenant
Observability — Four Things to Log
Request: model version, input token count, prompt identifier
No single control catches everything → deploy in series
Fail closed for safety controls — fail open = illusion of protection
Output screening judges TEXT, not ACTIONS — a side-effecting tool needs authorization before it runs
7.4 Fairness — Four Injection Points
Injection Point
How Skew Enters
Retrieval corpus
Over/under-represents groups
Prompt framing
Encodes assumptions pushing outcomes
Few-shot examples
Carry corpus skew
Downstream routing
Different groups routed differently
Fairness is an architectural property you instrument, not a model attribute you assume.
Decision Logging
Capture: inputs, retrieved context, model output, routing — keyed for replay. Three audiences: affected user (actionable explanation), regulator (consistency evidence), build team (full trace).
7.5 Review Routing — By Stakes, Not Volume
Route to a person when: low-confidence AND (irreversible OR high-cost)
Let through: confident, reversible, low-cost
When variables conflict, cost and reversibility win. Confidence sets volume.
Placement
Gives
Costs
Pre-action approval
Nothing irreversible unreviewed
Latency, doesn't scale
Post-action audit
Throughput high
Wrong action already landed
Sampled review
Quality monitoring
Bad decisions slip through
Reviewer must see 3 things: inputs, output, reason it was flagged.
Consent fatigue: Routing everything → reviewers click-approve → review collapses. Anthropic pattern: plan-level review, not per-step.
7.6 Compliance — Obligations to Controls with Evidence
Evidence/audit obligations (find in discovery, not legal review)
Translation Table
Stakeholder Statement
Implied Constraint
Architectural Decision
Assumption
"Seamless"
Latency budget, graceful failure
Set p95 target, safe failure state
Confirm "seamless" = responsiveness + continuity
"Clinicians will review anyway"
Licensed human must authorize
Human-in-the-loop as mandatory gate
Confirm authority and timing
"Be careful with data"
Privacy proof obligation
Audit-trail as core requirement
Confirm with compliance
Requirement vs assumption: Requirement traces to what stakeholder SAID. Unsourced assumption = most dangerous — nobody remembers deciding it.
Killer anti-pattern: Proposing architecture sketch mid-call. Plausibility ENDS the questions. Finish all four categories before proposing anything.
8.2 Tradeoffs & GTM
Three Elements (+1 Regulated)
What do we GAIN?
What do we GIVE UP?
What does REVERSAL cost? ← most skipped, most often changes the meeting
(Regulated)Compliance posture?
Reversal cost turns "what is the better technical answer?" into "what is the better business choice?"
Demo Design
Capabilities Demo
Scenario-Specific Demo
Answers
"What can this do?"
"What does this do with MY problem?"
Creates
Interest
Confidence
Limit placement: Decide in advance which 1-2 limitations to name, frame as intentional scope boundaries. Buyer discovering mid-demo drops confidence; naming early reads as discipline.
Three Objection Categories
Category
Response
Capability
Demonstrate capability
Governance & compliance
Show controls and evidence artifacts
Design-choice
Explain tradeoff + what alternative would have cost
Monitoring collects signals; feedback loop maps each signal to trigger + owner + action. A dashboard without that mapping = the eval-drift-for-12-weeks story.
SLA Names Three Things
What's measured? 2. What counts as breach? 3. What happens on breach?
Production volume = 1-2 orders of magnitude above pilot
Pre-empt: consumption forecast at production volume + spend-control posture BEFORE first invoice
Governance Table (Before Launch)
Signal
Trigger
Action
Regulated Checkpoint
Output quality
Score crosses threshold
Diagnose drift, iterate vs re-architect
Periodic output audit on schedule
Latency p95
Crosses UX budget
Investigate bottleneck
Usually none
Cost/interaction
Crosses budget
Identify driver, bring tradeoff to stakeholder
Usually none
Data-residency
Scheduled confirmation
Confirm posture, flag drift
Residency confirmation on schedule
Regulated reviews fire on a SCHEDULE, not a threshold.
8.4 Documentation — Handoff & Audit
Three Readers
Reader
What They Need
Handoff recipient
Decisions + REJECTED alternatives + why each was rejected
Compliance reviewer
Each obligation → control → owner → evidence artifact
Returning architect
Dated decisions, labeled assumptions, owned open items
Completeness Test
Can a competent Architect who was not in the room make a safe change after reading the document?
Why Rejected Alternatives Matter
Without them, a successor reverses the right decision for an understandable wrong reason — the financial-services postmortem: replacement switched context strategy to fix performance, reintroduced data-handling pattern violating residency. The diagram carried WHAT; the WHY left with the architect.
8.5 Outcome Document (6 Fields)
Use case with scope boundary
Metric BEFORE
Metric AFTER (same definition)
Auditable control (what makes comparison provable)
Measurement owner
Reuse potential
Volume, latency, and error rate are not business outcomes. Only before/after on the business metric proves what it's WORTH.
Champions first, then batches — NOT all-hands switch-on
Skills distribution
Four mechanisms (see below)
Spend posture
Model defaults, allowlists, effort guidance, spend/rate/per-user caps — BEFORE first bill
Champion-and-Batch Rollout
Champion per department — proves workflow, absorbs friction, builds local examples
Batch rollout — champion runs sessions for peers
Broad rollout — every department has working example, local expert, tuned config
Four Skills Distribution Mechanisms
Mechanism
Access
Versioning/Rollback
When
Org-provisioned Skill
Everyone in org
NO versioning/rollback
Capability needed by all, no governance
Plugin
Group/org targeting
Version-controlled, rollback
Scoped distribution with governance
Claude Code project Skills
Scoped to projects
Version with repository
Coding conventions for one team
API Skills
Programmatic
Explicit version pinning
Partner products calling programmatically
Classic failure: Skill shipped with no way back — one bad edit → wrong format across every team. When shared asset needs versioning/rollback → use org-managed plugin.
9.2 Dev Workflows
AI tooling pays off when it lives inside the existing workflow (editor, review, test loop), not a side chat window
Two failure modes:lumpy adoption (few heavy users, rest don't use → fix with champion/batch) and stalling at basic chat (Q&A box, never reaches tool use → access ≠ adoption)
Verification Checklist (Four Dimensions — Gates Before Production)
Dimension
Check
Correctness
Tests exist and pass; behavior matches requirement incl. edge cases
Security
No secrets, inputs validated, least-privilege calls
Maintainability
Reads clearly, follows conventions, no unexplained complexity
Human understanding
Developer can EXPLAIN what it does and why, including untested inputs
Judgment erosion: Green tests ≠ understanding. If author can't explain it, HOLD the merge.
9.3 Operational Support
Support = Translation, Not Firefighting
Team reports symptom → architect connects to architecture cause → teaches the path
Firefighting fixes one incident; a runbook entry fixes the class
Symptom → Cause Quick Map
Symptom
Likely Architecture Cause
Latency spike
Model tier mismatch, output length, context saturation
Output quality degradation (no code change)
Model/prompt drift, retrieval drift (corpus grew, index didn't keep pace)
Tool started failing
Tool API change, permission/scope issue, circuit breaker tripped
Cost spike
Volume increase, model tier escalation, caching not applied
Self-Sufficiency Artifacts
Runbook: captured symptom→cause→action paths — team resolves without Architect
Escalation path: named definition of who handles what and when
Goal: Team needs you only for NEW problems, not ones you've already taught them
10. Architecture Decision Frameworks
10.1 The Universal Decision Flow
1. Governance constraints → rule out entry points/routes BEFORE anything else
2. Decomposition → Claude / existing systems / humans
3. Pattern → augmented LLM / workflow / agent (simplest that works)
4. Reference architecture → don't invent; combine only when parts break differently
5. Model → start Sonnet, every swap = release gated by eval
6. Context strategy → progressive for production; budget for realistic, not ceiling
7. Entry point → by the work, not by what's on the shelf
10.2 Architecture Pattern Selection (7 Patterns)
Pattern
When
Failure Mode
Augmented LLM
Single well-defined task
Capability ceiling
Workflow (chaining)
Predictable multi-step, known decomposition
Rigidity
Routing
Distinct input categories
Misclassification cascades
Parallelization
Independent subtasks
Aggregation complexity
Orchestrator-workers
Unpredictable subtask structure
Blurred failure boundaries
Evaluator-optimizer
Clear eval criteria + iterative refinement
Loop cost; no exception path
Agent
Open-ended, unpredictable step count
Unbounded tool use, runaway cost
Rule: Use the simplest pattern that meets the requirement. Every added layer = cost + latency + failure surface.
Stable knowledge (was true yesterday, will be tomorrow)
RAG (retrieval from indexed corpus)
Live state (current value owned by a system)
Tool call to system of record
Exact terms, IDs, error codes
Hybrid search (BM25 + dense + RRF)
Relationship questions ("how does X affect Y and Z")
Graph RAG
Complex multi-step queries
Agentic RAG
10.6 Cost Optimization Decision Tree
Is system prompt long and stable? → Prompt caching (biggest lever)
Can steps use cheaper model? → Model tiering (Haiku for extraction, Sonnet/Opus for reasoning)
Need to tune within model? → Effort parameter
Async workload? → Batch API (50% off, stacks with caching)
Output too long? → Structured outputs, max_tokens budgets
Retrieval over-fetching? → Top-k tuning
Cost model from averages? → Switch to distribution modeling
Agent loops unbounded? → Per-turn budgets + stopping criteria
10.7 Safety Design Checklist
Drew boundary between trained behavior and application layer?
Every deployment-specific rule encoded in application layer (not assumed)?
Three control points: input screening, output screening, tool-call authorization?
Model-based and deterministic checks chained in series?
Fail closed for safety controls?
Indirect injection covered (screen retrieved content and tool outputs)?
Tool-call authorization BEFORE action executes (not after)?
Every blocked/failed gate logged?
10.8 Human-in-the-Loop Design
Is decision low-confidence AND (irreversible OR high-cost)? → Route to human
Is decision confident, reversible, low-cost? → Let through
When variables conflict → cost and reversibility win; confidence sets volume
Placement: pre-action (irreversible) / post-action (reversible) / sampled (monitoring)
Reviewer sees: inputs + output + flag reason (missing any = rubber-stamping)
Avoid consent fatigue: plan-level review, not per-step
POC → production sizing → volume × token DISTRIBUTION × tier; p95 not median; backoff+fallback+breaker from day one
11. Interview Q&A Quick Reference
Q: Explain transformer architecture at a high level
A: Transformers process input tokens in parallel (not sequentially like RNNs). Core mechanism is self-attention — each token attends to all other tokens, computing weighted relationships. Multi-head attention captures different relationship types simultaneously. Positional encoding adds sequence order. Feed-forward networks transform representations. The model predicts the next token by assigning probabilities across vocabulary. This architecture enables long-range dependencies and parallel training, which is why it scaled where RNNs/LSTMs couldn't.
Q: Fine-tuning vs RAG vs prompt engineering — when to use which?
RAG: When the model needs knowledge it doesn't have (proprietary docs, recent data, domain-specific). No model weight changes. Grounds answers in retrievable sources. Best for enterprise knowledge bases
Fine-tuning: Last resort. When you need the model to consistently follow a specific format, style, or behavior pattern that prompting can't reliably achieve. Changes model weights. Expensive, requires data, harder to iterate. Doesn't solve knowledge gaps — it shapes behavior
Q: Design a multi-tenant RAG system with fallback mechanisms
A:
Vector DB: Per-tenant collections/namespaces with RBAC. Metadata filtering at retrieval layer for ACL scoping
Embedding pipeline: Jenkins CI/CD for ingestion → chunking → embedding → storage. Same embedding model for indexing and querying
Retrieval: Hybrid search (BM25 + dense + RRF) for precision. Cross-encoder reranker for high-stakes queries
Fallback chain: Primary model (Sonnet) → fallback (Haiku) for latency spikes → cached response for API failures
Multi-tenancy: Separate API keys per tenant for attribution and isolation. IAM boundaries, VPC isolation
Reliability: Exponential backoff on 429/5xx. Circuit breaker at service boundary. Fallback chain in orchestration layer
Monitoring: Query latency, recall@k, index size, cost per tenant, retrieval quality in eval loop
Cost: Prompt caching for stable system prompts. Adaptive RAG routing. Batch API for async workloads
Q: Vector database trade-offs (Pinecone vs OpenSearch vs pgvector)
A:
Pinecone: Fully managed, auto-scaling, no infra to manage. Best when you want production without ops overhead. Cost scales with usage
OpenSearch (AWS): AWS-native, integrates with existing AWS infra. Hybrid search support. More control but more operational burden. Good when already on AWS
pgvector (PostgreSQL extension): Simplest if already running Postgres. Good for prototyping and moderate scale. No separate infrastructure. Best when vector search is secondary to relational queries
Drift detection: Monitor retrieval precision/recall over time. Corpus grows without reindexing = retrieval drift. Model behavior changes on stable inputs = model drift. Input distribution changes = data drift
SLOs for ML: Latency p95, task success rate, hallucination rate, cost per interaction. Error budgets apply
Business translation layer: Map task-success → first-contact-resolution, latency → handle-time. Build at design time
Feedback loop: Signals → Triage → Decide → Act → Review. Dashboard collects; feedback loop maps each signal to trigger + owner + action
Q: Design cost-optimized inference infrastructure on AWS
A:
Model serving: EKS with GPU autoscaling using KServe or Seldon Core. Scale to zero for low-traffic
Model tiering: Haiku for classification/extraction, Sonnet for general, Opus only where evals prove needed
Prompt caching: Static system prompts cached. Cache reads at 10% of input rate. Static content first, dynamic last
Batch API: 50% discount for async workloads. Stacks with caching
Guardrails: Three control points — input screening, output screening, tool-call authorization. Chain model-based and deterministic in series. Fail closed for safety. Screen retrieved content and tool outputs for indirect injection
Retrieval quality metrics: Precision@k, recall@k, MRR (mean reciprocal rank), NDCG. Keep in eval loop. Monitor for drift when corpus changes
Human-in-the-loop: Route by stakes (confidence × reversibility × cost), not volume. Reviewer sees inputs + output + flag reason
Q: How would you approach a stakeholder who says "we want AI in our product"?
A:
Don't propose a solution. Run structured discovery: Must DO, Must NOT do, Must COST, Must PROVE
Decompose: What Claude owns / existing systems own / humans own
Run feasibility through four AI properties
Present tradeoffs with three elements: gain, give up, reversal cost
Size the cost model before code: volume × token distribution × tier. Plan for heavy tail
Name limitations upfront — frame as intentional scope boundaries
Q: What's the difference between a workflow and an agent?
A:
Workflow: Fixed sequence of LLM calls. Predictable, auditable, cost-known per request. Use when you could write the steps in code
Agent: Model plans, acts via tools, observes results, iterates in a loop. Open-ended, variable step count. Needs guardrails: max tool calls, token budgets, stopping criteria
Decision rule: If you could have written the steps in code, use a workflow. Don't choose an agent because the task "feels" open-ended
Q: How do you handle model version updates in production?
A:
Pin model versions in configuration
Monitor deprecation page
Maintain version-update runbook
Every model swap re-runs the eval suite — treat it as a release
Never let versions roll forward silently
Fallback chains: Primary → alternative model → cached response
Q: Walk me through the safety stack for an AI deployment
A:
Four layers, each covering a different part of the request path:
Trained behavior (Anthropic): Broad harm classes, every request. Does NOT know your domain policy
System-prompt instruction (Architect): Role, tone, constraints. Guidance, NOT enforcement
Authorization (Architect): Whether this caller may take this action. Deterministic, auditable, BEFORE action executes
Key principle: Claude cannot enforce a rule it was never given. The dangerous failure is silent — assuming Claude enforces a rule that doesn't live in any layer.
Q: How do you design evals for an LLM system?
A:
Write eval suite BEFORE production code
Build golden dataset from REAL input population, including edge cases, counterexamples, adversarial inputs
Grading ladder: Code-based (cheapest) → LLM-as-judge (calibrated, different model) → Human review (last resort)
Set threshold from business requirement, not prototype performance
Eval = gate for every change: model swap, prompt revision, retrieval config
Keep eval suite current — out-of-date eval = false confidence
Multi-turn evals = separate category scoring full conversation sequences
Q: Explain data flywheel and reinforcement learning loops
A:
Data flywheel: System generates data through usage → data improves model → better model attracts more usage → more data. Key: capture user feedback (thumbs up/down, corrections, acceptance/rejection signals) and feed back into training/eval data
RL loops (RLHF): Human preferences on model outputs → reward model → policy optimization → improved model. In production: user feedback acts as implicit reward signal. Must be careful about reward hacking and distribution shift
Practical implementation: Log every interaction + outcome. Periodically retrain/fine-tune on high-quality interactions. Keep eval suite current to catch regressions. Automated rollback on metric degradation
12. Stakeholder Meeting Cheat Sheet
Before the Meeting
Run structured discovery (4 questions: Must DO, Must NOT do, Must COST, Must PROVE)
Translate preferences into constraints (write translation table)
Identify assumptions vs requirements (trace each to stakeholder statement)
Size the cost model (volume × token distribution × tier)
Prepare tradeoff presentation (gain, give up, reversal cost)
Prepare demo with buyer's data shape (scenario-specific, not capabilities)
Decide which 1-2 limitations to name upfront (frame as scope boundaries)
Notes compiled from: ai-agent.md, ai-engineering.md, cert.md, rag.md, vectordb.md, CCA-P module notes (Claude Platform, Enterprise Integration, Responsible AI, Stakeholder Engagement, Team Enablement), CCAR-P Master Prep Guide, and Combined Glossary. Verify model lineup and pricing at platform.claude.com before exams or production decisions.