CCA-P Domain 2: Enterprise Integration & Production
CCA-P Domain 2: Enterprise Integration & Production
Domain 2 of the Claude Certified Architect – Professional exam. Covers evals as acceptance criteria, the gap between a POC and a production system, use-case sizing and feasibility, the five enterprise integration layers, and structured A/B testing with observability at scale.
TL;DR
- Write the eval suite before production code — it forces measurable success criteria, exposes assumptions early, and becomes the gate for every model swap, prompt change, or retrieval update. If you can’t write an eval for a behavior, you have no reliable way to know it’s present.
- Grading ladder, cheapest reliable method first: code-based (schema, regex, exact match) → LLM-as-judge (calibrated against human labels, different model than the one being evaluated) → human review (last resort, high-stakes/novel). An uncalibrated judge is worse than no automated grade — it looks trustworthy but isn’t.
- A POC misleads on four dimensions invisible in a demo: cost (10-50 req/day hides the real bill), latency (single-request testing hides p95 under concurrent load), reliability (no retry/fallback/circuit-breaker until it’s forced), failure modes (only expected inputs get tested).
- Reliability controls sit at different layers: exponential backoff near the API call, fallback chains in orchestration, circuit breakers at the service boundary. Build these in from the start — retrofitting is much harder.
- Use-case sizing needs four inputs (call volume, token budget distribution, model tier, sensitivity parameters) and produces one of three feasibility verdicts: feasible as scoped / feasible with constraints / not feasible.
- Five integration layers, compliance first: compliance → identity & SSO → authorization & policy → data handling & PII → observability & audit. Getting compliance wrong means redesigning from scratch; the context window is not a data-governance boundary — anything passed in is transmitted.
- A/B tests need four components before running: a falsifiable hypothesis, randomized treatment/control assignment, one pre-specified primary metric, and a sample size calculated from minimum detectable effect (LLM output variance is higher than deterministic systems, so required samples are larger than intuition suggests).
- Observability has four layers — request-level tracing, metric aggregation, anomaly detection, change attribution — plus a translation layer mapping technical metrics to business metrics, built at design time, not after the first business review.
Core Concepts
Eval Workflow and Types
| Stage | What Happens | Output |
|---|---|---|
| 1. Define task | State behavior in measurable terms | Task spec with pass criteria |
| 2. Build golden dataset | Assemble inputs incl. edge cases | Labeled dataset |
| 3. Automated checks | Compare output vs. expected | Pass/fail per item |
| 4. Score with judge | Model-based judge for interpretive behaviors | Score + reasoning |
| 5. Interpret and act | Aggregate + per-category breakdown | Overall score |
| Type | How | Cost | Limitation |
|---|---|---|---|
| Code-based | Function checks output | Very low | Can’t assess interpretation |
| Model-based | Judge model + rubric | Medium-high | Inconsistent on borderline cases |
| Human-review | Human scores rubric | Highest | Slow, not scalable, own inconsistency |
Multi-turn evals score a full conversation sequence, not a single exchange. Favor volume over perfection — broad, cheap, automatically-gradable coverage catches more regressions than a few painstaking manual cases.
POC to Production
| Dimension | Why Invisible in Demo | Failure Mode |
|---|---|---|
| Cost | 10-50 req/day = negligible bill | Billing exceeds budget post-launch |
| Latency | One request at a time | p95 under load ≠ demo latency |
| Reliability | No retry/fallback/circuit-breaker | One transient failure takes down the workflow |
| Failure modes | Tested on expected inputs only | Silent degradation on edge cases |
Design latency for p95, not median — SLA breaches and abandonment come from the slow tail. Failure modes are architecture-specific: agents break on unbounded tool use and growing context (fix: per-turn token budgets, max tool-call counts, explicit stopping criteria); RAG breaks on retrieval quality drift (fix: monitor precision/recall, keep retrieval in the eval loop); document pipelines break on no exception path for low-confidence extractions (fix: confidence scoring → human review queue); orchestrator-worker systems break on blurred failure boundaries (fix: shared trace ID, coverage check at synthesis).
Use-Case Sizing and Feasibility
Four cost-model inputs: call volume (from the business owner, not developer intuition), token budget per request (model the distribution, not the average), model tier, sensitivity parameters (what if volume doubles?). Scoping sequence: business requirement → capability list → architecture sketch → boundary conditions → SOW.
| Verdict | Meaning |
|---|---|
| Feasible as scoped | All four AI properties favor Claude; cost/latency within ceiling; no compensating control needed |
| Feasible with constraints | Works under specific conditions (doc-length threshold, refresh schedule, human review gate) |
| Not feasible | A property limitation can’t be compensated within scope/budget |
ROI mapping (four steps): baseline in a business unit from the owner’s operational data (not intuition) → predicted post-deployment state in the same unit (include human-review cost if the design requires it) → subtract run cost → state payback period and sensitivity. The most common ROI error: projecting full automation when the design requires human review — labor is reduced, not eliminated, and the gap surfaces in the first operational period.
Enterprise Integration Layers
| Layer | Decision | What Breaks When Wrong |
|---|---|---|
| Compliance | Which routes/entry points survive the governing constraint? | Built on a route that fails legal/security review |
| Identity & SSO | Where does the identity boundary sit? | Claude can’t scope responses to authorized data |
| Authorization & policy | Which capabilities does this role have? | Users reach unauthorized data through an unguarded path |
| Data handling & PII | What data enters the context window? | PII surfaces in plaintext logs and audit |
| Observability & audit | What must be reconstructable later? | Unlogged path = invisible after an incident |
The context window is not a data-governance boundary — every field passed in is transmitted to the API; pass only what the language task actually needs, and redact non-essential PII server-side before the call. An action taken but not logged is, to a security reviewer, an action that cannot be allowed. Multi-tenant deployments need separate API keys per tenant for attribution — a shared key means no way to trace which tenant caused a rate-limit breach.
A/B Testing and Observability at Scale
| Component | Requires |
|---|---|
| Hypothesis | Falsifiable — names treatment, metric, threshold |
| Treatment/control | Random assignment, consistent per user/session |
| Primary metric | Single, defined before the experiment |
| Sample size | From minimum detectable effect, baseline, confidence level |
Shadow testing (run the new version in parallel, serve current version to all users, score offline) applies when a single bad output is too risky, traffic is too low for a live split, or exposure isn’t permissible in a regulated setting — the tradeoff is no downstream signal (acceptance, follow-ups).
| Observability Layer | What It Does |
|---|---|
| Request-level tracing | Model, version, tokens, latency, stop reason, tool calls |
| Metric aggregation | Cost/request, latency p50/p95, task success, error rate |
| Anomaly detection | Threshold alerts (cost spike, latency > SLA), drift comparison |
| Change attribution | Model drift vs. data drift vs. model-update effects |
Failure taxonomy: prompt failure (fix the prompt), hallucination (fix with grounding — retrieval, tool use, verification — not a stronger instruction), model mismatch (fix with eval-gated model selection), orchestrator-workers failure (fix with a shared trace ID across recoverable/unrecoverable boundaries).
Common Pitfalls
- Pitfall: Treating manual spot-checks as an eval substitute. Why: spot-checks confirm one input → output pair but say nothing about unknown inputs. Fix: build a golden dataset from the real distribution, including adversarial and edge cases.
- Pitfall: Sizing cost from average token usage. Why: skewed distributions mean a tail of long requests can consume 80% of spend, underestimating the bill by 2-3x. Fix: model the full distribution, not the mean.
- Pitfall: Retrofitting reliability after launch. Why: backoff, fallback, and circuit breakers are architectural decisions, not bolt-ons — retrofitting them after an outage is far more expensive than designing them in. Fix: build all three from day one, matched to their layer.
- Pitfall: Trusting a user-asserted role in the prompt. Why: anything in the user’s message is user-controlled and fakeable. Fix: verify identity server-side, before the Claude call, and inject only the verified role and authorized scope.
- Pitfall: Running an A/B test with an unspecified primary metric. Why: without pre-specifying the metric, teams gravitate to whichever number moved in the right direction — outcome-shopping. Fix: name the primary metric and sample size before the experiment starts.
Interview Questions
Q — Why is “evals present but out of date” the highest-risk moment for a production regression, worse than having no evals at all? An out-of-date eval suite creates false confidence — it looks like a safety net while actually measuring behavior the system no longer exhibits, so a genuinely breaking change can pass review cleanly. With no evals at all, at least nobody is misled into thinking the system was validated; with stale evals, the team believes it was.
Q — Walk through why a document-processing pipeline’s most common production failure isn’t accuracy — it’s the missing exception path. A pipeline built as a single evaluator-optimizer flow routes every document through the same path regardless of extraction confidence, so it produces wrong outputs on edge cases at the same rate it produces correct outputs on clean ones. The fix isn’t a better model — it’s adding a confidence-scoring step that routes low-confidence extractions to a human review queue, and including those edge cases in the eval set so the gap is visible before production, not after.
Q — In the 50,000-requests/month customer service scenario, why did turning on prompt caching do more work than switching models to hit both the cost ceiling and the latency target? The 5,000-token system prompt was stable across every request, so caching it eliminated the dominant input-cost driver at roughly 10% of the standard input rate — that single lever met both the $800/month ceiling and the 3-second p95 target simultaneously. Switching to Opus would have pushed cost over budget, and raising max_tokens would have made both cost and latency worse, so the highest-leverage fix was structural (cache the stable prefix), not a model-tier change.
Q — What must a production Claude system log, and why does the “context, outcome” pairing matter as much as the request/response pair? Four things: request (model version, input tokens, prompt identifier), response (output tokens, latency, stop reason), context (user role, session ID, whether caching applied), and outcome (did the downstream system accept the output, any rejection signal). The context and outcome fields matter because they’re what let you reconstruct why a decision happened and whether it actually worked downstream — request/response alone tells you what Claude produced, not whether it was correct or authorized.