CCA-P Domain 3: Responsible AI, Safety & Risk
CCA-P Domain 3: Responsible AI, Safety & Risk
Domain 3 of the Claude Certified Architect – Professional exam. Covers the four-layer safety stack, guardrail placement and failure modes, fairness as an instrumented architectural property, stakes-based human review routing, and turning compliance obligations into evidenced controls.
TL;DR
- Safety is a stack of layers, not a setting — each covers a different part of the request path with a blind spot the next layer must catch. The most common safety-design failure: assuming Claude enforces a rule it was never given.
- Four layers, Claude outward: trained behavior (Anthropic-owned, broad harm classes) → system-prompt instruction (architect-owned, guidance not enforcement — can be talked past) → runtime screening (architect-owned, input/output content checks) → authorization (architect-owned, whether a side-effecting action is permitted for this caller, this context).
- Three control points on the guarded path: input screening (before the model call), output screening (before the response reaches the user), tool-call authorization (before any side-effecting action — almost always deterministic, not model-based, because it must be auditable and replayable).
- Model-based and deterministic checks are chained, not chosen between — a classifier can be evaded, a deterministic rule is brittle; neither alone catches everything.
- Indirect prompt injection — instructions arriving via retrieved content or tool outputs — is the dominant enterprise vector because it bypasses user-input screening entirely; it needs its own control applied to retrieved/tool content before it enters context.
- For safety controls, fail closed is the deliberate choice — an operator-built guardrail that silently passes traffic when it errors is worse than one that blocks traffic, because it gives the appearance of protection without the function.
- Fairness enters at four injection points an architect controls: retrieval corpus, prompt framing, few-shot examples, downstream routing — it’s an architectural property you instrument, not a model attribute you assume.
- Review routing is set by stakes (reversibility + cost), not by volume — route low-confidence AND (irreversible OR high-cost) decisions to a person; routing everything causes consent fatigue, where reviewers click through without reading.
- A compliant entry point is a prerequisite, not proof — each compliance obligation must become a specific control, a named owner, and a living evidence artifact, or it’s a claim that can silently stop operating.
Core Concepts
The Safety Stack
| Layer | Covers | Doesn’t Cover | Owner |
|---|---|---|---|
| Trained behavior | Broad harmful/unsafe output classes, every request | Domain policy, data rules, authorization model | Anthropic |
| System-prompt instruction | Role, tone, stated constraints within one request | Anything adversarial input can talk Claude past | Architect |
| Runtime screening | Input/output content screening | Actions with side effects; novel attacks a classifier misses | Architect |
| Authorization | Whether a specific side-effecting action is permitted | Content quality and fairness | Architect |
Anthropic trains Claude against a constitution with a holistic priority order: broadly safe → ethical → compliant with guidelines → genuinely helpful. Training-time alignment reduces broad harm classes and is general by design; inference-time control enforces deployment-specific rules and is the architect’s job entirely.
Guardrails — Risks and Control Placement
Five recurring risk categories: direct prompt injection, indirect prompt injection, token-budget exhaustion, tool and action abuse, data exposure.
| Control Point | Runs | Model-Based Check | Deterministic Check |
|---|---|---|---|
| Input screening | Before model call | Ambiguous intent, jailbreak patterns | Blocklist, regex, length/format |
| Output screening | Before response reaches user | Toxicity, policy compliance | Known string, forbidden field, schema |
| Tool-call authorization | Before any side-effecting action | Rarely | Almost always — allowlist, identity, scope |
Indirect injection needs a separate control: screen retrieved content and tool outputs before they’re appended to context, using the same classifier applied to user input — the blind spot is different because the source is different. On refusal, the API returns stop_reason: "refusal" with a stop_details object carrying a policy category (cyber, bio, frontier_llm, reasoning_extraction); after a refusal, reset the conversation context before continuing, or the next request returns further refusals.
Fail open vs. fail closed: fail open passes traffic through unscreened on error; fail closed blocks until the control is healthy. Choose fail closed for safety controls — this applies to operator-built controls only, since Anthropic’s built-in model safety does not fail open.
Skill supply-chain defense (four steps): audit the bundle for anomalous calls and out-of-scope operations → runtime confinement in a least-privilege sandbox → trusted-source policy (vetted registry, verified publishers, signed releases) → verdict (approve/reject/remediate). Don’t assume the platform screens skills for you — verify what automated vetting actually exists.
Fairness — Injection Points and Explainability
| Injection Point | How Skew Enters |
|---|---|
| Retrieval corpus | Over/under-represents groups |
| Prompt framing | Encodes an assumption that pushes outcomes one way |
| Few-shot examples | Carries the same skew as the corpus |
| Downstream routing | Directs some groups down different paths |
Three audiences need different things from decision explanations: the affected user needs inputs + reason for the outcome in actionable terms; the regulator needs a durable, queryable record proving comparable cases were treated consistently; the build team needs a full trace (prompt, retrieved context, output, routing) to diagnose a flagged decision. Decision logging is the same observability instrumentation as production monitoring, pointed at “why did this specific decision happen” instead of “is the system healthy” — and the log itself is in scope for the compliance register when it contains sensitive personal data.
Review Routing — By Stakes, Not Volume
| Variable | What It Measures |
|---|---|
| Reversibility | How easily a wrong decision can be undone |
| Cost of wrong decision | What the mistake causes if uncorrected |
| Confidence | The system’s self-score — useful only if calibrated |
Routing rule: route to a person when low-confidence AND (irreversible OR high-cost); let confident, reversible, low-cost decisions through. When variables conflict, weight reversibility and cost higher — they determine consequences, while confidence only filters volume.
| Placement | Gives | Costs |
|---|---|---|
| Pre-action approval | Nothing irreversible happens unreviewed | Latency on every routed decision, needs a reviewer available |
| Post-action audit | High throughput | Wrong action has already taken effect |
| Sampled review | Monitors quality without slowing the process | A bad decision can slip through unsampled |
A reviewer must see three things: the inputs that drove the decision, the model’s output, and the reason it was flagged — without all three, review degrades into rubber-stamping. Consent fatigue (routing everything to review) collapses oversight into approval; Anthropic’s pattern is to reduce per-step approvals and move review to higher-value checkpoints (Claude Code uses plan-level review, not per-step).
Compliance — Obligations to Evidenced Controls
Each obligation must become three things: a specific technical control, a named owner, and an evidence artifact (signed agreement, configuration screen, authorization record, log query). A control named in a design document with no owner and no artifact is a claim, not proof. Training-use exclusion and data retention are two distinct claims — don’t conflate “not used for training” with “not retained for logging/abuse-prevention/legal purposes.” A control register maps every obligation to control + owner + evidence and is revalidated on a cadence — configurations drift, and an unowned control can go non-operational with no one noticing until an audit.
Common Pitfalls
- Pitfall: Assuming trained refusals cover deployment-specific policy. Why: Claude refusing dangerous prompts in testing says nothing about whether it knows the partner’s cross-business-unit access rules — those were never in training. Fix: explicitly encode deployment-specific rules in the application layer; never assume coverage from general safety behavior.
- Pitfall: Relying on a single output filter as “the guardrail.” Why: output screening judges text after a side-effecting tool has already executed — by the time the filter runs, the refund has already moved. Fix: authorize side-effecting tool calls before they run, independent of what the output filter later says about the response text.
- Pitfall: Screening only user input, missing indirect injection. Why: enterprise systems run on RAG and tool use, and malicious instructions in retrieved documents or tool outputs bypass input screening entirely. Fix: apply the same classifier to retrieved content and tool outputs before they enter context.
- Pitfall: Routing every decision to human review “to be safe.” Why: volume overwhelms reviewers, who click through without reading — consent fatigue means high-stakes and trivial decisions get the same rubber-stamp approval. Fix: route by stakes (reversibility + cost), not by volume; reserve human attention for the cases that need it.
- Pitfall: Treating a compliant delivery route as the finish line. Why: selecting a HIPAA-eligible or GDPR-compliant route is a prerequisite, not proof — a reviewer wants to know who owns each control and what evidence shows it’s still operating. Fix: build a control register mapping every obligation to a named owner and a living evidence artifact, revalidated on a schedule.
Interview Questions
Q — Why can a system-prompt instruction never be treated as an enforcement mechanism, and what has to sit on top of it? A system-prompt instruction shapes Claude’s behavior within a request, but an adversarial or unusual input can talk the model past it — instructions guide, they don’t guarantee. Enforcement requires runtime screening (catching disallowed content) and authorization (gating side-effecting actions) layered on top, because those are deterministic, auditable checks that don’t depend on the model choosing to comply.
Q — Why are model-based and deterministic checks chained rather than one being chosen over the other? A model-based classifier can be evaded by a user who phrases input to slip past it, while a deterministic rule is brittle — it blocks exactly what it was written to catch and nothing else, missing novel attacks and sometimes over-blocking legitimate lookalikes. Chaining them in series means each control’s blind spot is covered by the other, since no single control catches everything on its own.
Q — In the credit-decision fairness scenario, why did the team’s use of a bias-evaluated model fail to prevent unequal outcomes, and what should they have done instead? The model provider tested and published fairness evaluations for the base model, but the team’s own retrieval corpus over-represented certain cases — an injection point the vendor’s evaluations never touched and the team never monitored. They should have instrumented all four injection points (corpus, prompt framing, examples, routing) themselves and built decision-level logging, since fairness is a property of the whole system the architect assembles, not just the underlying model.
Q — Why does “confidence” filter volume rather than determine stakes in a review-routing design? Confidence is the system’s self-estimate of how likely its own output is to be wrong, but it says nothing about what happens if that output is wrong — a confident output can still be irreversible and high-cost, and a low-confidence output can be trivial to fix. Stakes come from reversibility and cost of a wrong decision; confidence only decides how much of that stakes-bucketed volume can safely be let through unreviewed.