Admission Controllers: The Last Gate Before etcd

TL;DR

• Admission webhooks: final gate BEFORE etcd; can validate (reject) or mutate (modify) requests after auth/authz.
• Order: Mutating webhooks first (modify request), then Validating webhooks (check final mutated object).
• failurePolicy: Fail blocks request if webhook down (strong policy); Ignore allows it (risky).
• Scope matters: broad scopes with failurePolicy: Fail = cluster outage if webhook fails. Keep scopes narrow.
• Policy engines: Gatekeeper (Rego), Kyverno (YAML native), Kubewarden (WebAssembly).

Core Concepts

Admission Webhook Types

MutatingAdmissionWebhook: runs first, modifies request (inject sidecar, add labels, set defaults).

ValidatingAdmissionWebhook: runs second, only accepts/rejects (checks final, fully-mutated object).

Request ──▶ Authentication ──▶ Authorization ──▶ Mutating Webhooks ──▶ Object Schema Validation ──▶ Validating Webhooks ──▶ etcd

Running mutation before validation is deliberate — you validate the final, fully-mutated object, not the raw one the client originally submitted, since a mutation could itself introduce something that needs validating.

2. Built-in Admission Controllers

Several ship with kube-apiserver and are typically already enabled:

• NamespaceLifecycle — prevents creating objects in a namespace that’s being terminated.
• LimitRanger — enforces LimitRange defaults/min/max.
• ResourceQuota — enforces namespace-level quota.
• PodSecurity — enforces Pod Security Standards (replaced the deprecated PodSecurityPolicy).
• DefaultStorageClass — auto-assigns the default StorageClass to PVCs that don’t specify one.

ps aux | grep kube-apiserver | grep enable-admission-plugins

3. Policy Engines: Gatekeeper and Kyverno

Rather than writing raw webhook server code, most teams use a policy engine that translates declarative policies into ValidatingAdmissionWebhook/MutatingAdmissionWebhook behavior automatically.

OPA Gatekeeper (policies written in Rego):

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-team-label
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Namespace"]
  parameters:
    labels: ["team"]

Kyverno (policies written in plain YAML, no separate policy language to learn):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-limits
    match:
      resources:
        kinds: ["Pod"]
    validate:
      message: "CPU and memory limits are required."
      pattern:
        spec:
          containers:
          - resources:
              limits:
                cpu: "?*"
                memory: "?*"

Kyverno’s YAML-native policies are generally faster to adopt for teams without Rego experience; Gatekeeper’s Rego is more expressive for genuinely complex logic.

4. Hands-on Command Cheat Sheet

List currently registered webhook configurations cluster-wide:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Check why an object was rejected by a policy engine:

kubectl describe pod my-pod
# Look for admission webhook denial messages in Events, or immediate error response on kubectl apply

Test what a Gatekeeper/Kyverno policy would do without enforcing yet (dry-run/audit mode):

validationFailureAction: Audit   # Kyverno: logs violations without blocking

Check enabled admission plugins on the API server directly:

cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep enable-admission-plugins

5. Architectural Interview Q&A

Q: A validating webhook that enforces “every pod must have resource limits” is deployed, but the webhook server itself goes down. What happens to new pod creation cluster-wide? A: This depends entirely on the webhook’s configured failurePolicy. Fail (often the safer default for security-critical policies) means the API server rejects all matching requests when the webhook is unreachable — effectively blocking all new pod creation cluster-wide until the webhook recovers, a serious but intentional fail-closed posture. Ignore means the API server proceeds as if the webhook had approved the request when it’s unreachable — fail-open, prioritizing availability over policy enforcement. Choosing between these is a genuine risk tradeoff that should be made deliberately per-policy, not left at a default without consideration.

Q: Why does mutation happen before validation in the admission chain, rather than validating the client’s original request first? A: If validation ran first against the pre-mutation object, a webhook could pass validation and then have a completely different, potentially non-compliant object slip through after mutation. Running mutations first ensures that what actually gets validated (and eventually stored in etcd) is the final, fully-realized object — the one that will actually run — rather than validating an intermediate representation that never actually exists in the cluster.

6. Common Pitfalls

[!WARNING] A Misconfigured failurePolicy: Fail Can Take Down the Entire Cluster If a validating webhook is scoped broadly (matching most or all resource types) and set to failurePolicy: Fail, any outage or misconfiguration of that webhook’s own backing service effectively halts all matching API operations cluster-wide — including, in worst cases, the ability to fix the problem via normal kubectl operations if the webhook’s own namespace/pods are somehow caught in its own match rules. Always scope webhooks as narrowly as possible (specific namespaces, specific resource types, exclude system namespaces explicitly) and test failurePolicy: Fail webhooks extremely carefully before enabling broadly in production.