Custom Resource Definitions: Extending the Kubernetes API

TL;DR

• CRD: extends K8s API with custom resources (e.g., Database, Cache); stored in etcd, accessed via kubectl, RBAC-governed like native resources.
• CRD alone is storage only — a separate controller/Operator must watch and reconcile custom resources to take action.
• Schema validation: CRD’s OpenAPI schema enforces field types, required fields, enums at admission time (before etcd).
• Multi-version CRDs: mark one version storage: true (what goes to etcd); others converted on-the-fly via conversion webhook if schemas differ.
• Deletion cascade: deleting a CRD deletes all instances cluster-wide (no orphans).

Core Concepts

CRD Structure

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com        # <plural>.<group>
spec:
  group: example.com
  names:
    kind: Database                    # CamelCase
    plural: databases
  scope: Namespaced                   # Or Cluster
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: [engine]
            properties:
              engine:
                type: string
                enum: [postgres, mysql]
              replicas:
                type: integer
                minimum: 1

Custom Resource Instance

apiVersion: example.com/v1
kind: Database
metadata:
  name: user-db
spec:
  engine: postgres
  replicas: 3

Configuration & Command Reference

Create and list:

kubectl apply -f crd.yaml
kubectl get crd databases.example.com
kubectl get db -A          # List custom resources

Common Pitfalls

• Pitfall: Created CRD but nothing happens when custom resource is created. Why: CRD is just storage; no controller watching it. Fix: implement controller/Operator.

• Pitfall: Deleted CRD expecting to keep custom resource instances. Why: CRD deletion cascades to all instances. Fix: migrate instances first.

Interview Questions

• Why is a controller required separate from the CRD? — Tests understanding that CRD is storage only.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames: ["db"]
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              engine:
                type: string
                enum: ["postgres", "mysql"]
              storageSize:
                type: string
            required: ["engine", "storageSize"]
        required: ["spec"]
    subresources:
      status: {}

Once applied, Database becomes a fully first-class API resource:

apiVersion: example.com/v1
kind: Database
metadata:
  name: payments-db
spec:
  engine: postgres
  storageSize: 50Gi

kubectl get databases
kubectl get db          # shortName works identically to built-ins

2. Schema Validation with OpenAPI v3

The schema block is enforced by the API server at admission time — invalid custom resources (wrong type, missing required field, value outside enum) are rejected before ever reaching etcd, exactly like built-in resource validation. This is a major improvement over the older CRD model (pre-1.16) which had no schema validation at all.

3. Multiple Versions and Conversion

A CRD can serve multiple API versions simultaneously (v1alpha1, v1beta1, v1), with exactly one marked storage: true (the version actually persisted in etcd). Requests for non-storage versions are converted on the fly:

versions:
- name: v1alpha1
  served: true
  storage: false
- name: v1
  served: true
  storage: true
conversion:
  strategy: Webhook
  webhook:
    clientConfig:
      service:
        name: conversion-webhook
        namespace: crd-system

Without a conversion webhook, only trivial (identical-schema) conversions between versions are possible — any real field renaming/restructuring requires implementing one.

4. Hands-on Command Cheat Sheet

List all installed CRDs cluster-wide:

kubectl get crd

Inspect a CRD’s full schema definition:

kubectl explain database.spec --recursive

Check the status/conditions of the CRD registration itself (not the custom resources it defines):

kubectl describe crd databases.example.com

Find every custom resource instance of a given CRD across all namespaces:

kubectl get databases --all-namespaces

5. Architectural Interview Q&A

Q: You kubectl apply a Database custom resource and it’s accepted, but nothing actually happens — no real database gets created. Why? A: A CRD only defines the schema and API surface for a custom resource type — it’s pure data storage with validation, with zero built-in behavior. Something actually watching that resource type and taking real action (provisioning infrastructure, calling a cloud API) requires a separate controller/operator process running in the cluster that watches for Database objects and reconciles the real world to match. The CRD and the controller are two entirely separate concerns — defining the shape of data versus acting on it.

Q: Why does exactly one version in a multi-version CRD need to be marked storage: true, rather than storing every served version separately? A: etcd only ever stores one canonical representation of each object — storing every version simultaneously would create data consistency problems (which version is authoritative if two are edited independently?) and massively bloat storage. Instead, the API server stores objects using the single storage: true version internally, and transparently converts to/from any other served version on read/write requests, giving clients the illusion of native multi-version support while etcd itself only ever holds one source of truth per object.

6. Common Pitfalls

[!WARNING] Deleting a CRD Deletes Every Instance of That Custom Resource Running kubectl delete crd databases.example.com doesn’t just remove the schema definition — it cascades and deletes every single Database custom resource object of that type across the entire cluster immediately, with no confirmation prompt beyond the standard delete warning. This is a common, high-blast-radius mistake when cleaning up or iterating on a CRD’s schema during development; always back up existing custom resource instances (kubectl get databases -A -o yaml > backup.yaml) before deleting or significantly modifying a CRD in any environment with real data.