Kubernetes Operators: Encoding Operational Knowledge as Code

TL;DR

• Operator: CRD + controller encoding domain-specific operational knowledge (backups, failover, scaling, upgrades) beyond generic deployment.
• Reconciliation loop: level-triggered (compares desired vs actual state), self-corrects from missed events, restarts, concurrent updates.
• Leader election: ensures only one active Operator instance (prevents split-brain conflicts during concurrent runs).
• Manual edits don’t stick: Operator reconciles back to desired state; always edit via CRD, not underlying resources.
• Examples: Prometheus Operator, Kafka Operator, MySQL Operator.

Core Concepts

Operator Pattern

Operator = CRD (desired state) + Controller (reconciliation loop)

CRD (PostgresCluster)  →  Operator watches it
  ↓
desired: replicas=3,      Reconciliation loop:
backup_schedule,          1. Get desired state (CRD)
failover_timeout          2. Get actual state (Deployments, etc.)
  ↓                       3. If drift, take action
  └─→ Creates/manages:
      - StatefulSet
      - Backups
      - Failover logic

Reconciliation Loop: Level-Triggered Design

Every cycle, ask: "desired state = actual state?"
│
├─ If YES: do nothing
│
└─ If NO: apply changes to converge
   └─ Self-corrects from missed events,
      restarts, concurrent changes

Why level-triggered (not edge-triggered)? Level-triggered compares current states, so it naturally self-corrects if an event is missed. Edge-triggered reacts to specific event sequences, so a missed event can permanently diverge.

Configuration & Command Reference

Check installed Operators (and their CRDs):

kubectl get crd | grep -v apiextensions  # Operators use CRDs
kubectl get csv -A                         # If using OLM

Watch Operator’s reconciliation loop:

kubectl logs -n operators deployment/database-operator -f | grep Reconcile

Common Pitfalls

• Pitfall: Manually edited a StatefulSet managed by an Operator; edits reverted. Why: Operator’s reconciliation loop detected drift and corrected. Fix: only edit via CRD; let Operator manage underlying resources.

Interview Questions

• Why is reconciliation loop level-triggered, not edge-triggered? — Tests understanding of robustness.

Every controller (built-in or custom) follows the same fundamental pattern:

   ┌─────────────────────────────────────────────┐
   │                                               │
   ▼                                               │
Watch for changes ──▶ Compare desired vs actual ──▶ Take action to converge
   (etcd via API)         state                        (create/update/delete)

This loop runs continuously and is level-triggered, not edge-triggered — the controller doesn’t care about the specific sequence of events that led to the current state, it only ever asks “given the desired state right now, and the actual state right now, what needs to change?” This design makes reconciliation naturally resilient to missed events, controller restarts, and out-of-order delivery.

// Simplified reconciliation loop pseudocode
func Reconcile(req Request) Result {
    desired := getCustomResource(req.Name)
    actual := getCurrentClusterState(req.Name)
    if actual != desired {
        applyChangesToConverge(actual, desired)
    }
    return Result{RequeueAfter: 30 * time.Second}
}

2. What Operators Do Beyond Basic CRUD

• Day-2 operations: automated backup scheduling, point-in-time restore, version upgrades with proper ordering (e.g. upgrade replicas before the primary).
• Self-healing beyond restart: detecting a failed primary in a database cluster and promoting a replica automatically, not just restarting a crashed pod.
• Complex scaling logic: adding a replica to a stateful cluster might require running ALTER statements, re-sharding, or updating a configuration file across all existing members — not just creating another pod.

This is the fundamental distinction from a plain Deployment: a Deployment’s controller only knows “keep N identical replicas running.” An Operator encodes domain-specific knowledge about that particular application’s operational lifecycle.

3. Building Operators: Frameworks

• Operator SDK (part of the Operator Framework) — scaffolds Go-based operators using controller-runtime, the same library Kubernetes’ own built-in controllers are built on.
• Kubebuilder — similar Go-based scaffolding, closely related to Operator SDK (they share underlying tooling).
• KUDO / Metacontroller — lower-code alternatives for simpler operational patterns without writing custom Go controllers.

# Typical Operator SDK scaffolding workflow
operator-sdk init --domain=example.com --repo=github.com/myorg/database-operator
operator-sdk create api --group=example --version=v1 --kind=Database --resource --controller

4. Hands-on Command Cheat Sheet

Check what CRDs (and therefore potentially what Operators) are installed:

kubectl get crd | grep -v "^NAME"

Check an Operator’s own controller pod logs for reconciliation activity/errors:

kubectl logs -n operators deployment/database-operator-controller-manager -f

Check a custom resource’s status conditions (Operators typically populate rich status, not just a phase string):

kubectl get database payments-db -o jsonpath='{.status.conditions}' | jq

Find OperatorHub/OLM-installed operators (if using Operator Lifecycle Manager):

kubectl get csv -A    # ClusterServiceVersion — OLM's installed-operator tracking object

5. Architectural Interview Q&A

Q: Why is a reconciliation loop deliberately designed to be level-triggered rather than edge-triggered (reacting to a specific sequence of events)? A: Edge-triggered systems are fragile — if an event is missed (controller was down, a watch connection dropped momentarily, an intermediate state changed twice before the controller processed the first change), the controller’s internal state can permanently diverge from reality with no self-correcting mechanism. Level-triggered reconciliation simply re-evaluates “desired vs actual, right now” on every pass regardless of history, meaning a missed event, a controller restart, or duplicate/out-of-order delivery all self-correct automatically on the very next reconciliation pass — no event history needs to be perfectly preserved or replayed.

Q: A database Operator promotes a replica to primary during a failover. What could go wrong if two reconciliation loop instances run concurrently against the same custom resource (e.g. during an Operator upgrade with a brief overlap)? A: Without proper concurrency control, both instances could simultaneously decide “no primary exists, promote a replica” and each independently promote a different replica, resulting in a split-brain scenario with two primaries accepting writes — a severe data-consistency incident. This is exactly why production Operators implement leader election (only one active reconciler instance at a time, using the same lease-based mechanism as kube-scheduler/kube-controller-manager HA) and often additional resource-level locking/status checks before taking any destructive or state-changing action.

6. Common Pitfalls

[!WARNING] Treating an Operator as a Black Box During Incidents Operators encode complex, sometimes opaque logic — during an incident, resist the urge to manually edit the underlying resources an Operator manages (a StatefulSet it owns, a Secret it generates) expecting the change to stick. The Operator’s reconciliation loop will typically detect the drift from its expected desired state and revert your manual change on its next pass, often within seconds, which is confusing and wastes critical incident-response time. Always work through the Operator’s own custom resource (the Database object, not the StatefulSet underneath it) or explicitly pause/scale down the Operator first if manual intervention is genuinely necessary.