GitOps Fundamentals

TL;DR

• GitOps makes Git the single source of truth for cluster state, reconciled continuously by a controller — not a one-time imperative action.
• Manifests are declarative (desired end state), so a controller can always converge toward them regardless of starting point or partial failures.
• Drift detection catches any divergence between live cluster state and Git (manual kubectl edits, out-of-band changes) and either auto-corrects it or flags it as OutOfSync.
• The strongest validation of a GitOps setup is deleting the cluster and rebuilding entirely from Git — a stronger test than restoring from backup.
• Biggest gotcha: OutOfSync is often treated as cosmetic, but it means Git no longer accurately describes reality — exactly the guarantee GitOps exists to provide.

Core Concepts

Declarative Deployments

Manifests in Git describe the desired end state — “there should be 3 replicas of this deployment, running this image” — not a sequence of imperative commands to run. This distinction matters mechanically: an imperative script (kubectl scale, then kubectl set image, then…) can partially fail midway and leave the cluster in an undefined intermediate state. A declarative reconciler instead keeps converging toward the described end state regardless of where it started or what failed along the way.

Desired State as the Core Model

GitOps inverts the traditional deployment mental model. In a traditional CI/CD pipeline, “deploy” is an action a person or script performs once, and success means that action completed. In GitOps, “deploy” is a continuous, ongoing process — a controller constantly compares live cluster state against what Git declares and corrects any gap. This is a meaningfully stronger guarantee than “the script ran and succeeded once,” because a one-time success only proves the state matched at that specific moment. It says nothing about five minutes later, after a manual change, a node failure resurrecting a stale pod spec, or an accidental kubectl edit.

Drift Detection

The mechanism that makes continuous reconciliation real. When live cluster state diverges from what’s declared in Git — a manual kubectl edit during an incident, someone scaling a deployment directly, a change applied by a different tool entirely — the controller detects the gap and, depending on configuration, either automatically reverts it back to the Git-declared state (self-heal) or flags it as OutOfSync for a human to reconcile deliberately. Either way, the manual change does not silently become the new unrecorded reality. OutOfSync should be treated the same as a monitoring alert, not a cosmetic label: it means Git, right now, does not accurately describe what’s actually running, and the gap needs to close before it becomes permanent, undocumented drift.

Validating the GitOps Guarantee

The real test of whether a GitOps setup delivers on its promise: can the cluster be deleted entirely and rebuilt from Git alone, with nothing else? This is a stronger test than restoring from a cluster backup — a backup restore proves the backup mechanism works; a from-Git rebuild proves Git is complete and accurate, which is the actual guarantee GitOps is supposed to provide. Running this drill commonly surfaces manually-applied resources that exist nowhere in Git — exactly the gap the drill exists to find before a real disaster does.

GitOps vs Traditional Push-Based CI/CD

Pull-Based vs Push-Based Reconciliation

Pull-based (GitOps) means the in-cluster controller initiates every change by pulling from Git — nothing outside the cluster needs write access to it, which shrinks the credential attack surface significantly. Push-based means an external system (CI pipeline) pushes changes into the cluster on its own schedule, requiring that system to hold cluster credentials and requiring some separate mechanism to catch state that drifts after the push completes. Pull-based reconciliation is also continuous by construction — the controller keeps checking and correcting indefinitely — whereas push-based deployment is a discrete event with no ongoing verification unless explicitly built.

Command / Configuration Reference

# Generic GitOps controller operations (concepts apply across ArgoCD/Flux)
kubectl get applications -n argocd              # list managed applications and sync status
kubectl describe application myapp -n argocd     # inspect drift / sync details for one app

# Example: declarative desired-state manifest tracked in Git
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3                # desired end state, not an imperative step
  template:
    spec:
      containers:
        - name: myapp
          image: myapp:v1.4.2

Common Pitfalls

• Pitfall: A manual kubectl change made during an incident becomes permanent, undocumented reality. Why: Nobody reconciled the resulting drift back into Git after the incident closed. Fix: Treat every emergency manual change as temporary by default; follow up with a Git commit that either codifies or reverts it.
• Pitfall: OutOfSync statuses accumulate and get ignored because the application “looks fine.” Why: The status is treated as cosmetic rather than as evidence that Git no longer matches reality. Fix: Alert on and investigate OutOfSync the same way as any other operational alert.
• Pitfall: A team confident in its GitOps setup discovers, during its first real “rebuild from Git” drill, that several production resources exist only as manually-applied leftovers. Why: No drill had ever tested whether Git was actually complete. Fix: Run periodic delete-and-rebuild-from-Git disaster recovery drills, not just backup restores.
• Pitfall: Auto-heal (automatic drift correction) silently reverts a legitimate emergency manual fix mid-incident. Why: Self-heal was enabled without the team being aware it would fight manual changes made during an incident. Fix: Make the self-heal vs. manual-reconciliation trade-off an explicit, documented decision per application, and disable self-heal temporarily during active incident response if needed.

Interview Questions

• Explain why GitOps is a stronger guarantee than “we have a deployment pipeline that runs kubectl apply.” — Tests whether the continuous-reconciliation-versus-one-time-action distinction is genuinely understood.
• An application shows OutOfSync but seems to be running fine. Why does this still matter? — Tests whether the erosion of Git-as-source-of-truth is understood as a real risk, not a cosmetic status.
• Design a disaster recovery drill that actually validates a GitOps setup is complete. — Tests whether “delete and rebuild from Git” is reached for over a simpler backup-restore test that proves less.
• Compare pull-based and push-based deployment models in terms of credential exposure and drift handling. — Tests understanding of why GitOps controllers pull rather than accept pushes.