Private Docker Registry

TL;DR

• Harbor is CNCF-graduated self-hosted private registry with vulnerability scanning, RBAC, and policy enforcement
• imagePullSecrets authenticate pods to private registries; attach to ServiceAccount for centralized rotation instead of per-pod
• Registry hostname in imagePullSecret must exactly match hostname in image reference (e.g., registry.example.com:5000)
• Self-signed certs in internal registries require adding CA certificate to node’s trusted CA bundle to avoid “x509: certificate signed by unknown authority” errors
• Harbor blocks deployment of images with critical CVEs if vulnerability policy is enabled (Trivy/Clair integration)
• kubectl create secret docker-registry creates .dockerconfigjson Secret type for registry authentication

Core Concepts

ImagePullSecret Mechanism

When kubelet pulls an image from a private registry, it reads imagePullSecrets from the pod spec or ServiceAccount. Secret contains registry hostname, username, and credentials (base64-encoded .dockerconfigjson).

kubelet uses these credentials to authenticate to the registry before pulling the image. Without matching credentials, the pull fails (ImagePullBackOff).

Harbor Architecture

Harbor is a self-hosted Docker registry with:

• Authentication: LDAP, OAuth, local accounts
• Authorization: RBAC per project/repository
• Vulnerability Scanning: Integrated with Trivy/Clair; scans images for CVEs
• Policy Enforcement: Blocks deployment of images with critical vulnerabilities
• Image Replication: Multi-site registry synchronization
• Content Trust: Image signing validation

Private Registry Hostname Matching

imagePullSecret credentials are indexed by registry hostname. If pod image reference uses a different hostname than the Secret’s registry field, authentication fails silently (no error, just ImagePullBackOff).

Example mismatch:

# imagePullSecret
---
kind: Secret
auths:
  registry.example.com:5000: { ... }

# Pod image reference
image: registry-internal.local:5000/myapp:v1  # Different hostname!
# Result: authentication fails; kubelet doesn't know which credentials to use

Self-Signed Certificates

Internal registries often use self-signed HTTPS certificates. kubelet rejects these by default (“x509: certificate signed by unknown authority”). Fix: Add registry’s CA certificate to the node’s trusted CA bundle, then kubelet trusts it.

ServiceAccount-Level imagePullSecrets

Attaching imagePullSecrets to a ServiceAccount (instead of every pod) centralizes credential management:

• Update Secret once → all pods using that ServiceAccount inherit new credentials
• Enables credential rotation without redeploying pods
• Single source of truth for registry access

Comparison Table

Command Reference

Create imagePullSecret

# Create docker-registry Secret
kubectl create secret docker-registry my-registry-secret \
  --docker-server=registry.example.com:5000 \
  --docker-username=myuser \
  --docker-password=mypassword \
  --docker-email=user@example.com \
  -n production

# Verify Secret created
kubectl get secret my-registry-secret -o yaml
# Should show: .dockerconfigjson (base64-encoded credentials)

Attach imagePullSecret to ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
imagePullSecrets:
- name: my-registry-secret

Pod Using ServiceAccount with imagePullSecret

apiVersion: v1
kind: Pod
metadata:
  name: myapp
  namespace: production
spec:
  serviceAccountName: app-sa  # Inherits imagePullSecrets from SA
  containers:
  - name: app
    image: registry.example.com:5000/myapp:v1.2.0

Configure Node CA Trust

# On node, add registry's CA cert to trusted bundle
sudo cp registry-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Restart container runtime
sudo systemctl restart containerd

Harbor API Usage

# Login to Harbor
docker login registry.example.com:5000

# List projects
curl -u username:password https://registry.example.com/api/v2.0/projects

# Query image vulnerabilities
curl -u username:password https://registry.example.com/api/v2.0/artifacts/project/image/scan-overview

# List artifacts in a repository
curl -u username:password https://registry.example.com/api/v2.0/projects/myproject/repositories/myimage/artifacts

Common Pitfalls

Pitfall: imagePullSecret Authentication Silently Fails Pod stuck in ImagePullBackOff; image pull fails even with correct credentials in Secret.

• Why: Registry hostname in imagePullSecret doesn’t exactly match hostname in image reference. Or Secret is in wrong namespace, unrelated to pod’s ServiceAccount.
• Fix: Verify hostname match: kubectl describe secret <secret-name> shows registry.example.com:5000; pod image must use identical hostname. Verify imagePullSecret is referenced: kubectl describe pod <pod> shows imagePullSecrets. Verify Secret is in same namespace as pod.

Pitfall: Self-Signed Certificate Rejection Pod fails: x509: certificate signed by unknown authority when pulling from internal registry.

• Why: kubelet validates certificate chain. Self-signed certs (common in private registries) fail validation by default.
• Fix: Add registry’s CA certificate to node’s trusted bundle: sudo cp ca.crt /etc/pki/ca-trust/source/anchors/ (Ubuntu/CentOS), then sudo update-ca-trust. Restart container runtime. Verify: openssl s_client -connect registry:5000 -CAfile /path/to/ca.crt.

Pitfall: Per-Pod imagePullSecrets Complicates Rotation Registry credential rotation requires updating dozens of pod manifests.

• Why: imagePullSecrets specified at pod spec level couples credentials to every deployment.
• Fix: Move imagePullSecrets to ServiceAccount. All pods using that ServiceAccount inherit credentials. Rotation becomes a single Secret update, not manifest changes.

Pitfall: Harbor Vulnerability Policy Blocks Expected Images Deployment fails; Harbor rejects image due to critical CVE.

• Why: Harbor’s vulnerability policy is enabled; images with critical vulnerabilities are blocked from deployment.
• Fix: Update image to patched version (upstream image provider publishes fixed tag). Or disable policy for testing (not recommended for production). Check Harbor UI for scan results: which CVE, severity.

Pitfall: Private Registry Not Accessible from Cluster Pod cannot pull image; network connectivity issue.

• Why: Internal registry’s hostname doesn’t resolve from cluster pods, or network policy blocks egress to registry.
• Fix: Verify DNS: kubectl run -it debug --image=alpine -- nslookup registry.example.com. Verify network connectivity: kubectl run -it debug --image=alpine -- curl -v https://registry.example.com:5000/v2/. Check network policies: kubectl get networkpolicies -A | grep registry.

Interview Questions

Q — Why should imagePullSecrets be attached to ServiceAccounts instead of directly to pod specs? Attaching imagePullSecrets to ServiceAccounts centralizes credential management. All pods using that ServiceAccount automatically inherit the credentials. When rotating registry credentials, a single Secret update propagates to all pods without redeploying them. Per-pod imagePullSecrets couples credentials to every pod manifest, making rotation require updating dozens of deployments and pod specs—operationally expensive and error-prone.

Q — A pod fails with “x509: certificate signed by unknown authority” when pulling from a private registry with a self-signed cert. How would you fix this? Self-signed certificates fail X.509 validation by default. Add the registry’s CA certificate to the node’s trusted CA bundle. On the node: sudo cp registry-ca.crt /etc/pki/ca-trust/source/anchors/ (path varies by OS), then sudo update-ca-trust. Restart the container runtime. Verify with openssl s_client -connect registry:5000 -CAfile /path/to/ca.crt. kubelet now trusts the cert and can pull images.

Q — What is the most common cause of imagePullSecret authentication failure, and how would you diagnose it? Most common: registry hostname mismatch. imagePullSecret credentials are indexed by exact hostname (e.g., registry.example.com:5000). If pod image reference uses a different hostname (e.g., registry-internal.local:5000), kubelet doesn’t know which credentials to use; pull fails silently. Diagnose: kubectl describe secret <secret-name> shows the indexed hostnames; verify pod’s image reference uses identical hostname. Verify Secret is in the same namespace and referenced by the pod/ServiceAccount.

Q — What role does Harbor play in preventing vulnerable images from being deployed, and how does it work? Harbor integrates with vulnerability scanners (Trivy, Clair). When an image is pushed, Harbor scans it for known CVEs. If policy is enabled, Harbor blocks deployment of images with critical/high CVEs. This prevents known-vulnerable code from reaching production. Admins review scan results in Harbor UI (Project → Artifacts → scan overview) and either apply patches (image provider publishes fixed tag) or explicitly approve the image if risk is acceptable.