Container Platform Architecture

TL;DR

• Running a container and running a container platform are different problems: the platform layer is identity (IRSA), scaling (Karpenter), deployment safety (progressive traffic shifting), and service connectivity (Service Connect/App Mesh) done deliberately, not improvised per team.
• Node-level IAM instance profiles grant every pod on that node the same permissions — IRSA/Pod Identity scopes credentials to the Kubernetes service account instead, and skipping this is the most common security gap in EKS deployments.
• Karpenter replaces the Auto Scaling Group abstraction Cluster Autoscaler depends on, provisioning right-sized nodes directly from pending pod requirements — faster scale-up, better bin-packing.
• Instant, all-or-nothing blue/green traffic cutovers turn every bad deploy into a full-blown incident; progressive, health-gated shifting with automated rollback contains the blast radius.
• ECR’s value isn’t being the only registry that works — it’s IAM-native auth and image pulls plus built-in vulnerability scanning, removing a separately-managed credential surface.

Core Concepts

ECS vs. EKS as a platform commitment

Covered in the compute architecture module as a technology decision; in the platform context, the more important discipline is committing to one and building real platform conventions around it — identity model, deployment pipeline, observability — rather than half-supporting both because different teams independently chose different orchestrators.

ECR

ECR’s value is not that it’s the only registry technically capable of storing images — it’s what it removes. Pull/push authorization runs through IAM directly, with no separate credential management to rotate or leak. Integration with EKS/ECS image pulls is native, and vulnerability scanning is wired into the same identity model as the rest of the platform. A separately-managed third-party registry is an additional credential surface and an additional integration to maintain, for no capability most AWS-native platforms actually need.

IRSA / EKS Pod Identity

IAM Roles for Service Accounts (IRSA), or its newer replacement EKS Pod Identity, is the single most important security control in EKS platform design and one of the most frequently skipped under deadline pressure. Without it, IAM permissions live on the node’s instance profile: every pod scheduled on that node, regardless of team or workload ownership, inherits those permissions. IRSA scopes credentials to the Kubernetes service account instead, so a pod receives only the AWS permissions its own workload requires. Mechanically, IRSA works by federating the cluster’s OIDC identity provider with IAM — a service account is annotated with an IAM role ARN, and the EKS control plane injects a projected service account token that the AWS SDK exchanges for temporary credentials scoped to that role via AssumeRoleWithWebIdentity. Pod Identity simplifies this further by removing the OIDC trust-policy configuration step, associating roles to service accounts through the EKS API directly. Skipping either mechanism turns a single compromised container into a compromise of every workload co-located on that node.

Cluster Autoscaler vs. Karpenter

Cluster Autoscaler scales through the Auto Scaling Group abstraction — it adjusts the desired count of predefined node groups, meaning capacity is quantized to whatever instance types and sizes those groups were configured with. Karpenter bypasses that abstraction: it observes pending pod resource requirements directly and provisions right-sized nodes on demand, choosing instance types dynamically from a defined set of constraints. This produces faster scale-up (no ASG-level launch coordination) and materially better bin-packing (fewer partially-utilized nodes sitting idle).

Service Connect vs. App Mesh

Service Connect is ECS-native and handles straightforward service discovery and basic traffic management (e.g., simple weighted routing) with minimal configuration. A full service mesh — App Mesh, or a CNCF option like Istio/Linkerd on EKS — earns its added operational complexity when fine-grained mTLS between services, sophisticated traffic shaping for canary releases, or consistent observability across a heterogeneous, multi-platform environment is genuinely required. The complexity of a service mesh (sidecar injection, control-plane operation, certificate rotation) should be justified by a concrete requirement Service Connect’s simpler model cannot meet, not adopted because it is the more sophisticated-sounding option.

Blue/Green deployment safety

An instant, all-or-nothing traffic cutover is the most common near-miss in ECS/EKS deployment design: a bad deploy receives 100% of production traffic immediately, and the blast radius is the entire user base until a human notices and rolls back. The fix is progressive, health-gated traffic shifting — a small percentage of traffic routed to the new version first, with automated rollback triggered by a CloudWatch alarm breach (elevated error rate, latency regression) — so a bad deploy fails on a small fraction of traffic and is caught by automation before it becomes a paging incident.

Cluster Autoscaler vs. Karpenter

Service Connect vs. App Mesh

Command / Configuration Reference

# EKS: IRSA annotation on a Kubernetes service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: team-a-app
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/team-a-s3-read

# Associate a role with a service account via EKS Pod Identity (no OIDC trust policy needed)
aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace default \
  --service-account team-a-app \
  --role-arn arn:aws:iam::123456789012:role/team-a-s3-read

# Enable ECR image scanning on push
aws ecr put-image-scanning-configuration \
  --repository-name my-app \
  --image-scanning-configuration scanOnPush=true

# Karpenter NodePool: constrain instance types and enable consolidation
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]

# ECS CodeDeploy blue/green with linear traffic shifting (10% every minute)
aws deploy create-deployment \
  --application-name my-app \
  --deployment-group-name my-app-dg \
  --deployment-config-name CodeDeployDefault.ECSLinear10PercentEvery1Minutes

Common Pitfalls

• Pitfall: Granting AWS permissions via the node’s instance profile instead of IRSA/Pod Identity. Why: IAM permissions attached to an instance profile apply to every pod scheduled on that node, not just the intended workload. Fix: Use IRSA or Pod Identity to scope credentials to the specific Kubernetes service account.
• Pitfall: Staying on Cluster Autoscaler at meaningful scale despite slow scale-up and poor bin-packing. Why: The ASG abstraction quantizes capacity to predefined node groups rather than actual pending pod requirements. Fix: Migrate to Karpenter for direct, right-sized node provisioning.
• Pitfall: Adopting a full service mesh for a simple, single-platform ECS deployment. Why: Service mesh operational complexity (sidecars, control plane, cert rotation) is taken on without a concrete requirement it uniquely solves. Fix: Default to Service Connect and adopt a mesh only when mTLS, advanced traffic shaping, or cross-platform observability is a genuine requirement.
• Pitfall: Shipping blue/green deployments as an instant, all-or-nothing cutover. Why: A bad deploy receives 100% of production traffic immediately, with no automated safeguard. Fix: Use progressive, health-gated traffic shifting with automated rollback tied to CloudWatch alarms.
• Pitfall: Using an unscanned, separately-authenticated third-party container registry. Why: It introduces an additional credential surface to manage and removes integrated vulnerability scanning from the deployment pipeline. Fix: Use ECR with scan-on-push enabled, integrated into the same IAM identity model as the rest of the platform.

Interview Questions

• “Explain exactly how IRSA scopes permissions differently from a node instance profile, and walk through the failure mode if a team skips it.” — tests real understanding of the OIDC federation mechanism, not just name recognition.
• “Your EKS cluster is slow to scale under sudden load and has poor node utilization. Diagnose and propose a fix.” — tests whether Karpenter is the reasoned answer, not just a buzzword drop.
• “Design a deployment strategy for a customer-facing service that can’t tolerate more than 1% of users seeing a bad deploy.” — tests whether progressive traffic shifting with automated rollback is the instinct, not just “blue/green” as a label.
• “When would you introduce a service mesh instead of staying with ECS Service Connect?” — tests whether the answer is grounded in a concrete requirement (mTLS, cross-platform observability) rather than general sophistication.
• “Walk through what happens end to end when a pod using IRSA calls an AWS API.” — tests depth of understanding of the token exchange and AssumeRoleWithWebIdentity flow.