Enterprise Networking

TL;DR

• Networking decisions — CIDR allocation, hub-and-spoke topology, choice of connectivity primitive — are among the most expensive to reverse once workloads depend on them; they warrant more upfront rigor than typical application design decisions.
• Central IPAM before VPC creation prevents CIDR overlap; overlapping CIDRs cannot be peered or routed between without re-IPing one side.
• Full-mesh VPC Peering scales as N(N-1)/2 connections and becomes unmanageable well before 20-30 VPCs; Transit Gateway/Cloud WAN centralize this into one attachment per VPC.
• NAT Gateway bills hourly and per-GB processed — VPC Gateway Endpoints for S3/DynamoDB remove that traffic from the NAT path at no additional cost.
• PrivateLink/VPC Lattice expose a single service at the service layer, avoiding CIDR overlap concerns and shrinking blast radius versus peering an entire VPC.
• Direct Connect and Site-to-Site VPN solve different constraints (bandwidth/latency guarantees vs. provisioning speed) and are typically paired, not chosen exclusively.

Core Concepts

CIDR planning and IPAM

The foundational decision, and the one that most needs to happen before any VPC is created: CIDR allocation. IPAM centralizes allocation across accounts and Regions so independently-provisioned VPCs cannot pick overlapping ranges. Overlapping CIDRs cannot be peered or routed between — the only fix once discovered is re-IPing one of the VPCs, which is operationally disruptive in production. A central IPAM pool enforced at VPC-creation time removes this failure mode structurally rather than relying on manual coordination between teams.

IPv4 vs IPv6

Private IPv4 space inside a growing multi-account fleet is a genuinely finite resource — /16 blocks run out faster than expected once VPCs are provisioned per-team, per-environment. IPv6 (or dual-stack) removes private-range exhaustion entirely, and AWS service support for it has matured enough that new landing zones should have a deliberate adoption plan, even where a full cutover isn’t immediate or necessary.

NAT Gateway and Gateway Endpoints

NAT Gateway bills both hourly and per-GB of data processed, so a workload pulling meaningful data volume through it — large S3 downloads, chatty external API calls — pays twice for that traffic. VPC Gateway Endpoints for S3 and DynamoDB remove that traffic from the NAT path entirely, at no additional hourly or per-GB charge; routing to these two services through a Gateway Endpoint instead of NAT is close to a free win and typically a small routing-table change.

Transit Gateway and Cloud WAN vs VPC Peering

VPC Peering is a point-to-point connection between exactly two VPCs, with no transitive routing — a peering connection to VPC B and one to VPC C does not let B reach C. This is fine for two or three VPCs but grows quadratically as connectivity requirements expand: full-mesh peering across N VPCs requires N(N-1)/2 connections, so 30 VPCs need 435 individual peering connections, each with its own route table entries. Transit Gateway replaces this with a hub-and-spoke model: one attachment per VPC, centralized route tables, transitive routing between spokes through the hub. Cloud WAN extends this further into a managed global network with policy-based routing across Regions and on-premises connections, intended for organizations outgrowing a single hub-and-spoke TGW design.

PrivateLink and VPC Lattice

Solve a different problem than peering or Transit Gateway: exposing a specific service, not an entire network. When a producer team wants to offer an API without granting the consumer full network-level reachability into its VPC — and without CIDR overlap being a concern at all — PrivateLink (and its more flexible successor, VPC Lattice) connects at the service layer rather than the network layer. The blast radius is meaningfully smaller: a compromised consumer can reach only the exposed service, not the producer’s entire subnet.

Direct Connect and VPN

The two on-premises connectivity options carry a genuine trade-off. Direct Connect provides dedicated, predictable bandwidth and lower latency but requires weeks to provision physically. Site-to-Site VPN is available within minutes over the public internet but with variable latency and lower guaranteed throughput. Production hybrid architectures typically use Direct Connect as the primary path with VPN configured as an automatic failover, rather than relying on either exclusively.

VPC Peering vs Transit Gateway vs Cloud WAN

PrivateLink/VPC Lattice vs VPC Peering

Direct Connect vs Site-to-Site VPN

Command / Configuration Reference

# Create an IPAM pool to centralize CIDR allocation across accounts
aws ec2 create-ipam-pool --ipam-scope-id <scope-id> --address-family ipv4

# Create a VPC Gateway Endpoint for S3 — removes S3 traffic from the NAT path
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0123456789 \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-0123456789 \
  --vpc-endpoint-type Gateway

# Create a Transit Gateway and attach a VPC
aws ec2 create-transit-gateway --description "hub-tgw"
aws ec2 create-transit-gateway-vpc-attachment \
  --transit-gateway-id tgw-0123456789 \
  --vpc-id vpc-0123456789 \
  --subnet-ids subnet-0123456789

# Create a PrivateLink endpoint service to expose one service to consumer VPCs
aws ec2 create-vpc-endpoint-service-configuration \
  --network-load-balancer-arns arn:aws:elasticloadbalancing:...

Common Pitfalls

• Pitfall: No central IPAM pool before VPC creation. Why: Independently-provisioned VPCs can pick overlapping CIDRs with no coordination mechanism to prevent it. Fix: Enforce a central IPAM pool as the only source of CIDR allocation for new VPCs.
• Pitfall: NAT Gateway costs silently balloon. Why: NAT Gateway bills hourly and per-GB processed, and nobody wired up S3/DynamoDB Gateway Endpoints to remove that traffic from the NAT path. Fix: Route S3/DynamoDB traffic through VPC Gateway Endpoints; audit NAT data-processed metrics against actual necessary internet egress.
• Pitfall: Full-mesh VPC Peering used as the topology past a handful of VPCs. Why: Peering connections scale as N(N-1)/2, turning into unmanageable route-table sprawl well before 30 VPCs. Fix: Migrate to Transit Gateway (or Cloud WAN for multi-Region policy routing) once VPC count and growth trajectory outpace simple peering.
• Pitfall: Exposing an entire VPC via peering when the actual requirement was one team calling one API. Why: Peering grants network-level reachability into the whole peered CIDR, far beyond what’s needed. Fix: Use PrivateLink or VPC Lattice to expose only the specific service.
• Pitfall: Direct Connect with no VPN failover. Why: A single physical circuit is a single point of failure that looks fine until it fails. Fix: Configure Site-to-Site VPN as an automatic failover path alongside Direct Connect.

Interview Questions

• “Design connectivity for 40 VPCs across 3 Regions that all need to reach a shared services VPC, with the fewest moving parts.” — tests whether Transit Gateway/Cloud WAN is reached for instead of defaulting to peering out of familiarity.
• “A NAT Gateway bill tripled this month with no new EC2 spend. Walk through how to investigate.” — tests operational debugging instinct, not just terminology recall.
• “When would you choose VPC Lattice over Transit Gateway for connecting two teams’ services?” — tests whether network-layer versus service-layer connectivity is understood as genuinely different tools, not interchangeable ones.
• “Two teams provisioned VPCs independently with the same CIDR and now need to peer them. What’s the fix, and how do you prevent recurrence?” — tests both the immediate remediation and the structural fix (IPAM).
• “Explain the cost model of NAT Gateway and how a Gateway Endpoint changes it.” — tests concrete understanding of AWS network billing mechanics, not just the name of the feature.