Networking Basics

TL;DR

• Network problems are diagnosed fastest by moving methodically up the stack: ICMP reachability → TCP port connectivity → TLS handshake → application response — not by guessing at the layer where the symptom appeared.
• A successful ping only confirms ICMP/network-layer reachability; it says nothing about whether a specific port is open or a service is listening.
• DNS changes are gated by TTL and caching — lower TTL before a planned cutover, since caches you don’t control can’t be forced to drop early.
• A hanging connection usually means a firewall silently DROPped the packet; an immediate refusal means REJECT (explicit RST/ICMP unreachable) or nothing listening.
• Server-side HTTP clients don’t trust internal/private CAs by default the way browsers trust public CAs — the fix is adding the CA to the client’s trust bundle, never disabling verification.
• Load balancer health checks failing for all targets simultaneously right after a network change usually means the health-check path itself got blocked, not that the application broke.

Core Concepts

TCP/IP layering and reachability

ping operates at the network layer using ICMP echo request/reply — a successful ping confirms the host is reachable and responding at that layer only. It says nothing about any specific TCP/UDP port: a host can respond to ping while every application port is closed, firewalled, or the owning process has crashed. Testing actual service reachability requires a transport-layer check (attempting a TCP connection to the specific port, e.g. with nc or telnet) in addition to, or instead of, ping.

The behavior of a blocked connection attempt is itself diagnostic:

• A DROP rule discards the packet with no response at all. The client has no way to know the packet was rejected, so it waits for a TCP handshake reply that will never arrive, until its own connection-timeout fires — this produces a hang, not an immediate error.
• A REJECT rule sends back an explicit refusal — a TCP RST for TCP traffic, or an ICMP “port unreachable”/“host unreachable” for others — causing the client to fail fast instead of hanging.

A hang during triage is a strong signal that something is silently discarding traffic (firewall, security group, network ACL); an immediate refusal signals either an active REJECT rule or a reachable host with nothing listening on that port.

DNS resolution, TTL, and caching

DNS resolvers (recursive resolvers, OS-level stub resolvers, and clients themselves) cache a record’s answer for the duration of its TTL (time-to-live). Updating a DNS record to a new IP does not propagate instantly — every resolver and client holding a cached copy continues to use the old answer until that cached copy’s TTL expires, and there is no way to force an arbitrary downstream resolver to drop a value it has already cached.

The operationally correct sequence for a planned DNS cutover is to lower the TTL well before the change (giving existing long-TTL cached copies time to expire naturally), make the IP change once the shorter TTL is in effect, and only raise the TTL back up after the cutover has stabilized. Lowering TTL after the change is already made does nothing to speed up records that were cached under the old, longer TTL.

HTTP/HTTPS and SSL/TLS certificate trust

TLS establishes an encrypted, authenticated channel; authentication depends on the certificate presented being signed by a Certificate Authority (CA) the connecting client trusts. Browsers ship with a large, actively maintained default set of trusted public CAs. Server-side HTTP clients (curl, a language’s HTTP library, an SDK) typically use a much smaller or different default trust store and, critically, do not automatically trust a private/internal CA unless that CA’s certificate is explicitly added to the client’s trusted CA bundle.

This produces a specific, common failure signature: an HTTPS endpoint using an internally-issued certificate works fine when opened in a browser (which may have been separately configured to trust the internal CA, or the browser’s OS-level trust store was updated) but fails with a certificate verification error from a server-to-server call. The correct fix adds the internal CA’s certificate to the calling service’s trust bundle. Disabling TLS verification entirely removes protection against man-in-the-middle attacks and should not be used as a substitute.

SSH connectivity behavior

SSH connections follow the same DROP-vs-REJECT diagnostic pattern as any other TCP service: a hang during connection attempt indicates something is silently discarding packets in the path (firewall, security group, network ACL) rather than the SSH daemon itself rejecting the connection; an immediate “connection refused” indicates the port is reachable at the network level but nothing is listening (sshd not running, wrong port) or a REJECT rule is actively firing.

Load balancers and health checks

A load balancer periodically probes each backend target using a configured health check (protocol, port, path, expected response) independent of real user traffic. If a network-layer change — most commonly a security group or firewall rule update — blocks the load balancer’s health-check traffic specifically (wrong source, wrong port, wrong protocol), every backend target can be marked unhealthy simultaneously even though the application itself is running correctly and would serve real traffic fine if it could be reached. This is a distinct failure mode from an actual application crash, and confirming the health-check path (source IP/security group, port, protocol) before assuming an application-layer bug avoids a large class of wasted investigation.

Firewalls and security groups

Security groups (cloud-native, typically stateful, attached to instances/interfaces) and traditional firewalls (iptables/nftables, network ACLs) both function as an access-control layer that determines, per rule, whether traffic is allowed, silently dropped, or explicitly rejected. The DROP-vs-REJECT choice should be deliberate rather than left to whatever a platform’s default happens to be, since it directly determines whether a blocked legitimate client experiences a hang or a fast, clear failure.

TLS termination architecture

Terminating TLS at the load balancer/edge and running plain HTTP between the load balancer and backend instances is a common pattern that reduces per-backend certificate management and CPU overhead. It is a reasonable trade-off when the internal network segment carrying that plaintext HTTP traffic is genuinely isolated and trusted — for example, a private VPC subnet with tightly scoped security groups and no untrusted tenants sharing the segment. It becomes a real risk in a shared, multi-tenant, or otherwise less-trusted internal network, where traffic between the load balancer and backend could be intercepted by another workload on the same segment. The design question that has to be answered explicitly is whether the internal trust boundary is actually enforced (network isolation, access controls) or simply assumed because the traffic is “internal.”

TCP vs. UDP

DROP vs. REJECT (firewall rule behavior)

Command / Configuration Reference

ping -c 4 <host>                     # ICMP reachability only, not port-level
nc -zv <host> <port>                 # TCP port connectivity check
telnet <host> <port>                 # manual TCP connection test
traceroute <host>                    # hop-by-hop path, useful for locating a DROP

dig <domain>                         # DNS lookup with TTL shown
dig +trace <domain>                  # full resolution path from root nameservers
nslookup <domain>                    # basic DNS lookup

curl -v https://<host>               # verbose HTTPS request, shows TLS handshake detail
curl --cacert internal-ca.pem https://<internal-host>   # trust a specific internal CA
openssl s_client -connect <host>:443 -showcerts          # inspect the presented certificate chain

ssh -v user@host                     # verbose SSH connection, shows where it hangs/fails

iptables -L -n -v                    # list firewall rules (legacy)
nft list ruleset                     # list firewall rules (nftables)

Common Pitfalls

• Pitfall: An “application is down” incident is investigated for an extended period before the load balancer’s health check itself is found to be blocked. Why: A security group change blocked the health-check traffic specifically, while the application remained healthy and reachable for normal traffic. Fix: Check the health-check path (source, port, protocol) immediately after any network-layer change, before assuming an application failure.
• Pitfall: A DNS cutover causes stale traffic to the old server for an extended window. Why: TTL was lowered only after the record change, so already-cached long-TTL answers had no way to expire sooner. Fix: Lower TTL proactively, well before the planned cutover.
• Pitfall: A server-to-server TLS integration is “fixed” by disabling certificate verification. Why: The client doesn’t trust the internal CA by default, and disabling verification is the fastest way to make the error disappear. Fix: Add the internal CA’s certificate to the calling service’s trusted CA bundle.
• Pitfall: A hanging connection is debugged as if it were an active rejection. Why: DROP and REJECT produce different client-side symptoms (hang vs. fast failure) that point to different root causes. Fix: Treat a hang as evidence of silent packet discard and check firewall/security-group rules for DROP behavior specifically.
• Pitfall: An internal, HTTP-only network segment is assumed safe without verification. Why: “Internal” is not automatically synonymous with “isolated and trusted” — a shared or multi-tenant segment can still allow traffic interception. Fix: Explicitly confirm network isolation and access controls before relying on plaintext HTTP internally.

Interview Questions

• A service is unreachable — ping succeeds, but the port times out. What does that narrow the problem down to, and what are the next steps? — Tests methodical, layer-by-layer triage instinct.
• All backend targets on a load balancer went unhealthy immediately after a security group change — what’s checked first? — Tests whether health-check-path blocking is considered before assuming an application-layer failure.
• Why does “it works in the browser but fails from the backend service” happen specifically with TLS/certificates? — Tests understanding of the difference between browser and server-side client CA trust stores.
• Explain the difference in client-side behavior between a firewall rule that DROPs versus REJECTs a connection. — Tests precise understanding of TCP handshake timeout behavior versus explicit refusal.
• Why is lowering DNS TTL before a cutover effective, but lowering it after is not? — Tests understanding of caching and TTL expiration mechanics, not just “DNS takes time to propagate.”