Runtime Security

TL;DR

• Runtime security monitors what happens after containers are running — the threat class that build-time scanning, signing, and admission control cannot structurally see.
• Falco is the standard open-source tool for monitoring container behavior (syscalls, processes, network, file access) and alerting on deviations.
• Behavioral detection (a shell unexpectedly spawning, unexpected outbound from a database pod) catches novel attacks that signature databases have never seen.
• A clean, signed, zero-CVE image at build time offers no guarantee against zero-days, supply-chain compromises, or runtime misconfigurations exploited after deployment.
• Tuning discipline is essential; an untuned, generic ruleset produces false positives that cause alert fatigue and real threats getting missed.
• Biggest gotcha: assuming build-time and admission-time controls provide complete security, with zero runtime monitoring to catch compromises that occur after deployment.

Core Concepts

Pre-deploy controls vs. runtime threats

Build-time scanning, image signing, and admission control policies all evaluate artifacts and configurations before deployment — a structural pre-limitation that prevents them from seeing what happens after a container starts. Runtime security watches what actually happens: unexpected processes, shell spawns, suspicious file writes, unexpected network connections. This is a distinct threat class that only runtime monitoring can address.

Falco: behavioral monitoring

Falco is the widely-adopted open-source tool for runtime behavior monitoring. It observes syscalls, process activity, network connections, and file access patterns, comparing them against defined rules. When behavior deviates from expected norms, Falco alerts. The core value: detecting behavioral anomalies that may indicate compromise, rather than relying on known vulnerability signatures.

Shell spawning: anomaly detection

A shell (/bin/sh, /bin/bash) spawning inside a production application container that never does this during normal operation is a strong behavioral signal. This isn’t because shells are inherently malicious, but because interactive shell access inside a running production container is rarely legitimate normal application behavior — it’s a common attacker pattern after gaining code execution. Detecting this anomaly catches an active compromise that build-time scanning could not have prevented, because the original image may have contained nothing vulnerable.

Why “clean” images still need runtime monitoring

A fully scanned, signed, zero-known-CVE image at build time doesn’t guarantee safe behavior forever. Zero-days with no assigned CVE yet, supply-chain compromises not flagged by any scanner, or misconfigurations exploited at runtime can lead to malicious activity inside a container that passed every static check. Runtime detection catches behavior — what’s actually happening — not just known vulnerability signatures, which is fundamentally different and complementary.

Behavioral vs. signature-based detection

A rule like “unexpected outbound network connection from a database pod” doesn’t require knowing a specific malware signature in advance — it detects deviation from expected normal behavior (a database pod legitimately has no reason to make arbitrary outbound connections). This can catch genuinely novel attack techniques that no signature database has catalogued. Behavioral detection is thus a meaningfully different layer from signature-based scanning.

Tuning and alert fatigue

An untuned, generic ruleset produces significant false positives — genuinely normal-but-unusual-looking application behavior triggering alerts constantly. High false-positive volume causes alert fatigue: on-call learns to distrust the system, real threats get missed or delayed. Careful tuning against actual observed baseline behavior — “what does this specific application legitimately do?” — keeps runtime detection a trusted signal instead of background noise.

Falco Detection Types

Configuration / Command Reference

# Install Falco (via Helm or package manager)
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco --namespace falco --create-namespace

# Example Falco rule: detect shell spawning in production containers
- rule: Unexpected shell in production
  desc: Shell spawned in production container
  condition: >
    spawned_process and container and
    proc.name in (sh, bash) and
    container.privileged = false and
    container.image.repository != "debugging-image"
  output: >
    Unexpected shell in container
    (user=%user.name container=%container.name image=%container.image.repository)
  priority: WARNING

# Query Falco alerts
journalctl -u falco -f  # or grep in /var/log/falco/falco.log

# Test a Falco rule by triggering the condition
kubectl exec -it <pod-name> -- /bin/sh  # triggers shell-spawn detection

Common Pitfalls

• Pitfall: An organization confident in its “complete” security posture (build-time + admission-time controls) with zero runtime monitoring, missing an active compromise occurring after deployment. Why: Pre-deploy controls are assumed to be sufficient; runtime threat class is overlooked. Fix: Treat runtime security as a distinct, required layer that pre-deploy controls cannot structurally address.
• Pitfall: A shell spawn inside a production container going undetected for weeks because no runtime behavioral monitoring exists. Why: Without runtime monitoring, there’s no way to observe what happened after deployment. Fix: Deploy runtime behavior monitoring (Falco) to catch post-deployment anomalies.
• Pitfall: A broad, untuned generic ruleset generating so many false positives that on-call stops taking alerts seriously. Why: Generic rules trigger on behavior that’s normal for this specific application; alert fatigue causes real threats to be missed. Fix: Tune rules against actual observed baseline behavior specific to each workload.
• Pitfall: Assuming a fully scanned, signed, zero-known-CVE image is immune to compromise. Why: Build-time scans only catch known vulnerabilities; zero-days and runtime exploits are not covered. Fix: Implement runtime behavioral detection as the complementary layer for post-deployment threats.

Interview Questions

• Your organization has thorough build-time and admission-time security controls but no runtime monitoring. What threat class are you still exposed to? — Tests whether the structural coverage gap is correctly identified.
• Explain why a shell spawning inside a production container is a strong security signal, even without knowing what caused it. — Tests understanding of behavioral anomaly detection as a distinct detection mechanism.
• A runtime detection system generates hundreds of false positives per day and on-call has stopped taking the alerts seriously. What’s the actual fix? — Tests whether rule tuning against real baseline behavior is the core solution, not just “add more alerting.”
• How would you design a runtime security baseline for an application that legitimately uses shells for specific operational tasks? — Tests understanding of tuning discipline and the challenge of balancing detection with operational reality.