Performance Testing

TL;DR

• Test design matters more than the tool; an instant-ramp test to peak load with only average response time reports almost nothing about real user experience.
• Percentile latency (p95, p99) reveals tail degradation the average hides; users at the tail are the ones experiencing the worst service.
• Realistic failure modes in load tests (errors, retries, malformed input) are more resource-intensive than happy-path requests; happy-path-only tests overestimate capacity.
• Infrastructure representative of production is essential; staging at 1/10th production’s scale can hide bottlenecks that only emerge at production capacity.
• Spike testing (sudden traffic discontinuity) exercises auto-scaling and cache behavior differently than gradual ramps; both scenarios are necessary.
• k6 (JavaScript-based, PR-reviewable code) vs. JMeter (GUI-authored XML plans) differ in authoring ergonomics and CI integration, not core capability.

Core Concepts

Load test design vs. tool choice

The most common performance testing mistake is designing a test that answers the wrong question. “Handles 10,000 concurrent users” based on an instant spike and average response time reveals almost nothing about actual user experience or realistic traffic patterns. Design comes first; tool choice is secondary.

Realistic load patterns: ramp vs. spike

Real traffic grows gradually in most scenarios (a gradual ramp tests steady-state capacity and auto-scaling ramp-up) but can also spike suddenly (marketing campaign, viral moment). These are fundamentally different scenarios:

• Gradual ramp: exercises auto-scaling warm-up, resource initialization, connection pool behavior under gradual increase
• Spike: exercises auto-scaling reaction time, whether stateful components (caches, connection pools) handle sharp discontinuities, connection storms

A system fine with a gradual ramp can fail badly on a spike.

Percentile analysis vs. average latency

Average response time hides tail latency. If 99% of requests complete in 100ms but 1% take 10 seconds, the average might be 200ms — which looks healthy — while that 1% (the actual users having the worst experience) suffer badly. p95 (95th percentile) and p99 (99th percentile) reveal this tail degradation. These are what actually matter for user experience.

Realistic failure modes in test traffic

Production traffic includes failures, retries, and malformed inputs. Error handling and retry logic are often more resource-intensive than successful requests (retry means doing the work twice; error handling means logging, alerting, cleanup). Load tests exercising only happy-path requests systematically overestimate real capacity. A failure-inclusive test catches the load pattern that emerges during partial outages or degraded dependencies.

Infrastructure representativeness

Running load tests against staging at 1/10th production’s capacity can hide bottlenecks that only emerge at production’s real scale. Performance characteristics (resource contention, connection pool limits, I/O bandwidth) don’t scale linearly. A test passing cleanly in staging can fail in production due to this gap.

k6 vs. JMeter: authoring and CI integration

k6 tests are written in JavaScript, version-controlled, reviewable via PR, designed for CLI/CI. JMeter traditionally uses a GUI-authored XML plan, harder to review as a diff, less naturally suited to code-first pipelines (though JMeter can run headless in CI). The difference is authoring ergonomics and reviewability, not CI capability.

Gradual Ramp vs. Spike Testing

Configuration / Command Reference

# k6 load test script (JavaScript)
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },  // Gradual ramp to 100 VUs
    { duration: '5m', target: 100 },  // Hold for 5 min
    { duration: '2m', target: 200 },  // Ramp to 200
    { duration: '1m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500', 'p(99)<1000'],  // p95 < 500ms, p99 < 1s
    http_req_failed: ['rate<0.1'],                    // Error rate < 10%
  },
};

export default function () {
  const res = http.get('https://example.com');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
  sleep(1);
}

# Run k6 test
k6 run script.js

# Spike test variant
export const options = {
  stages: [
    { duration: '30s', target: 5000 },  // Spike to 5000 in 30 sec
    { duration: '2m', target: 5000 },   // Hold for 2 min
    { duration: '30s', target: 0 },     # Ramp down
  ],
};

# JMeter (GUI-authored, can export to headless CLI)
jmeter -n -t test-plan.jmx -l results.jtl -Dusers=100 -Drampup=60 -Dduration=600

# View percentile latency in results
jmeter -g results.jtl -o report/
# Output includes p50, p95, p99 in HTML report

Common Pitfalls

• Pitfall: Load test showing healthy average response time at 10,000 users, but p99 is badly degraded for a fraction of real users. Why: Average hides tail latency; only percentile analysis reveals user experience degradation. Fix: Always report and analyze p95/p99, not just average.
• Pitfall: Happy-path-only load test dramatically overestimating capacity; real partial outage’s retry storm behaves nothing like the test predicted. Why: Failure handling is more resource-intensive; happy-path doesn’t test it. Fix: Include realistic error rates, retries, and malformed input in test scenarios.
• Pitfall: Performance test results from staging (1/10th production scale) fail to predict production bottleneck at actual production capacity. Why: Performance characteristics don’t scale linearly; staging scale can hide bottlenecks. Fix: Test against infrastructure genuinely representative of production, or explicitly account for the scale gap.
• Pitfall: System handles gradual traffic growth fine but fails badly during a sudden marketing-driven spike; sustained-ramp-only test design never exercised this scenario. Why: Gradual and sudden traffic exercise different failure modes. Fix: Include both ramp and spike scenarios in test design.
• Pitfall: Team invests heavily in JMeter GUI-based test authoring when k6’s code-first approach would integrate better with their PR review workflow. Why: Tool choice without evaluating actual authoring/review needs. Fix: Evaluate k6 and JMeter based on team’s CI/review workflow, not just raw load-generation capability.

Interview Questions

• A load test shows healthy average response time at 10,000 concurrent users, but customers report slowness. What’s likely missing from the test? — Tests whether percentile latency is correctly identified as the missing analysis.
• Why might a load test that passes cleanly in a smaller staging environment fail to predict a real production performance problem? — Tests understanding of infrastructure-scale effects on performance characteristics.
• Design a load testing strategy that would have caught a system’s poor behavior during a sudden viral traffic spike. — Tests whether spike and ramp testing are understood as distinct, complementary scenarios.
• Explain the difference between happy-path-only load testing and realistic failure-inclusive load testing, and why it matters. — Tests understanding of resource consumption differences in error handling.