Nginx Test Harness: Leak Testing For Nginx & Angie Modules

Your nginx module can leak one file descriptor per request for a year, and ASan will never say a word. Neither will valgrind. A leaked fd isn’t a memory error, so the tools everyone trusts to catch leaks are structurally blind to a whole class of them. The kernel counts anyway. Somewhere around worker_rlimit_nofile the worker stops accepting connections, and you learn about it from monitoring at 3 a.m. instead of a red line in CI.

nginx-test-harness is the nginx test harness that came out of learning this the annoying way, while building the modules we ship at deb.myguard.nl. It does functional and leak testing for nginx and Angie modules, in C, with no Perl anywhere in the stack. It’s public now, BSD-2-Clause, and this post is the tour: what it catches, how the two halves fit together, and the traps that will otherwise eat an afternoon of your life.

nginx-test-harness: functional and leak testing for nginx and Angie modules in C

Table of Contents

The leaks your sanitizers can’t see

Sanitizers watch memory. That’s the deal. An fd, a slab page in a shared-memory zone, a slot in a connection table: none of those are memory errors, so none of them show up. Your CI is green. Your module is eating the process alive.

There’s a second blind spot, and it’s meaner. nginx gives every request its own pool and frees the whole thing when the request ends. Leak a per-request allocation on the request pool and from outside the process nothing happened: memory goes up during the request, comes back down after, repeat forever. The graph is a flat line. So the probe doesn’t measure the request pool at all. It measures the cycle pool, the one that lives as long as the worker does, where nothing in normal request handling should allocate. On that pool, any nonzero delta across a request is unbounded growth for the life of the worker. Not “worth investigating”. A bug, every time.

That’s the whole idea of the nginx test harness in one sentence: take a snapshot of worker state before a request, take one after, and assert the difference is exactly zero.

Two halves, one JSON document

The nginx test harness is two pieces of C that meet over HTTP.

src/ngx_test_probe.c is the probe. It compiles into the module under test and renders worker state as JSON: pid, connection counters, open fds counted from /proc, cycle-pool statistics, plus the shared-memory zone’s name, size and slab page accounting. Things the HTTP response never reveals, which is why response-only test suites keep missing this class of bug.

prober/ is the prober, a standalone binary that knows nothing about your module. It reads rule files, talks raw sockets, parses the probe’s JSON with an RFC 8259-strict reader, and reports TAP. That reader is the oracle every single assertion runs through, so it carries 24 self-tests and refuses to run if any of them fail. A lax JSON parser doesn’t make a test suite wrong, it makes it unable to fail, which is worse. A suite that can’t fail is a lie with a green checkmark.

nginx-test-harness architecture: prober sends HTTP to nginx worker with ngx_test_probe, JSON snapshot feeds delta oracle

One nginx test harness, two servers

Here’s a fun one. Stock Test::Nginx::Socket, the Perl framework most module test suites are built on, probes nginx -V and requires the output to start with nginx version:. Angie answers Angie version: Angie/1.12.0. The suite bails before running a single test. Which means an entire ecosystem of modules that build fine against Angie has, in practice, zero functional coverage on it.

Deciding whether software works by parsing its version banner is astrology. The prober reads no banner at all: it sends requests, checks answers, and the same rule files run against both servers unchanged. Inside the probe, Angie detection uses ANGIE_VERSION rather than NGINX_VERSION, because Angie defines both and checking the wrong one tells you every server is nginx. Ask me how I know.

Rule files and the delta oracle

A test case in the nginx test harness is a few lines of text:

name    a passed-through request leaks no fd and no cycle-pool memory
from    127.0.0.20
send    GET /unguarded HTTP/1.1\r\n
send    Host: prober\r\nConnection: close\r\n\r\n
expect  status=502
delta   fds == 0
delta   pool.cycle_used == 0

probe lines assert on a single snapshot; delta lines assert on the change across the case. The probe request wraps around your real request, and the subtraction happens in the prober where the rule can see it.

The from line binds the source address, and it’s load-bearing. Anything keyed on the client IP (rate limits, reputation tables, per-IP fault arming) needs the address to actually vary between cases. Without it, a per-IP fault never fires and the case passes for the wrong reason. A test that passes for the wrong reason is the most expensive kind of green.

My favourite trap in the whole project lives right here. When /proc is unreadable, the probe reports fds as -1, a deliberate sentinel so direct assertions fail loudly. But run that through a delta: -1 minus -1 is 0, and delta fds == 0 passes on a box where fds were never counted at all. The prober rejects the sentinel under a delta explicitly. It costs four lines of C and it closes the gap between “we measured no leak” and “we measured nothing”.

The reverse direction fails loudly too: a delta on a field the server never rendered reports “delta path not present” instead of quietly passing, so new rules against an old build tell you they’re vacuous.

Making malloc fail on purpose

malloc does not fail in CI. Ever. Your runner has gigabytes free, so the branch that handles a full slab or a failed ngx_palloc has never once executed before it runs for real on a box under pressure, at which point it either works or takes the worker down with it. Those branches are exactly where the crash bugs live, because nobody has ever watched them run.

The probe’s second hook, fault_set, lets a rule arm a fault injector at those sites: the next N allocations at a tagged site fail, on demand, deterministically. Now the out-of-memory path is just another test case.

Fault tests collide with another gate, though. By default the runner treats any [alert], [crit] or [emerg] line in the error log as a failure, because a worker that segfaults, finalizes a request twice, or reuses a busy buffer logs at one of those levels and then carries on serving. Without the log gate, the suite passes while the bug ships. But exhausting a slab on purpose makes nginx log at [crit], legitimately. PROBER_ALLOW_LOG takes an extended regex and exempts matching lines, one line at a time: exempting no memory for still fails the run on a segfault two lines later.

One lesson from wiring this into our shield module: we assumed for two sessions that its fault tests would need the exemption. Measured it. They didn’t; the module logs its zone-full path at [warn], below the gate. Scrape the actual log on an actual fault run before you write an allowlist, or your blanket exemption will cover the real [crit] the day it appears.

Wiring the nginx test harness into a module

Add the harness as a git submodule, compile the probe alongside your module when testing is enabled, and register hooks for what the probe can’t know generically:

static const ngx_test_probe_hooks_t  my_hooks = {
    .zone_render = my_zone_render,   /* append module fields to the JSON */
    .fault_set   = my_fault_set,     /* arm allocation faults */
};

ngx_test_probe_register(&my_hooks);

Both hooks are optional. Register neither and you still get the whole generic document: pid, connections, fds, cycle-pool stats, and the zone’s slab accounting. That’s already enough for fd and memory leak assertions without one line of module-specific C, and it works on any module because every nginx shared-memory zone begins with an ngx_slab_pool_t, so slab occupancy reads the same everywhere. What you do supply is the HTTP surface: a directive and a tiny content handler that calls ngx_test_probe_json(), about 60 lines, since an nginx module can’t inherit another module’s command table.

One hard requirement: worker_processes 1 in the test config. The pid oracle asserts the worker pid is stable across consecutive probes, which is how a crash-and-respawn gets caught. With four workers, a perfectly healthy server answers each probe from a different pid and every case fails while pointing nowhere near the cause. The runner parses your rendered config and bails out before the first case rather than let you discover this one red test at a time.

The runner refuses to start in two other situations: when the prober binary is older than its sources, and when the module binary doesn’t actually contain the probe directive. Each of those gates exists because its absence once produced a green run that proved nothing. There are only so many times you’ll debug a passing test suite before you make “refuse to run” the default.

Never ship the probe

Everything compiles out unless NGX_TEST_HARNESS is defined, and in packaged builds it must stay that way. The probe walks queues under the slab mutex, scans /proc, and answers to anyone who can reach the port, unauthenticated by design because test rigs don’t do login flows. In a test container that’s a feature. In production it’s a diagnostic backdoor with a JSON API. The packages on deb.myguard.nl are built without it, and yours should be too.

Anyway. Grep your release .so for the string ngx_test_probe before you ship it. Takes five seconds, and the day it saves you is the day you’ll buy me a beer.

FAQ

Does the probe end up in production builds?

No. Everything is behind the NGX_TEST_HARNESS define and compiles out entirely unless you define it in the test build. Packaged builds from deb.myguard.nl never define it. The probe exposes internal worker state unauthenticated, so keeping it out of release binaries is a hard rule, not a recommendation.

Does the nginx test harness work with Angie?

Yes, that is one of its reasons to exist. The prober never parses a version banner, so the same rule files run against nginx and Angie unchanged. Perl-based suites built on Test::Nginx::Socket bail out on Angie because its version string does not start with nginx version.

Do I need to write C to test my module with it?

Almost none. Registering zero hooks still gets you pid, connection, fd, cycle-pool and slab-zone statistics, which covers fd and memory leak assertions. You only supply a probe directive and a small content handler, roughly 60 lines behind an ifdef, plus optional hooks for module-specific zone fields and fault injection.

Why does it require worker_processes 1?

The pid oracle asserts the worker pid stays stable across consecutive probe requests, which is how a crashed and respawned worker gets detected. With multiple workers a healthy server answers each probe from a different process, so every case would fail on a false crash signal. The runner checks the rendered config and bails out early instead.

Why measure the cycle pool instead of the request pool?

Request pools are freed wholesale when the request ends, so a per-request leak inside one is invisible from outside the process. The cycle pool lives as long as the worker, and normal request handling should never allocate on it, so any nonzero delta across a request is real unbounded growth.

Does the nginx test harness replace ASan and valgrind?

No. The nginx test harness covers what they cannot see: leaked file descriptors, slab pages in shared memory zones, and cycle-pool growth, none of which are memory errors. Run both. One practical note: nginx never frees its configuration pool, so run ASan with detect_leaks=0 or LeakSanitizer reports the whole config parse as a leak.

nginx-test-harness: Leak Testing for nginx and Angie Modules