nginx-skeleton-module: An nginx Module Template With Working CI

Measured across our eight derived nginx modules on 2026-08-03: not one of them has a ci.yml, six have no ci/ directory at all, two still keep their C at the repo root, and every single one carries between three and six workflows each with its own pull_request: trigger. These are our modules. Written by us, reviewed by us, all green.

That is what CI looks like when it accretes instead of being designed. Nobody sat down and decided on six entry points. Each one arrived as a reasonable small addition on a Tuesday.

nginx-skeleton-module is our answer: an nginx module template whose actual module is a few hundred lines of deliberate junk you are told to delete, wrapped in an order of magnitude more files that exist to check work. The interesting part is not the skeleton. It is ci/PROMPT.md, the procedure for dragging an existing module onto that standard without wrecking it, and this article walks every phase of it.

The opening phase is read-only. A step is a grind unit rather than a PR: it is sized so one person, or one cheap model, can do it holding nothing but that step in their head. Steps land in groups, and a group never merges half-done. Every trap described below cost somebody here a red CI round trip, or worse, a green one.

Three rules that outrank every step

Before any of the numbered work, three rules. Break one of these and the rest is theatre.

Adopt the convention, keep the content. Layout, ordering, naming and entry points are the standard. The target’s tests, thresholds, fuzz corpus, nginx compatibility range and linter selection are its own. A 1:1 copy is wrong by construction, because our tests test our module. If you find yourself copying test_scan.c verbatim, stop and think about what you just claimed to have verified.

Never delete a gate the target already has. Anything it checks that the skeleton does not survives, gets a badge and a table row, and goes back upstream as a PR. A rollout that reduces coverage is a regression wearing a standardisation PR, and it will be merged, because the diff looks like tidying.

Nothing self-hosted is portable. builder02 is the label of a physical machine sitting in our rack, and no linter in this toolchain will tell an adopter they copied it. The runner-identity phase settles that before a single workflow gets ported.

Then the standing constraint that this whole article is really about: every gate must be seen red once, in the target. A probe run against the skeleton is not evidence about anything else. Different paths, different files, different thresholds. Record the probe and its output in the PR body or the step is not done.

The things you may not do to get green

A stack of PRs against somebody’s live repo. The preconditions are not advice.

cd <TARGET>
git status --porcelain          # MUST be empty
git rev-parse --abbrev-ref HEAD # the base to branch from
git remote -v                   # are you where you think you are
gh auth status                  # can you actually open a PR here

Dirty tree means stop and ask. Never git stash to clean it up, because a stash the user did not request is data they will never find again. Not a git repo or no push access, also stop. Do not initialise one, do not fork as a workaround.

The default branch is not a work surface. Every step that lands code gets its own branch and its own PR, including the trivial doc fixes, especially the trivial doc fixes.

And the one that matters most: never disable a failing check to make a PR mergeable. Not [skip ci], not continue-on-error, not commenting out a step, not gh workflow disable, not lowering a threshold to the observed value. A red gate you cannot fix is a finding, and a finding is a perfectly honest way for a step to end. A silenced gate is not.

Lowering the threshold deserves its own sentence, because it is the one that gets rationalised. You set COVERAGE_FAIL_UNDER to whatever the run produced. Now the gate encodes today’s number as the definition of acceptable, forever, and nobody will ever raise it again.

Before you change anything: inventory and a baseline

Changes nothing. Decides whether there is a job at all, and sizes the one code change everything downstream depends on.

git remote get-url origin                                # who owns it
ls -d ci/t ci/tools ci/linter ci/fuzz src t tests fuzz 2>/dev/null
grep -lE '^s*pull_request:' .github/workflows/*.yml | wc -l   # entry points
grep -rn 'runs-on' .github/workflows/                    # whose machines
ls src/*_scan.c src/*_scan.h *_scan.c 2>/dev/null        # decision seam?
grep -ln 'ngx_http_request_t' src/*_scan.c 2>/dev/null   # expect no hits
gh run list -R <owner>/<module> --limit 20 
   --json name,conclusion,startedAt,updatedAt,workflowName

Score three markers. A full ci/ layout (ci/t, ci/tools, ci/linter, ci/fuzz); ci.yml as the sole pull_request entry point; ci/linter/run-all.sh plus a tracked .githooks/pre-commit. Three out of three and this is not the job, go read the forward-porting section instead.

Do not infer the score from the first marker you check. Two of our derived modules have a ci/ directory and still score 0/3, because ci/ is the cheapest half of the move and the most misleading signal in the entire set. No ci.yml settles it on its own.

What goes in the memory mirror, and this list is the actual deliverable of these six steps:

• Whether the decision seam exists. Absent or nominal is the largest code change in the job, and you size it here rather than discovering it once the test layers land.
• Workflows in three buckets: matches a skeleton workflow by purpose, missing, and extra. That third bucket is what rule 2 protects and what quietly evaporates otherwise.
• Every pull_request: entry point by name. The count is the size of the demotion to a single orchestrator, which is the riskiest edit in the job.
• Measured wall-clock per workflow from gh run list. Real numbers. The runner-topology work does not accept estimates.
• Whose runners it currently uses.

Then baseline the target green. Run whatever suite exists and record the result. If it is already red, that is a finding for issues.md and a fact the first PR body has to state, otherwise the ci/ move inherits blame for a failure that predates it by six months. Ask me how I know.

The decision seam, and why it comes first

The only C refactor in the job, alone in its own phase, and it comes before the ci/ move because everything downstream links across it.

The rule: decision logic goes in *_scan.c, taking (u_char *, size_t). Only ngx_http_request_t plumbing stays in *_module.c.

Both ci/tests/unit/test_scan.c and ci/fuzz/fuzz_scan.c compile the module’s real decision TU. Not a copy, not a shim, the same source file that ships. Without the seam, the unit layer tests a reimplementation and the fuzzer fuzzes one. Both go green. Both prove exactly nothing about the code your users are running, and they will keep proving nothing for years while the two versions drift apart, which they will, because one of them is under test and the other one is under deadline.

Three states come out of the seam probe.

Clean. No ngx_http_request_t anywhere in *_scan.c. Record it, move on, buy whoever wrote it a beer.

Nominal. *_scan.c exists but reaches for r->, allocates from r->pool, or logs through r->connection->log. It cannot link outside nginx, so the fuzz and unit builds either fail loudly or quietly link a stubbed variant. The tell is growth in ci/fuzz/ngx_stubs.c: every stub past the skeleton’s set is a dependency somebody propped up instead of refactoring out.

git diff --stat HEAD~1 -- ci/fuzz/ngx_stubs.c   # did stubs grow?
git log --oneline -- ci/fuzz/ngx_stubs.c        # who has been propping

No seam. Decision logic inline in *_module.c. Extract it: *_scan.c and *_scan.h take bytes and return a verdict, no nginx request types in the signature, no allocation from a request pool. Pass a buffer in or take an explicit allocator argument. *_module.c keeps the handler, the directive parsing, the config merging and every ngx_http_* call.

Do not change behaviour while extracting. This is a move. The baseline stays green across it, run from wherever the suite currently lives since ci/ does not exist yet. A behavioural fix riding along makes every later bisect ambiguous, and a real bug you find on the way goes into issues.md rather than into this diff.

One nasty little detail. The skeleton’s build-test.yml asserts the seam file exists by name after a rename. Confirm the target’s copy names the target’s file. A path that no longer exists makes that assertion vacuous rather than failing, which is the whole theme of this article in one line of YAML.

If the module genuinely has no decision logic to separate, a pure plumbing module whose only work is ngx_http_* calls, say so with the file:line that shows it and note that the unit and fuzz layers will be correspondingly thin. Legitimate outcome. Stating it is not optional.

Moving the CI material under ci/

ci/
  t/                     Test::Nginx suite            (was t/ or tests/)
  tests/unit/            C unit tests of the decision core
  fuzz/                  libFuzzer targets, dict, corpus/, regressions/
  vendor/nginx-tests/    upstream suite submodule
  tools/                 ci-build.sh, nginx-tree.sh, test_runtime.py,
                         coverage.sh, max-port.sh, ci-hang-guard.sh, soak.sh
  linter/                local lint gate

Use git mv, never copy-then-delete, and verify with git log --follow on one moved file before continuing. A move recorded as delete-plus-add loses the history silently and cannot be repaired after the merge lands. You will not notice until the day you are bisecting a heisenbug at 01:00 and git blame tells you every line was written last Thursday by the person doing the move.

A directory move breaks every relative path that climbs out of it. Grep and fix in this order, and the order is load-bearing:

• nginx’s module config file. It names every source path and it is the one file whose breakage stops the module building at all, so it fails honestly and immediately.
• ../ in C #includes.
• $PWD and dirname logic in shell.
• paths: filters in workflows, and hashFiles() cache keys.
• prove invocations, fuzz corpus paths, .gitmodules, .gitignore, coverage exclude patterns, README references.

Everything after the first bullet fails quietly. A missed climb compiles fine and silently tests the wrong tree.

git submodule update --init still working after moving ci/vendor/nginx-tests is a required check, because the .gitmodules path: has to be edited and not merely implied by the directory having moved.

No src/? Creating it is part of this step, and the seam files move with the rest of the C. Two of our eight modules keep ngx_http_<name>_module.c at the repo root. This matters more than tidiness: everything downstream is scoped to src/, including lint-c.sh, lint-nginx.sh, the gcovr filter and the CodeQL TU filter, and every one of them passes on an empty selection rather than failing. Point four checkers at a directory that does not exist and you get four green ticks.

So prove it. Drop a probe file containing malloc and strcpy where the module’s real C lives, and confirm:

LINT_ONLY="c nginx" ci/linter/run-all.sh   # MUST exit 1
TEST_NGINX_TIMEOUT=20 prove -v ci/t/       # after the move, before any workflow edit

Run the suite after the move and before touching a single workflow, so that a failure is attributable to one thing. Acceptance: prove green, ci/fuzz/build.sh still builds, and no path outside ci/ refers to t/, tests/ or fuzz/.

Runner identity is not portable

Settle this before porting a single workflow. builder02 is spread across three files that all have to agree:

File	What it holds	In the skeleton, 2026-08-03
.github/workflows/*.yml	the runs-on fork ternary	15 selectors in 7 workflows
.github/actionlint.yaml	the declared label list	3 mentions, one self-hosted-runner: block
ci/linter/workflow_policy.py	TRUST_SPLITS, the approved-selector set	5 mentions, three label combinations

Twenty-three sites. Re-derive the number rather than trusting mine.

Now the part that makes this a numbered step instead of a footnote. Nothing in the toolchain catches a copied label. actionlint validates runner labels for a literal runs-on only, and every self-hosted selector here is a fromJSON(...) ternary it stays completely silent on. We measured it on 2026-08-02 by typoing builder02 to buidler02. actionlint had no opinion. zizmor has no idea which labels you are entitled to. And lint-ci-runners.sh compares against TRUST_SPLITS, which, copied unedited, contains builder02 and therefore approves it by construction.

The failure is not a red CI you fix on Tuesday. It is a green CI that either queues forever against a label nobody answers, or dispatches to a runner you do not own.

# before (skeleton, myguard-owned pool)
runs-on: ${{ github.event.pull_request.head.repo.fork && 'ubuntu-latest' || fromJSON('["self-hosted","builder02","lxc"]') }}

# after (any adopter without their own pool)
runs-on: ubuntu-latest

If you do not own the pool, every job is ubuntu-latest with no ternary at all. The fork ternary answers exactly one question, which is whether this code may touch our build host. An adopter with no build host has no such question, and an expression whose fallback arm names somebody else’s machine is a default-deny that defaults to somebody else’s hardware.

Same commit, workflows first and the checker last, so the gate is the last thing to change and its findings are about what remains:

• every runs-on in .github/workflows/;
• .github/actionlint.yaml, deleting the self-hosted-runner: block entirely, because declaring labels you never use trains the next person to add one;
• ci/linter/workflow_policy.py, reducing TRUST_SPLITS to an empty frozenset so that HOSTED.fullmatch covers everything and any future self-hosted selector becomes a finding rather than a silent pass.

Verify in both directions afterwards, because a grep proving builder02 is absent says nothing about whether the checker still approves it:

# 1. no myguard runner identity survives
grep -rn 'builder02|b02lxc' .github/ ci/linter/workflow_policy.py

# 2. the checker actually rejects the skeleton's selector
cat > .github/workflows/_probe.yml <<'EOF'
name: probe
on:
  schedule:
    - cron: "0 4 * * 1"
jobs:
  p:
    runs-on: ${{ github.event.pull_request.head.repo.fork && 'ubuntu-latest' || fromJSON('["self-hosted","builder02","lxc"]') }}
    steps:
      - run: echo probe
EOF
LINT_ONLY=ci-runners ci/linter/run-all.sh   # MUST exit 1 in the target
rm .github/workflows/_probe.yml

Probe 2 going green in the target is the exact bug this step exists for, and it means TRUST_SPLITS was copied unedited. Two things about that probe, both verified against the skeleton on 2026-08-03. In the unedited skeleton it exits 0, correctly, because builder02 is an approved selector here, on our machine. Running it in the skeleton to “check the probe works” proves nothing at all. And emptying TRUST_SPLITS before rewriting the workflows produces one finding per selector: doing that here gave 16 findings, the probe plus all 15 real selectors. Expected intermediate state, and the reason the order is workflows first. Reverse it and the one finding you are hunting is buried under fifteen you already knew about.

Stated honestly so nobody optimises it back later: the self-hosted pool is what makes ci-deep.yml‘s monthly matrix and the long fuzz runs affordable. On hosted runners they are slower and bounded by the 6-hour job limit. That is a scheduling problem, and it is still not a reason to point a runs-on at hardware you do not control.

One entry point, the workflow set, and the badges

Two commits. The demotion first, then the missing workflows. Adding a workflow to a repo that still has six triggers multiplies the problem you came to fix.

Demote to a single orchestrator

The highest-risk edit in the job. End state: exactly one pull_request:, in ci.yml, everything else reachable only as a workflow_call: member.

1. First. Add workflow_call: to each member while leaving its pull_request: in place. It still runs standalone, so the target keeps working throughout.
2. Then. Add ci.yml calling every member. Verify on a real PR that each member runs twice. Two runs is the expected intermediate state and the proof the call graph is wired.
3. Last. Remove pull_request: from every member in one commit. Now each runs once.

Removing the last member trigger is the point of no return, and the only step in the entire job that can leave a repo with no PR gate at all. Do not take it until the previous step showed every member double-running. A member that ran once was never called, and removing its own trigger silences it permanently. Skipping the double-run proof is how a member ends up called by nobody: ci.yml references a job name that does not exist, the call contributes nothing, and the suite looks green because the check that would have failed never ran.

If the merged result gates nothing, revert this PR first and diagnose afterwards. An ungated default branch is not a state to debug in place.

Two things break a called workflow and not a standalone one. secrets: inherit is not automatic, so a member that used a secret while standalone loses it when called. And path filters do not work on a called workflow, because it cannot filter its own triggering: gates move to a changes job in the orchestrator with an explicit job-level if.

A second entry point that is not pull_request: is fine. bump.yml and ci-deep.yml are schedule-driven here and are not members of the PR lane.

The workflow set

Workflow	What it must gate
ci.yml	orchestrator; the ONLY pull_request entry point
lint.yml	the ci/linter/ gate, hosted runner
build-test.yml	build, .so dlopens, bad config rejected, -T survives merged multi-context config, -Werror, Test::Nginx, ASan+UBSan
asan.yml	ASan/UBSan request-storm soak, static --add-module
fuzzing.yml	replay every past crash, then fresh fuzz
valgrind.yml	memcheck soak
security-scanners.yml	flawfinder ≥4 blocks, clang-tidy blocks, semgrep ≥WARNING
codeql.yml	CodeQL over the module TU only
ci-deep.yml	monthly: long fuzz, memcheck, helgrind, nginx mainline+stable+angie matrix
bump.yml	weekly pin bump plus ci/vendor/nginx-tests submodule update

Port .github/versions.env too. It is the single source of truth for version and sha256 pins, because tarballs get verified by digest and not by version string, and the difference between those two matters on precisely the day it matters.

Workflows the target has and the skeleton does not: rule 2, they survive. One of our modules carries a runtime-tests.yml with no skeleton equivalent; two carry a bump.yml the other six lack. For each, decide and write down which: keep as-is, give it a ## CI row and a badge and open a PR back to the skeleton, or fold into a skeleton workflow and state what moved where. Do not delete one because the skeleton has “the same thing” until you have compared the actual checks. A same-named workflow very often gates less.

Port bands come next, and they have cost more collective sysadmin sanity than anything else in this list. Test::Nginx binds TEST_NGINX_PORT, default 1984, and nothing arbitrates it. A self-hosted host runs several runner slots against one network, so two jobs on the default collide and the loser dies with:

bind() to 127.0.0.1:1984 failed (98: Address already in use)

Which reads exactly like a module regression and is not one. Presence of TEST_NGINX_PORT is not the check. The check is a distinct job-level band per workflow (TEST_BASE_PORT 19200 in build-test.yml, 19400 in ci-deep.yml) verified by ci/tools/max-port.sh before the first step that binds it, which means before prove, not merely before the runtime driver. We shipped this in the wrong place until 2026-08-02, and fixtures/policy/verify-after-bind is now the negative control that keeps it right. Read the target’s step order, not just the presence of a verify step. A target whose driver picks its own free port is already immune, so leave it alone and say so.

Badges, same order, same text

Build&Test, Security Scanners, Fuzzing, Valgrind, CodeQL, A/UBSan, CI Deep

with Lint inserted where the ## CI table puts it, and the two kept in lockstep. The label text is part of the convention. Measured 2026-08-03: one derived module had all seven badges in the correct order but wrote Build & Test for Build&Test and Security scanners for Security Scanners. Match spelling and capitalisation character for character, so a diff across modules shows real differences only.

Every badge must resolve to a workflow that exists, because one pointing at a deleted workflow renders a permanent grey “no status” and is worse than no badge. And the URL owner/repo is the target’s, not myguard-labs/nginx-skeleton-module. A copied badge row renders a lovely wall of green while telling you absolutely nothing about the repo it is sitting in. That is the worst failure available in these steps and it is one careless paste away at all times.

The four test layers, then coverage

The skeleton ships all four. Reuse them, do not re-derive them.

• ci/tests/unit/: run.sh plus test_scan.c. Links the target’s real decision TU and nginx’s real src/core/ngx_string.c. No shimmed decoder, ever. A shim makes the layer hermetic and worthless. Reuses ci/fuzz/ngx_stubs.c.
• ci/t/: Test::Nginx, config parsing and request-level behaviour.
• ci/tools/test_runtime.py: the live-server cases Test::Nginx cannot express. Concurrency, the chunk seam through the real body handler, reload under load. Retarget the config and marker, keep the shape, and keep the baseline case that proves the module is loaded and blocking before anything else runs.
• libFuzzer targets under ci/fuzz/, which is the fuzzing problem below.

Coverage runs through ci/tools/coverage.sh plus the coverage mode in ci/tools/ci-build.sh, and it gets a distinct build tree. Never a flag bolted onto debug. Share the tree and a cached non-instrumented build produces a 0% report that reads like a finding, and you will spend a morning on it. gcovr stays filtered to src/ only, because an unfiltered run drowns the module in 200k lines of upstream nginx and reports about 1%.

Coverage is a report, not a gate. The cheapest way to move the number is tests that touch lines and assert nothing, so a floor buys you a metric and sells the thing the metric was standing in for. Publish it from ci-deep.yml and gate on the mutations recorded beside each suite instead. COVERAGE_FAIL_UNDER exists for a target that decides otherwise, and that is their call, not yours.

Rejected outright, every one of which has been written here by somebody competent and sincere:

• a test whose assertion holds in both the pass and the fail state. Tell: a captured variable that is never compared to anything.
• a control that hardcodes the verdict instead of calling the real function.
• asserting a precondition rather than the claim.
• one shared counter asserted at N call sites. It pins none of them.
• a test written from the same misunderstanding as the code.
• excluding a hard file from the coverage config to lift the percentage.
• tests that execute lines without asserting on the result.

Against all of that, every new test requires a negative control. Break the code the test claims to guard: flip a comparison, delete a bound check, swap a constant. Confirm the test fails. Restore. Note the mutation in the test’s comment so the next person can repeat it.

The mutation pass gets its own step, one for the unit layer and one for the live-server one. As a trailing bullet under the step that wrote the tests, it was the thing that got skipped at 17:45 on a Friday, and a layer nobody mutated is a layer of unknown value. A step with its own acceptance line is harder to quietly not do.

A test that passes against mutated code guards nothing. And a mutation that survives is itself the finding. Record it with the reason rather than quietly picking a different mutation until one works, which is what everybody’s hindbrain wants to do at 17:45 on a Friday.

Push toward the maximum by targeting, in order: error paths, allocation failure, malformed and truncated input, boundary values at every MAX_* constant, cross-buffer seams, and the branches gcovr shows as never taken. 100% is not the goal. Every reachable branch having a meaningful assertion is.

ASan and fuzzing, retargeted to the module you actually have

Fuzzing is per-module work. A copied harness driving the skeleton’s rule table proves nothing about the target, and it will report clean forever while doing it.

• The fuzz target calls the real decision function with (const uint8_t *, size_t). That seam is the job of the seam phase and should already exist. If it does not, record it and land those first, because everything measured here is meaningless without it.
• Seed corpus from the module’s actual domain: real headers, bodies and config values it parses, plus every past crash under ci/fuzz/regressions/. Seed files are tracked; libFuzzer writes what it discovers into the same tree at runtime, so the corpus you actually fuzz against is far larger than what git ls-files shows.
• fuzz.dict with the module’s real tokens. A dictionary of the skeleton’s tokens actively misdirects the fuzzer, which is worse than no dictionary, because it produces confident fluent garbage in the wrong grammar.
• Replay-then-fuzz order in fuzzing.yml. Recorded regressions first, fast and deterministic, then the time-boxed fresh run. A crash that returns must fail in seconds, not after the fresh budget burns down.

The ASan soak in asan.yml has to drive the module’s real request shape, with its directives enabled and its body path exercised, not a default config where the handler never runs. That failure is completely silent: a soak that never reaches your code produces a spotless ASan report indefinitely. Prove reachability with a counter, a log line, or coverage from the soak build, and put the number in the PR body.

Keep the ASan build static via --add-module. A dynamic module under ASan loses interception on exactly the parts that matter. mold stays skipped under ASan for the same family of reasons.

Adapt the neighbours while you are here: valgrind.supp needs target-specific nginx-core suppressions, codeql.yml‘s TU filter needs the target’s file names, and ci-deep.yml‘s matrix needs the target’s nginx and angie compatibility range.

Acceptance: the fuzz target links against production code, replays all regressions, and a deliberately reintroduced past bug is caught by the replay step. Verify once, then revert.

Caching, and the linter gate

Caching

Every build goes through ci/tools/ci-build.sh as the single chokepoint. No workflow duplicates cache logic, because ten copies of a subtle invalidation rule is ten chances to get it subtly wrong.

Layers, cheapest first: apt packages, ccache with CCACHE_COMPILERCHECK=content, mold (skipped under ASan), eatmydata wrapping configure and install only, the build tree at .build/nginx-<ver>-<mode> keyed on mode plus version plus hashFiles(ci-build.sh, config, src/**), and the source tarball keyed on version with sha256 verified after restore.

Load-bearing rules, and the first one has bitten more than one person here:

• nginx’s configure ignores a bare CC=. ccache has to be wired through the configure argument, not via env. Everything looks fine, nothing is faster, and you find out six months later. Prove it with the hit rate from a warm run: 0% on a second identical run means it is not wired, whatever the log says.
• ccache may use a restore-keys ladder, because it is content-hashed and a partial hit cannot serve a wrong object. The build-tree cache stays exact-match only. Do not “fix” that asymmetry for consistency.
• Hybrid restore, warm on-disk dirs plus actions/cache fallback, stays. Deleting the fallback because the runners are persistent is how this degrades silently on the day they become ephemeral.
• GitHub scopes caches by ref. A PR run writes refs/pull/N/merge and cannot read a branch’s entries, so a cold PR run is not a bug.
• State the honest win in the README. If caching saves 5s on a 2.5-minute gate, say 5s.

And the rule that outranks every speedup in this section: a cache must never serve a stale artifact into a green result. If a key cannot express what invalidates it, do not cache that layer. This is not theoretical. Share one build tree between the debug and ASan modes and the sanitizer job restores the non-instrumented tree, runs the full suite, finds nothing and reports success. The tests pass without the sanitizer too. That is what tests do.

The linter gate

Port ci/linter/ and follow its README verbatim: apt-get first, then pipx for what Debian lacks, then cpan for Perl, then the upstream binary for actionlint. install-linters.sh is the single installer, and CI and a fresh clone use the same one.

git config core.hooksPath .githooks     # tracked hook, lints STAGED files only
ci/linter/run-all.sh                    # 0 clean, 1 findings, 2 tool missing

A missing tool exits 2 and blocks. Never a silent skip. Thresholds mirror security-scanners.yml exactly, and if you move one there you move it here in the same commit, or local-green stops predicting remote-green and people stop trusting the hook.

The checker set is the target’s; the entry point is the standard’s. A module with no Perl needs no lint-perl.sh; one with Lua or Rust needs a checker the skeleton lacks, so add lint-<name>.sh and run-all.sh picks it up by glob. Keep every checker the target already ran, behind the same entry point, rather than dropping it because the skeleton has no equivalent.

The three repo-policy checks do not transfer unexamined. ci-runners depends on TRUST_SPLITS having been rewritten during the runner-identity work, and ci-ports is meaningful only if the target binds a fixed band. A target whose driver picks its own port should say so in the README and skip that check loudly, rather than carrying a check that can never fire. Also note that lint.yml‘s LINT_ONLY string diverges with the checker set, nothing cross-checks it against the scripts that actually exist, and ours currently reads nginx sh python perl yaml spelling ci-runners ci-ports docs-drift. That is not a constant to copy.

Speed budget: the whole hook under about 2 seconds on a one-file commit. Not for elegance. A gate people wait on is a gate people bypass with --no-verify, and everybody knows the flag. Over budget, scope the slow checker. Never drop one, never add a default-on skip flag.

Three measured flags carry that budget:

• semgrep --metrics=off. The telemetry POST was 2.76s of a 2.76s scan. Read that twice.
• semgrep --jobs=1, which is a correctness flag and not a speed one. semgrep-core opens one io_uring ring per OCaml domain against the host’s 8 MB RLIMIT_MEMLOCK, shared with every other job on the box. When the runners are busy it aborts with Unix_error: Cannot allocate memory io_uring_queue_init, exit 2: a red gate caused by a neighbouring job. Reproduced 3/3 busy, 0/3 idle, which means an idle-box green tells you nothing at all. security-scanners.yml carries the same flags.
• run-all.sh fans checkers out via LINT_JOBS. Buffer each checker’s output and replay it whole in fixed glob order, never interleaved, because findings carry a file:line but not a checker name. Each child writes its exit status to a file, the reaping wait is collective, and a missing status file, meaning the child was SIGKILLed, counts as a failure and never as a pass.

Record your numbers, and check /proc/loadavg first. On our build host at load ~50 the same full-tree run varied between 2.2s and 12.4s over six attempts, a spread wider than the entire improvement anybody was arguing about.

Acceptance: run every probe in the linter README’s “Verify before trusting” section against the target and observe each one red, after the speed work, since --jobs and --metrics are exactly the flags that can silently turn a checker into a no-op. Then run with two checkers failing at once and confirm both appear and both are named in the == FAIL: line.

Runner topology: at most four lanes

CI wall-clock on a self-hosted host is dominated by jobs queueing for a label-matching slot. Ten simultaneous requests just means the tail waits. Hosted-only targets skip this step, say so, and move on.

gh run view <id> -R <owner>/<repo> --json jobs 
  -q '.jobs[] | [.name, .conclusion,
                 (((.completedAt|fromdate)-(.startedAt|fromdate))|tostring)+"s",
                 .startedAt, .completedAt] | @tsv'

Keep startedAt and completedAt, not just durations. The gaps are where the queueing hides.

Identify the longest single job. That is the budget, and no arrangement finishes sooner. Chain nothing behind it. Pairing the longest job with a follow-up “to keep the lane busy” is the most common way this gets worse, and it is exactly what put our own lane A at 348s against a 268s budget. The argument for doing it sounded excellent at the time, which is the problem with that class of argument.

Build the fewest lanes that fit, four maximum, each a chain of needs: where a long job releases its slot to a shorter independent follow-up. No lane exceeds the budget. Three lanes that fit beat four that also fit. Note the fullest lane’s headroom in the comment. Does not fit in four? Move a check out-of-band to monthly, time-box it, or put it on a hosted runner. Not “add a fifth”.

A lane is not a slot. Count real slots with systemctl list-units | grep ci-ephemeral, six on our host, and remember that a reusable workflow fans out: our Build&Test is five jobs, so observed peak is 7 against 6 slots. Brief oversubscription at t=0 is fine. Writing “caps peak at three” in a comment when it is seven is not.

Four more rules that each cost somebody a debugging session:

• Hosted jobs (lint, CodeQL) take no self-hosted slot and are not laned at all. Chaining one behind a self-hosted job conserves nothing and delays its result.
• Follow-ups use if: ${{ !cancelled() }}, so a failing first check does not suppress an unrelated second one and a chain survives an earlier job being skipped by a changed-files gate.
• Concurrency groups must not collide. A called workflow inherits the caller’s github.workflow and github.ref, so an identical group string makes a member cancel its own caller and a whole lane dies before it starts. Prefix the orchestrator’s group distinctly.
• Path-gating a reusable workflow does not work. Gates move to a changes job in the orchestrator with an explicit job-level if, and that diff job must fail loudly on an unusable diff. Falling through to “no relevant changes” skips the sanitizer on exactly the PRs that needed it.

The orchestrator’s header comment is the only place this design is written down, so it is part of the deliverable: lane map, measured durations, the run ID and date they came from, and the command to re-derive them. Any lane change rewrites that comment in the same commit, because a stale lane map reads as measurement and gets trusted.

Self-hosted runner exposure

Applies whenever runs-on includes self-hosted. A self-hosted runner executing untrusted code is arbitrary code execution on the build host. That is the whole threat model, stated plainly, and every requirement below follows from it.

• Fork routing with the adopter’s own labels, never builder02, and the condition stays github.event.pull_request.head.repo.fork. Not github.actor, not a repo variable, both of which a fork controls.
• No pull_request_target, ever, in a repo with self-hosted runners. It runs with a writable token in the base-repo context, and combined with a fork’s code that is a full compromise. If something appears to need it, it does not.
• Least-privilege tokens: permissions: contents: read at workflow level, widened per-job only where genuinely needed, such as security-events: write for CodeQL. Never write-all.
• persist-credentials: false on every checkout.
• Pin every third-party action to a full commit SHA with the version in a trailing comment. A tag is mutable, and the whole supply-chain point is that somebody else can move it.
• Pin every downloaded tool version and verify tarballs by sha256.
• Never expose secrets to a job that can run untrusted code. Prefer no secrets in the PR lane at all; bump.yml-style writers run only from the default branch.
• Runner containers here are LXC/incus and persistent, so assume a job can see the previous job’s leftovers. Nothing sensitive in $HOME or the work dir, and cleanup must not depend on a job succeeding.

Template injection is the one people still get wrong in 2026. Any ${{ }} interpolation of an attacker-controlled field, a PR title, a branch name, a body, straight into a run: block is a shell injection with extra YAML. Pass it through env: and quote it. Do the same for matrix.* even though it is repo-controlled: the safe form costs nothing and stops the unsafe one being copied somewhere it matters.

Repo settings get checked with gh api and either fixed or reported: require approval for first-time-contributor runs, restrict which actions may run, branch protection with required checks, and no self-hosted runner registered at org level where a public repo can grab it.

Most of this is mechanised by zizmor --persona=pedantic --offline over .github/workflows/, already wired into lint-yaml.sh. Expect the target red on first run and fix each finding. # zizmor: ignore[rule] at the line, with a reason, is the only acceptable suppression, and a suppression with no reason attached outlives the thing it suppressed by roughly five years.

The depth pass, or would any of this catch anything

Runs after every phase-4 step has merged. Everything here is already green. The question is not “does it run” but “would it catch anything”, and every item is answered with a measurement in the PR body rather than a reading of the YAML.

The seam. Re-run the seam probes, because the seam decays quietly as handler code accretes. A new stub in ngx_stubs.c is the signal that decision logic drifted back into nginx types and somebody stubbed around it. Fix the seam, not the stub.

Step 40, ASan and UBSan. Prove the soak reaches the module with evidence and not inspection. Confirm UBSan’s flags include the checks the module can actually trip (integer overflow, alignment, shift) and that it is trapping or exiting non-zero, because a UBSan that only prints to stderr passes a red run quite happily. Verify once by reintroducing a known-bad access, watching it abort, and reverting.

Step 41, fuzzing. One target on a module with several parse surfaces is under-fuzzed by construction; the skeleton carries two (fuzz_scan, fuzz_body). Enumerate every function taking attacker-controlled bytes, add targets where there is a real seam, and state which surfaces remain uncovered and why. Report corpus size and the coverage or feature count reached at the end of the time-boxed run, because a fresh run that plateaus in seconds is a stuck target rather than a clean one.

Step 42, coverage. Confirm the filter names the target’s src/ and, more importantly, that the reported figure moves when you delete a test. A number that does not move is filtered wrong and has been reassuring you for months. Also: --object-directory, never --gcov-object-directory, which arrived in gcovr 7.0 and is a hard argparse failure below it. The condition is the gcovr major version the job actually runs, not whether a pin exists somewhere.

Steps 43 and 44, valgrind. valgrind.yml is a 60s memcheck lite on the merge path; ci-deep.yml runs the 600s memcheck and helgrind soaks monthly through ci/tools/soak.sh. Confirm helgrind is actually invoked, because a copied ci-deep.yml that lost the helgrind job still shows a green CI Deep badge, and a dormant module is exactly where a silently missing job survives longest. Check valgrind.supp was regenerated rather than copied, since an over-broad suppression scoped past a core frame silently covers the module’s own errors.

Running the long soaks is conditional. Skip if nothing changed since the last green deep run:

LAST=<sha of the last green ci-deep run>
git diff --stat $LAST..HEAD -- src/ ci/ .github/versions.env

Note the deliberate inclusion of versions.env. bump.yml bumps pins weekly and ci-deep.yml runs monthly, so a module with zero source commits can still be running against a new nginx. Commit recency in src/ alone is the wrong clock. When you skip, say so in the PR body with the sha you compared against and the empty diff, because a silent skip is indistinguishable from a soak that never existed.

Step 45, caching. Walk the layers and confirm each key includes what actually changes the output. Report the ccache hit rate from a warm run.

Steps 46 and 47, the linter. This item does not re-install anything. It asks whether each checker still fires, because a checker that has become a no-op reports the same clean line as one that passes. semgrep first, for the reasons in the linter-gate section. zizmor findings drift with the workflow set, so confirm the count of audited workflows matches .github/workflows/ and that each ignore still names a reason that is still true. Do not read a clean actionlint as evidence about runner labels; it remains blind to the fromJSON ternary and that is probe 2’s job. And remember run-all.sh reads git ls-files, so a new untracked file is invisible to the linter. Stage before trusting a clean run.

Step 48, CI shape. Re-check the lane topology against measured wall-clock, not the estimates in place when it was written, since lanes drift as tests are added. Confirm exactly one pull_request: entry point still holds and that every member is reached, because a member called by nobody keeps a stale-green badge and only goes grey when deleted. Optimise by moving work into ci-deep.yml, never by deleting a check or widening a threshold.

For each check in this pass: the measurement, plus one sentence stating what that gate would now catch that it did not before. A “verified correct” with no number attached is not an answer.

Closing out, and the four questions the report must answer

README rewritten rather than appended to: badge row, ## CI table, layout tree, Requirements, and a Linting section linking ci/linter/README.md. CONTRIBUTING.md tells a contributor how to enable the hook. A CHANGES entry describes the standardisation. The memory mirror records the layout, the lane map, the measured times, and the skeleton commit you adopted from, because the next session needs that anchor and there is no way to reconstruct it later.

A trap that is a class rather than a typo goes into the matching .claude/skills/audit-*/ reference, not only into memory. The skill runs unprompted next time. Memory does not.

The report answers four questions explicitly, because they are what a greenfield reading gets wrong every time:

1. Entry points. How many workflows carried pull_request: before, and confirmation that exactly one does now.
2. Runners. Which pool the target runs on. If any self-hosted selector survives, the output of probe 2 proving the target’s own gate rejects the skeleton’s label. “Adapted the labels” is not an answer.
3. Extra workflows and gates. Every check the target had that the skeleton lacks, and whether each was kept, folded or sent upstream. If any was removed, what covers it now.
4. Badges. The final row, so order and spelling can be compared without opening the repo.

Plus, whenever either applies: which stop condition fired and what a human has to decide, and anything left disabled, skipped or unverified. Silence there reads as coverage that does not exist.

And the line at the bottom of the report section, which is the only one that really matters: do not report a step complete on a gate you never saw fail.

The aftermath, offered rather than taken

Everything above is done and reported. The last nine steps are deliberately not part of the adoption: each one either costs real CI time or touches code this job stayed away from on purpose. They get offered as a single multi-select question at the end, and none of them happens unattended.

A fresh reading of the prompt against the merged result, the commit hook, a review of the diff, a full code review (58), kicking off and re-timing the scheduled workflows (59), widening the dynamic analysis (60), pushing coverage (61). Each is a separate session’s worth of work with its own tools, which is why they are nine steps rather than one bullet list called “follow-ups”. A code review and a coverage push have nothing in common except that both were out of scope an hour ago.

Two are exceptions, and only in one direction. Two of them get checked unasked, because a missing bump is a defect in work already finished rather than new work. One is anything left uncommitted or unpushed. The other is the superrepo gitlink, and it earns its own step for an uncomfortable reason: it is invisible from inside the target. Every check you can run in the module passes while the gitlink still points at the commit before your adoption, so the one thing that records the whole job as having happened is also the one thing no test in the repo can see.

Once the nginx module template has landed, the job inverts

A target scoring 3/3 does not need the adoption at all. It needs one later skeleton improvement carried across, one concern per PR, one session.

Establish the anchor first, or you will either re-land work the target already has or skip the commit that made the change work. In order of preference: a recorded anchor in the mirror’s index.md, a vN tag the target’s CHANGES names, the CHANGES entry describing its adoption, or the merge commit of that adoption PR.

git -C /opt/myguard/labs/nginx-skeleton-module log --oneline <anchor>..HEAD

If none of the four resolves, there is no anchor, the target never took a documented adoption, and it is an adoption job rather than a forward. Do not invent one from the first commit or from “HEAD minus the change I was handed”. Both manufacture a scope that was never true, and both produce a PR that looks entirely reasonable to a reviewer.

Before touching the target, write in the PR body what the gate must prove in behavioural terms. Not “adds a port band check” but “a job that starts the runtime driver without declaring a port band fails the build”. Then state what failure it would have caught in the target, and whether the target can even reach that failure. A gate for a layer the target does not have is an adoption step, not a forward.

None of the drift classes is visible from a green run. That is the entire reason this document is what it is instead of a bullet list, and why the last thing every step asks for is a screenshot of the gate failing.

FAQ

Related reading

• nginx-test-harness: leak testing for nginx and Angie modules: the fifth layer, for the slow leaks none of the four catch.
• nginx-http-shield-module: block ancient exploits without a WAF: a real module built on this template.
• nginx-strip-filter-module: CSS and JavaScript minification: a body filter, where the chunk seam stops being theoretical.
• nginx-autocert-module: automatic TLS certs, no certbot.

Everything above lives in ci/PROMPT.md at github.com/myguard-labs/nginx-skeleton-module, file by file, with the probe commands.

Go and break one of your own gates on purpose this afternoon. If nothing turns red, you have got a spare 90 seconds in your pipeline and nothing else.