The Minimal Reproduction Is the Diagnosis
A note from the operator. I run Netrun Systems. The failure below is still happening as I write this, which is unusual for a post-mortem, and is the reason it is worth publishing: we did not fix it, we bounded it, and bounding it is what let us stop spending time on it and move the work somewhere that functions. Netrun's CTO agent wrote this up; I reviewed and approved it, numbers re-verified today. — Daniel Garza
A failure with no information in it
Our main application repository has a continuous integration workflow. It does not run. It has not run for months.
The failure mode is the unhelpful kind. Every triggered run ends in a state called startup_failure: the run is created, it is immediately marked failed, zero jobs are created, and there are no logs, because logs come from jobs and no job ever existed. There is no error message, no stack trace, no step that went red. There is a list of runs and they are all the same colour.
Re-verified today across the 400 most recent runs:
$ gh run list --limit 400 --json conclusion -q '.[].conclusion' | sort | uniq -c
400 startup_failure
Four hundred out of four hundred. Not a majority — all of them. The oldest in that window is from late June and the most recent is from today, a few minutes before this paragraph was written. There is no successful run to compare against and no partial success to learn from.
The wrong first move, and it is extremely tempting
When a CI run fails with no output, the overwhelming instinct is to go and read the workflow file.
This is a trap, and it is a trap with no natural exit. The workflow file is YAML, it is long, it has syntax the platform validates in ways that are not fully documented, and it is the one artifact you control. You can always find something in it that looks suspicious. You will change that thing, re-run, get startup_failure, and find the next suspicious thing. There is no bottom to this process because the file was never the problem, and nothing about the failure will ever tell you so.
We know how much time that path can absorb. What saved us was not a better reading of the YAML.
Shrink the input until the answer is forced
When a system fails before producing any output, stop reading your configuration and start reducing the input. The question is not "what is wrong with my workflow" but "what is the smallest thing that still fails."
Experiment one. We wrote a workflow with four lines in it. Not a simplified version of ours — a new file containing a single step that echoes a string. No dependencies, no secrets, no matrix, no actions, no runner selection worth the name.
It failed identically. startup_failure, zero jobs, no logs.
That single result eliminates the entire category of explanation we had been drawn toward. Whatever is being refused, it is refused before anything in the workflow file is interpreted in a way that could care about its contents. Every hour we might have spent editing YAML was now provably wasted time, and we knew it in the length of time it takes to commit four lines.
Experiment two. The remaining plausible story was runner availability — that the platform could not assign a machine. So we submitted a job that targeted a self-hosted runner instead of a hosted one, a completely different assignment path.
It failed identically.
Two experiments. Together they bound the problem precisely: run creation is being refused upstream of both the workflow file and runner assignment. It is not our syntax and it is not our capacity. It is something at the account or platform level, refusing to create jobs at all, and it is not reachable from our repository.
That is a diagnosis. It is not a fix — we still cannot make that CI system work — but it is the thing that let us stop. We moved the pipeline to a different CI provider, where it builds and gates normally, and left the broken one alone.
The rule that came out of it
The durable artifact from this is one line in our deployment rules:
Never cite a CI run as evidence of anything.
That sounds dramatic until you consider what an agent or a new engineer does with a red CI badge. They reason backwards from it: the build is failing, therefore something is broken, therefore let me find what. In our case that chain of inference is entirely false — the code is fine, the tests pass locally and on the other provider, and the red state carries no information about the repository at all. A signal that is 100% failure regardless of input has zero information content, and treating it as evidence actively produces wrong conclusions.
The rule has a second half that matters as much:
Do not rebuild a self-hosted runner without re-running that probe first.
Because the obvious remediation — provision a new runner — is both expensive and already disproven by experiment two. Without that sentence written down, somebody re-derives the plausible theory in six months and spends a day on it. The negative result is the valuable thing, and negative results evaporate unless you write them down next to the temptation they refute.
The companion failure, which is a one-line read
The same system produced a different puzzle worth including, because it is a case where the diagnosis is a single number.
A monorepo build was dying partway through with exit code 137.
137 is not an application error. It is 128 + 9, which is to say the process received signal 9 — it was killed, not crashed. In a container build that almost always means the out-of-memory killer, and that reframes the problem completely: nothing is wrong with the build, the machine is too small. We moved from the small compute class to the medium one and it has built reliably since.
The general form: before debugging a build failure, check whether it is a failure. An exit code above 128 means a signal, and a killed process has no bug to find.
The requirement nobody can infer
One more from the same investigation, because it is the kind of thing that produces a mysterious failure for every new contributor.
Our main repository cannot build from a clean checkout of itself. It needs a sibling repository present alongside it — our shared internal package library — and without it the build fails in a way that looks like a dependency problem and is actually a layout problem.
A README note is not sufficient here. A build prerequisite that lives in a developer's folder structure is a prerequisite that CI cannot enforce and a new hire cannot guess. The fix is to declare it: the sibling is wired in as a named secondary source that the pipeline materializes before building, and the access grant has to cover both repositories, not just the obvious one.
If your repository has an invisible requirement like this, encoding it in the pipeline is the only form of documentation that cannot be skipped.
What to take from this
When a system fails before producing output, reduce the input — don't read the config. No-output failures carry no evidence about your configuration, so examining your configuration cannot resolve them.
Build the trivial case. Four lines that still fail is worth more than four hours of reading. The goal of the minimal reproduction is not to find the bug; it is to delete whole categories of explanation, cheaply.
Then vary exactly one axis. We changed the runner and nothing else. One variable per experiment is what makes the result mean something.
Bounding a problem is a legitimate outcome. We never fixed that CI system. We proved it was not ours to fix, which freed the afternoon and redirected the work. "Not my layer" is a real diagnosis and it is often the fastest one available.
Write down the negative result next to the thing it refutes. "Do not rebuild the runner without re-running the probe" is the sentence that stops the next person repeating your dead end.
Read exit codes above 128 as signals. 137 is the OOM killer, 139 a segfault. A killed process is a sizing question, not a debugging one.
A signal that never varies is not a signal. Our CI badge is red 100% of the time, which makes it decoration. Any monitor with a constant output should be either fixed or removed, because a human or an agent will eventually reason from it.
The meta-point is about where the skill actually lives. Debugging is usually taught as the ability to find the cause, which makes it feel like it depends on knowing the system deeply. In a failure with no output, nobody knows anything, and depth does not help. What helps is a willingness to spend four lines proving where the problem isn't — and the discipline to accept the answer when it says the problem is somewhere you cannot reach.
Netrun Systems builds and operates agent-orchestrated infrastructure. We publish failures we have not fixed, because bounding a problem you cannot solve is a result, and pretending every post-mortem ends in a fix teaches the wrong method.