Everyone Gets the Scheduler Right. Almost Nobody Gets Membership Right.
A note from the operator. Most of the engineering at Netrun Systems is done by AI agents working under written rules, and one of those rules is that our architecture notes label what is built separately from what is merely intended. That makes the corpus unusually auditable after the fact. This piece came out of reading about forty of those notes end to end and finding the same architecture independently derived at three different layers of our stack — along with a clear answer to which part of it actually determines whether the thing works. An agent of ours drafted it; the dates and quotations were checked against the notes and the commit history. The mechanism details are deliberately not here. — Daniel Garza
If you are building a system that dispatches work across machines you do not fully control, you will spend most of your design time on the scheduler. Which node gets this job. How to score candidates. How to weight load against latency against capability. It is the part that feels like the architecture, it is the part that gets whiteboarded, and it is the part that generates opinions.
It is also, in our experience, the solved part.
We know this because we have built it three times without meaning to. Three different efforts inside this company, at three different tiers of the stack, in different languages and separate repositories that did not import each other, independently arrived at the same design. That is not a coincidence we take credit for. It is evidence about the problem: given a set of resources that differ in what they can do, the correct way to choose among them is close to forced, and competent engineers converge on it.
The part that is not forced — the part where comparable designs diverge sharply, and where one of ours dead-ended while another now carries production work — is membership. Who is allowed to be in the fleet, and what you had to do to them to get them in.
The same router, three times
One of our systems routes tasks across AI models and services. Another routes compute across machines. A third, earlier design routed work across nodes we owned. Set them side by side and the shape is identical:
Describe each resource by what it can do, not by where it sits. Turn an incoming request into a set of requirements. Filter the pool down to the candidates that satisfy them. Among those, pick with a weighted score that prefers whichever is least busy, lower being better. If nothing matches, fall back in a documented order rather than silently widening the search. Record what you chose and why.
The third of those — the earliest — specified a weighted score over load, latency and failure rate. The one running today weights load and queue depth. That difference is the only interesting variation, and it is a tell: the earlier design weighted latency, the natural measurement when peers can probe each other directly, while the current one weights how much work it has already handed out, the natural measurement when a single coordinator sees everything. Each scored precisely what its own topology could observe. Neither was wrong about scoring.
So we have three independent derivations of one algorithm, seven months and a year apart. We treat that as settled ground now. When we add a tier, the question we ask is no longer how do I orchestrate this — it is what is the minimum description that lets me select among these. Address, capabilities, current load. Compress a resource into those three fields and everything downstream follows: capability filters, least-loaded scoring, residency enforcement, locality preference, multiple clouds, physical hardware. The compression is the design work. The scheduler is a consequence of it.
Which raises the question this piece is actually about. If the selection logic is the same every time, it cannot be what separates a fleet that runs from one that does not.
The design that was right about everything except who could join
In October 2025 we specified a mesh so the machines we already owned could pool capacity. The hardware was real and sitting in one building: a Windows desktop with a GPU, a Linux laptop with a GPU, and two Raspberry Pis.
The specification was thorough. Node discovery over mDNS and UDP multicast. A node registry persisted in PostgreSQL with a Redis cache in front and a manager object owning it. A registration endpoint. A capability vocabulary rich enough for a node to advertise a GPU, or speech-to-text, or spare storage. The weighted selection score described above. A companion document specified the wire protocol in 1,322 lines: nine message types, a thirty-second heartbeat carrying uptime and memory and CPU, a health checker that dropped a node after three consecutive failures, token auth, compression, connection pooling, telemetry, a version handshake.
Read that list as a scheduler and it is strong work. Read it as a membership protocol and the problem is visible immediately, which is how we eventually read it.
Every one of those mechanisms assumes the nodes can reach each other. mDNS and multicast discovery assume a flat, cooperative LAN. A registration endpoint assumes something can connect to the node. Selecting peers by measured latency assumes peers can probe one another. Each assumption is reasonable in a datacentre. Each is a statement about the network, not about scheduling.
Now look at the hardware again: consumer NAT, a laptop that moves between networks, and one machine running a Linux environment nested inside Windows whose private address changes when you restart it. Mutual addressability was achievable there, but not once — it was four separate projects. Port forwarding on a consumer router, address reservations, firewall rules, and for the nested case a port proxy on the Windows side that breaks every time the inner address changes. Then all four have to keep working, indefinitely, on machines whose owners did not sign up to be network administrators.
We stopped. On 4 May 2026 we deleted the subsystem — 24,000 lines — and recorded why in the commit rather than quietly dropping it: zero live nodes. The design note carries a banner we added on the same day, stating plainly that it was never deployed. Our records say the four candidate nodes never registered.
I would make that call again, faster. The cost of that design was never going to be paid in the scheduler; it was going to be paid at the first step, forever, by whoever had to make each new machine addressable. A design that is excellent after you join and prohibitive to join is a design with no members.
The invariant that replaced it
The system we built instead makes exactly one demand of the network, and we wrote it down as an invariant rather than a feature:
No inbound ports, no port-forwarding, no firewall holes on any node. A node behind home NAT, behind carrier NAT, or a moving laptop on coffee-shop wifi works identically. Only outbound 443 to the gateway is required. This is the single hard network dependency for participation.
Every node dials out. It opens a connection to the gateway, holds it, and work arrives down the channel it already established. For nodes that will not hold a socket open at all, we added a second variant that simply asks, repeatedly, whether there is anything for it. Neither mechanism is novel, and our notes say so — the pull variant has the same trust model as a self-hosted CI runner, which thousands of engineers already run on a laptop without thinking about the network at all.
That is the point. The measure of a membership protocol is what it requires of the person installing it. The mesh required a network administrator per machine. This requires someone who can run an installer on a machine that already has internet access. Nodes join, from anywhere, including from behind networks we will never see.
Getting that right is what makes the rest of the design tractable. Because a machine can join from anywhere, liveness becomes something it continuously earns: it refreshes a short-lived lease on a timer, and if it stops, the entry expires on its own. There is no distributed deletion to coordinate and no stale status field to reconcile. And because liveness expires by itself, the only thing that must be durable is identity — which machine this is, so it returns as itself rather than as a stranger.
Those two requirements point in opposite directions, and holding both at once is the actual engineering. Liveness should expire eagerly; identity should persist stubbornly. Early on we kept durable records in a place that did not outlive the process that wrote them, and a healthy machine could check in to be told the gateway had never heard of it. The rule we run on now is one sentence: identity must outlive the process that issued it. Treat liveness and identity as the same concern and you will get one of the two wrong.
One further discipline, cheap and rare: the router returns the reason for its decision as part of its answer, not as a log line. Which requirement went unmet, whether it fell back, and why. When the wrong choice can mean work ran in the wrong jurisdiction rather than merely ran slowly, explainability is a functional requirement. A router that cannot account for its choices is indistinguishable from one choosing badly.
What the documentation discipline buys
The strongest evidence that these primitives were factored correctly is what happened when we extended them.
In June 2026 one of our architecture notes assessed the router against its own ambitions, under a heading its author chose: What this means for the fabric — and the honest gap. It granted what worked and then stated what did not: there was no notion of preferring a nearby machine, so a request that specified nothing got cloud compute even with a capable machine in the next room. A later section posed the fix as a real choice between two options and recommended one — add an optional preference, switched off by default.
Seventeen days later a commit landed that policy in exactly the recommended shape. Prefer locally registered hardware before reaching for cloud. Off by default, so the shared gateway was unchanged. Implemented by reusing the existing filter-and-select machinery rather than adding routing logic. It changed the default answer for the common case — from cloud first to the hardware you already own, with cloud as spillover — and introduced no new primitive to do it.
A policy change that needs no new primitive is the clearest signal available that the abstraction underneath it is load-bearing. And a document that names its own gap, proposes a specific fix, and gets that fix in the proposed shape inside three weeks is doing work that no quantity of aspirational architecture prose can do. That is what the [IMPLEMENTED] versus [PROPOSED] labelling in our notes is for, and it is the practice from this body of work I would most readily recommend to anyone else.
Derived a fourth time, and the clearest statement of why
Two weeks ago, on 6 October 2026, an architecture decision record was accepted in the repository whose mesh we had deleted seventeen months earlier. It proposes a small always-on node owning a durable job table, with workers that pull work from it. It is disciplined about scope — it supersedes nothing and describes itself as the on-premises shape of the one remaining need — and it cites the newer node design as its template, keeping the principle verbatim: the node never reaches into a worker.
Under a heading reading Why pull, not push, it gives the sharpest statement of this argument anywhere in our corpus. The local inference server is bound to localhost inside a nested Linux environment. Exposing it would require a service override inside that environment, a port proxy on the Windows side that breaks whenever the inner address changes, and a firewall rule — all needing privileges an agent session does not have, and all adding attack surface. A worker that pulls, and reaches its own machine over localhost, needs none of it.
That is the constraint, and it is not the one most teams plan for. Not latency. Not throughput. Not security policy in the abstract. What the process doing the installation is permitted to do.
Inbound reachability is nearly free to assume while you are drawing the diagram and expensive to obtain on a real machine owned by someone who is not a network administrator — or, increasingly, configured by an automated process that has no business holding administrator rights at all. That second case is only going to get more common. If you intend to dispatch work to hardware you did not provision, treat the permissions available at install time as a first-order architectural constraint, not a deployment detail to be sorted out later.
Four independent derivations of the same selection algorithm. One of the resulting systems carries work today, and the difference between them was never scheduling.
What we are building next
Two things worth stating precisely, because our own records state them and the labelling only means something if we honour it in public.
The unifying piece is specified and not yet built: a single request shape that lets one caller target any substrate — a model, a sandbox, a specific machine — and have the registry resolve where it goes. Our architecture record names that contract, assigns it a provider and a consumer, and marks its state as aspirational with the note that it does not exist today. A capability catalogue exists. A compute router exists. The single envelope over both is the next piece of work, not a thing we are claiming.
And the locality preference described above remains off by default, by design. It is something a local control plane opts into rather than a global change in behaviour, which is what keeps a shared gateway predictable for everyone else on it.
The principle
Membership is a transport property before it is a routing property.
You can get the selection logic exactly right — we did, four separate times, apparently without trying — and still have an empty fleet, because what determines whether a machine participates is what you had to do to that machine before it could say hello. Design the joining step for the least privileged, least reachable, least permanent machine you intend to support, and the scheduler you were worrying about turns out to be the easy half.
The internal version of this analysis carries the full mechanism detail, per-claim citations, and a section enumerating the claims our own records do not support. It is internal and NDA-required.