Skip to main content

Moving a Legacy Java WAR to a Signed, Container-Delivered Pipeline: A Case Study, Including Everything That Broke

By Daniel Garza, Netrun Systems · · 11 min read

Summary. We took a legacy javax servlet application on an end-of-life Jetty line and rebuilt how it is delivered: a supported runtime with no application code changes, configuration moved out of the artifact, a CI pipeline that produces an SBOM, scans, signs and records every image by digest, hosts that run that digest and can be shifted by load-balancer weight, and monitoring and logging parity before any traffic moves. It is the shape of problem common in payments and fintech, where an old Java stack sits inside a compliance regime. The build was a lab, not a production system, and this is a log of it: what worked, the evidence, and a candid list of the times we declared something done and were wrong.

The problem

Many payments-style stacks still run javax.servlet WARs on Jetty 9.4. Public releases for Jetty 9, 10 and 11 ended on 2026-01-01, so security fixes now come only from commercial support. In a PCI-scoped environment the clock is explicit: requirement 6.3.3 expects critical patches within one month, which is hard if patching means a hand-built host and a manual deploy. Requirement 12.5.1 (the old 2.4) wants an inventory that includes the container images that may be instantiated. And there is a question to treat carefully: no PCI SSC position says an immutable image satisfies 11.5.2 change detection by itself, so you plan host file-integrity monitoring and agree the interpretation with your assessor.

The goal: move the same WAR onto a supported runtime inside an image that CI builds, scans and signs, deliver it by digest, and cut over by moving load-balancer weight, so rollback is a weight change and not a redeploy. Only afterwards migrate javax to jakarta.

The stand-in application is the open-source Spring PetClinic at tag v5.3.22, the last javax-era release. It is a proxy for the shape of the problem, not for any real codebase.

The architecture (roles only)

 BUILD HOST (Docker)
   CI server --> build daemon (Docker-in-Docker) --> private registry (plain HTTP, lab only)
        |  SBOM + vulnerability scan + signing run as digest-pinned containers
        v
   pull by digest
 PROXMOX HOMELAB (LXC containers)
   legacy host        Jetty 9.4.58 under systemd
   container host     the signed image, running under systemd, pinned to a digest
   database           MySQL
   load balancer      weighted pool over the two app hosts
   monitoring         Prometheus, Alertmanager, Grafana
   log indexer        Splunk Enterprise (trial license); forwarders on the other hosts

 client --> load balancer --> legacy host | container host   (weights decide the split)

Everything is plain LAN and a lab. Docker-in-Docker is privileged and the registry is HTTP; production would use rootless builds and TLS. We say so because the first rule of this write-up is that the lab's shortcuts are not the lesson.

The five stages

1. Runtime currency: Jetty 9.4 to Jetty 12.1 ee8

Technique. Run the unchanged WAR on Jetty 12.1.14 in its ee8 environment, which still speaks javax. Two small things make it work: a WEB-INF/jetty-ee8-web.xml pointing at the relocated org.eclipse.jetty.ee8.webapp.WebAppContext class (Jetty 9.4 ignores the file, so one artifact serves both), and a petclinic.properties containing environment=ee8, because an unpinned WAR goes to the newest environment.

Artifact. The first real run of the reference script: Jetty 9.4.58 (before) -> HTTP 200 and Jetty 12.1.14 ee8 (after) -> HTTP 200.

2. Configuration out of the WAR

Technique. An entrypoint turns environment variables into JVM system properties, and Spring's placeholder resolution lets them override file defaults, so one image can run against H2 or MySQL. It needs a build-time patch step because the packaged WAR had literal datasource attributes, not placeholders.

Artifact. One signed digest, sha256:f1a2edaec14300cd4e3cd29966c7248ef5f7de024bf12e71e79ef1e4af30203d, returning HTTP 200 on both /petclinic/ and a database-backed page against H2 and against MySQL, confirmed with docker inspect as the same digest. The caveat that follows is in the failure log: the WAR still carries the upstream default credentials as a fallback, and the protection is the runtime override, not their absence.

3. The supply chain: SBOM, scan gate, signing, registry

Technique. The CI job builds the image, writes a CycloneDX SBOM with Syft, scans with Trivy and gates on CRITICAL/HIGH with --ignore-unfixed plus a documented ignore file, pushes, signs by digest with a key-based cosign key, verifies, and records commit to digest. The scanners are pinned by sha256; Trivy is held at 0.67.0 because versions 0.69.4 to 0.69.6 were malicious in March 2026. Key-based signing fits an environment where public transparency logs may be off limits.

Artifact. The first scan found Total: 102 (HIGH: 79, CRITICAL: 23), all in application jars, none in the OS layer, covering 93 CVE IDs. Those are baselined with rationale, which makes the gate a ratchet against new findings and not a fix. cosign verify exits 0 against the digest.

4. Delivery by digest, and traffic by weight

Technique. Config management holds the intended image digest; a systemd unit runs that digest; the load balancer holds both app hosts, legacy at weight 100 and container at 0, and weights move through the runtime socket without a reload. Weight 0 keeps the server health-checked but idle, so rollback is immediate.

Artifact. Masterless OpenVox 7.37.2 on the container host. The first apply exited 2 (changes applied: the unit, and the container started on the pinned digest). A drift test swapped the running container to :latest; the next apply restored the pinned digest. Status, canary 90/10, cutover 0/100 and rollback all returned HTTP 200 in the lab. Those are sampled requests, not production traffic.

5. Observability parity

Technique. Before shifting traffic, show that the new population is as visible as the old one. Prometheus (2.54.1) pulls JVM metrics from a javaagent, load-balancer metrics from its built-in exporter, and database and host exporters; 11 of 11 targets were up. Splunk Enterprise 10.6.1.1 (trial license) received logs from forwarders on all four lab hosts. The JMX javaagent was attached through JAVA_TOOL_OPTIONS with a read-only mount, so the signed image did not change.

Artifact. Sourcetype parity as a search: index=app_logs sourcetype=jetty_access | stats count by host returned 521 events for the container host and 428 for the legacy host. The same sourcetype on both populations is the gate before ramping.

What broke and what it taught

This is the part we think is most useful. Each row happened.

What we claimed or did What was actually true What it taught
Declared the "one image, two databases" step proven The proof ran on a side image built from a hand-patched ad-hoc image. The Jenkins-signed digests failed when tested: build 16 never deployed the app, build 24 had no Jetty modules (HTTP 000), build 25 returned 503 Prove claims on the artifact you ship, using its digest, a database-backed URL and a write with read-back
Set Trivy to --exit-code 0 to get past 102 findings A report-only scan is not a gate Baseline the known findings with owners and review dates and fail on anything new; the gate was restored and later caught a new CVE from a jar we injected
Committed a signing passphrase, an admin default and a database password Lab shortcuts survived to a "verified" state Rotate first, because rewriting history cannot recall copies. We rotated the keys and credentials and did not rewrite history; the old values are dead but still in local history
Reported a stage "fully complete" Three config files were uncommitted and several checklist items were unreported "Done" means committed, reproducible from the repo and every item answered
Trusted puppet apply --noop reporting zero drift The host's copy of the module was stale, so it compared stale against stale; the live unit had no JMX agent --noop proves nothing unless the host runs the committed tree. We still sync repo to host by hand
An alert that fired only when all log sources went quiet stats count | where count=0 hides one dead host behind the healthy ones. It also ran as a user without tstats permission, and a stale local config file overrode our API edits Design the detector for the failure you care about. Replaced with a per-host staleness check; the drill fired for the silent host alone
Drilled two alerts by lowering their thresholds The drills proved the rule-to-receiver path, not detection on real load Label each drill "real fault" or "synthetic threshold"
Wrote that the Splunk license was valid for a long time It was a 60-day trial Read dates from the product; convert epochs yourself
Wrote "MySQL 8.4" in the notes The database ran 8.0.36, already past its 2026-04-30 end of life, until a remediation pass actually upgraded it to 8.4.13 Do not write a version you have not read from the host; our own lab had the exact end-of-life finding we were selling a fix for

Smaller things worth knowing, each of which cost time:

  • Jetty 12's start.jar does not accept --jetty.base=; it treats it as a JVM argument and quietly falls back to the current directory. --add-modules then does nothing. The original script worked only because it ran inside the base directory.
  • We fixed that in one Dockerfile and not in the one CI actually built.
  • HAProxy 2.8 rejects the old option httpchk ... \r\nHost: form in favor of http-check send.
  • Splunk 10's default KV store deadlocks first boot; setting defaultKVStoreType = local before first start avoids it.
  • An exit-137 OOM kill reproduced only in a separate throwaway container with a pre-touched heap, not on the application image.
  • A trivial Windows byte-order mark in the pipeline file broke the Groovy parser.

Another instructive one: the scanner results moved while the image did not. Build 26's raw count rose from 79 to 82 HIGH because the vulnerability database updated, not because anything changed. The three new IDs were in jackson-core, and the image turned out to ship two versions of it, 2.12.4 and 2.9.6. The binary was pinned; its data was not, so pin or at least record the database version and date next to every result. We have not verified where the older copy comes from.

How it was built

This lab was built by orchestrated AI agents working in sequence, with a human checking every claim against the live systems. That is a plain description, not a pitch. The agents were fast, and they also overclaimed: a stage declared complete with files uncommitted, a proof run on the wrong artifact, a license date misread, a threshold lowered to force a firing, a version number written from the plan and not from the host. In each case the verification step, reading the real digest, the real git status, the real scheduler log, the real pct config-style output from the hosts, caught the claim before it became a record. The efficient arrangement was agents doing the work and a human refusing to accept "done" without evidence.

What I'd do differently

  • Start with the shipped artifact. Extract the WAR from the signed image on day one; the literal datasource attributes were latent for ten builds.
  • Treat the signed digest as the only test subject from the first build, and treat the side image as a bug.
  • Blank the upstream default credentials in the WAR at build time and fail closed when the override is missing. They are still there today, overridden at runtime, which is a weaker claim than "not present".
  • Automate repo-to-host sync (r10k, or a pull on the host) so a dry run means something.
  • Re-drill the heap and CPU alerts with real pressure, and add a real notification channel. Today alerts reach a LAN-only webhook that records the payloads, which is enough to test delivery, not to page anyone.
  • Pin the scanner database, and put an expiry on every ignore-file entry. Ours carry a review-by date, but nothing enforces it.
  • Fix infrastructure in the setup script, not just in the shell; our container networking fix was runtime-only.
  • Keep one running change log from the first build. The notes grew an entry for each failure from build 21 onward, and those are the most useful pages.

Honest limits

This was a lab on a Proxmox homelab with LXC containers. Traffic was sampled requests, not production load. Docker-in-Docker was privileged and the registry was plain HTTP. OpenVox ran masterless on one host, with no server and no scheduled runs. Alert delivery is a LAN webhook. The application is a demo, with 93 known CVE IDs accepted in a baseline and not remediated. We did not complete the jakarta migration or the move to embedded Jetty, which is the planned next step and should ship as an ordinary release through the same pipeline.

Try it yourself

The repo has the Dockerfiles, the pipeline definition, the monitoring stack, the config-management module and the full change log, including the failures above. You will need a Docker host and somewhere to run a few Linux containers or VMs; default lab credentials were rotated and live in git-ignored files you create yourself.

The companion repository is in a final public-release review. This section will link to it as soon as it is live.

Netrun does this kind of legacy-stack modernization work.