Case study · Ingress

The fleet-wide 502 that no backend was causing

At peak, every app behind one edge started returning 502 Connection refused. Not constantly — just often enough to be maddening. Auth, checkout, the API tier, all at once, and yet every backend was healthy and every health check was green. The cause was five layers down: a proxy stuck in an OOM loop, a node that had quietly stopped serving, and a load balancer that never noticed either. Here is how an agent ran it down on emisar — reading freely, and changing nothing in production without an approval.

The stack

Five bare-metal Debian nodes in one colo, each running Traefik as a Nomad system job bound to :443. Every node also raises the same anycast VIP — 10.x.x.250/32 on a dummy interface — and advertises it to the edge firewall over BGP. The firewall ECMP-hashes each incoming connection across whoever is advertising. Five nodes, five proxies, five advertisers: that is the design, and it works right up until a node keeps advertising after it stops serving.

client → CDN (per-app pull zone)
       → edge firewall WAN VIP (DNAT)
       → anycast 10.x.x.250:443   ← ECMP across all advertisers (BGP/FRR + BFD)
   ┌───────────┬───────────┬───────────┬───────────┬───────────┐
 hvn01       hvn02       hvn03       hvn04       hvn05
 Traefik     Traefik     Traefik     Traefik     Traefik   ← Nomad system job, host net :443 (one per node)

Everything below ran through emisar's stock traefik, nomad, consul, and frr packs. The edge was already covered; nobody had to write a tool to read it.

It wasn't the app

The page blamed one service — a health check 404-ing on a URL with a doubled slash. Two problems with that. The doubled-slash path matches no Traefik route, so the edge returns its default 404 page not found; the same backend answers 200 on its real path, and Consul had it registered and passing the whole time. The dashboards weren't much help either: 47 targets up, a 5xx ratio rounding to zero, and a p99 of 9.8 seconds that no single service accounted for. The one thing every symptom shared was the edge they all sat behind. So that is where the agent started.

Reading down to the edge

Reads run freely on emisar — each scoped to a runner and logged with its reason — so the agent could walk the whole edge from one surface instead of four CLIs and four sets of credentials.

# Claude, over MCP → emisar. Each call is a declared, scoped, logged read.

consul.service_health {"service": "store-api"}
→ Server http://10.x.x.101:22504  status=UP
  HTTP GET /healthcheck: 200 OK   {"commit":"3433150"}
  # the named app is registered and healthy. it is not the app.

frr.bgp_summary {"runners": ["nomad-hvn01 … nomad-hvn05"]}
→ every node, including hvn05:   PfxSnt = 1  →  upstream-edge-fw
  # the smoking gun: a node with no working ingress is STILL advertising
  # the anycast /32. the advertisement is not tied to Traefik's health.

traefik.ping {"runners": ["nomad-hvn01 … nomad-hvn05"]}
→ hvn01–04: "404 page not found"    hvn05: curl (7) connection refused :8080
  # four nodes are serving; hvn05 is a black hole — advertising, not serving.

nomad.job_resources {"job": "traefik"}
→ group traefik · task traefik · cpu 1000 MHz · memory 1024 MiB
  # the cap, confirmed: 1 GiB — read it before touching it.

nomad.alloc_status {"alloc_id": "1c11be31… (hvn04 traefik)"}
→ Terminated  Exit Code: 137  "OOM Killed"
  Total Restarts = 23   Memory  956 MiB / 1.0 GiB
  # and the four live nodes are OOM-looping on that 1 GiB cap, ~every 70s.

nomad.alloc_status {"alloc_id": "f746f2a3… (hvn05 traefik)"}
→ Setup Failure: pre-run hook "consul" failed
  Post "http://127.0.0.1:8500/v1/acl/login": context deadline exceeded
  # hvn05 can't even start Traefik: its local Consul won't issue a token.

consul.members
→ nomad-hvn05   10.x.x.105:8301   left   server
  # the root of hvn05: its Consul agent wedged after a reboot and never
  # rejoined the gossip — so the token derivation above times out.

consul.autopilot_state
→ Healthy = true   FailureTolerance = 0   Voters = nomad-hvn01, nomad-hvn03
  # "healthy" only describes the shrunken raft: hvn05 leaving cost a voter, so
  # two remain. one more loss and Consul loses quorum.

linux.uptime {"runners": ["nomad-hvn01 … nomad-hvn05"]}
→ hvn02 and hvn05 booted ~02:00; the other three, weeks ago
  # not a one-off — an overnight unattended-upgrades reboot wave. hvn05 is
  # the one node that didn't come back clean.

debian.kernel_info {"runners": ["nomad-hvn01 … nomad-hvn05"]}
→ all five: 6.12.x-amd64 · reboot-required: no
  # rules out a kernel regression — every peer booted the same kernel and is
  # fine, so hvn05's failure is node-local, not a fleet-wide landmine.

vm.query_instant {"query":
  "topk(5, sum by(service) (rate(traefik_service_requests_total[2m])))"}
→ match-api 1751 req/s   web 179   auth 122   feed 91   chat 65
  # one single-replica backend drives ~1.75k req/s — the load that tips
  # the four surviving nodes over the 1 GiB cap at peak. it is also the
  # main producer of the 504s mixed into the report.

Three faults, stacked

On its own, none of these takes down a fleet. Stacked, they did.

Traefik was capped at 1 GiB, and under real traffic it didn't fit. The cgroup killed it — Exit 137 — and Nomad restarted it, twenty-plus times a node. What pushed it over was one single-replica backend (match-api) pulling ~1.75k req/s; with a node already gone, four proxies were carrying five nodes' worth of load.

The missing node was hvn05. An overnight reboot had dropped it to the initramfs on an LVM race. It came back — but a later test reboot left its Consul agent left from the gossip, and Traefik needs a Consul token to start. A wedged Consul never issues one, so hvn05 ran no proxy at all. Nobody caught it, because the only thing checked after the reboot was Nomad's server quorum.

The amplifier is the part worth remembering. Every node advertises the VIP whether or not its Traefik is up, so the firewall kept hashing connections onto nodes with nothing listening — hvn05 constantly, plus whichever proxies were mid-restart — and each of those connections got a reset. That is the whole shape of it: intermittent because ECMP picks per connection, fleet-wide because every app shares the edge, and self-healing because a recovered proxy rejoined on its own. A probe at the VIP settled it — about 40% of connections refused, while the very next one went through.

Stopping the bleed — two approvals

Two changes would stop it, and both touch production, so both waited for a human. The agent's job was to line them up and say why; the yes belonged to someone else. Start with the weird node: hvn05 isn't serving because its Consul is wedged, so restart Consul — not pull the node from the anycast. A node you're about to heal isn't one you drain.

linux.systemctl_restart {"unit": "consul", "runners": ["nomad-hvn05"], "reason":
  "hvn05 Consul wedged (left the gossip) after a reboot — restart so it rejoins and Traefik's token derivation succeeds"}
⏸ pending approval — linux.systemctl_restart is risk:high; a human approves in the portal
✓ approved by you · one use · audit event recorded
→ consul rejoined the gossip · quorum restored · Traefik starts on hvn05 — it serves again, no drain needed

nomad.task_resources_set {"job": "traefik", "group": "traefik", "task": "traefik",
  "memory": 4096, "reason": "Traefik OOM-looping at the 1 GiB cap under VIP load; raise to 4 GiB"}
⏸ pending approval — nomad.task_resources_set is risk:high
✓ approved · audit event recorded
→ jobspec patched at JobModifyIndex (MemoryMB 1024 → 4096) · rolling update places 4 GiB on all five

Neither is a shell. The Consul restart isn't a special case — linux.systemctl_restart takes the unit by a name that can't carry shell metacharacters, and which units a runner may touch is a policy decision, high-risk and approval-gated by default. The memory change refuses to write if the jobspec moved under it, and it edits only that one task's limit. Two real changes to production, each one named, reasoned, and on the record.

The fix that actually holds

Restarting Consul and raising the cap stopped the pain, but neither is the fix. A node can wedge again tomorrow, and a running memory change evaporates the next time the job is re-registered. Two of these belong in the repo; the third belongs in the load balancer's idea of "healthy."

# traefik.nomad.hcl — persist the cap so a re-register keeps it:
  resources {
-   memory = 1024   # below observed RSS under load → cgroup OOM kills
+   memory = 4096   # headroom for connection state, metrics, log buffers
  }

# the real fix: health-gate the anycast. a tiny per-node unit that probes
# the local Traefik and adds/withdraws the /32 on the dummy interface — so
# FRR pulls the route within a BFD interval (~900ms) and ECMP drops the node.
while true; do
  if traefik_serving_on :443; then ip addr add 10.x.x.250/32 dev lb-anycast
  else                             ip addr del 10.x.x.250/32 dev lb-anycast
  fi
  sleep 1
done
# now an OOMing, rebooting, or Consul-wedged node withdraws ITSELF
# instead of black-holing one connection in five.

And the wedge itself has a one-line cure. A Consul server should never leave on a planned reboot — leaving is what dropped hvn05 out of raft to begin with. Tell it to stop, not leave (leave_on_terminate = false, skip_leave_on_interrupt = true), and a rebooted server rejoins on its own.

And about that anycast: pulling a node out of it by hand is now a single gated action. frr.bgp_neighbor_shutdown drops a node's BGP advertisement so ECMP routes around it — the break-glass for a node that can't come back fast. It didn't come to that here; restarting Consul brought hvn05 back outright, and a node you've just healed isn't one you pull from rotation. But when the fix isn't a thirty-second restart, the lever is on the governed path now — not an SSH session and a vtysh prompt.

With Consul healthy and 4 GiB rolled to every node, the probes came back boring: nothing refused at the VIP, every node serving, both quorums whole.

What emisar didn't do

Worth being plain about the limits. emisar didn't prevent this outage, and the page shouldn't pretend otherwise. The three things that caused it — a boot that lost a race with udev, a Consul server that left when it should have stopped, a VIP that advertises without checking — are design flaws, and you fix design flaws in the repo, not with a tool call.

What it changed is the hour in the middle: the part where someone is staring at a fleet-wide 502 with no failing service, hunting for the thread. Every read was one call away and on the record. Every change that touched production waited for a yes. Nobody held a shell on the edge, and nothing happened that you can't go read back later. Different failure, same fleet, same loop — the night a storage driver reformatted a live LUN and erased 33 hours of metrics ran exactly this way.