This is a writeup of building a required CI gate for degraded-network behavior. The system under test is a robotics fleet substrate, but the finding applies to anyone shaping networks in CI.
Ganglion exists to reach robots on networks nobody controls. Warehouse Wi-Fi, carrier CGNAT, a hospital VLAN, a customer firewall that was configured once in 2019 and has not been touched since.
Until this week that claim was a sentence on a website. CI ran on clean loopback, everything was green, and the failure modes that actually matter in the field were the exact ones the test suite could never produce.
That is now a required gate. Every push to main runs the full deploy, invoke and verify round trip over the relay against five shaped network profiles, and all five have to pass before anything merges.
I build Ganglion, so treat the enthusiasm accordingly. The part worth your time is not that it went green. It is what I got wrong on the way there.
The five profiles
- clean: baseline, no shaping. If this one fails, something else is broken.
- lossy: packet loss with light reordering.
- high-latency: 250ms round trip.
- asymmetric: plentiful downlink, starved uplink. This is the one nobody tests and the one teleop actually dies on, because control acknowledgements go the starved direction.
- nat-relay: endpoints with no route to each other at all, forcing hole punching to fail and relay fallback to carry the session.
The last two are the ones I care about. Loss and latency are what people imagine a bad network is. Asymmetry and no-direct-route are what a bad network usually is.
What I got wrong
The original design assumed you can pin netem's seed and get a repeatable lossy run. Two profiles: a pinned-seed one that gates the build, and a nastier randomized one that runs nightly and is allowed to fail.
You cannot pin netem's seed. Its loss and jitter draw from the kernel RNG and there is no seed parameter to set. A "deterministic lossy netem profile" is not a thing that exists.
This matters more than a wrong detail usually would, because the whole design rested on it. A gate that fails randomly is worse than no gate at all. It does not catch regressions, it teaches everyone to re-run the job until it passes, and after a month nobody reads red as meaning anything.
What actually reproduces
The fix was to build the gate out of only the mechanisms that reproduce exactly, and to be honest that this is a smaller set than netem advertises:
- Fixed netem delay. Constant delay, zero jitter. Deterministic.
- tbf rate caps. Bandwidth ceilings are a token bucket, not a distribution. Deterministic.
-
iptables -m statistic --mode nthfor loss. This drops precisely every Nth packet rather than N percent on average. Every 33rd packet is roughly three percent loss, and critically it is the same three percent every run. - Route blocking for the nat-relay case. Either a route exists or it does not.
Randomized netem still runs. It just runs nightly, in a separate non-blocking job, and a failure opens an issue instead of stopping a merge.
The nightly job generates its parameters deterministically from a recorded seed, so a failing run can be replayed. Worth being precise about what that buys, because it is easy to oversell: replaying reproduces the impairment distribution, not the packet-level draw. You get the same shape of bad network, not the same individual dropped packets. That is documented in the README rather than glossed over, and it is exactly why chaos never blocks a merge.
Riding the harness that already existed
The obvious implementation is a fresh veth pair or a pair of network namespaces built specifically for shaping. I did not do that, and I think the reason generalizes.
There was already an end-to-end dispatch harness running the real deploy, invoke and verify round trip through the relay. Building a second rig would have given me a network test that did not exercise the actual product path, which is the failure mode where CI is green and the thing still breaks.
So the matrix rides the existing harness. Shaping is applied inside the robot and operator containers before the agent starts. netem inside Docker was already proven green in CI by an existing mobile-CGNAT scenario, so the risky part was already de-risked. One rig, one new axis.
Record what you shaped
Every run writes a JSON artifact: mode, seed, the exact shaping commands issued, duration, and result.
This came out of a conversation on ROS Discourse where someone building replay tooling for robot fleets made the point better than I would have. Injected faults that are not recorded produce failures you can see and cannot get back. The fault has to be part of the run's record, or a genuine bug becomes a flaky ghost and gets closed as unreproducible.
Shape at the qdisc, record what you shaped. The two compose.
Keeping CI cheap
A five-profile matrix is an easy way to quadruple your CI bill. What kept it small:
- One required-gate job, not five parallel ones. The profiles run sequentially inside it.
- One build, reused across all five. Compiling five times to test networking is pure waste.
- Main-push only. Not every push to every branch.
- Early exit on docs-only pushes.
That lands around 15 to 25 minutes per push. The nightly chaos run is scheduled separately and blocks nothing.
The de-flaking pass, which is the part everyone skips
First full run was not clean, and the failure was not in the shaping. The nat-relay profile had a DNS race: containers with no route between them were resolving each other inconsistently at startup, and the test failed in a way that looked like a shaping bug. Fixed with static compose IPs.
The gate profiles also retry once and attach compose logs on failure. That is a deliberate concession. A gate that is right in principle and flaky in practice gets disabled within two weeks, and then you have neither.
Where it landed
Five of five deterministic profiles passing on main. Clean, lossy, high-latency, asymmetric, nat-relay.
The honest framing is not that Ganglion now works on bad networks. It is that a specific set of bad-network behaviors is now regression-tested, and if I break relay fallback under an asymmetric link, the build tells me instead of a customer telling me.
That is a smaller claim than the marketing sentence. It is also the first version of it I can point at.
Ganglion is Apache-2.0. The harness lives in test-harness/degraded-link/, including the determinism contract and the replay command: https://github.com/RobotDen/ganglion
Originally published at automaton.run. Ganglion is built for ROS 2 robot fleets, but the substrate underneath is transport-agnostic: outbound-only reachability, signed WASM tooling, and default-deny policy work the same for any distributed system on networks you don't control.
