The Lab Is a Network

The cause is the lab. The fix is to measure it. Both deserve a closer look.

What the lab adds

On a single machine, every link in the emulation is virtual. Two satellite pods running on the same Linux box are connected by a veth pair, a virtual cable with both ends inside the same kernel. Traffic between them never leaves the kernel. The physical latency of that cable is essentially zero. Whatever delay the platform tells the kernel to add is exactly what the packet experiences. 10.7 milliseconds in, 10.7 milliseconds out.

Spread those pods across two physical machines and the veth pair stops working, because a veth pair only exists inside one kernel. The platform instead builds a tunnel between the two machines, with one end of the tunnel landing inside each pod. Each pod still sees what looks like an ordinary network interface, with the same delay applied. But the path between those two interfaces is no longer purely virtual. It runs across the physical network between the two machines, which is the lab.

FIG.01 — Substrate latency δ_lab = 0.42 ms · measured per-pair

The lab is a network. It has switches and cables and queues. It has latency.

A switch between two server racks adds a small but real amount of latency. A misconfigured one adds more. Whatever it adds, the packet you sent through the tunnel experiences it on top of whatever the platform already asked the kernel to add. The orbital delay was supposed to be 10.7 milliseconds. The lab added another 1.8 on its own. The total is 12.5. The routing protocol inside the pod, watching its propagation delay, now believes the other satellite is farther away than it actually is.

That is the lie.

Measure the lab. Subtract it from the math.

In NodalArc, every Node Agent runs a small piece of code we call the substrate monitor. Its job is to measure the actual physical latency between its own machine and every other machine in the cluster, continuously, in the background. Once a minute, for each peer, it sends ten quick ICMP probes across the lab network, takes the median round-trip time, and publishes the result onto the bus.

The Scheduler subscribes to those measurements. When the Scheduler computes the delay for a link whose two endpoints are on different physical machines, it does not simply take the orbital range divided by the speed of light. It takes that, subtracts the substrate latency it has measured between the two machines, and sends the result to the Node Agent as the value the kernel should actually add.

The arithmetic is small. The consequence is not. The packet now experiences kernel delay (orbital minus substrate), plus the physical traversal (substrate). The two terms sum to the orbital latency. The emulator tells the truth again.

The measurement is continuous because the lab can change. A switch reboot puts in a jump. A misbehaving queue introduces drift. A noisy neighbor on a shared link adds jitter at the wrong moment. The substrate monitor keeps the numbers fresh, the Scheduler keeps the math current, and the emulation stays honest under whatever the lab is doing on any given day.

Why this matters more than it looks

Without substrate compensation, every number an experiment produces is mixed with the lab. Convergence times include the lab's idle ping. Path latency between two ground stations depends on which physical rack the pods happened to land on. A comparison between two routing protocols on the same constellation can end up measuring the lab as much as the protocols.

With substrate compensation, the kernel delay plus the physical traversal sum to the orbital latency. The lab is still in the path. It is no longer in the numbers.