An orbital visibility event is not a cable.
That sounds obvious until you try to build the lab.
The OME can say two satellites can see each other. It can give you the time, the range, the delay, the terminal type, and the fact that the antenna can still track the other bird. That is orbital truth.
But a router does not form an adjacency with orbital truth.
A router forms an adjacency on an interface.
Something has to stand between the sky and the kernel. Something has to take a physical fact, decide whether that fact should become part of the running network, and then make Linux prove it happened.
That is the work of this post.
post 003 (The Future Is Computable) ended with a visibility event. This post follows that event down to the machine.
Not conceptually. Actually.
Can, should, did
The clean way to think about NodalArc is three questions.
Can this link exist?
Should this link exist?
Did the kernel make it real?
Those are different questions. They belong to different parts of the system.
The OME answers can. It owns orbital truth. Line of sight, range, elevation, terminal capacity, slew rate, visibility windows, propagation delay.
The Scheduler answers should. It owns desired versus actual link state. It decides which visible links should be active, which active links should be torn down, and which live links need their latency changed.
The Node Agent answers did. It owns the kernel on one Kubernetes node. It creates interfaces, drives carrier, builds VXLAN when peers live on different hosts, applies tc netem and tc tbf, and reports proof.
That split is the whole machine.
Can is not should
A visibility event says the sky permits a link.
It does not say the platform should bring that link up.
That distinction matters.
Two satellites may be mutually visible while every terminal is already busy. A ground station may see three satellites while it only has two terminals. A scenario may intentionally suppress a link so a researcher can watch the routing recover. A future experiment may choose not to use a visible link because a better path is already planned.
Visibility is permission, not policy.
That is why the Scheduler exists.
It reads the facts coming from the OME and keeps a map of what the network should look like right now. Not what just happened. Not what the last event said. What should be true now.
Then it compares that map to what NodalArc has already made active.
Desired on one side. Actual on the other.
The gap is the work.
Terrestrial network engineers already know this pattern, even if they do not use the word every day. A routing protocol does not reinstall every route every time it hears an update. It keeps a table, compares new information against old information, and changes what needs changing. Kubernetes does the same thing with pods. NodalArc does it with links.
Events move the map
A naive controller would treat every visibility event like a command.
Visible true: bring the link up.
Visible false: bring the link down.
That works in a demo. It fails in a real system.
Restart the controller in the middle of an orbital window and it has missed part of the story. Drop a message and the kernel may keep believing in a link the sky has already taken away. Replay an old event at the wrong moment and you can resurrect a link from the dead.
The problem is not that the controller reacted too slowly.
The problem is that it trusted events more than state.
NodalArc uses events to move the map, but the map is what it protects. Visibility events update desired state. Link snapshots restore desired state when a component restarts or catches up. Scenario commands modify desired state when the experiment needs an artificial wound.
Everything flows into the same place.
Then one worker reconciles.
No special path for visibility. No special path for a scenario. No hidden shortcut for startup recovery. The Scheduler builds a dispatch intent, puts it on a queue, and the dispatch worker drains to the newest intent before touching the Node Agent.
That sounds like plumbing because it is plumbing.
It is also what keeps the machine honest.
The dispatch is concrete
At the boundary between the Scheduler and the Node Agent, the idea becomes a command.
A link-up request looks roughly like this:
command: BatchLinkUp
session_id: iridium-66-isis-flat
wiring_generation: 7
target_sim_time: "2026-01-01T00:05:12Z"
interfaces:
- node_id: sat-P02S03
interface_name: isl1
peer_node_id: sat-P02S04
peer_interface_name: isl3
link_type: isl
latency_ms: 10.68
bandwidth_mbps: 100
range_km: 3200.4
locality: LOCALThis is not orbital mechanics anymore.
This is an instruction to Linux.
Bring up this interface. Connect it to that peer. Make the delay 10.68 milliseconds. Shape the bandwidth to 100 megabits. Do it for this session, this wiring generation, this moment in simulated time.
The fencing matters. A stale command from an old session should not be able to touch a new kernel. A command from an old wiring generation should not be able to operate on interfaces that no longer mean what it thinks they mean.
A moving network already has enough motion in it. The control plane does not get to add ghosts.
A wiring generation is the identity of the interface layout for a session. If the pods are rebuilt, the old interface names and PIDs may no longer point at the same things. NodalArc puts the generation on every command so an old command cannot mutate a new world.
Make the kernel obey
The Scheduler does not touch the kernel.
That is not an implementation preference. It is a boundary.
The Scheduler knows which links should exist. It knows which Kubernetes node hosts each endpoint. It knows whether a link is local to one host or crosses to another host. It knows enough to send the right command to the right Node Agent.
It does not know how to perform netlink surgery.
That job belongs to the Node Agent.
One Node Agent runs on each Kubernetes node that participates in the emulation. It is privileged because it has to be. It creates host-side veths, moves pod-side interfaces into network namespaces, creates VXLAN tunnels for cross-node links, attaches ground links through host-side bridge plumbing, and applies traffic control.
The router inside the pod sees the result.
It does not see the OME.
It does not see the Scheduler.
It does not see the host plumbing.
It sees carrier.
Carrier comes up and the daemon behaves like a router. IS-IS sends hellos. OSPF sends hellos. The adjacency forms. Routes move.
Carrier drops and the daemon behaves like a router again. The neighbor vanishes. The topology changes. SPF runs if SPF needs to run.
That is the point of emulation. The protocol is not being modeled. It is being given a world to live in.
Carrier is the part the routing daemon can feel. A Linux interface may exist but have no usable link underneath it. When NodalArc drives carrier up or down, the daemon sees the same kind of signal it would see from a real cable, optic, radio, or virtual interface. That is why adjacency behavior is real instead of simulated.
Proof, not optimism
The Scheduler should not believe a link is active because it asked nicely.
It should believe a link is active only after the Node Agent proves the kernel did the work.
That proof is part of the contract.
A successful response is not just "done."
It is closer to this:
response: BatchLinkUpResult
success: true
dirty_kernel: false
interfaces:
- node_id: sat-P02S03
interface_name: isl1
success: true
verified: true
proof:
- pod interface exists
- host-side veth is up
- carrier is present
- netem delay matches 10.68ms
- bandwidth qdisc is installedThe exact wire format is not the lesson. The shape is.
Every requested entry gets a result. Success must be verified. If cleanup fails, rollback fails, or the kernel ends up in a state the Node Agent cannot prove, the response marks the kernel dirty.
A dirty kernel is not a warning light you ignore.
It means the ground under the experiment is no longer trustworthy. The Scheduler stops dispatching for that generation instead of manufacturing active state from hope.
That may sound severe. It is not.
If you are measuring routing convergence, path stretch, or failure behavior, a false link is poison. A link that exists in the Scheduler's memory but not in Linux is poison. A link that exists in Linux but not in the Scheduler's memory is poison.
The lab has to be more honest than the network it is testing.
Local and cross-node are not the same lie
On one machine, a link can be host-mediated plumbing. Two pods live on the same Kubernetes node. The Node Agent can wire local interfaces and shape the path with traffic control.
Across machines, there is a real network underneath the lab.
Now the lie has a second layer.
The orbital link may want 10.68 milliseconds of one-way delay, but the two Kubernetes nodes may already be 0.6 milliseconds apart, or 3 milliseconds apart, or worse. If NodalArc blindly adds 10.68 milliseconds of netem, the emulated satellite link is too slow by whatever the lab network already contributed.
So the system has to compensate.
The Node Agents measure substrate latency between hosts. The Scheduler subtracts that measured substrate delay from the orbital delay before asking the Node Agent to apply netem.
The packet still crosses the real lab network. It still pays that physical cost. NodalArc only adds the missing part.
If the substrate latency is unknown, stale, or from the wrong wiring generation, dispatch blocks.
No measurement, no lie.
The substrate is the physical network underneath the emulation: the Ethernet, switches, NICs, host stacks, and Kubernetes node-to-node path. In a single-node lab it mostly disappears. In a multi-node lab it becomes part of every packet's delay unless the emulator accounts for it.
The router may react
The reconcile-versus-react argument has been about the platform.
It is not the same argument for the router.
The router is allowed to react. In an IGP session, that is exactly what we want. Carrier comes up, the daemon reacts. Carrier goes down, the daemon reacts. We are not trying to spare the router from reality. We are trying to give it a better reality, one where orbital motion has become ordinary interface state.
That is how you get a fair measurement.
The platform knows the appointment.
The router feels the cable.
Between those two facts sit the Scheduler and the Node Agent.
One decides what should exist.
One proves what does exist.
The sky is not enough. The kernel is not enough. The useful work happens between them.
The boundary is the design
The OME knows the sky.
It does not know the kernel.
The Scheduler knows what links should exist.
It does not mutate Linux.
The Node Agent knows the kernel on its own host.
It does not reason about orbital visibility.
The router knows routing.
It does not know why the cable appeared.
Those boundaries are not ceremony. They are what make the system changeable.
Swap the orbital propagator for one that uses real TLEs and the Node Agent should not notice.
Change the Scheduler so it batches more aggressively and the OME should not notice.
Replace FRR with another routing daemon and the OME, Scheduler, and Node Agent should keep doing the same work: publish facts, decide link state, make carrier real.
That is the architectural bet.
Let each piece do the one thing it is built to do. Stitch the pieces together with facts and proof. Keep the magic in the pattern, not in a monolith.
In post 005 (The Lab Is a Network), that boundary gets harder. A single machine can lie cleanly. A multi-node lab cannot. Once packets cross the lab network underneath the emulation, the lab itself has latency, loss, and topology. The lab becomes part of the network.