Experiment: sim/experiments/backbone_chain.py · Tests: sim/tests/test_backbone_chain.py ·
Builds on: multicell_backbone.py (608e2cf), backbone-latency-optimization.md, the #60/#63
half-frame slot-fit invariant (sim/tests/test_slot_fit.py). All airtime from phy.FLEET
(SF9/BW125/CR4:5/16-preamble) — the same numbers firmware + infra static_assert against.
Run: PYTHONPATH=sim python3 -m experiments.backbone_chain --minutes 120 --seed 1.
A linear chain of cells linked by single-radio, half-duplex backbone relays bottlenecks
end-to-end throughput because each relay's FRAME_T/2 half-frame holds exactly ONE message-sized
forward grant. That caps a relay at 1 forwarded message per FRAME_T per direction = 6.8 msg/min/dir.
The busiest middle relay of an 8-cell all-cross-chain chain carries ~0.286 of all cross traffic in one
direction, so the chain saturates between 0.5 and 1.0 msg/min/user — delivery ratio falls off a cliff
(1.00 → 0.70 → 0.39 → 0.21) and p95 latency blows up to thousands of seconds. The earlier prototype found
this qualitatively with a magic FWD_PER_FRAME=2; this note derives the real cap from airtime and pins
where the wall is. A dedicated 2nd-channel relay radio (SF7/BW250) raises the cap 26× and removes the
bottleneck across the whole tested range (deliv/offer stays 1.00 to 8 msg/min/user, p95 ≈ 7 s).
The as-built backbone is a two-coordinator half-frame stagger on one channel: a role-B relay's beacon
is offset by FRAME_T/2, so each relay owns one half of the FRAME_T = 8800 ms superframe. Into its
half it must pack: beacon (525) + guard (3) + 3 telemetry slots (3×320) + the granted forward message
slots (900 each) + a message-sized downlink (873) + the identity advert (751) + margin (150). This is the
exact worst_half_span_ms shape the firmware/infra slot-fit guard enforces (test_slot_fit.py):
| GRANT_MAX | worst half-span | fits FRAME_T/2 = 4400 ms? |
|---|---|---|
| 0 | 3262 ms | yes |
| 1 | 4162 ms | yes |
| 2 | 5062 ms | no — overruns |
| 3 | 5962 ms | no |
So a single-radio relay can forward at most 1 message-sized frame per half-frame per direction:
Per-relay forward cap = 1 msg / 8.8 s = 0.114 msg/s = 6.8 msg/min per direction.
This is the ceiling. It does not change with chain length — every relay has it, and the middle relays must pass all cross-chain traffic flowing through their cut.
5 clients/cell, all traffic cross-chain (worst case), light 0.5 msg/min/user to isolate the
length effect. Phase count P from greedy interference-graph coloring (spatial reuse → P=2 holds from
2 to 8 cells, confirming the prototype's Q1).
| cells | P | offered | delivered | deliv/offer | mean lat | p95 | backlog |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 2.53/m | 2.53/m | 1.00 | 0.0 s | 0.0 s | 0 |
| 2 | 2 | 5.31/m | 5.31/m | 1.00 | 9.6 s | 22.0 s | 0 |
| 3 | 2 | 7.89/m | 7.85/m | 1.00 | 11.9 s | 26.4 s | 4 |
| 5 | 2 | 12.29/m | 12.29/m | 1.00 | 18.1 s | 44.0 s | 1 |
| 8 | 2 | 19.76/m | 19.70/m | 1.00 | 40.0 s | 110.0 s | 8 |
At this light load the chain still delivers everything, but latency grows ~linearly with hops (store-and-forward: a message waits for each relay's window, one hop per frame). p95 climbs from 22 s (2 cells) to 110 s (8 cells). This matches the prototype's Q2 directionally; the absolute seconds are larger here because this model uses the real grant cap (1, not 2) and the bench-calibrated "wait for the next frame's slot" behaviour (backbone-latency-optimization.md §calibration).
Per-relay cap = 6.8 msg/min/dir. The busiest directional cut of an 8-cell chain carries 0.286 of all
cross-chain messages (combinatorial, verified separately). At offered rate R msg/min/user × 5 users × 8
cells, the busiest relay-direction sees 0.286 × 40 × R msg/min. The cap predicts saturation when that
exceeds 6.8 — i.e. at R ≈ 0.6 msg/min/user. The sim lands exactly there:
| load/user | offered | delivered | deliv/offer | mean lat | p95 | backlog |
|---|---|---|---|---|---|---|
| 0.25/m | 9.8/m | 9.81/m | 1.00 | 18.9 s | 39.6 s | 2 |
| 0.50/m | 19.8/m | 19.70/m | 1.00 | 40.0 s | 110.0 s | 8 |
| 1.00/m | 39.5/m | 27.62/m | 0.70 | 908 s | 3018 s | 1425 SAT |
| 2.00/m | 80.3/m | 31.47/m | 0.39 | 1488 s | 4488 s | 5855 SAT |
| 4.00/m | 159.4/m | 34.15/m | 0.21 | 1831 s | 4765 s | 15031 SAT |
Independent airtime check: busiest relay-direction at 0.5/min/user = 5.7 msg/min < 6.8 → delivers all; at 1.0/min/user = 11.4 msg/min > 6.8 → collapses. The knee is exactly the airtime cap, not a tuning artifact. Note delivered throughput plateaus near ~30–34 msg/min regardless of how much more is offered — it is bounded by the relay cut, so the chain degrades from "slow" to "lossy", and pushing harder only grows the backlog. Seed-robust: deliv/offer = 0.70/0.70/0.69 across seeds 1/2/3 at the 1.0/min knee.
This is a backbone-CAPACITY limit, not the #60 slot-fit one. The on-demand intra-cell fix (#63) does not touch it — it shortens the cell frame, but cross-chain forwarding is bounded by the half-frame backbone grant budget.
Move cross-cell relay onto a separate radio/channel (the coordinators already carry a second radio — LR1121 capacity cell). Two compounding wins: a 162 B relay frame at SF7/BW250 is 135 ms (vs 873 ms in-band, 6.5×), and the dedicated channel carries only relay traffic — no beacon/telemetry/advert/ client-downlink competing for the half-frame. A short 2.0 s backbone-only TDMA frame then holds 6 relay grants per half → forward cap 180 msg/min/dir = 26× the in-band cap.
Same 8-cell, all-cross-chain chain, in-band vs 2nd-channel:
| load/user | in-band deliv/offer | in-band p95 | 2nd-ch deliv/offer | 2nd-ch p95 |
|---|---|---|---|---|
| 0.5/m | 1.00 | 110 s | 1.00 | 7.0 s |
| 1.0/m | 0.70 | 3018 s | 1.00 | 7.0 s |
| 2.0/m | 0.39 | 4488 s | 1.00 | 7.0 s |
| 4.0/m | 0.21 | 4765 s | 1.00 | 7.0 s |
| 8.0/m | 0.11 | 5372 s | 1.00 | 7.0 s |
The 2nd channel delivers all traffic with constant ~7 s p95 across the entire range that collapsed the in-band backbone. This is the recommended scaling architecture: in-band SX1262 cell for clients + a dedicated SF7/BW250 backbone channel for relay. The backbone channel runs its own small TDMA among adjacent coordinators (same 2–3 phase spatial-reuse result, so it scales).
FRAME_TIf a second radio is unavailable, a longer FRAME_T amortizes the fixed per-half overhead
(~2.4 s of beacon+telemetry+downlink+advert) over more 900 ms grant slots, so the cap rises
super-linearly at first — but it inflates the latency floor (the period itself grew) and eats spectrum:
| FRAME_T | grants/half | fwd cap | 8c @ 1/min deliv/offer | p95 |
|---|---|---|---|---|
| 8.8 s | 1 | 6.8/m/d | 0.70 | 3018 s |
| 13.2 s | 3 | 13.6/m/d | 0.99 | 72.6 s |
| 17.6 s | 6 | 20.5/m/d | 1.00 | 61.6 s |
| 26.4 s | 11 | 25.0/m/d | 0.99 | 92.4 s |
Doubling the frame to 17.6 s clears the 1 msg/min knee (deliv/offer 1.00) but at a 62 s p95 floor — ~9× worse than the 2nd-channel's 7 s, and every idle cell now pays a 17.6 s superframe. Useful as a fallback knob, not the primary fix. (Other unmodeled levers: topology — prefer short chains/trees and multiple backbones over one long chain; admission/backpressure so the backbone sheds load gracefully instead of building a 15 000-deep backlog.)
Grounded: all airtimes from phy.FLEET; the per-relay cap is derived from the same
worst_half_span_ms half-frame budget the firmware/infra enforce (GRANT_MAX=1 in-band, matching
test_slot_fit); store-and-forward is FIFO per (relay, direction) with the bounded grants; spatial-reuse
phase count from interference-graph coloring; the saturation knee is cross-checked against the
combinatorial busiest-cut load (0.286).
Assumed/idealized: frame-stepped MC at one phase-window granularity, so absolute latency is approximate (bench calibration says scale in-band absolutes by ~1.4× — relative results and the deliv/offer cliff are unaffected); no PHY collisions on the backbone (phase coloring assumed to remove co-channel overlap) — this is the optimistic case, so the bottleneck found here is a floor, real RF would be worse, not better; traffic offered uniformly with all-cross-chain (worst case). The 2nd-channel frame (2.0 s, 6 grants) and its SF7/BW250 relay ToA (135 ms) are computed from the PHY model, but the dedicated-channel TDMA and its inter-coordinator coordination are not yet a full protocol — that is the firmware/spec follow-up before deployment.
FRAME_T is a fallback (clears the knee, ~9× the latency floor) when a 2nd radio isn't
available, plus topology (short chains/trees, multiple backbones) and backpressure to fail gracefully.