smarm/ROADMAP.md

smarm — Roadmap

Shipped (compacted — full cycle plans and deviation records live in git history)

Cycles before v0.7 (v0.4 actor primitives, v0.5 runtime decomposition & pluggable run queue, v0.6 actor ergonomics): see git log ROADMAP.md.

v0.7 — select, on epoch-stamped consuming wakes ✅

A 24-bit park-epoch packed into the slot word gives every wait an identity: registrations carry (pid, epoch), every successful wake consumes the epoch, stale wakes die at one failed CAS. Subsumed the per-primitive wait seqs (channel, mutex, timer); the only wildcard wake left is request_stop (terminal). On top: select/select_timeout — ready-index wait over many receivers, priority order, no cancellation pass. Loom theorems re-proved on the new word. Notable deviations: retire_wait for the no-park exit path (plan missed it); the Drop-guard deregistration was superseded by relaxed single-receiver asserts; a closed arm is ready forever (documented gotcha). Commits 4913835…00128f3, a0a93b6.

v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅

Spent select on the server loop: static info arms (type Info, ServerBuilder::with_info) and dynamic monitor forwarding (ServerCtx/Watcher + a control arm), priority downs → control → infos → inbox. Closed the v0.2 fd hole: a drop guard in wait_fd DELs the kernel registration on unwind (the leak was worse than documented — a stale waiters entry permanently poisoned the fd). Deviation: the plain-inbox fast path narrowed; servers holding a Watcher select forever. Commits e5d1b3b, 24b95c9, f6969e5.

---

Decision record — queue topology 🔒 CLOSED (2026-06-10)

The run-queue shootout (harness 6d9f369, 24-core sweep 1–24 schedulers, report bench_report_rq_shootout.html) landed in RFC 005's World 3: the three queue variants are within 10–15% of each other in rq_runtime at every scheduler count ≥ 4, on all three workloads. Consequences:

• rq-mutex stays the default — simplest correct, no capacity constraints, locking model already integrated.
• Feature plumbing stays as is. All three variants keep compiling in every build; rq-mpmc/rq-striped remain selectable for benching.
• Reopening is benchmark-driven only. The report documents the conditional upgrade paths if a future workload qualifies: mpmc for message-passing-dominant loads at N ≤ 8; striped for high-contention balanced push/pop at N ≥ 16. Neither is a scheduler workload as measured.
• Effort redirects to the wake path: RFC 005 (billed as a latency patch, per its own World 3 framing), RFC 004, and eventually per-switch cost.

---

v0.9 — Wake-path latency

Goal: attack per-wake latency — the measured ceiling — with the two specified mechanisms, each benched against a clean baseline before the next lands. Order: RFC 005 first (the slot is measured against the just-benched idle policy), then RFC 004 (measured against the slot-accepted baseline), then an interaction pass.

Full specs: rfc_005-wake-slot.md, rfc_004-tunable-scheduler-idle-policy.md (artefacts.kalsbeek.dev).

1. Wake slot (RFC 005)

Per-scheduler, thread-local, capacity-one wake cache, checked before the shared queue. Runtime-selected via Config { wake_slot: bool }, default off until accepted (one binary benches both arms).

• Push policy: slot-eligible iff the wake originates from actor context (current_pid().is_some()). Scheduler-context wakes (timer/IO drain) and spawns always go shared.
• Displacement: newest wake takes the slot, occupant pushed shared (Go semantics); the branch swap is the fallback if benches look pathological.
• Pop order: slot, then shared. Slot-popped actors inherit the waker's remaining timeslice — a handoff chain is bounded by one slice, so the shared queue is consulted at least once per slice per scheduler (the starvation bound, zero new counters).
• Invariant care: the slot push replaces run_queue.push at the tail of the Parked → Queued CAS; at-most-once-enqueued holds verbatim (pid in slot ⊕ shared queue). Stall blast radius grows by exactly one actor.
• Counters: slot_hits, slot_displacements.

2. Slot shootout

Extend rq_runtime/bench_rq.sh with the slot on/off dimension (cheap — it's a Config knob, not a feature rebuild). Same sweep as the rq shootout.

• ping-pong-pairs — target metric; expect the win, single- and multi-scheduler.
• yield-storm — regression guard; yields never touch the slot, any delta is pop-path overhead.
• spawn-storm — neutrality check; spawns bypass the slot by policy. Acceptance flips the default on and re-baselines for RFC 004. This is done, see results annotated in RFC005.

3. Spinning workers (RFC 004)

Bounded spin-before-park for idle schedulers, killing the ~100 µs thread::sleep worst case on cross-thread handoff. Two Config knobs: spin_budget_cycles (0 recovers today's behaviour) and max_spinners (default N/2); one new atomic n_spinning. No run-queue changes — lands on the frozen rq-mutex substrate, independent of the slot mechanically. Credit for the idea: Dennis Gustafsson - Parallelizing the physics solver BSC 2025.

4. RFC 004 bench + interaction pass

Re-run the sweep with spinning enabled, slot on and off. The known interaction: once idle pickup latency ≪ slice, the slot's latency trade (occupant waits out the waker's slice while other schedulers idle) may stop paying — RFC 005's documented escape hatch is Go's "spinners exist → push shared instead" check. In scope only if the data demands it.

---

Later

Highest priority

Process groups: the primitive pubsub & channels should have sat on

Context: urus is a webserver written on top of smarm to provide a testing target. A named pid→multiset map with monitor-backed removal: registry.rs generalised from name↔pid bimap to name→multiset, the death hook reused verbatim. Local first. The point is that urus pubsub collapses into a pg consumer (subscribe = join, broadcast = send-to-members) instead of being a bespoke mechanism, and the same group set reads two ways — fan-out (all members) vs discovery/pool (one member), with different netsplit consequences. Urus shipped pubsub/channels predate this and want reframing on top of it. Foundational, so early in the post-v0.9 stack. Needs an RFC.

Per-switch cost (context shims, epoch protocol)

The shootout's residual: per-wake latency is 0.16–0.18 µs at N=1 and 0.8–1.2 µs at N=8+. The N=1 profiling spike is done (docs/perswitch-profile-n1.md): it revises the old shim-first framing. At N=1 the context shims + TLS sp accessors are ~1% of self-time, not the dominant cost — the shim hypothesis was always a many-core one (its only evidence was the N=1→N=8 jump, blamed on TLS access mode and cross-core coherency on the sp/epoch words), and nothing at N=1 on one core can confirm or refute it. The N=1 round-trip is instead dominated by the scheduler core (schedule_loop + run-queue, ~33%) plus a removable wake-nobody futex_wake.

That futex_wake is now fixed: it was an interaction between the RFC 004 spin defaults and the N=1 pool-size cap of 0 — spinning was gated on budget > 0 alone, so at N=1 spinning was enabled while the cap forbade any spinner from enlisting, pinning n_spinning at 0 and firing a wake-nobody futex_wake on every enqueue. spinning is now gated on budget > 0 && max_spinners > 0 (a debug_assert guards the invariant). On the 1-core sandbox this cut N=1 p50 from ~466 ns to ~186 ns — the syscall was ~60% of the round-trip in wall-clock terms (the handoff's "~12%" was perf self-time, not latency share).

Two separable targets remain: (a) the N=1 floor — schedule_loop + run-queue, the irreducible scheduler core, no obvious cheap win left; (b) the N→8 slope — shim / TLS / coherency, which needs the 5900X (HW-counter profiling; this sandbox has no PMU). Still "the whole game" alongside v0.9. Next code chunk is a remote mode for benches/switch_cost.rs (wake straddling two schedulers) — writable in the sandbox, only measurable on the box. Do not optimise the shim until the N=8 HW-counter data confirms it is the cost. Then an RFC; the spinning-gating fix above may also warrant a small RFC since it touches the RFC-004-settled futex path.

send_after / cancel_timer

Message-delivery timer on the existing min-heap (timer.rs): deliver a value to a channel at a deadline, cancellable. Unlocks the gen_server idioms with no clean expression today — heartbeat, debounce, retry backoff, session expiry. Small;

Introspection — process_info / get_state / tree dump

trace.rs is the seed. What an actor is parked on, queue depth, stack size; a gen_server state snapshot; a supervision-tree walk. The native edge over tokio — actors already carry pid, name, parent where tokio tasks are anonymous — so it costs little and differentiates a lot. Needs an RFC.

Worker pool behaviour

Supervised, interchangeable workers with restart semantics over a shared inbox (poolboy / NimblePool shape) — distinct from connection pools (bb8/deadpool), which pool resources, not supervised processes. Sits on supervisor.rs + gen_server.rs. Needs an RFC.

Medium Priority

Demand-driven pipelines — GenStage / Broadway shape

Supervised producer/consumer stages where consumers signal demand upstream, with batching, ack, partitioning. The clearest thing hex has and crates.io lacks (stream combinators and bounded channels are not a supervised demand-contract stage graph), and the natural fit for ingestion-shaped workloads. Builds on channels + gen_server

• supervisor. Needs an RFC.

Unwakeable idle sleep when io is absent (terminal-wake residual)

The (Some(deadline), None) idle branch — timers pending, io subsystem never initialized — blocks in thread::sleep with no wake mechanism at all. The terminal wake (writes the wake pipe at AllDone) cannot reach it: no io, no pipe. Same stall as the fixed bug, in any no-io runtime: a sibling that blocked on an orphaned deadline sleeps it out in full after everything else finished. Candidates, mutually exclusive: (a) clamp the sleep (cheap, but turns idle into periodic wakeups), or (b) park the branch on a condvar/futex the AllDone path signals — and at that point consider making the condvar the idle primitive for the no-io runtime generally (a cross-thread unpark could signal it too, see below). Decide before any no-io deployment.

Cross-thread unpark

RuntimeInner::enqueue does not wake idle sibling schedulers — only io completions write the wake pipe. Mid-flight this is masked (the enqueuing thread is awake and eats the work itself), but it costs parallelism: work enqueued by a busy thread waits until the sibling's idle poll times out. Needs bench evidence (does the shared-queue handoff latency actually show up?) before a mechanism is picked.

Unbounded / configurable-bounded actor count

Fixed slab with a loud assert (Config::max_actors(n), default 16 384). Revisit with a segmented slab (array of AtomicPtr<Segment>, doubling segment sizes, append-only) once the cap is actually hit. Do not let it calcify.

arm-port validation & merge

arm-port branch carries an AAPCS64 context-switch backend, never run on hardware. Build + run full test suite on an aarch64 device; check chained_spawn / yield_many bench medians; merge and update README.

Low priority

gen_statem — postponement + state timeouts only

A thin layer over gen_server, not a new behaviour. The (state, event) dispatch matrix is free from the type system and not worth porting. The two mechanisms that are: event postponement (defer events in the wrong state, replay on transition — selective receive, codified) and state timeouts (auto-cancel on state change). Device-connection FSMs are the canonical use. Wants send_after underneath. Needs an RFC.

Clustering — distribution epic

Sequenced deliberately after v0.9 and the per-switch-cost spike. A fat stack of RFCs, not one. Spine settled in discussion; decisions still open:

• Explicit remote boundary, never transparency. Serialization colours edges (channel types), not functions — local edges stay zero-copy Send, only remote edges take a RemoteRef<T: Serialize + DeserializeOwned>. No hidden latency when a peer migrates; the refactor is visible by construction.
• One binary, role as runtime config (ROLE=… REGION=… SEEDS=…); a build-hash handshake enforces same-binary type identity and sidesteps cross-version type agreement. Roles select which supervision subtree mounts.
• Distributed pg falls out of local pg + a membership/gossip layer, and distributed pubsub falls out of that for free; per-member metadata (region, load) enables fly-style nearest-member routing.
• Migratable gen_servers as a sub-layer: only behaviours migrate (a raw actor's stack is opaque; a gen_server between callbacks is just its State), gated by Serialize bounds + an on_arrive reacquire hook, addressed by name not pid. The BEAM can't do this — leaning on the behaviour layer is what buys it. Requires State to be serializable, so should probaby spec a trait MigratableGenServer: GenServer where Self::State: Migratable , or something to that extent, so we can lean on the type system to make sure we don't accidentally make state that cannot be serialised.
• CRDT presence is the high-value, genuinely-hard layer above distributed pg, kept out of the pg primitive (the pg2 strong-consistency lesson). Furthest out. Can maybe defer to rust ecosystem

---

Look into

app actors block AllDone; no external stop path

Agent working on urus (see same git server as smarm) reported a lazily spawned actor never returning, blocking program shutdown. Maybe we should do something about it. Agent worked around it by giving the actor an atomic bool to spin on. See urus example crud for exact impl.

---

Invariants & gotchas (respect these across all cycles)

• Shared mutex is non-reentrant. Sender::send can call unpark → with_shared. Never send on a channel while holding the shared lock. Pattern: mem::take data under the lock, send after releasing. See finalize_actor.
• finalize_actor order: take stack/waiters/monitors under lock + set Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs → unpark joiners → reclaim slot if outstanding_handles==0. Death notifications always precede reclamation.
• Slot lifecycle reset in THREE places: Slot::vacant(), reclaim_slot() (runtime.rs), slot-init block in spawn_under (scheduler.rs). Any new Slot field must be reset in all three.
• Pid = (index, generation). Stale handles caught by generation mismatch in slot()/slot_mut(). The monitor NoProc path relies on this.
• The only wildcard wake is request_stop, and it is terminal. Every registration-based waker (channel sends, mutex grants, wait-timers, io completions, joiner wakes, select arms) carries the wait's park-epoch and wakes through unpark_at; every successful wake consumes the epoch. Wakes are therefore meaningful: one-shot park sites interpret them without loops, and select needs no cancellation pass. When adding a new waker, decide which form it is — if its registration handle can outlive the wait it was created for, it MUST be epoch-stamped; a wait that can exit without parking MUST retire_wait first (see slot_state.rs).
• select exists; a unified per-process mailbox still does not. The supervisor keeps its single supervisor_channel funnel; recv_match stays per-channel. select composes channels at the wait, not into one queue — gen_server's handle_info/handle_down (v0.8) are built on exactly that composition, with documented arm priority (downs → control → infos → inbox) instead of mailbox FIFO. A hot higher-priority arm starves lower ones by design; that's the contract.
• Cooperative-only. Preemption and cancellation both depend on the actor reaching check!()/yield/alloc/blocking points.
• Lock order is Leaf → Channel, one of each at most (debug-asserted in raw_mutex.rs). Leaf = cold locks / free list / stack pool / registry, mutual leaves. A channel lock may be taken under a Leaf (finalize/monitor clone senders living in slots); nothing may be locked under a channel lock.
• Queue ops require preemption disabled. A producer suspended mid-publish stalls every consumer — livelock. with_runtime, with_shared, and RawMutex guards all disable preemption for their span.
• run() is single-thread (Config::exact(1)); tests rely on deterministic single-thread ordering. Multi-thread via runtime::init(Config…).