smarm/ROADMAP.md

smarm — Roadmap

v0.4 — Actor primitives ✅ DONE

1. Cooperative cancellation ✅ (a8ddb4a)

Stop flag on Actor behind Arc<AtomicBool>. Observation points: maybe_preempt/check!() + the wakeup side of park_current/yield_now. Sentinel = StopSentinel (zero-size), caught in trampoline → Outcome::Stopped. join() on a stopped actor returns Ok(()).

Known gap (by design): a tight no-alloc loop without check!() cannot be stopped, same as preemption.

2. one_for_all / rest_for_one + ordered shutdown ✅ (351dc9c)

Strategy::{OneForOne,OneForAll,RestForOne} via .strategy(). Triggering child's Restart policy decides whether anything restarts; strategy decides which siblings are cycled. Survivors stopped in reverse start order, awaited on the existing supervisor_channel funnel, restarted in start order.

Signal::Stopped(pid) + DownReason::Stopped added (distinct from Exit).

Also fixed a latent bug (e80334b): orphaned timers were gating shutdown. live == 0 is now the shutdown condition; timer heap cleared on exit.

3. Links + trap_exit ✅ (6581484)

Slot.links: Vec<Pid> (bidirectional). On finalize: abnormal death → request_stop(peer) unless peer traps, in which case deliver ExitSignal message. Normal exit does not propagate. trap_exit() returns a dedicated Receiver<ExitSignal> (distinct from monitor Down channel).

4. Selective receive ✅ (03f3875)

Receiver::recv_match(pred) + try_recv_match. Scans VecDeque front-to-back, removes+returns first match, parks and re-scans on new arrivals. pred is Fn(&T) -> bool (not FnMut; runs under the channel lock — keep it cheap).

5. Grab-bag ✅

• demonitor (5181037): monitor() → Monitor { id, target, rx }. demonitor(&m) -> Option<MonitorId>. Dropped Sender released post-lock (non-reentrant shared mutex discipline).
• gen_server (55221e9): GenServer trait, call/cast over one Envelope inbox. terminate hook via drop guard. No handle_info, no call timeout (deferred — needs cross-channel mailbox merge or per-recv deadline).
• Named registry: register/whereis/send_by_name — not yet done. HashMap in SharedState. No dependencies; can land any time.
• arm-port branch: aarch64 context-switch backend extracted into src/arch/. Untested on hardware. Merge only after on-device validation.

---

v0.5 — Runtime decomposition & run-path scaling ✅ DONE

Goal: dismantle the single Mutex<SharedState> along the run path so the runtime scales to dozens of cores, with crash isolation hardened so a torn write or a stray cancellation can never poison shared runtime state.

Guiding rules established this cycle:

• Lock count that matters is hot-path locks. Cold lifecycle paths may keep a small per-slot lock; the run path targets zero locks.
• Lock ordering: io-before-shared, slot locks are leaves (never hold two slot locks at once).
• No unwind inside a runtime critical section. Stop sentinel only fires at lock-free observation points.
• Never hold a thread-local guard across a possible switch point. An actor can be preempted at any allocation and resume on a different OS thread. with_runtime, with_shared, and every RawMutex guard disable preemption for this reason.

Phase 1 — State decomposition ✅ (3c7e26b)

next_monitor_id → AtomicU64. timers → own Mutex<Timers>. io → own Mutex<Option<IoThread>>. pending_closures → folded into Slot::pending_closure. check_cancelled() gated behind PREEMPTION_ENABLED (poison fix).

SharedState now holds only: slots, free_list, run_queue, root_pid.

Phase 2 — Slot table split ✅ (5e0c9d4)

Fixed slab Box<[Slot]>. Generation packed into state word: one AtomicU64 = (gen << 32) | state — gen check is atomic with every transition. Per-slot CAS state machine Vacant/Queued/Running/RunningNotified/Parked/Done. Per-slot RawMutex for cold collections. Free list → RawMutex<Vec<u32>>. live_actors atomic; termination = io_out==0 && queue empty && live==0.

Phase 3 — Pluggable run queue ✅ (1b3b618)

Compile-time selected via feature flags. Variants:

• rq-mutex — Mutex<VecDeque>, baseline.
• rq-mpmc — hand-rolled Vyukov bounded MPMC ring.
• rq-striped — M Vyukov rings, fetch-add ticket distribution.

All variants compile in every build; feature only picks the alias. Bounded rings are sound via slab cap + at-most-once-enqueued invariant.

Queue ops require preemption disabled (a producer suspended mid-publish stalls every consumer behind its cell — livelock).

Phase 4 — Bench harness ✅ (6d9f369)

benches/rq_micro.rs: raw structure sweep. benches/rq_runtime.rs: yield-storm, ping-pong-pairs, spawn-storm. scripts/bench_rq.sh: rebuilds per feature, aggregates to bench_results/summary.csv.

Phase 5 — Safety hardening & model checking ✅ (039703d)

State machine extracted to src/slot_state.rs for loom checking. Loom on the rings via src/sync_shim.rs. NoPreempt/no-unwind/TLS-guard audit complete. Invariant-assertion sweep: every StateWord transition self-asserts its precondition.

---

v0.6 — Actor ergonomics ✅ DONE

1. Channel mutex migration ✅ (8ff6cf4)

channel::Inner<T> migrated from std::sync::Mutex to RawMutex, closing the cross-thread pthread-release hole (guard held with preemption enabled → timeslice switch → unlock from a different OS thread, UB) and removing poison.

The strict leaf rule did not survive contact: finalize clones the supervisor/trap senders and monitor() clones the Down sender, all under a cold lock, and those senders live in the slot — the nesting is structural. The check generalized to two classes: Leaf → Channel, at most one of each, both violation directions debug-asserted at the acquisition site. Channel critical sections call only the lock-free unpark protocol, so the order is acyclic. recv_match's user predicate still runs under the channel lock — cheap/pure/channel-free, as documented.

2. Named registry ✅ (2c7cf0b)

register / whereis / unregister + the inverse name_of(pid): a bimap (two HashMaps in exact inverse, debug-asserted) under one Leaf RawMutex in RuntimeInner — the planned "HashMap in SharedState" predated the v0.5 decomposition. Erlang semantics: one name per pid, one pid per name, NameTaken only against a live holder. Cleanup is lazy: liveness rides the generation-checked slot word (pids are never reused), dead bindings behave as absent and are pruned on contact — no finalize_actor hook, no Slot field, no reset-in-three-places exposure.

send_by_name was dropped from scope deliberately: smarm has no per-pid mailbox, so there is nothing generic to send to. The registry yields a Pid for monitor/link/request_stop and for routing user-owned channels.

3. gen_server: call timeout ✅ (134ff52, 90b7040)

Landed as two primitives, without the sketched monitor machinery:

• Receiver::recv_timeout(Duration) — the per-recv deadline, on the existing timer::Reason::WaitTimeout/TimerTarget machinery exactly as Mutex::lock_timeout uses it (per-wait seq; stale heap entries no-op on seq mismatch; message-first race resolution; Disconnected distinct from Timeout).
• ServerRef::call_timeout = send + recv_timeout on the reply channel. Server death is already observable there (reply sender drops as the server unwinds), so monitor + demonitor-on-timeout had nothing left to guarantee: no registration exists, so nothing can leak. Timed-out requests are still handled; the late reply's send fails harmlessly (Erlang semantics).

(handle_info still deferred — needs cross-channel mailbox merge; see Later.)

---

v0.7 — select, on epoch-stamped consuming wakes ✅ DONE

Goal: cross-channel wait (select) without surrendering the precise-wake invariant. Decision record: the alternative ("wakes are hints", loop-and- recheck at every park site) was evaluated and rejected — it distributes the soundness obligation across every present and future park site and turns every stale wake into a wasted reschedule. Instead the slot protocol itself gains wait identity: a park-epoch packed into the spare bits of the slot word, so that a stale unpark dies at one failed CAS in the waker.

Protocol (the consuming-wake rules):

• Word becomes (gen << 32) | (epoch << 8) | state — 24-bit epoch in the previously unused bits; every transition still one CAS, gen check still atomic with it.
• begin_wait (actor-side, before registering with any waker) bumps the epoch and returns it; registrations carry (pid, epoch).
• Every successful wake transition consumes the epoch (bumps it): Parked(e) → Queued(e+1), Running(e) → RunningNotified(e+1). A second unpark stamped with the same epoch can never land — at most one wake per wait, by construction. This is what makes select's loser arms harmless without a cancellation pass: their stale registrations fire one futile CAS on next send and self-clean.
• unpark_at(pid, epoch) = epoch-matched wake. Bare unpark(pid) remains as the wildcard and is reserved for terminal wakes (request_stop), which never return control to the code that parked.
• One-shot park sites (Mutex::lock_timeout, sleep, block_on_io, wait_fd) stay exactly as written: precise wakes are preserved, so a wake still means something.

1. Slot-word epoch + consuming unparks ✅ (4913835)

slot_state.rs: repack, begin_wait, epoch-matched/consuming unpark, epoch-preserving claim/yield/park transitions. Loom: existing theorems re-proved on the new word + a new stale-epoch theorem (an unpark stamped with a consumed epoch can never enqueue or notify). Plumbing only — no caller stamps yet; behaviour identical.

2. Stamp every registration-based wake; retire per-primitive wait seqs ✅ (400854a)

The epoch IS the wait identity, so channel::Inner.{cur_wait,next_wait_seq} and mutex's per-wait seq are subsumed: registrations become (pid, epoch), TimerTarget::on_timeout(pid, epoch), wakers call unpark_at. Sleep timers, io completions (Blocking + FdReady) and join waiters stamped likewise. End-state invariant, auditable in one sentence: the only wildcard wake is request_stop.

3. select — ready-index wait over multiple receivers ✅ (00128f3)

select(&[&dyn Selectable]) -> usize: register self in each arm's parked_receiver (sequentially — the park protocol makes cross-arm atomicity unnecessary), park once, wake, scan, return the first ready index; caller try_recvs it. Sound as ready-index because channels are single-receiver: nobody can steal between ready and recv. A closed arm counts as ready (its try_recv reports the closure). Arms are scanned in order — priority order, documented, like BEAM receive clauses; no fairness promise. No cancel pass (see protocol above). Drop-guard deregistration on the unwind path only, to keep tests' single-parked-receiver asserts honest.

4. select_timeout ✅ (a0a93b6)

A WaitTimeout timer stamped with the select's epoch, whose target is a stateless unpark_at. The woken selector classifies the wake from state alone: some arm ready → that arm; none ready → with precise wakes the timer is the only remaining stamped waker → None. No flag, no shared waiter struct, no registration to leak. The timer entry expires as a stale-epoch no-op when an arm wins, per the no-cancellation convention in timer.rs.

5. Docs + integration proof ✅ (this commit)

README module table, gotchas rewrite (the "No select" bullet becomes the consuming-wake invariant), and an integration test of the motivating pattern: a server selecting over its inbox and a monitor Down channel (tests/select.rs::motivating_pattern_inbox_plus_monitor_down).

Unblocked but deliberately out of scope: gen_server::handle_info (now an API-design question, not a runtime question — see Later).

Deviations from the plan

• The no-park exit needed a protocol piece the plan missed. A select returning an arm ready at registration time leaves earlier arms holding live-epoch registrations — a sender could still set RunningNotified and fault the actor's next one-shot park. Landed as scheduler::retire_wait (bump the epoch, eat a landed notification via the new StateWord::clear_notify, re-observe the stop flag — in that order) plus the loom theorem retire_eats_late_arm_notification. The parked path needed nothing: the winning wake's consume already retires the epoch.
• No Drop guard. The planned unwind-path deregistration was superseded: the single-receiver debug_asserts relax to "none, or my own pid", so a leftover own-registration (consumed epoch — inert by theorem) is simply overwritten by the receiver's next wait on that channel. Less machinery, and the asserts still catch the real misuse (two actors on one channel).
• A closed arm is ready forever — callers must drop it from the arm set once observed, or priority order starves every higher-indexed arm (documented on select; found the hard way by a livelocking test).

---

v0.8 — gen_server: handle_info / handle_down + io fd hygiene

Goal: spend the select primitive on the server loop — out-of-band messages (handle_info) and monitor Downs (handle_down) dispatched alongside calls and casts — and close the v0.2 fd-hygiene hole in io.rs.

Design decisions (settled up front):

• Info channels are static. type Info: Send + 'static on the trait; a Vec<Receiver<G::Info>> handed over at start. Users enum their own Info. Dynamic info subscription is deliberately out: revisit against a consumer.
• Down forwarding is dynamic, because monitors are. You monitor a worker you just spawned inside a handler, so down arms cannot ride the static list. Mechanism: the loop owns a control arm (Receiver<Monitor>); init grows a &ServerCtx parameter (breaking, no-op default body) whose clonable Watcher the state stores and calls watch(monitor) from any handler thereafter. Handler signatures otherwise untouched.
• Priority order: downs → control → infos → inbox. System messages preempt; a hot inbox cannot starve a death notice. Within each class, declaration order (select's documented priority semantics).
• Closed arms drop silently (the closed-arm-is-ready-forever gotcha). Closed inbox still means graceful shutdown, unchanged.

1. handle_info — static info arms in the server loop

type Info + handle_info (no-op default) on GenServer; ServerBuilder (Server::new(state).with_info(rx).under(sup).start()) so start variants don't multiply; start/start_under stay as wrappers. Loop becomes select over [infos…, inbox], dispatching by arm index.

2. handle_down — ServerCtx::watcher + control arm

handle_down (no-op default), ServerCtx/Watcher, the control arm, and the loop's live Vec<Monitor>. A Down retires its arm (monitors are one-shot). Integration test: the motivating worker-pool pattern — a server that spawns workers in a handler, watches them, and reacts to their deaths.

3. io fd hygiene — DEL on the unwind path

A stopped actor unwinding out of wait_fd's park leaks its waiters entry and the kernel registration; the stale entry then fails every future wait_* on that fd with AlreadyExists (the defensive bare-DEL in epoll_register sits behind the contains_key check, so it never runs). Fix where the invariant breaks: a drop guard in wait_fd, armed after a successful register, disarmed on normal wake; on unwind it removes the waiters entry iff it is still (me, epoch) and only then DELs the fd (an entry consumed by a racing FdReady means the fd may already be re-registered by someone else — leave it alone).

---

Later

Tunable scheduler idle policy (RFC 004)

Today idle schedulers fall back to thread::sleep(100µs), putting ~100µs worst-case latency on every cross-thread actor handoff. RFC 004 specifies a bounded spinning-worker policy (Go/Tokio model): idle schedulers spin on a cheap atomic for a configurable budget before parking, so a burst of work is picked up in cache-coherency time rather than after a kernel reschedule.

Two new Config knobs: spin_budget_cycles (RDTSC cycles to spin; 0 recovers today's behaviour) and max_spinners (cap on concurrent spinners; default N/2). Single new atomic n_spinning. No change to the run-queue data structure — lands on the current Mutex<VecDeque> and is independent of any future queue topology work.

Full spec: rfc_004-tunable-scheduler-idle-policy.md (artefacts.kalsbeek.dev).

Unbounded / configurable-bounded actor count

Fixed slab with a loud assert (Config::max_actors(n), default 16 384). Revisit with a segmented slab (array of AtomicPtr<Segment>, doubling segment sizes, append-only) once the cap is actually hit. Do not let it calcify.

arm-port validation & merge

arm-port branch carries an AAPCS64 context-switch backend, never run on hardware. Build + run full test suite on an aarch64 device; check chained_spawn / yield_many bench medians; merge and update README.

---

Invariants & gotchas (respect these across all cycles)

• Shared mutex is non-reentrant. Sender::send can call unpark → with_shared. Never send on a channel while holding the shared lock. Pattern: mem::take data under the lock, send after releasing. See finalize_actor.
• finalize_actor order: take stack/waiters/monitors under lock + set Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs → unpark joiners → reclaim slot if outstanding_handles==0. Death notifications always precede reclamation.
• Slot lifecycle reset in THREE places: Slot::vacant(), reclaim_slot() (runtime.rs), slot-init block in spawn_under (scheduler.rs). Any new Slot field must be reset in all three.
• Pid = (index, generation). Stale handles caught by generation mismatch in slot()/slot_mut(). The monitor NoProc path relies on this.
• The only wildcard wake is request_stop, and it is terminal. Every registration-based waker (channel sends, mutex grants, wait-timers, io completions, joiner wakes, select arms) carries the wait's park-epoch and wakes through unpark_at; every successful wake consumes the epoch. Wakes are therefore meaningful: one-shot park sites interpret them without loops, and select needs no cancellation pass. When adding a new waker, decide which form it is — if its registration handle can outlive the wait it was created for, it MUST be epoch-stamped; a wait that can exit without parking MUST retire_wait first (see slot_state.rs).
• select exists; a unified per-process mailbox still does not. The supervisor keeps its single supervisor_channel funnel; recv_match stays per-channel. select composes channels at the wait, not into one queue — the mailbox-merge decision remains open (see Later: gen_server handle_info).
• Cooperative-only. Preemption and cancellation both depend on the actor reaching check!()/yield/alloc/blocking points.
• Lock order is Leaf → Channel, one of each at most (debug-asserted in raw_mutex.rs). Leaf = cold locks / free list / stack pool / registry, mutual leaves. A channel lock may be taken under a Leaf (finalize/monitor clone senders living in slots); nothing may be locked under a channel lock.
• Queue ops require preemption disabled. A producer suspended mid-publish stalls every consumer — livelock. with_runtime, with_shared, and RawMutex guards all disable preemption for their span.
• run() is single-thread (Config::exact(1)); tests rely on deterministic single-thread ordering. Multi-thread via runtime::init(Config…).