diff --git a/ROADMAP.md b/ROADMAP.md index 94b9783..4716dbd 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,331 +1,131 @@ # smarm — Roadmap -## v0.4 — Actor primitives ✅ DONE +## Shipped (compacted — full cycle plans and deviation records live in git history) -### 1. Cooperative cancellation ✅ (`a8ddb4a`) -Stop flag on `Actor` behind `Arc`. Observation points: -`maybe_preempt`/`check!()` + the wakeup side of `park_current`/`yield_now`. -Sentinel = `StopSentinel` (zero-size), caught in trampoline → `Outcome::Stopped`. -`join()` on a stopped actor returns `Ok(())`. +Cycles before v0.7 (v0.4 actor primitives, v0.5 runtime decomposition & +pluggable run queue, v0.6 actor ergonomics): see `git log ROADMAP.md`. -Known gap (by design): a tight no-alloc loop without `check!()` cannot be -stopped, same as preemption. +### v0.7 — select, on epoch-stamped consuming wakes ✅ +A 24-bit park-epoch packed into the slot word gives every wait an identity: +registrations carry `(pid, epoch)`, every successful wake **consumes** the +epoch, stale wakes die at one failed CAS. Subsumed the per-primitive wait +seqs (channel, mutex, timer); the only wildcard wake left is `request_stop` +(terminal). On top: `select`/`select_timeout` — ready-index wait over many +receivers, priority order, no cancellation pass. Loom theorems re-proved on +the new word. Notable deviations: `retire_wait` for the no-park exit path +(plan missed it); the Drop-guard deregistration was superseded by relaxed +single-receiver asserts; a closed arm is ready *forever* (documented gotcha). +Commits `4913835`…`00128f3`, `a0a93b6`. -### 2. one_for_all / rest_for_one + ordered shutdown ✅ (`351dc9c`) -`Strategy::{OneForOne,OneForAll,RestForOne}` via `.strategy()`. Triggering -child's `Restart` policy decides whether anything restarts; strategy decides -which siblings are cycled. Survivors stopped in reverse start order, awaited on -the existing `supervisor_channel` funnel, restarted in start order. - -`Signal::Stopped(pid)` + `DownReason::Stopped` added (distinct from Exit). - -Also fixed a latent bug (`e80334b`): orphaned timers were gating shutdown. -`live == 0` is now the shutdown condition; timer heap cleared on exit. - -### 3. Links + trap_exit ✅ (`6581484`) -`Slot.links: Vec` (bidirectional). On finalize: abnormal death → -`request_stop(peer)` unless peer traps, in which case deliver `ExitSignal` -message. Normal exit does not propagate. `trap_exit()` returns a dedicated -`Receiver` (distinct from monitor `Down` channel). - -### 4. Selective receive ✅ (`03f3875`) -`Receiver::recv_match(pred)` + `try_recv_match`. Scans `VecDeque` front-to-back, -removes+returns first match, parks and re-scans on new arrivals. `pred` is -`Fn(&T) -> bool` (not `FnMut`; runs under the channel lock — keep it cheap). - -### 5. Grab-bag ✅ -- **demonitor** (`5181037`): `monitor()` → `Monitor { id, target, rx }`. - `demonitor(&m) -> Option`. Dropped `Sender` released post-lock - (non-reentrant shared mutex discipline). -- **gen_server** (`55221e9`): `GenServer` trait, `call`/`cast` over one - `Envelope` inbox. `terminate` hook via drop guard. No `handle_info`, no call - timeout (deferred — needs cross-channel mailbox merge or per-recv deadline). -- **Named registry**: `register`/`whereis`/`send_by_name` — **not yet done**. - HashMap in `SharedState`. No dependencies; can land any time. -- **arm-port branch**: aarch64 context-switch backend extracted into - `src/arch/`. Untested on hardware. Merge only after on-device validation. +### v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅ +Spent `select` on the server loop: static info arms (`type Info`, +`ServerBuilder::with_info`) and dynamic monitor forwarding +(`ServerCtx`/`Watcher` + a control arm), priority downs → control → infos → +inbox. Closed the v0.2 fd hole: a drop guard in `wait_fd` DELs the kernel +registration on unwind (the leak was worse than documented — a stale waiters +entry permanently poisoned the fd). Deviation: the plain-inbox fast path +narrowed; servers holding a `Watcher` select forever. +Commits `e5d1b3b`, `24b95c9`, `f6969e5`. --- -## v0.5 — Runtime decomposition & run-path scaling ✅ DONE +## Decision record — queue topology 🔒 CLOSED (2026-06-10) -Goal: dismantle the single `Mutex` along the run path so the -runtime scales to dozens of cores, with crash isolation hardened so a torn -write or a stray cancellation can never poison shared runtime state. +The run-queue shootout (harness `6d9f369`, 24-core sweep 1–24 schedulers, +report `bench_report_rq_shootout.html`) landed in **RFC 005's World 3**: the +three queue variants are within 10–15% of each other in `rq_runtime` at every +scheduler count ≥ 4, on all three workloads. Striped's real micro-bench +dominance (24.8 M ops/s at 24t, 4.3× over mutex) does not propagate to the +runtime because the queue is not the hot path in any measured workload. USL +fits put σ at 5.5–7.6 (super-serial) for ping-pong: the ceiling is per-wake +latency through the park/unpark protocol, not queue push/pop. -Guiding rules established this cycle: -- **Lock count that matters is hot-path locks.** Cold lifecycle paths may keep a - small per-slot lock; the run path targets zero locks. -- **Lock ordering: io-before-shared, slot locks are leaves** (never hold two - slot locks at once). -- **No unwind inside a runtime critical section.** Stop sentinel only fires at - lock-free observation points. -- **Never hold a thread-local guard across a possible switch point.** An actor - can be preempted at any allocation and resume on a different OS thread. - `with_runtime`, `with_shared`, and every `RawMutex` guard disable preemption - for this reason. - -### Phase 1 — State decomposition ✅ (`3c7e26b`) -`next_monitor_id` → `AtomicU64`. `timers` → own `Mutex`. `io` → own -`Mutex>`. `pending_closures` → folded into `Slot::pending_closure`. -`check_cancelled()` gated behind `PREEMPTION_ENABLED` (poison fix). - -`SharedState` now holds only: `slots`, `free_list`, `run_queue`, `root_pid`. - -### Phase 2 — Slot table split ✅ (`5e0c9d4`) -Fixed slab `Box<[Slot]>`. Generation packed into state word: one -`AtomicU64 = (gen << 32) | state` — gen check is atomic with every transition. -Per-slot CAS state machine `Vacant/Queued/Running/RunningNotified/Parked/Done`. -Per-slot `RawMutex` for cold collections. Free list → `RawMutex>`. -`live_actors` atomic; termination = io_out==0 && queue empty && live==0. - -### Phase 3 — Pluggable run queue ✅ (`1b3b618`) -Compile-time selected via feature flags. Variants: -- `rq-mutex` — `Mutex`, baseline. -- `rq-mpmc` — hand-rolled Vyukov bounded MPMC ring. -- `rq-striped` — M Vyukov rings, fetch-add ticket distribution. - -All variants compile in every build; feature only picks the alias. Bounded rings -are sound via slab cap + at-most-once-enqueued invariant. - -Queue ops require preemption disabled (a producer suspended mid-publish stalls -every consumer behind its cell — livelock). - -### Phase 4 — Bench harness ✅ (`6d9f369`) -`benches/rq_micro.rs`: raw structure sweep. `benches/rq_runtime.rs`: -yield-storm, ping-pong-pairs, spawn-storm. `scripts/bench_rq.sh`: rebuilds per -feature, aggregates to `bench_results/summary.csv`. - -### Phase 5 — Safety hardening & model checking ✅ (`039703d`) -State machine extracted to `src/slot_state.rs` for loom checking. Loom on the -rings via `src/sync_shim.rs`. `NoPreempt`/no-unwind/TLS-guard audit complete. -Invariant-assertion sweep: every `StateWord` transition self-asserts its -precondition. +Consequences: +- **`rq-mutex` stays the default** — simplest correct, no capacity + constraints, locking model already integrated with the preemption-disabled + queue-op invariant. +- **Feature plumbing stays as is.** All three variants keep compiling in + every build; `rq-mpmc`/`rq-striped` remain selectable for benching. +- **Reopening is benchmark-driven only.** The report documents the + conditional upgrade paths if a future workload qualifies: mpmc for + message-passing-dominant loads at N ≤ 8 (1.3× lower σ, 0.16 µs ping-pong + at 1t); striped for high-contention balanced push/pop at N ≥ 16 — neither + is a scheduler workload as measured. +- **Effort redirects to the wake path**: RFC 005 (billed as a latency patch, + per its own World 3 framing), RFC 004, and eventually per-switch cost. --- -## v0.6 — Actor ergonomics ✅ DONE +## v0.9 — Wake-path latency -### 1. Channel mutex migration ✅ (`8ff6cf4`) -`channel::Inner` migrated from `std::sync::Mutex` to `RawMutex`, closing -the cross-thread pthread-release hole (guard held with preemption enabled → -timeslice switch → unlock from a different OS thread, UB) and removing poison. +Goal: attack per-wake latency — the measured ceiling — with the two specified +mechanisms, each benched against a clean baseline before the next lands. +Order: RFC 005 first (the slot is measured against the just-benched idle +policy), then RFC 004 (measured against the slot-accepted baseline), then an +interaction pass. -The strict leaf rule did not survive contact: finalize clones the -supervisor/trap senders and `monitor()` clones the Down sender, all under a -cold lock, and those senders *live in the slot* — the nesting is structural. -The check generalized to two classes: **Leaf → Channel**, at most one of -each, both violation directions debug-asserted at the acquisition site. -Channel critical sections call only the lock-free unpark protocol, so the -order is acyclic. `recv_match`'s user predicate still runs under the channel -lock — cheap/pure/channel-free, as documented. +Full specs: `rfc_005-wake-slot.md`, `rfc_004-tunable-scheduler-idle-policy.md` +(artefacts.kalsbeek.dev). -### 2. Named registry ✅ (`2c7cf0b`) -`register` / `whereis` / `unregister` + the inverse `name_of(pid)`: a -**bimap** (two HashMaps in exact inverse, debug-asserted) under one Leaf -`RawMutex` in `RuntimeInner` — the planned "HashMap in SharedState" predated -the v0.5 decomposition. Erlang semantics: one name per pid, one pid per -name, NameTaken only against a *live* holder. Cleanup is lazy: liveness -rides the generation-checked slot word (pids are never reused), dead -bindings behave as absent and are pruned on contact — no `finalize_actor` -hook, no Slot field, no reset-in-three-places exposure. +### 1. Wake slot (RFC 005) +Per-scheduler, thread-local, capacity-one wake cache, checked before the +shared queue. Runtime-selected via `Config { wake_slot: bool }`, default off +until accepted (one binary benches both arms). +- **Push policy:** slot-eligible iff the wake originates from actor context + (`current_pid().is_some()`). Scheduler-context wakes (timer/IO drain) and + spawns always go shared. +- **Displacement:** newest wake takes the slot, occupant pushed shared (Go + semantics); the branch swap is the fallback if benches look pathological. +- **Pop order:** slot, then shared. Slot-popped actors inherit the waker's + remaining timeslice — a handoff chain is bounded by one slice, so the + shared queue is consulted at least once per slice per scheduler (the + starvation bound, zero new counters). +- **Invariant care:** the slot push replaces `run_queue.push` at the tail of + the `Parked → Queued` CAS; at-most-once-enqueued holds verbatim (pid in + slot ⊕ shared queue). Stall blast radius grows by exactly one actor. +- Counters: `slot_hits`, `slot_displacements`. -`send_by_name` was dropped from scope deliberately: smarm has no per-pid -mailbox, so there is nothing generic to send *to*. The registry yields a -`Pid` for monitor/link/request_stop and for routing user-owned channels. +### 2. Slot shootout +Extend `rq_runtime`/`bench_rq.sh` with the slot on/off dimension (cheap — +it's a Config knob, not a feature rebuild). Same sweep as the rq shootout. +- **ping-pong-pairs** — target metric; expect the win, single- and + multi-scheduler. +- **yield-storm** — regression guard; yields never touch the slot, any delta + is pop-path overhead. +- **spawn-storm** — neutrality check; spawns bypass the slot by policy. +Acceptance flips the default on and re-baselines for RFC 004. -### 3. gen_server: call timeout ✅ (`134ff52`, `90b7040`) -Landed as two primitives, *without* the sketched monitor machinery: +### 3. Spinning workers (RFC 004) +Bounded spin-before-park for idle schedulers, killing the ~100 µs +`thread::sleep` worst case on cross-thread handoff. Two Config knobs: +`spin_budget_cycles` (0 recovers today's behaviour) and `max_spinners` +(default N/2); one new atomic `n_spinning`. No run-queue changes — lands on +the frozen `rq-mutex` substrate, independent of the slot mechanically. -- `Receiver::recv_timeout(Duration)` — the per-`recv` deadline, on the - existing `timer::Reason::WaitTimeout`/`TimerTarget` machinery exactly as - `Mutex::lock_timeout` uses it (per-wait seq; stale heap entries no-op on - seq mismatch; message-first race resolution; `Disconnected` distinct from - `Timeout`). -- `ServerRef::call_timeout` = send + `recv_timeout` on the reply channel. - Server death is already observable there (reply sender drops as the server - unwinds), so monitor + demonitor-on-timeout had nothing left to guarantee: - no registration exists, so nothing can leak. Timed-out requests are still - handled; the late reply's send fails harmlessly (Erlang semantics). - -(`handle_info` still deferred — needs cross-channel mailbox merge; see Later.) - ---- - -## v0.7 — select, on epoch-stamped consuming wakes ✅ DONE - -Goal: cross-channel wait (`select`) without surrendering the precise-wake -invariant. Decision record: the alternative ("wakes are hints", loop-and- -recheck at every park site) was evaluated and rejected — it distributes the -soundness obligation across every present and future park site and turns -every stale wake into a wasted reschedule. Instead the slot protocol itself -gains *wait identity*: a park-epoch packed into the spare bits of the slot -word, so that a stale unpark dies at one failed CAS in the waker. - -Protocol (the consuming-wake rules): -- Word becomes `(gen << 32) | (epoch << 8) | state` — 24-bit epoch in the - previously unused bits; every transition still one CAS, gen check still - atomic with it. -- `begin_wait` (actor-side, before registering with any waker) bumps the - epoch and returns it; registrations carry `(pid, epoch)`. -- Every successful wake transition **consumes** the epoch (bumps it): - `Parked(e) → Queued(e+1)`, `Running(e) → RunningNotified(e+1)`. A second - unpark stamped with the same epoch can never land — at most one wake per - wait, by construction. This is what makes select's *loser* arms harmless - without a cancellation pass: their stale registrations fire one futile CAS - on next send and self-clean. -- `unpark_at(pid, epoch)` = epoch-matched wake. Bare `unpark(pid)` remains as - the wildcard and is reserved for **terminal** wakes (`request_stop`), which - never return control to the code that parked. -- One-shot park sites (`Mutex::lock_timeout`, `sleep`, `block_on_io`, - `wait_fd`) stay exactly as written: precise wakes are preserved, so a wake - still *means* something. - -### 1. Slot-word epoch + consuming unparks ✅ (`4913835`) -`slot_state.rs`: repack, `begin_wait`, epoch-matched/consuming `unpark`, -epoch-preserving claim/yield/park transitions. Loom: existing theorems -re-proved on the new word + a new stale-epoch theorem (an unpark stamped with -a consumed epoch can never enqueue *or* notify). Plumbing only — no caller -stamps yet; behaviour identical. - -### 2. Stamp every registration-based wake; retire per-primitive wait seqs ✅ (`400854a`) -The epoch IS the wait identity, so `channel::Inner.{cur_wait,next_wait_seq}` -and `mutex`'s per-wait `seq` are subsumed: registrations become -`(pid, epoch)`, `TimerTarget::on_timeout(pid, epoch)`, wakers call -`unpark_at`. Sleep timers, io completions (`Blocking` + `FdReady`) and join -waiters stamped likewise. End-state invariant, auditable in one sentence: -**the only wildcard wake is `request_stop`.** - -### 3. `select` — ready-index wait over multiple receivers ✅ (`00128f3`) -`select(&[&dyn Selectable]) -> usize`: register self in each arm's -`parked_receiver` (sequentially — the park protocol makes cross-arm atomicity -unnecessary), park once, wake, scan, return the first ready index; caller -`try_recv`s it. Sound as ready-index *because channels are single-receiver*: -nobody can steal between ready and recv. A closed arm counts as ready (its -`try_recv` reports the closure). Arms are scanned in order — priority order, -documented, like BEAM receive clauses; no fairness promise. No cancel pass -(see protocol above). Drop-guard deregistration on the unwind path only, to -keep tests' single-parked-receiver asserts honest. - -### 4. `select_timeout` ✅ (`a0a93b6`) -A `WaitTimeout` timer stamped with the select's epoch, whose target is a -stateless `unpark_at`. The woken selector classifies the wake from state -alone: some arm ready → that arm; none ready → with precise wakes the timer -is the only remaining stamped waker → `None`. No flag, no shared waiter -struct, no registration to leak. The timer entry expires as a stale-epoch -no-op when an arm wins, per the no-cancellation convention in `timer.rs`. - -### 5. Docs + integration proof ✅ (this commit) -README module table, gotchas rewrite (the "No `select`" bullet becomes the -consuming-wake invariant), and an integration test of the motivating -pattern: a server selecting over its inbox and a monitor `Down` channel -(`tests/select.rs::motivating_pattern_inbox_plus_monitor_down`). - -Unblocked but deliberately out of scope: `gen_server::handle_info` (now an -API-design question, not a runtime question — see Later). - -### Deviations from the plan -- **The no-park exit needed a protocol piece the plan missed.** A `select` - returning an arm ready at *registration* time leaves earlier arms holding - live-epoch registrations — a sender could still set RunningNotified and - fault the actor's next one-shot park. Landed as `scheduler::retire_wait` - (bump the epoch, eat a landed notification via the new - `StateWord::clear_notify`, re-observe the stop flag — in that order) plus - the loom theorem `retire_eats_late_arm_notification`. The parked path - needed nothing: the winning wake's consume already retires the epoch. -- **No Drop guard.** The planned unwind-path deregistration was superseded: - the single-receiver debug_asserts relax to "none, or my own pid", so a - leftover own-registration (consumed epoch — inert by theorem) is simply - overwritten by the receiver's next wait on that channel. Less machinery, - and the asserts still catch the real misuse (two actors on one channel). -- **A closed arm is ready *forever*** — callers must drop it from the arm - set once observed, or priority order starves every higher-indexed arm - (documented on `select`; found the hard way by a livelocking test). - ---- - -## v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅ DONE - -Goal: spend the `select` primitive on the server loop — out-of-band messages -(`handle_info`) and monitor `Down`s (`handle_down`) dispatched alongside -calls and casts — and close the v0.2 fd-hygiene hole in `io.rs`. - -Design decisions (settled up front): -- **Info channels are static.** `type Info: Send + 'static` on the trait; a - `Vec>` handed over at start. Users enum their own Info. - Dynamic info subscription is deliberately out: revisit against a consumer. -- **Down forwarding is dynamic, because monitors are.** You monitor a worker - you just spawned inside a handler, so down arms cannot ride the static - list. Mechanism: the loop owns a control arm (`Receiver`); - `init` grows a `&ServerCtx` parameter (breaking, no-op default body) whose - clonable `Watcher` the state stores and calls `watch(monitor)` from any - handler thereafter. Handler signatures otherwise untouched. -- **Priority order: downs → control → infos → inbox.** System messages - preempt; a hot inbox cannot starve a death notice. Within each class, - declaration order (select's documented priority semantics). -- **Closed arms drop silently** (the closed-arm-is-ready-forever gotcha). - Closed inbox still means graceful shutdown, unchanged. - -### 1. handle_info — static info arms in the server loop ✅ (`e5d1b3b`) -`type Info` + `handle_info` (no-op default) on `GenServer`; `ServerBuilder` -(`Server::new(state).with_info(rx).under(sup).start()`) so start variants -don't multiply; `start`/`start_under` stay as wrappers. Loop becomes -`select` over `[infos…, inbox]`, dispatching by arm index. - -### 2. handle_down — ServerCtx::watcher + control arm ✅ (`24b95c9`) -`handle_down` (no-op default), `ServerCtx`/`Watcher`, the control arm, and -the loop's live `Vec`. A `Down` retires its arm (monitors are -one-shot). Integration test: the motivating worker-pool pattern — a server -that spawns workers in a handler, watches them, and reacts to their deaths. - -### 3. io fd hygiene — DEL on the unwind path ✅ (`f6969e5`) -A stopped actor unwinding out of `wait_fd`'s park leaks its `waiters` entry -and the kernel registration; the stale entry then fails every future -`wait_*` on that fd with `AlreadyExists` (the defensive bare-DEL in -`epoll_register` sits behind the `contains_key` check, so it never runs). -Fix where the invariant breaks: a drop guard in `wait_fd`, armed after a -successful register, disarmed on normal wake; on unwind it removes the -waiters entry iff it is still `(me, epoch)` and only then DELs the fd -(an entry consumed by a racing `FdReady` means the fd may already be -re-registered by someone else — leave it alone). - -### Deviations from the plan -- **The plain-inbox fast path narrowed.** It now also requires the control - arm closed, so every server pays one select on its first iteration (the - ctx drops after `init`; a state that didn't clone the Watcher closes the - arm there and then). Servers that DO hold a Watcher select forever — the - price of dynamic watch. -- **The fd leak was worse than the v0.2 caveat admitted.** Not "at most one - stale wakeup": the stale `waiters` entry failed every future `wait_*` on - that fd with AlreadyExists, permanently (the defensive DEL sat behind the - `contains_key` check). The regression test exhibits it as a hang. +### 4. RFC 004 bench + interaction pass +Re-run the sweep with spinning enabled, slot on and off. The known +interaction: once idle pickup latency ≪ slice, the slot's latency trade +(occupant waits out the waker's slice while other schedulers idle) may stop +paying — RFC 005's documented escape hatch is Go's "spinners exist → push +shared instead" check. In scope only if the data demands it. --- ## Later -### Tunable scheduler idle policy (RFC 004) -Today idle schedulers fall back to `thread::sleep(100µs)`, putting ~100µs -worst-case latency on every cross-thread actor handoff. RFC 004 specifies a -bounded spinning-worker policy (Go/Tokio model): idle schedulers spin on a -cheap atomic for a configurable budget before parking, so a burst of work is -picked up in cache-coherency time rather than after a kernel reschedule. - -Two new `Config` knobs: `spin_budget_cycles` (RDTSC cycles to spin; `0` -recovers today's behaviour) and `max_spinners` (cap on concurrent spinners; -default `N/2`). Single new atomic `n_spinning`. No change to the run-queue -data structure — lands on the current `Mutex` and is independent of -any future queue topology work. - -Full spec: `rfc_004-tunable-scheduler-idle-policy.md` (artefacts.kalsbeek.dev). +### Per-switch cost (context shims, epoch protocol) +The shootout's residual: per-wake latency is 0.16–0.18 µs at N=1 and +0.8–1.2 µs at N=8+, dominated by the context-switch shims and the epoch +protocol, not the queue. On current evidence this is the larger constant — +"the whole game" alongside the v0.9 work — but there is no spec yet. Needs a +profiling spike (where do the cycles actually go per park/unpark round-trip) +and then an RFC before it can be scheduled. ### Unbounded / configurable-bounded actor count Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384). Revisit with a segmented slab (array of `AtomicPtr`, doubling segment sizes, append-only) once the cap is actually hit. Do not let it calcify. - - ### arm-port validation & merge `arm-port` branch carries an AAPCS64 context-switch backend, never run on hardware. Build + run full test suite on an aarch64 device; check