smarm/ROADMAP.md

# smarm — Roadmap

## v0.4 — Actor primitives  ✅ DONE

### 1. Cooperative cancellation  ✅ (`a8ddb4a`)
Stop flag on `Actor` behind `Arc<AtomicBool>`. Observation points:
`maybe_preempt`/`check!()` + the wakeup side of `park_current`/`yield_now`.
Sentinel = `StopSentinel` (zero-size), caught in trampoline → `Outcome::Stopped`.
`join()` on a stopped actor returns `Ok(())`.

Known gap (by design): a tight no-alloc loop without `check!()` cannot be
stopped, same as preemption.

### 2. one_for_all / rest_for_one + ordered shutdown  ✅ (`351dc9c`)
`Strategy::{OneForOne,OneForAll,RestForOne}` via `.strategy()`. Triggering
child's `Restart` policy decides whether anything restarts; strategy decides
which siblings are cycled. Survivors stopped in reverse start order, awaited on
the existing `supervisor_channel` funnel, restarted in start order.

`Signal::Stopped(pid)` + `DownReason::Stopped` added (distinct from Exit).

Also fixed a latent bug (`e80334b`): orphaned timers were gating shutdown.
`live == 0` is now the shutdown condition; timer heap cleared on exit.

### 3. Links + trap_exit  ✅ (`6581484`)
`Slot.links: Vec<Pid>` (bidirectional). On finalize: abnormal death →
`request_stop(peer)` unless peer traps, in which case deliver `ExitSignal`
message. Normal exit does not propagate. `trap_exit()` returns a dedicated
`Receiver<ExitSignal>` (distinct from monitor `Down` channel).

### 4. Selective receive  ✅ (`03f3875`)
`Receiver::recv_match(pred)` + `try_recv_match`. Scans `VecDeque` front-to-back,
removes+returns first match, parks and re-scans on new arrivals. `pred` is
`Fn(&T) -> bool` (not `FnMut`; runs under the channel lock — keep it cheap).

### 5. Grab-bag  ✅
- **demonitor** (`5181037`): `monitor()` → `Monitor { id, target, rx }`.
  `demonitor(&m) -> Option<MonitorId>`. Dropped `Sender` released post-lock
  (non-reentrant shared mutex discipline).
- **gen_server** (`55221e9`): `GenServer` trait, `call`/`cast` over one
  `Envelope` inbox. `terminate` hook via drop guard. No `handle_info`, no call
  timeout (deferred — needs cross-channel mailbox merge or per-recv deadline).
- **Named registry**: `register`/`whereis`/`send_by_name` — **not yet done**.
  HashMap in `SharedState`. No dependencies; can land any time.
- **arm-port branch**: aarch64 context-switch backend extracted into
  `src/arch/`. Untested on hardware. Merge only after on-device validation.

---

## v0.5 — Runtime decomposition & run-path scaling  ✅ DONE

Goal: dismantle the single `Mutex<SharedState>` along the run path so the
runtime scales to dozens of cores, with crash isolation hardened so a torn
write or a stray cancellation can never poison shared runtime state.

Guiding rules established this cycle:
- **Lock count that matters is hot-path locks.** Cold lifecycle paths may keep a
  small per-slot lock; the run path targets zero locks.
- **Lock ordering: io-before-shared, slot locks are leaves** (never hold two
  slot locks at once).
- **No unwind inside a runtime critical section.** Stop sentinel only fires at
  lock-free observation points.
- **Never hold a thread-local guard across a possible switch point.** An actor
  can be preempted at any allocation and resume on a different OS thread.
  `with_runtime`, `with_shared`, and every `RawMutex` guard disable preemption
  for this reason.

### Phase 1 — State decomposition  ✅ (`3c7e26b`)
`next_monitor_id` → `AtomicU64`. `timers` → own `Mutex<Timers>`. `io` → own
`Mutex<Option<IoThread>>`. `pending_closures` → folded into `Slot::pending_closure`.
`check_cancelled()` gated behind `PREEMPTION_ENABLED` (poison fix).

`SharedState` now holds only: `slots`, `free_list`, `run_queue`, `root_pid`.

### Phase 2 — Slot table split  ✅ (`5e0c9d4`)
Fixed slab `Box<[Slot]>`. Generation packed into state word: one
`AtomicU64 = (gen << 32) | state` — gen check is atomic with every transition.
Per-slot CAS state machine `Vacant/Queued/Running/RunningNotified/Parked/Done`.
Per-slot `RawMutex` for cold collections. Free list → `RawMutex<Vec<u32>>`.
`live_actors` atomic; termination = io_out==0 && queue empty && live==0.

### Phase 3 — Pluggable run queue  ✅ (`1b3b618`)
Compile-time selected via feature flags. Variants:
- `rq-mutex` — `Mutex<VecDeque>`, baseline.
- `rq-mpmc` — hand-rolled Vyukov bounded MPMC ring.
- `rq-striped` — M Vyukov rings, fetch-add ticket distribution.

All variants compile in every build; feature only picks the alias. Bounded rings
are sound via slab cap + at-most-once-enqueued invariant.

Queue ops require preemption disabled (a producer suspended mid-publish stalls
every consumer behind its cell — livelock).

### Phase 4 — Bench harness  ✅ (`6d9f369`)
`benches/rq_micro.rs`: raw structure sweep. `benches/rq_runtime.rs`:
yield-storm, ping-pong-pairs, spawn-storm. `scripts/bench_rq.sh`: rebuilds per
feature, aggregates to `bench_results/summary.csv`.

### Phase 5 — Safety hardening & model checking  ✅ (`039703d`)
State machine extracted to `src/slot_state.rs` for loom checking. Loom on the
rings via `src/sync_shim.rs`. `NoPreempt`/no-unwind/TLS-guard audit complete.
Invariant-assertion sweep: every `StateWord` transition self-asserts its
precondition.

---

## v0.6 — Actor ergonomics  ✅ DONE

### 1. Channel mutex migration  ✅ (`8ff6cf4`)
`channel::Inner<T>` migrated from `std::sync::Mutex` to `RawMutex`, closing
the cross-thread pthread-release hole (guard held with preemption enabled →
timeslice switch → unlock from a different OS thread, UB) and removing poison.

The strict leaf rule did not survive contact: finalize clones the
supervisor/trap senders and `monitor()` clones the Down sender, all under a
cold lock, and those senders *live in the slot* — the nesting is structural.
The check generalized to two classes: **Leaf → Channel**, at most one of
each, both violation directions debug-asserted at the acquisition site.
Channel critical sections call only the lock-free unpark protocol, so the
order is acyclic. `recv_match`'s user predicate still runs under the channel
lock — cheap/pure/channel-free, as documented.

### 2. Named registry  ✅ (`2c7cf0b`)
`register` / `whereis` / `unregister` + the inverse `name_of(pid)`: a
**bimap** (two HashMaps in exact inverse, debug-asserted) under one Leaf
`RawMutex` in `RuntimeInner` — the planned "HashMap in SharedState" predated
the v0.5 decomposition. Erlang semantics: one name per pid, one pid per
name, NameTaken only against a *live* holder. Cleanup is lazy: liveness
rides the generation-checked slot word (pids are never reused), dead
bindings behave as absent and are pruned on contact — no `finalize_actor`
hook, no Slot field, no reset-in-three-places exposure.

`send_by_name` was dropped from scope deliberately: smarm has no per-pid
mailbox, so there is nothing generic to send *to*. The registry yields a
`Pid` for monitor/link/request_stop and for routing user-owned channels.

### 3. gen_server: call timeout  ✅ (`134ff52`, `90b7040`)
Landed as two primitives, *without* the sketched monitor machinery:

- `Receiver::recv_timeout(Duration)` — the per-`recv` deadline, on the
  existing `timer::Reason::WaitTimeout`/`TimerTarget` machinery exactly as
  `Mutex::lock_timeout` uses it (per-wait seq; stale heap entries no-op on
  seq mismatch; message-first race resolution; `Disconnected` distinct from
  `Timeout`).
- `ServerRef::call_timeout` = send + `recv_timeout` on the reply channel.
  Server death is already observable there (reply sender drops as the server
  unwinds), so monitor + demonitor-on-timeout had nothing left to guarantee:
  no registration exists, so nothing can leak. Timed-out requests are still
  handled; the late reply's send fails harmlessly (Erlang semantics).

(`handle_info` still deferred — needs cross-channel mailbox merge; see Later.)

---

## v0.7 — select, on epoch-stamped consuming wakes  ✅ DONE

Goal: cross-channel wait (`select`) without surrendering the precise-wake
invariant. Decision record: the alternative ("wakes are hints", loop-and-
recheck at every park site) was evaluated and rejected — it distributes the
soundness obligation across every present and future park site and turns
every stale wake into a wasted reschedule. Instead the slot protocol itself
gains *wait identity*: a park-epoch packed into the spare bits of the slot
word, so that a stale unpark dies at one failed CAS in the waker.

Protocol (the consuming-wake rules):
- Word becomes `(gen << 32) | (epoch << 8) | state` — 24-bit epoch in the
  previously unused bits; every transition still one CAS, gen check still
  atomic with it.
- `begin_wait` (actor-side, before registering with any waker) bumps the
  epoch and returns it; registrations carry `(pid, epoch)`.
- Every successful wake transition **consumes** the epoch (bumps it):
  `Parked(e) → Queued(e+1)`, `Running(e) → RunningNotified(e+1)`. A second
  unpark stamped with the same epoch can never land — at most one wake per
  wait, by construction. This is what makes select's *loser* arms harmless
  without a cancellation pass: their stale registrations fire one futile CAS
  on next send and self-clean.
- `unpark_at(pid, epoch)` = epoch-matched wake. Bare `unpark(pid)` remains as
  the wildcard and is reserved for **terminal** wakes (`request_stop`), which
  never return control to the code that parked.
- One-shot park sites (`Mutex::lock_timeout`, `sleep`, `block_on_io`,
  `wait_fd`) stay exactly as written: precise wakes are preserved, so a wake
  still *means* something.

### 1. Slot-word epoch + consuming unparks  ✅ (`4913835`)
`slot_state.rs`: repack, `begin_wait`, epoch-matched/consuming `unpark`,
epoch-preserving claim/yield/park transitions. Loom: existing theorems
re-proved on the new word + a new stale-epoch theorem (an unpark stamped with
a consumed epoch can never enqueue *or* notify). Plumbing only — no caller
stamps yet; behaviour identical.

### 2. Stamp every registration-based wake; retire per-primitive wait seqs  ✅ (`400854a`)
The epoch IS the wait identity, so `channel::Inner.{cur_wait,next_wait_seq}`
and `mutex`'s per-wait `seq` are subsumed: registrations become
`(pid, epoch)`, `TimerTarget::on_timeout(pid, epoch)`, wakers call
`unpark_at`. Sleep timers, io completions (`Blocking` + `FdReady`) and join
waiters stamped likewise. End-state invariant, auditable in one sentence:
**the only wildcard wake is `request_stop`.**

### 3. `select` — ready-index wait over multiple receivers  ✅ (`00128f3`)
`select(&[&dyn Selectable]) -> usize`: register self in each arm's
`parked_receiver` (sequentially — the park protocol makes cross-arm atomicity
unnecessary), park once, wake, scan, return the first ready index; caller
`try_recv`s it. Sound as ready-index *because channels are single-receiver*:
nobody can steal between ready and recv. A closed arm counts as ready (its
`try_recv` reports the closure). Arms are scanned in order — priority order,
documented, like BEAM receive clauses; no fairness promise. No cancel pass
(see protocol above). Drop-guard deregistration on the unwind path only, to
keep tests' single-parked-receiver asserts honest.

### 4. `select_timeout`  ✅ (`a0a93b6`)
A `WaitTimeout` timer stamped with the select's epoch, whose target is a
stateless `unpark_at`. The woken selector classifies the wake from state
alone: some arm ready → that arm; none ready → with precise wakes the timer
is the only remaining stamped waker → `None`. No flag, no shared waiter
struct, no registration to leak. The timer entry expires as a stale-epoch
no-op when an arm wins, per the no-cancellation convention in `timer.rs`.

### 5. Docs + integration proof  ✅ (this commit)
README module table, gotchas rewrite (the "No `select`" bullet becomes the
consuming-wake invariant), and an integration test of the motivating
pattern: a server selecting over its inbox and a monitor `Down` channel
(`tests/select.rs::motivating_pattern_inbox_plus_monitor_down`).

Unblocked but deliberately out of scope: `gen_server::handle_info` (now an
API-design question, not a runtime question — see Later).

### Deviations from the plan
- **The no-park exit needed a protocol piece the plan missed.** A `select`
  returning an arm ready at *registration* time leaves earlier arms holding
  live-epoch registrations — a sender could still set RunningNotified and
  fault the actor's next one-shot park. Landed as `scheduler::retire_wait`
  (bump the epoch, eat a landed notification via the new
  `StateWord::clear_notify`, re-observe the stop flag — in that order) plus
  the loom theorem `retire_eats_late_arm_notification`. The parked path
  needed nothing: the winning wake's consume already retires the epoch.
- **No Drop guard.** The planned unwind-path deregistration was superseded:
  the single-receiver debug_asserts relax to "none, or my own pid", so a
  leftover own-registration (consumed epoch — inert by theorem) is simply
  overwritten by the receiver's next wait on that channel. Less machinery,
  and the asserts still catch the real misuse (two actors on one channel).
- **A closed arm is ready *forever*** — callers must drop it from the arm
  set once observed, or priority order starves every higher-indexed arm
  (documented on `select`; found the hard way by a livelocking test).

---

## v0.8 — gen_server: handle_info / handle_down + io fd hygiene

Goal: spend the `select` primitive on the server loop — out-of-band messages
(`handle_info`) and monitor `Down`s (`handle_down`) dispatched alongside
calls and casts — and close the v0.2 fd-hygiene hole in `io.rs`.

Design decisions (settled up front):
- **Info channels are static.** `type Info: Send + 'static` on the trait; a
  `Vec<Receiver<G::Info>>` handed over at start. Users enum their own Info.
  Dynamic info subscription is deliberately out: revisit against a consumer.
- **Down forwarding is dynamic, because monitors are.** You monitor a worker
  you just spawned inside a handler, so down arms cannot ride the static
  list. Mechanism: the loop owns a control arm (`Receiver<Monitor>`);
  `init` grows a `&ServerCtx` parameter (breaking, no-op default body) whose
  clonable `Watcher` the state stores and calls `watch(monitor)` from any
  handler thereafter. Handler signatures otherwise untouched.
- **Priority order: downs → control → infos → inbox.** System messages
  preempt; a hot inbox cannot starve a death notice. Within each class,
  declaration order (select's documented priority semantics).
- **Closed arms drop silently** (the closed-arm-is-ready-forever gotcha).
  Closed inbox still means graceful shutdown, unchanged.

### 1. handle_info — static info arms in the server loop
`type Info` + `handle_info` (no-op default) on `GenServer`; `ServerBuilder`
(`Server::new(state).with_info(rx).under(sup).start()`) so start variants
don't multiply; `start`/`start_under` stay as wrappers. Loop becomes
`select` over `[infos…, inbox]`, dispatching by arm index.

### 2. handle_down — ServerCtx::watcher + control arm
`handle_down` (no-op default), `ServerCtx`/`Watcher`, the control arm, and
the loop's live `Vec<Monitor>`. A `Down` retires its arm (monitors are
one-shot). Integration test: the motivating worker-pool pattern — a server
that spawns workers in a handler, watches them, and reacts to their deaths.

### 3. io fd hygiene — DEL on the unwind path
A stopped actor unwinding out of `wait_fd`'s park leaks its `waiters` entry
and the kernel registration; the stale entry then fails every future
`wait_*` on that fd with `AlreadyExists` (the defensive bare-DEL in
`epoll_register` sits behind the `contains_key` check, so it never runs).
Fix where the invariant breaks: a drop guard in `wait_fd`, armed after a
successful register, disarmed on normal wake; on unwind it removes the
waiters entry iff it is still `(me, epoch)` and only then DELs the fd
(an entry consumed by a racing `FdReady` means the fd may already be
re-registered by someone else — leave it alone).

---

## Later

### Tunable scheduler idle policy (RFC 004)
Today idle schedulers fall back to `thread::sleep(100µs)`, putting ~100µs
worst-case latency on every cross-thread actor handoff. RFC 004 specifies a
bounded spinning-worker policy (Go/Tokio model): idle schedulers spin on a
cheap atomic for a configurable budget before parking, so a burst of work is
picked up in cache-coherency time rather than after a kernel reschedule.

Two new `Config` knobs: `spin_budget_cycles` (RDTSC cycles to spin; `0`
recovers today's behaviour) and `max_spinners` (cap on concurrent spinners;
default `N/2`). Single new atomic `n_spinning`. No change to the run-queue
data structure — lands on the current `Mutex<VecDeque>` and is independent of
any future queue topology work.

Full spec: `rfc_004-tunable-scheduler-idle-policy.md` (artefacts.kalsbeek.dev).

### Unbounded / configurable-bounded actor count
Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384).
Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling segment
sizes, append-only) once the cap is actually hit. Do not let it calcify.



### arm-port validation & merge
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
hardware. Build + run full test suite on an aarch64 device; check
`chained_spawn` / `yield_many` bench medians; merge and update README.

---

## Invariants & gotchas (respect these across all cycles)

- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
  `with_shared`. Never send on a channel while holding the shared lock. Pattern:
  `mem::take` data under the lock, send after releasing. See `finalize_actor`.
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
  Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs →
  unpark joiners → reclaim slot if `outstanding_handles==0`. Death notifications
  always precede reclamation.
- **Slot lifecycle reset in THREE places:** `Slot::vacant()`, `reclaim_slot()`
  (runtime.rs), slot-init block in `spawn_under` (scheduler.rs). Any new `Slot`
  field must be reset in all three.
- **Pid = (index, generation).** Stale handles caught by generation mismatch in
  `slot()/slot_mut()`. The monitor `NoProc` path relies on this.
- **The only wildcard wake is `request_stop`, and it is terminal.** Every
  registration-based waker (channel sends, mutex grants, wait-timers, io
  completions, joiner wakes, `select` arms) carries the wait's park-epoch
  and wakes through `unpark_at`; every successful wake consumes the epoch.
  Wakes are therefore *meaningful*: one-shot park sites interpret them
  without loops, and `select` needs no cancellation pass. When adding a new
  waker, decide which form it is — if its registration handle can outlive
  the wait it was created for, it MUST be epoch-stamped; a wait that can
  exit without parking MUST `retire_wait` first (see slot_state.rs).
- **`select` exists; a unified per-process mailbox still does not.** The
  supervisor keeps its single `supervisor_channel` funnel; `recv_match`
  stays per-channel. `select` composes channels at the wait, not into one
  queue — the mailbox-merge decision remains open (see Later:
  gen_server handle_info).
- **Cooperative-only.** Preemption and cancellation both depend on the actor
  reaching `check!()`/yield/alloc/blocking points.
- **Lock order is Leaf → Channel, one of each at most** (debug-asserted in
  `raw_mutex.rs`). Leaf = cold locks / free list / stack pool / registry,
  mutual leaves. A channel lock may be taken under a Leaf (finalize/monitor
  clone senders living in slots); nothing may be locked under a channel lock.
- **Queue ops require preemption disabled.** A producer suspended mid-publish
  stalls every consumer — livelock. `with_runtime`, `with_shared`, and
  `RawMutex` guards all disable preemption for their span.
- **`run()` is single-thread** (`Config::exact(1)`); tests rely on deterministic
  single-thread ordering. Multi-thread via `runtime::init(Config…)`.