smarm/ROADMAP.md

# smarm — Roadmap

## Shipped (compacted — full cycle plans and deviation records live in git history)

Cycles before v0.8 (v0.4 actor primitives, v0.5 runtime decomposition &
pluggable run queue, v0.6 actor ergonomics, v0.7 select on epoch-stamped
consuming wakes): see `git log ROADMAP.md`.

### v0.8 — gen_server: handle_info / handle_down + io fd hygiene  ✅
Spent `select` on the server loop: static info arms (`type Info`,
`ServerBuilder::with_info`) and dynamic monitor forwarding
(`ServerCtx`/`Watcher` + a control arm), priority downs → control → infos →
inbox. Closed the v0.2 fd hole: a drop guard in `wait_fd` DELs the kernel
registration on unwind (the leak was worse than documented — a stale waiters
entry permanently poisoned the fd). Deviation: the plain-inbox fast path
narrowed; servers holding a `Watcher` select forever.
Commits `e5d1b3b`, `24b95c9`, `f6969e5`.

### v0.9 — Wake-path latency  ✅
Attacked per-wake latency with the RFC 005 **wake slot**: a per-scheduler,
thread-local, capacity-one wake cache checked before the shared queue, pushed
only from actor context, slot-then-shared pop with the waker's residual slice
as the starvation bound (`slot_hits`/`slot_displacements`). Benched via the
slot on/off dimension of `rq_runtime` — ping-pong-pairs (win), yield-storm
(regression guard), spawn-storm (neutrality); results annotated in RFC 005.
The RFC 004 spinning-workers experiment, originally scoped here, was evaluated
and **excised** (not worth the code cost; preserved on branch
`rfc-004-spinning`). Also a false-sharing fix (`align(64)` on `SchedulerStats`)
and a termination wake for idle siblings.
Commits `2708042`, `37d9319`, `eddf3fe`.

---

## Decision record — queue topology  🔒 CLOSED (2026-06-10)

The run-queue shootout (harness `6d9f369`, 24-core sweep 1–24 schedulers,
report `bench_report_rq_shootout.html`) landed in **RFC 005's World 3**: the
three queue variants are within 10–15% of each other in `rq_runtime` at every
scheduler count ≥ 4, on all three workloads. 
Consequences:
- **`rq-mutex` stays the default** — simplest correct, no capacity
  constraints, locking model already integrated.
- **Feature plumbing stays as is.** All three variants keep compiling in
  every build; `rq-mpmc`/`rq-striped` remain selectable for benching.
- **Reopening is benchmark-driven only.** The report documents the
  conditional upgrade paths if a future workload qualifies: mpmc for
  message-passing-dominant loads at N ≤ 8; striped for high-contention balanced push/pop at N ≥ 16. Neither is a scheduler workload as measured.
- **Effort redirects to the wake path**: RFC 005 (billed as a latency patch,
  per its own World 3 framing), RFC 004, and eventually per-switch cost.

---

## Typed addressable mailboxes  ✅ SHIPPED (RFC 013 + RFC 014)

The unblocker several later items quietly assumed: a `Pid` is now messageable.
RFC 013 reworked `registry.rs` from a name↔pid *bimap* into a name→**live
mailbox** directory off the cold leaf; RFC 014 layered the typed addressing and
producers on top. Two addressing modes — `Pid<A>` (direct, identity-bound) and
`Name<M>` (durable, re-resolving, location-transparent) — compile-time typing
preserved via phantom tokens over *contained* `Box<dyn Any>` erasure (the
global-enum alternative was rejected: it breaks library-extensibility for
out-of-crate actors). Channel store keyed by message `TypeId` in every path.

Delivered surface:
- **Sends:** `send_to` (`Pid<A>`), `send` (`Name<M>`), `send_dyn` (bare-pid
  escape hatch, names the message type) — earlier 014 phases.
- **Producers & discovery** (final phase, `a866e34`): `spawn_addr` (typed-path
  producer; parent-side inbox publish so an immediate `send_to` resolves, no
  race on the body); `lookup_as` / `pick_as` / `members_as` (unchecked-but-sound
  re-type of an erased pid — a wrong `A` degrades to `NoChannel`, never
  misdelivery); `dispatch` (pick-a-live-member-and-send, `SendError::NoMember`
  on empty pool).
- **By-name gen_servers** (final phase): `ServerName<G>` over the existing
  typed-channel store (keyed by `TypeId::of::<Envelope<G>>()`, so `Envelope`
  stays private and no separate directory is needed); type-state
  `NamedServerBuilder<G>` (fallible `start`, `NameTaken`) leaving the infallible
  `ServerBuilder::start` untouched; free `call` / `cast` / `whereis_server`;
  `ServerRef::shutdown` + free `shutdown` as the sys-style synchronous stop.
- **Root-exit teardown** (final phase): the run's initial actor is the root;
  when it exits, the scheduler's idle verdict stops the parked-forever remainder
  (deferred past the queue drain, so actors with in-flight work finish rather
  than unwinding on the stop). Closes the "app actor blocks AllDone" stall — see
  Look into, below.

Extends — does not retire — the "select exists; a unified per-process mailbox
still does not" invariant: 014 adds addressable *delivery*, not a unified inbox;
multi-port stays `select` composition over named channels. Examples:
`examples/{typed_actor,named_genserver,worker_pool}.rs`. Earlier-phase commits
and the full 013/014 history in git.

---

## gen_server time-related patterns  ✅ SHIPPED (RFC 015)
*(layer 2; seven commits `47d75d1`→`f454a91`, each a reviewable chunk. Builds on
the `send_after` substrate below.)*

The OTP time vocabulary against the v0.8 server loop, with **no handler-signature
change** — the capability is a handle stashed on `self` (the `Watcher` pattern),
not a return-directive or a `&ctx` threaded through handlers. New trait surface:
`type Timer` (server's own scheduled payload, `()` if unused, kept distinct from
the external `type Info`), `handle_timer`, `handle_idle` (both no-op defaults).

- **One-shot / debounce / retry-backoff** — `ctx.timer()` hands out a clonable
  `TimerHandle`; `arm_after(d, msg)` arms, `cancel(id)` carries the substrate's
  race bool. Debounce/backoff are just arm-and-`cancel` against the latest event
  (no special mechanism).
- **Periodic tick / heartbeat** — `tick_every(d, msg)`: loop-managed sugar over
  the one-shot substrate (re-arm at `now + d`), one stable id, `cancel` stops the
  re-arm *and* the pending instance. Requires `Timer: Clone` (method-level bound
  only); the loop re-delivers via a stored factory, so the bound never leaks onto
  `type Timer` or the loop.
- **Idle / receive timeout** — *not* a channel: it is the timeout on the loop's
  `select_timeout` / `recv_timeout`. `ctx.idle_after(d)` (set once in `init`)
  fixes the window; reset on any dispatched message; `None` from the wait ⇒
  `handle_idle`; re-arms steady. No generation-tag race — there is no token.

Mechanics: control (monitor intake) + armed timers fold into one loop-internal
`Sys` channel selected above the inbox, so **armed timers outrank infos** (a
heartbeat can't be starved). One substrate addition — `send_after_to` (a
channel-targeting sibling of `send_after`, lands the fire on the loop's own arm).
Exit is leak-free: the drop guard (same one that runs `terminate`) drains and
cancels every live timer id, then `debug_assert!`s none survive. `call_timeout`
is unchanged — it's a *client-side* call deadline, disambiguated in docs from the
server-side idle timeout (same word, two axes). `gen_statem` state timeouts
(deferred, Low) will reuse the loop-owned idle deadline.

---

## Process groups — the primitive pubsub & channels should have sat on  ✅ SHIPPED (RFC 012)
*(`src/pg.rs`, four commits `b78311b`→`56f2fc5`; see HANDOFF + git history. Context retained below.)*

Context: urus is a webserver written on top of smarm to provide a testing target.
A named pid→multiset map with monitor-backed removal: `registry.rs` generalised
from name↔pid *bimap* to name→*multiset*, the death hook reused verbatim. Local
first. The point is that urus pubsub collapses into a pg consumer (`subscribe` =
join, `broadcast` = send-to-members) instead of being a bespoke mechanism, and the
same group set reads two ways — fan-out (all members) vs discovery/pool (one
member), with different netsplit consequences. Urus shipped pubsub/channels predate this
and want reframing on top of it. Foundational, so early in the post-v0.9 stack.
Needs an RFC.

---

## Later
### Highest priority
#### Per-switch cost (context shims, epoch protocol)
The shootout's residual: per-wake latency is 0.16–0.18 µs at N=1 and
0.8–1.2 µs at N=8+, dominated by the context-switch shims and the epoch
protocol, not the queue. On current evidence this is the larger constant —
"the whole game" alongside the v0.9 work — but there is no spec yet. Needs a
profiling spike (where do the cycles actually go per park/unpark round-trip)
and then an RFC before it can be scheduled.

#### send_after / cancel_timer
✅ SHIPPED (`61520bf`) — message-delivery timer on the `timer.rs` min-heap:
deliver a value to an address (`Pid<A>` via `send_to`, `Name<M>` via `send`),
resolved *on fire* so a dead target / restarted name is observed at fire time;
failed resolve dropped (Erlang `erlang:send_after`). `Reason::Send { fire }`
carries delivery type-erased; cancellation is an `armed` set keyed on entry
`seq` (only `Send` uses it — `Sleep`/`WaitTimeout` stay inert-stale), exposed as
an opaque `TimerId`; `cancel` is unscoped and returns the race signal.
`peek_deadline` relaxed to "≤ true next deadline" for a future timing wheel. The
gen_server time layer (RFC 015, shipped above) lands on this, adding only the
channel-targeting `send_after_to` sibling.

#### Introspection — process_info / get_state / tree dump
✅ SHIPPED (RFC 016) — runtime introspection & observability, superseding the
RFC 000 / 006 / 009 sketches. The mechanism is an internal synchronous read,
not a C ABI: `snapshot()` / `actor_info(pid)` return owned data (pid, names,
fine scheduling state, parent edge, trap, monitor/link/joiner counts, mailbox
depth) carrying `SNAPSHOT_FORMAT_VERSION` (Chunk 1, ps-semantics tearing);
`tree()` folds that into a parentage forest with orphan re-rooting (Chunk 3).
Per-actor counters — timeslice overruns, messages-received, and a feature-gated
approximate time budget — ride hot `AtomicU64` slot fields (Chunk 2). The live
`observer` gen_server is the read transport over that primitive, behind the
off-by-default `observer` feature (Chunk 4); it is the read half of the future
RFC 003 control plane. Wedged-runtime dumps stay gdb's job, and park-reason
detail / C ABI are explicit non-goals.

#### Worker pool behaviour
Supervised, interchangeable workers with restart semantics over a shared inbox
(poolboy / NimblePool shape) — distinct from connection pools (bb8/deadpool), which
pool *resources*, not *supervised processes*. Sits on `supervisor.rs` +
`gen_server.rs`. Needs an RFC.

### Medium Priority

#### Demand-driven pipelines — GenStage / Broadway shape
Supervised producer/consumer stages where consumers signal demand upstream, with
batching, ack, partitioning. The clearest thing hex has and crates.io lacks (stream
combinators and bounded channels are not a supervised demand-contract stage graph),
and the natural fit for ingestion-shaped workloads. Builds on channels + gen_server
+ supervisor. Needs an RFC.
#### Unwakeable idle sleep when io is absent (terminal-wake residual)
The `(Some(deadline), None)` idle branch — timers pending, io subsystem never
initialized — blocks in `thread::sleep` with no wake mechanism at all. The
terminal wake (writes the wake pipe at AllDone) cannot reach it: no io, no
pipe. Same stall as the fixed bug, in any no-io runtime: a sibling that
blocked on an orphaned deadline sleeps it out in full after everything else
finished. Candidates, mutually exclusive: (a) clamp the sleep (cheap, but
turns idle into periodic wakeups), or (b) park the branch on a condvar/futex
the AllDone path signals — and at that point consider making the condvar the
idle primitive for the no-io runtime generally (a cross-thread unpark could
signal it too, see below). Decide before any no-io deployment.
#### Cross-thread unpark
`RuntimeInner::enqueue` does not wake idle sibling schedulers — only io
completions write the wake pipe. Mid-flight this is masked (the enqueuing
thread is awake and eats the work itself), but it costs parallelism: work
enqueued by a busy thread waits until the sibling's idle poll times out. Needs bench evidence (does the shared-queue handoff latency actually show up?) before a mechanism is picked.
#### Unbounded / configurable-bounded actor count
Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384).
Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling segment
sizes, append-only) once the cap is actually hit. Do not let it calcify.
#### arm-port validation & merge
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
hardware. Build + run full test suite on an aarch64 device; check
`chained_spawn` / `yield_many` bench medians; merge and update README.

### Low priority
#### gen_statem — postponement + state timeouts only
A thin layer over gen_server, not a new behaviour. The (state, event) dispatch
matrix is free from the type system and not worth porting. The two mechanisms that
are: event **postponement** (defer events in the wrong state, replay on transition
— selective receive, codified) and **state timeouts** (auto-cancel on state
change). Device-connection FSMs are the canonical use. Wants send_after underneath.
Needs an RFC. 

#### Clustering — distribution epic
Sequenced deliberately after v0.9 and the per-switch-cost spike. A fat stack of
RFCs, not one. Spine settled in discussion; decisions still open:
- **Explicit remote boundary, never transparency.** Serialization colours *edges*
  (channel types), not functions — local edges stay zero-copy `Send`, only remote
  edges take a `RemoteRef<T: Serialize + DeserializeOwned>`. No hidden latency when
  a peer migrates; the refactor is visible by construction.
- **One binary, role as runtime config** (`ROLE=… REGION=… SEEDS=…`); a build-hash
  handshake enforces same-binary type identity and sidesteps cross-version type
  agreement. Roles select which supervision subtree mounts.
- **Distributed pg falls out of local pg + a membership/gossip layer**, and
  distributed pubsub falls out of that for free; per-member metadata (region, load)
  enables fly-style nearest-member routing.
- **Migratable gen_servers** as a sub-layer: only behaviours migrate (a raw actor's
  stack is opaque; a gen_server *between callbacks* is just its `State`), gated by
  `Serialize` bounds + an `on_arrive` reacquire hook, addressed by name not pid. The
  BEAM can't do this — leaning on the behaviour layer is what buys it. Requires `State` to be serializable, so should probaby spec a `trait MigratableGenServer: GenServer where Self::State: Migratable` , or something to that extent, so we can lean on the type system to make sure we don't accidentally make state that cannot be serialised. (The "addressed by name not pid" half is RFC 013's `Name<M>` durable address — its local form is the foundation this remote layer extends.)
- **CRDT presence** is the high-value, genuinely-hard layer above distributed pg,
  kept *out* of the pg primitive (the pg2 strong-consistency lesson). Furthest out. Can maybe defer to rust ecosystem


--- 

## Look into 

### app actors block AllDone; no external stop path  — ADDRESSED (RFC 014 root-exit teardown)
Agent working on urus (see same git server as smarm) reported a lazily spawned actor never returning, blocking program shutdown. Maybe we should do something about it. Agent worked around it by giving the actor an atomic bool to spin on. See urus example crud for exact impl.

**Update (RFC 014):** root-exit teardown stops the parked-forever remainder when
the root actor exits, so a lazily spawned daemon no longer wedges shutdown on
`live_actors > 0`; `ServerRef::shutdown` (+ free `shutdown`) is the explicit stop
path the atomic-bool workaround stood in for. Re-check the urus crud repro to
confirm the workaround can be retired (the teardown is cooperative — an actor in
a tight loop with no observation point still can't be stopped).

---

## Invariants & gotchas (respect these across all cycles)

- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
  `with_shared`. Never send on a channel while holding the shared lock. Pattern:
  `mem::take` data under the lock, send after releasing. See `finalize_actor`.
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
  Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs →
  unpark joiners → reclaim slot if `outstanding_handles==0`. Death notifications
  always precede reclamation.
- **Slot lifecycle reset in THREE places:** `Slot::vacant()`, `reclaim_slot()`
  (runtime.rs), slot-init block in `spawn_under` (scheduler.rs). Any new `Slot`
  field must be reset in all three.
- **Pid = (index, generation).** Stale handles caught by generation mismatch in
  `slot()/slot_mut()`. The monitor `NoProc` path relies on this.
- **The only wildcard wake is `request_stop`, and it is terminal.** Every
  registration-based waker (channel sends, mutex grants, wait-timers, io
  completions, joiner wakes, `select` arms) carries the wait's park-epoch
  and wakes through `unpark_at`; every successful wake consumes the epoch.
  Wakes are therefore *meaningful*: one-shot park sites interpret them
  without loops, and `select` needs no cancellation pass. When adding a new
  waker, decide which form it is — if its registration handle can outlive
  the wait it was created for, it MUST be epoch-stamped; a wait that can
  exit without parking MUST `retire_wait` first (see slot_state.rs).
- **`select` exists; a unified per-process mailbox still does not.** The
  supervisor keeps its single `supervisor_channel` funnel; `recv_match`
  stays per-channel. `select` composes channels at the wait, not into one
  queue — gen_server's `handle_info`/`handle_down` (v0.8) are built on
  exactly that composition, with documented arm priority (downs → control
  → infos → inbox) instead of mailbox FIFO. A hot higher-priority arm
  starves lower ones by design; that's the contract.
- **Cooperative-only.** Preemption and cancellation both depend on the actor
  reaching `check!()`/yield/alloc/blocking points.
- **Lock order is Leaf → Channel, one of each at most** (debug-asserted in
  `raw_mutex.rs`). Leaf = cold locks / free list / stack pool / registry,
  mutual leaves. A channel lock may be taken under a Leaf (finalize/monitor
  clone senders living in slots); nothing may be locked under a channel lock.
- **Queue ops require preemption disabled.** A producer suspended mid-publish
  stalls every consumer — livelock. `with_runtime`, `with_shared`, and
  `RawMutex` guards all disable preemption for their span.
- **`run()` is single-thread** (`Config::exact(1)`); tests rely on deterministic
  single-thread ordering. Multi-thread via `runtime::init(Config…)`.