docs(roadmap): restructure around the rq shootout verdict
Shootout (6d9f369 harness, 24-core sweep) landed in RFC 005's World 3:
queue variants indistinguishable in rq_runtime; per-wake latency is the
ceiling. Record the queue-topology closure (rq-mutex frozen as default,
benchmark-driven reopening only), define v0.9 as the wake-path latency
cycle (RFC 005 slot -> bench -> RFC 004 spinning workers -> bench +
interaction pass), and add per-switch cost to Later pending a profiling
spike + RFC.
Completed cycles compacted to the two trailing minors (v0.7, v0.8);
earlier history lives in git.
This commit is contained in:
Claude
+98
-298
@@ -1,331 +1,131 @@
|
|||||||
# smarm — Roadmap
|
# smarm — Roadmap
|
||||||
|
|
||||||
## v0.4 — Actor primitives ✅ DONE
|
## Shipped (compacted — full cycle plans and deviation records live in git history)
|
||||||
|
|
||||||
### 1. Cooperative cancellation ✅ (`a8ddb4a`)
|
Cycles before v0.7 (v0.4 actor primitives, v0.5 runtime decomposition &
|
||||||
Stop flag on `Actor` behind `Arc<AtomicBool>`. Observation points:
|
pluggable run queue, v0.6 actor ergonomics): see `git log ROADMAP.md`.
|
||||||
`maybe_preempt`/`check!()` + the wakeup side of `park_current`/`yield_now`.
|
|
||||||
Sentinel = `StopSentinel` (zero-size), caught in trampoline → `Outcome::Stopped`.
|
|
||||||
`join()` on a stopped actor returns `Ok(())`.
|
|
||||||
|
|
||||||
Known gap (by design): a tight no-alloc loop without `check!()` cannot be
|
### v0.7 — select, on epoch-stamped consuming wakes ✅
|
||||||
stopped, same as preemption.
|
A 24-bit park-epoch packed into the slot word gives every wait an identity:
|
||||||
|
registrations carry `(pid, epoch)`, every successful wake **consumes** the
|
||||||
|
epoch, stale wakes die at one failed CAS. Subsumed the per-primitive wait
|
||||||
|
seqs (channel, mutex, timer); the only wildcard wake left is `request_stop`
|
||||||
|
(terminal). On top: `select`/`select_timeout` — ready-index wait over many
|
||||||
|
receivers, priority order, no cancellation pass. Loom theorems re-proved on
|
||||||
|
the new word. Notable deviations: `retire_wait` for the no-park exit path
|
||||||
|
(plan missed it); the Drop-guard deregistration was superseded by relaxed
|
||||||
|
single-receiver asserts; a closed arm is ready *forever* (documented gotcha).
|
||||||
|
Commits `4913835`…`00128f3`, `a0a93b6`.
|
||||||
|
|
||||||
### 2. one_for_all / rest_for_one + ordered shutdown ✅ (`351dc9c`)
|
### v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅
|
||||||
`Strategy::{OneForOne,OneForAll,RestForOne}` via `.strategy()`. Triggering
|
Spent `select` on the server loop: static info arms (`type Info`,
|
||||||
child's `Restart` policy decides whether anything restarts; strategy decides
|
`ServerBuilder::with_info`) and dynamic monitor forwarding
|
||||||
which siblings are cycled. Survivors stopped in reverse start order, awaited on
|
(`ServerCtx`/`Watcher` + a control arm), priority downs → control → infos →
|
||||||
the existing `supervisor_channel` funnel, restarted in start order.
|
inbox. Closed the v0.2 fd hole: a drop guard in `wait_fd` DELs the kernel
|
||||||
|
registration on unwind (the leak was worse than documented — a stale waiters
|
||||||
`Signal::Stopped(pid)` + `DownReason::Stopped` added (distinct from Exit).
|
entry permanently poisoned the fd). Deviation: the plain-inbox fast path
|
||||||
|
narrowed; servers holding a `Watcher` select forever.
|
||||||
Also fixed a latent bug (`e80334b`): orphaned timers were gating shutdown.
|
Commits `e5d1b3b`, `24b95c9`, `f6969e5`.
|
||||||
`live == 0` is now the shutdown condition; timer heap cleared on exit.
|
|
||||||
|
|
||||||
### 3. Links + trap_exit ✅ (`6581484`)
|
|
||||||
`Slot.links: Vec<Pid>` (bidirectional). On finalize: abnormal death →
|
|
||||||
`request_stop(peer)` unless peer traps, in which case deliver `ExitSignal`
|
|
||||||
message. Normal exit does not propagate. `trap_exit()` returns a dedicated
|
|
||||||
`Receiver<ExitSignal>` (distinct from monitor `Down` channel).
|
|
||||||
|
|
||||||
### 4. Selective receive ✅ (`03f3875`)
|
|
||||||
`Receiver::recv_match(pred)` + `try_recv_match`. Scans `VecDeque` front-to-back,
|
|
||||||
removes+returns first match, parks and re-scans on new arrivals. `pred` is
|
|
||||||
`Fn(&T) -> bool` (not `FnMut`; runs under the channel lock — keep it cheap).
|
|
||||||
|
|
||||||
### 5. Grab-bag ✅
|
|
||||||
- **demonitor** (`5181037`): `monitor()` → `Monitor { id, target, rx }`.
|
|
||||||
`demonitor(&m) -> Option<MonitorId>`. Dropped `Sender` released post-lock
|
|
||||||
(non-reentrant shared mutex discipline).
|
|
||||||
- **gen_server** (`55221e9`): `GenServer` trait, `call`/`cast` over one
|
|
||||||
`Envelope` inbox. `terminate` hook via drop guard. No `handle_info`, no call
|
|
||||||
timeout (deferred — needs cross-channel mailbox merge or per-recv deadline).
|
|
||||||
- **Named registry**: `register`/`whereis`/`send_by_name` — **not yet done**.
|
|
||||||
HashMap in `SharedState`. No dependencies; can land any time.
|
|
||||||
- **arm-port branch**: aarch64 context-switch backend extracted into
|
|
||||||
`src/arch/`. Untested on hardware. Merge only after on-device validation.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## v0.5 — Runtime decomposition & run-path scaling ✅ DONE
|
## Decision record — queue topology 🔒 CLOSED (2026-06-10)
|
||||||
|
|
||||||
Goal: dismantle the single `Mutex<SharedState>` along the run path so the
|
The run-queue shootout (harness `6d9f369`, 24-core sweep 1–24 schedulers,
|
||||||
runtime scales to dozens of cores, with crash isolation hardened so a torn
|
report `bench_report_rq_shootout.html`) landed in **RFC 005's World 3**: the
|
||||||
write or a stray cancellation can never poison shared runtime state.
|
three queue variants are within 10–15% of each other in `rq_runtime` at every
|
||||||
|
scheduler count ≥ 4, on all three workloads. Striped's real micro-bench
|
||||||
|
dominance (24.8 M ops/s at 24t, 4.3× over mutex) does not propagate to the
|
||||||
|
runtime because the queue is not the hot path in any measured workload. USL
|
||||||
|
fits put σ at 5.5–7.6 (super-serial) for ping-pong: the ceiling is per-wake
|
||||||
|
latency through the park/unpark protocol, not queue push/pop.
|
||||||
|
|
||||||
Guiding rules established this cycle:
|
Consequences:
|
||||||
- **Lock count that matters is hot-path locks.** Cold lifecycle paths may keep a
|
- **`rq-mutex` stays the default** — simplest correct, no capacity
|
||||||
small per-slot lock; the run path targets zero locks.
|
constraints, locking model already integrated with the preemption-disabled
|
||||||
- **Lock ordering: io-before-shared, slot locks are leaves** (never hold two
|
queue-op invariant.
|
||||||
slot locks at once).
|
- **Feature plumbing stays as is.** All three variants keep compiling in
|
||||||
- **No unwind inside a runtime critical section.** Stop sentinel only fires at
|
every build; `rq-mpmc`/`rq-striped` remain selectable for benching.
|
||||||
lock-free observation points.
|
- **Reopening is benchmark-driven only.** The report documents the
|
||||||
- **Never hold a thread-local guard across a possible switch point.** An actor
|
conditional upgrade paths if a future workload qualifies: mpmc for
|
||||||
can be preempted at any allocation and resume on a different OS thread.
|
message-passing-dominant loads at N ≤ 8 (1.3× lower σ, 0.16 µs ping-pong
|
||||||
`with_runtime`, `with_shared`, and every `RawMutex` guard disable preemption
|
at 1t); striped for high-contention balanced push/pop at N ≥ 16 — neither
|
||||||
for this reason.
|
is a scheduler workload as measured.
|
||||||
|
- **Effort redirects to the wake path**: RFC 005 (billed as a latency patch,
|
||||||
### Phase 1 — State decomposition ✅ (`3c7e26b`)
|
per its own World 3 framing), RFC 004, and eventually per-switch cost.
|
||||||
`next_monitor_id` → `AtomicU64`. `timers` → own `Mutex<Timers>`. `io` → own
|
|
||||||
`Mutex<Option<IoThread>>`. `pending_closures` → folded into `Slot::pending_closure`.
|
|
||||||
`check_cancelled()` gated behind `PREEMPTION_ENABLED` (poison fix).
|
|
||||||
|
|
||||||
`SharedState` now holds only: `slots`, `free_list`, `run_queue`, `root_pid`.
|
|
||||||
|
|
||||||
### Phase 2 — Slot table split ✅ (`5e0c9d4`)
|
|
||||||
Fixed slab `Box<[Slot]>`. Generation packed into state word: one
|
|
||||||
`AtomicU64 = (gen << 32) | state` — gen check is atomic with every transition.
|
|
||||||
Per-slot CAS state machine `Vacant/Queued/Running/RunningNotified/Parked/Done`.
|
|
||||||
Per-slot `RawMutex` for cold collections. Free list → `RawMutex<Vec<u32>>`.
|
|
||||||
`live_actors` atomic; termination = io_out==0 && queue empty && live==0.
|
|
||||||
|
|
||||||
### Phase 3 — Pluggable run queue ✅ (`1b3b618`)
|
|
||||||
Compile-time selected via feature flags. Variants:
|
|
||||||
- `rq-mutex` — `Mutex<VecDeque>`, baseline.
|
|
||||||
- `rq-mpmc` — hand-rolled Vyukov bounded MPMC ring.
|
|
||||||
- `rq-striped` — M Vyukov rings, fetch-add ticket distribution.
|
|
||||||
|
|
||||||
All variants compile in every build; feature only picks the alias. Bounded rings
|
|
||||||
are sound via slab cap + at-most-once-enqueued invariant.
|
|
||||||
|
|
||||||
Queue ops require preemption disabled (a producer suspended mid-publish stalls
|
|
||||||
every consumer behind its cell — livelock).
|
|
||||||
|
|
||||||
### Phase 4 — Bench harness ✅ (`6d9f369`)
|
|
||||||
`benches/rq_micro.rs`: raw structure sweep. `benches/rq_runtime.rs`:
|
|
||||||
yield-storm, ping-pong-pairs, spawn-storm. `scripts/bench_rq.sh`: rebuilds per
|
|
||||||
feature, aggregates to `bench_results/summary.csv`.
|
|
||||||
|
|
||||||
### Phase 5 — Safety hardening & model checking ✅ (`039703d`)
|
|
||||||
State machine extracted to `src/slot_state.rs` for loom checking. Loom on the
|
|
||||||
rings via `src/sync_shim.rs`. `NoPreempt`/no-unwind/TLS-guard audit complete.
|
|
||||||
Invariant-assertion sweep: every `StateWord` transition self-asserts its
|
|
||||||
precondition.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## v0.6 — Actor ergonomics ✅ DONE
|
## v0.9 — Wake-path latency
|
||||||
|
|
||||||
### 1. Channel mutex migration ✅ (`8ff6cf4`)
|
Goal: attack per-wake latency — the measured ceiling — with the two specified
|
||||||
`channel::Inner<T>` migrated from `std::sync::Mutex` to `RawMutex`, closing
|
mechanisms, each benched against a clean baseline before the next lands.
|
||||||
the cross-thread pthread-release hole (guard held with preemption enabled →
|
Order: RFC 005 first (the slot is measured against the just-benched idle
|
||||||
timeslice switch → unlock from a different OS thread, UB) and removing poison.
|
policy), then RFC 004 (measured against the slot-accepted baseline), then an
|
||||||
|
interaction pass.
|
||||||
|
|
||||||
The strict leaf rule did not survive contact: finalize clones the
|
Full specs: `rfc_005-wake-slot.md`, `rfc_004-tunable-scheduler-idle-policy.md`
|
||||||
supervisor/trap senders and `monitor()` clones the Down sender, all under a
|
(artefacts.kalsbeek.dev).
|
||||||
cold lock, and those senders *live in the slot* — the nesting is structural.
|
|
||||||
The check generalized to two classes: **Leaf → Channel**, at most one of
|
|
||||||
each, both violation directions debug-asserted at the acquisition site.
|
|
||||||
Channel critical sections call only the lock-free unpark protocol, so the
|
|
||||||
order is acyclic. `recv_match`'s user predicate still runs under the channel
|
|
||||||
lock — cheap/pure/channel-free, as documented.
|
|
||||||
|
|
||||||
### 2. Named registry ✅ (`2c7cf0b`)
|
### 1. Wake slot (RFC 005)
|
||||||
`register` / `whereis` / `unregister` + the inverse `name_of(pid)`: a
|
Per-scheduler, thread-local, capacity-one wake cache, checked before the
|
||||||
**bimap** (two HashMaps in exact inverse, debug-asserted) under one Leaf
|
shared queue. Runtime-selected via `Config { wake_slot: bool }`, default off
|
||||||
`RawMutex` in `RuntimeInner` — the planned "HashMap in SharedState" predated
|
until accepted (one binary benches both arms).
|
||||||
the v0.5 decomposition. Erlang semantics: one name per pid, one pid per
|
- **Push policy:** slot-eligible iff the wake originates from actor context
|
||||||
name, NameTaken only against a *live* holder. Cleanup is lazy: liveness
|
(`current_pid().is_some()`). Scheduler-context wakes (timer/IO drain) and
|
||||||
rides the generation-checked slot word (pids are never reused), dead
|
spawns always go shared.
|
||||||
bindings behave as absent and are pruned on contact — no `finalize_actor`
|
- **Displacement:** newest wake takes the slot, occupant pushed shared (Go
|
||||||
hook, no Slot field, no reset-in-three-places exposure.
|
semantics); the branch swap is the fallback if benches look pathological.
|
||||||
|
- **Pop order:** slot, then shared. Slot-popped actors inherit the waker's
|
||||||
|
remaining timeslice — a handoff chain is bounded by one slice, so the
|
||||||
|
shared queue is consulted at least once per slice per scheduler (the
|
||||||
|
starvation bound, zero new counters).
|
||||||
|
- **Invariant care:** the slot push replaces `run_queue.push` at the tail of
|
||||||
|
the `Parked → Queued` CAS; at-most-once-enqueued holds verbatim (pid in
|
||||||
|
slot ⊕ shared queue). Stall blast radius grows by exactly one actor.
|
||||||
|
- Counters: `slot_hits`, `slot_displacements`.
|
||||||
|
|
||||||
`send_by_name` was dropped from scope deliberately: smarm has no per-pid
|
### 2. Slot shootout
|
||||||
mailbox, so there is nothing generic to send *to*. The registry yields a
|
Extend `rq_runtime`/`bench_rq.sh` with the slot on/off dimension (cheap —
|
||||||
`Pid` for monitor/link/request_stop and for routing user-owned channels.
|
it's a Config knob, not a feature rebuild). Same sweep as the rq shootout.
|
||||||
|
- **ping-pong-pairs** — target metric; expect the win, single- and
|
||||||
|
multi-scheduler.
|
||||||
|
- **yield-storm** — regression guard; yields never touch the slot, any delta
|
||||||
|
is pop-path overhead.
|
||||||
|
- **spawn-storm** — neutrality check; spawns bypass the slot by policy.
|
||||||
|
Acceptance flips the default on and re-baselines for RFC 004.
|
||||||
|
|
||||||
### 3. gen_server: call timeout ✅ (`134ff52`, `90b7040`)
|
### 3. Spinning workers (RFC 004)
|
||||||
Landed as two primitives, *without* the sketched monitor machinery:
|
Bounded spin-before-park for idle schedulers, killing the ~100 µs
|
||||||
|
`thread::sleep` worst case on cross-thread handoff. Two Config knobs:
|
||||||
|
`spin_budget_cycles` (0 recovers today's behaviour) and `max_spinners`
|
||||||
|
(default N/2); one new atomic `n_spinning`. No run-queue changes — lands on
|
||||||
|
the frozen `rq-mutex` substrate, independent of the slot mechanically.
|
||||||
|
|
||||||
- `Receiver::recv_timeout(Duration)` — the per-`recv` deadline, on the
|
### 4. RFC 004 bench + interaction pass
|
||||||
existing `timer::Reason::WaitTimeout`/`TimerTarget` machinery exactly as
|
Re-run the sweep with spinning enabled, slot on and off. The known
|
||||||
`Mutex::lock_timeout` uses it (per-wait seq; stale heap entries no-op on
|
interaction: once idle pickup latency ≪ slice, the slot's latency trade
|
||||||
seq mismatch; message-first race resolution; `Disconnected` distinct from
|
(occupant waits out the waker's slice while other schedulers idle) may stop
|
||||||
`Timeout`).
|
paying — RFC 005's documented escape hatch is Go's "spinners exist → push
|
||||||
- `ServerRef::call_timeout` = send + `recv_timeout` on the reply channel.
|
shared instead" check. In scope only if the data demands it.
|
||||||
Server death is already observable there (reply sender drops as the server
|
|
||||||
unwinds), so monitor + demonitor-on-timeout had nothing left to guarantee:
|
|
||||||
no registration exists, so nothing can leak. Timed-out requests are still
|
|
||||||
handled; the late reply's send fails harmlessly (Erlang semantics).
|
|
||||||
|
|
||||||
(`handle_info` still deferred — needs cross-channel mailbox merge; see Later.)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## v0.7 — select, on epoch-stamped consuming wakes ✅ DONE
|
|
||||||
|
|
||||||
Goal: cross-channel wait (`select`) without surrendering the precise-wake
|
|
||||||
invariant. Decision record: the alternative ("wakes are hints", loop-and-
|
|
||||||
recheck at every park site) was evaluated and rejected — it distributes the
|
|
||||||
soundness obligation across every present and future park site and turns
|
|
||||||
every stale wake into a wasted reschedule. Instead the slot protocol itself
|
|
||||||
gains *wait identity*: a park-epoch packed into the spare bits of the slot
|
|
||||||
word, so that a stale unpark dies at one failed CAS in the waker.
|
|
||||||
|
|
||||||
Protocol (the consuming-wake rules):
|
|
||||||
- Word becomes `(gen << 32) | (epoch << 8) | state` — 24-bit epoch in the
|
|
||||||
previously unused bits; every transition still one CAS, gen check still
|
|
||||||
atomic with it.
|
|
||||||
- `begin_wait` (actor-side, before registering with any waker) bumps the
|
|
||||||
epoch and returns it; registrations carry `(pid, epoch)`.
|
|
||||||
- Every successful wake transition **consumes** the epoch (bumps it):
|
|
||||||
`Parked(e) → Queued(e+1)`, `Running(e) → RunningNotified(e+1)`. A second
|
|
||||||
unpark stamped with the same epoch can never land — at most one wake per
|
|
||||||
wait, by construction. This is what makes select's *loser* arms harmless
|
|
||||||
without a cancellation pass: their stale registrations fire one futile CAS
|
|
||||||
on next send and self-clean.
|
|
||||||
- `unpark_at(pid, epoch)` = epoch-matched wake. Bare `unpark(pid)` remains as
|
|
||||||
the wildcard and is reserved for **terminal** wakes (`request_stop`), which
|
|
||||||
never return control to the code that parked.
|
|
||||||
- One-shot park sites (`Mutex::lock_timeout`, `sleep`, `block_on_io`,
|
|
||||||
`wait_fd`) stay exactly as written: precise wakes are preserved, so a wake
|
|
||||||
still *means* something.
|
|
||||||
|
|
||||||
### 1. Slot-word epoch + consuming unparks ✅ (`4913835`)
|
|
||||||
`slot_state.rs`: repack, `begin_wait`, epoch-matched/consuming `unpark`,
|
|
||||||
epoch-preserving claim/yield/park transitions. Loom: existing theorems
|
|
||||||
re-proved on the new word + a new stale-epoch theorem (an unpark stamped with
|
|
||||||
a consumed epoch can never enqueue *or* notify). Plumbing only — no caller
|
|
||||||
stamps yet; behaviour identical.
|
|
||||||
|
|
||||||
### 2. Stamp every registration-based wake; retire per-primitive wait seqs ✅ (`400854a`)
|
|
||||||
The epoch IS the wait identity, so `channel::Inner.{cur_wait,next_wait_seq}`
|
|
||||||
and `mutex`'s per-wait `seq` are subsumed: registrations become
|
|
||||||
`(pid, epoch)`, `TimerTarget::on_timeout(pid, epoch)`, wakers call
|
|
||||||
`unpark_at`. Sleep timers, io completions (`Blocking` + `FdReady`) and join
|
|
||||||
waiters stamped likewise. End-state invariant, auditable in one sentence:
|
|
||||||
**the only wildcard wake is `request_stop`.**
|
|
||||||
|
|
||||||
### 3. `select` — ready-index wait over multiple receivers ✅ (`00128f3`)
|
|
||||||
`select(&[&dyn Selectable]) -> usize`: register self in each arm's
|
|
||||||
`parked_receiver` (sequentially — the park protocol makes cross-arm atomicity
|
|
||||||
unnecessary), park once, wake, scan, return the first ready index; caller
|
|
||||||
`try_recv`s it. Sound as ready-index *because channels are single-receiver*:
|
|
||||||
nobody can steal between ready and recv. A closed arm counts as ready (its
|
|
||||||
`try_recv` reports the closure). Arms are scanned in order — priority order,
|
|
||||||
documented, like BEAM receive clauses; no fairness promise. No cancel pass
|
|
||||||
(see protocol above). Drop-guard deregistration on the unwind path only, to
|
|
||||||
keep tests' single-parked-receiver asserts honest.
|
|
||||||
|
|
||||||
### 4. `select_timeout` ✅ (`a0a93b6`)
|
|
||||||
A `WaitTimeout` timer stamped with the select's epoch, whose target is a
|
|
||||||
stateless `unpark_at`. The woken selector classifies the wake from state
|
|
||||||
alone: some arm ready → that arm; none ready → with precise wakes the timer
|
|
||||||
is the only remaining stamped waker → `None`. No flag, no shared waiter
|
|
||||||
struct, no registration to leak. The timer entry expires as a stale-epoch
|
|
||||||
no-op when an arm wins, per the no-cancellation convention in `timer.rs`.
|
|
||||||
|
|
||||||
### 5. Docs + integration proof ✅ (this commit)
|
|
||||||
README module table, gotchas rewrite (the "No `select`" bullet becomes the
|
|
||||||
consuming-wake invariant), and an integration test of the motivating
|
|
||||||
pattern: a server selecting over its inbox and a monitor `Down` channel
|
|
||||||
(`tests/select.rs::motivating_pattern_inbox_plus_monitor_down`).
|
|
||||||
|
|
||||||
Unblocked but deliberately out of scope: `gen_server::handle_info` (now an
|
|
||||||
API-design question, not a runtime question — see Later).
|
|
||||||
|
|
||||||
### Deviations from the plan
|
|
||||||
- **The no-park exit needed a protocol piece the plan missed.** A `select`
|
|
||||||
returning an arm ready at *registration* time leaves earlier arms holding
|
|
||||||
live-epoch registrations — a sender could still set RunningNotified and
|
|
||||||
fault the actor's next one-shot park. Landed as `scheduler::retire_wait`
|
|
||||||
(bump the epoch, eat a landed notification via the new
|
|
||||||
`StateWord::clear_notify`, re-observe the stop flag — in that order) plus
|
|
||||||
the loom theorem `retire_eats_late_arm_notification`. The parked path
|
|
||||||
needed nothing: the winning wake's consume already retires the epoch.
|
|
||||||
- **No Drop guard.** The planned unwind-path deregistration was superseded:
|
|
||||||
the single-receiver debug_asserts relax to "none, or my own pid", so a
|
|
||||||
leftover own-registration (consumed epoch — inert by theorem) is simply
|
|
||||||
overwritten by the receiver's next wait on that channel. Less machinery,
|
|
||||||
and the asserts still catch the real misuse (two actors on one channel).
|
|
||||||
- **A closed arm is ready *forever*** — callers must drop it from the arm
|
|
||||||
set once observed, or priority order starves every higher-indexed arm
|
|
||||||
(documented on `select`; found the hard way by a livelocking test).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅ DONE
|
|
||||||
|
|
||||||
Goal: spend the `select` primitive on the server loop — out-of-band messages
|
|
||||||
(`handle_info`) and monitor `Down`s (`handle_down`) dispatched alongside
|
|
||||||
calls and casts — and close the v0.2 fd-hygiene hole in `io.rs`.
|
|
||||||
|
|
||||||
Design decisions (settled up front):
|
|
||||||
- **Info channels are static.** `type Info: Send + 'static` on the trait; a
|
|
||||||
`Vec<Receiver<G::Info>>` handed over at start. Users enum their own Info.
|
|
||||||
Dynamic info subscription is deliberately out: revisit against a consumer.
|
|
||||||
- **Down forwarding is dynamic, because monitors are.** You monitor a worker
|
|
||||||
you just spawned inside a handler, so down arms cannot ride the static
|
|
||||||
list. Mechanism: the loop owns a control arm (`Receiver<Monitor>`);
|
|
||||||
`init` grows a `&ServerCtx` parameter (breaking, no-op default body) whose
|
|
||||||
clonable `Watcher` the state stores and calls `watch(monitor)` from any
|
|
||||||
handler thereafter. Handler signatures otherwise untouched.
|
|
||||||
- **Priority order: downs → control → infos → inbox.** System messages
|
|
||||||
preempt; a hot inbox cannot starve a death notice. Within each class,
|
|
||||||
declaration order (select's documented priority semantics).
|
|
||||||
- **Closed arms drop silently** (the closed-arm-is-ready-forever gotcha).
|
|
||||||
Closed inbox still means graceful shutdown, unchanged.
|
|
||||||
|
|
||||||
### 1. handle_info — static info arms in the server loop ✅ (`e5d1b3b`)
|
|
||||||
`type Info` + `handle_info` (no-op default) on `GenServer`; `ServerBuilder`
|
|
||||||
(`Server::new(state).with_info(rx).under(sup).start()`) so start variants
|
|
||||||
don't multiply; `start`/`start_under` stay as wrappers. Loop becomes
|
|
||||||
`select` over `[infos…, inbox]`, dispatching by arm index.
|
|
||||||
|
|
||||||
### 2. handle_down — ServerCtx::watcher + control arm ✅ (`24b95c9`)
|
|
||||||
`handle_down` (no-op default), `ServerCtx`/`Watcher`, the control arm, and
|
|
||||||
the loop's live `Vec<Monitor>`. A `Down` retires its arm (monitors are
|
|
||||||
one-shot). Integration test: the motivating worker-pool pattern — a server
|
|
||||||
that spawns workers in a handler, watches them, and reacts to their deaths.
|
|
||||||
|
|
||||||
### 3. io fd hygiene — DEL on the unwind path ✅ (`f6969e5`)
|
|
||||||
A stopped actor unwinding out of `wait_fd`'s park leaks its `waiters` entry
|
|
||||||
and the kernel registration; the stale entry then fails every future
|
|
||||||
`wait_*` on that fd with `AlreadyExists` (the defensive bare-DEL in
|
|
||||||
`epoll_register` sits behind the `contains_key` check, so it never runs).
|
|
||||||
Fix where the invariant breaks: a drop guard in `wait_fd`, armed after a
|
|
||||||
successful register, disarmed on normal wake; on unwind it removes the
|
|
||||||
waiters entry iff it is still `(me, epoch)` and only then DELs the fd
|
|
||||||
(an entry consumed by a racing `FdReady` means the fd may already be
|
|
||||||
re-registered by someone else — leave it alone).
|
|
||||||
|
|
||||||
### Deviations from the plan
|
|
||||||
- **The plain-inbox fast path narrowed.** It now also requires the control
|
|
||||||
arm closed, so every server pays one select on its first iteration (the
|
|
||||||
ctx drops after `init`; a state that didn't clone the Watcher closes the
|
|
||||||
arm there and then). Servers that DO hold a Watcher select forever — the
|
|
||||||
price of dynamic watch.
|
|
||||||
- **The fd leak was worse than the v0.2 caveat admitted.** Not "at most one
|
|
||||||
stale wakeup": the stale `waiters` entry failed every future `wait_*` on
|
|
||||||
that fd with AlreadyExists, permanently (the defensive DEL sat behind the
|
|
||||||
`contains_key` check). The regression test exhibits it as a hang.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Later
|
## Later
|
||||||
|
|
||||||
### Tunable scheduler idle policy (RFC 004)
|
### Per-switch cost (context shims, epoch protocol)
|
||||||
Today idle schedulers fall back to `thread::sleep(100µs)`, putting ~100µs
|
The shootout's residual: per-wake latency is 0.16–0.18 µs at N=1 and
|
||||||
worst-case latency on every cross-thread actor handoff. RFC 004 specifies a
|
0.8–1.2 µs at N=8+, dominated by the context-switch shims and the epoch
|
||||||
bounded spinning-worker policy (Go/Tokio model): idle schedulers spin on a
|
protocol, not the queue. On current evidence this is the larger constant —
|
||||||
cheap atomic for a configurable budget before parking, so a burst of work is
|
"the whole game" alongside the v0.9 work — but there is no spec yet. Needs a
|
||||||
picked up in cache-coherency time rather than after a kernel reschedule.
|
profiling spike (where do the cycles actually go per park/unpark round-trip)
|
||||||
|
and then an RFC before it can be scheduled.
|
||||||
Two new `Config` knobs: `spin_budget_cycles` (RDTSC cycles to spin; `0`
|
|
||||||
recovers today's behaviour) and `max_spinners` (cap on concurrent spinners;
|
|
||||||
default `N/2`). Single new atomic `n_spinning`. No change to the run-queue
|
|
||||||
data structure — lands on the current `Mutex<VecDeque>` and is independent of
|
|
||||||
any future queue topology work.
|
|
||||||
|
|
||||||
Full spec: `rfc_004-tunable-scheduler-idle-policy.md` (artefacts.kalsbeek.dev).
|
|
||||||
|
|
||||||
### Unbounded / configurable-bounded actor count
|
### Unbounded / configurable-bounded actor count
|
||||||
Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384).
|
Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384).
|
||||||
Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling segment
|
Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling segment
|
||||||
sizes, append-only) once the cap is actually hit. Do not let it calcify.
|
sizes, append-only) once the cap is actually hit. Do not let it calcify.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### arm-port validation & merge
|
### arm-port validation & merge
|
||||||
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
|
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
|
||||||
hardware. Build + run full test suite on an aarch64 device; check
|
hardware. Build + run full test suite on an aarch64 device; check
|
||||||
|
|||||||
Reference in New Issue
Block a user