feat(runtime): phase 1 — peel timers/io/monitor-id out of SharedState; gate stop sentinel behind PREEMPTION_ENABLED

- next_monitor_id -> AtomicU64 on RuntimeInner
- timers -> own Mutex<Timers>; io -> own Mutex<Option<IoThread>> (lock order: io-before-shared)
- pending_closures Vec folded into Slot::pending_closure
- termination check reads io liveness before shared; ordering argument documented
- poison fix: check_cancelled() no longer fires while preemption is disabled,
  so a cancellation unwind can never poison a runtime/channel mutex
- regression test: tests/poison_stop.rs
- ROADMAP_v0.5.md added
This commit is contained in:
Claude
2026-06-09 18:56:17 +00:00
parent d908eb3f95
commit 3c7e26bc98
6 changed files with 303 additions and 108 deletions
+109
View File
@@ -0,0 +1,109 @@
# smarm v0.5 — Runtime decomposition & run-path scaling
Goal: dismantle the single `Mutex<SharedState>` along the run path so the
runtime scales to dozens of cores, with crash isolation hardened so a torn
write or a stray cancellation can never poison shared runtime state.
Guiding rules established this cycle:
- **Lock count that matters is hot-path locks.** Cold lifecycle paths
(spawn/join/monitor/link/finalize) may keep a small per-slot lock; the run
path (yield/park/unpark/pop/resume) targets zero locks.
- **Lock ordering: io-before-shared, and slot locks are leaves** (never hold
two slot locks at once). Any new lock states its position in this order.
- **No unwind inside a runtime critical section.** The stop sentinel only
fires at lock-free observation points.
---
## Phase 1 — State decomposition: easy peel-offs ✅ DONE
Split independent concerns out of `SharedState` so the global lock guards less.
- [x] `next_monitor_id``AtomicU64` on `RuntimeInner` (lock-free id minting).
- [x] `timers` → own `Mutex<Timers>` (only drain-winner + blocking prims touch it).
- [x] `io` → own `Mutex<Option<IoThread>>` (completion queue already self-locked).
- [x] `pending_closures: Vec` → folded into `Slot::pending_closure` (per-actor data).
- [x] Termination check split: read io liveness *before* `shared`; ordering
argument documented in `schedule_loop`.
- [x] **Poison fix:** gate `check_cancelled()` in `maybe_preempt` behind
`PREEMPTION_ENABLED`, so a cancellation sentinel can never unwind while a
`std::sync::Mutex` (shared/channel) is held. Regression test:
`tests/poison_stop.rs`.
`SharedState` now holds only: `slots`, `free_list`, `run_queue`, `root_pid`.
## Phase 2 — Slot table split (NEXT, the big correctness step)
Make slot lookup lock-free and per-slot state independently mutable.
- [ ] Fixed slab: `Box<[SlotCell; max_actors]>`, slots never move → stable
addresses, lock-free index. **Assert on exhaustion** with a panic message
naming `Config::max_actors(n)` as the fix. Default `max_actors = 16_384`.
(Mental note: must become unbounded or configurable-bounded later — see
"Deferred" below. Do not let the fixed cap calcify into an assumption.)
- [ ] Per-slot `generation: AtomicU32` outside the lock for cheap stale-PID
rejection.
- [ ] Per-slot CAS state machine replacing `state` + `pending_unpark`:
`Parked / Queued / Running / RunningNotified / Done` in one `AtomicU8`.
`unpark` = CAS Parked→Queued (then push) or Running→RunningNotified;
park-return = CAS Running→Parked, else re-queue. Eliminates the
lost-wakeup bool by turning the race window into a state.
- [ ] `sp` → relaxed `AtomicUsize` (happens-before carried by queue
release-push / acquire-pop).
- [ ] Per-slot raw non-poisoning mutex (spin-then-futex, Linux; embedded-
friendlier than `std::sync::Mutex`) for the cold collections: `waiters`,
`monitors`, `links`, `supervisor_channel`, `outcome`, `pending_io_result`,
`outstanding_handles`. Guard enters `NoPreempt`.
- [ ] Free list → its own structure (lock-free Treiber stack or striped).
- [ ] `finalize_actor` link cascade: lock peers one at a time (leaf rule);
write the acyclicity argument next to it.
- [ ] `live_actors` / `io_outstanding` → atomics; termination = both zero +
queue empty. Finalize decrements `live` strictly *after* all wakeups it
produced are enqueued — document this ordering as the correctness crux.
## Phase 3 — Pluggable run queue + bench shootout
- [ ] Extract `RunQueue` behind a `type RunQueue = ...` alias selected at
compile time by mutually-exclusive features, with a `compile_error!`
guard if zero or >1 is set. No runtime dispatch.
- `rq-mutex``Mutex<VecDeque>`, the control/baseline.
- `rq-mpmc` — single hand-rolled Vyukov bounded MPMC ring
(per-cell seq numbers). Strict FIFO; "one hot cache line".
- `rq-striped` — M Vyukov rings, fetch-add ticket distribution. Relaxed
FIFO, reordering bounded by ~M. Predicted winner @20c.
- [ ] Bounded rings are sound because the slab cap + at-most-once-enqueued
invariant bound occupancy ≤ `max_actors`; size Σcapacity > `max_actors`
so push is infallible (probe next stripe when full; terminates).
- [ ] All hand-rolled, dependency-free (portability incl. possible embedded).
## Phase 4 — Bench harness
- [ ] Raw-structure microbench: N threads, push/pop throughput vs thread count,
sweeping producer:consumer ratios. Isolates the data structure.
- [ ] Runtime-level: yield-storm, ping-pong-pairs, spawn-storm, all sweeping
scheduler count. Reuses existing `benches/` harness style.
- [ ] Driver script rebuilding per `rq-*` feature to compare variants in one go.
- [ ] Sandbox validates correctness via oversubscribed `Config::exact(N)` on 1
core; real contention numbers come from the 20-core box.
## Phase 5 — Safety hardening & model checking
- [ ] `loom` (dev-dependency only, x86 Linux) model tests for the slot state
machine and each ring variant.
- [ ] `NoPreempt` / no-unwind-under-lock audit across all internal critical
sections; assert the invariant in debug builds where feasible.
---
## Fast follow (post-v0.5, written down so it isn't lost)
- **Channel mutex migration.** `channel::Inner<T>` is `Arc<Mutex<_>>` of the
same poison class as the old shared lock; `recv_match` even runs a user
predicate under it. The Phase-1 `check_cancelled` gating already removes the
unwind *source* globally, so channels are safe today — but migrating the
channel to the raw non-poisoning mutex (Phase 2) is a clean follow-up rather
than bundled into the run-path rework.
## Deferred / explicitly out of scope for v0.5
- **Unbounded / configurable-bounded actor count.** v0.5 ships a fixed slab
with a loud assert. Revisit with a segmented slab (array of
`AtomicPtr<Segment>`, doubling segment sizes, append-only) once we actually
hit the cap or a workload demands it.
- **Idle-wakeup eventcount.** Idle scheduler threads keep the current 100µs
poll-sleep. A futex-based eventcount is a later optimization if benches show
idle latency matters.
- **User-facing safe data structures** (ArcSwap-style cells, structurally-
shared persistent maps). Context for the runtime work, not in scope here.
- **IO fd hygiene on actor death** (pre-existing v0.2 TODO in `io.rs`).