docs(roadmap): consolidate task.md + ROADMAP_v0.5.md into ROADMAP.md
Single source of truth: v0.4 history, v0.5 phases, v0.6 open tracks, and invariants/gotchas. Old task.md and ROADMAP_v0.5.md removed.
This commit is contained in:
+186
@@ -0,0 +1,186 @@
|
||||
# smarm — Roadmap
|
||||
|
||||
## v0.4 — Actor primitives ✅ DONE
|
||||
|
||||
### 1. Cooperative cancellation ✅ (`a8ddb4a`)
|
||||
Stop flag on `Actor` behind `Arc<AtomicBool>`. Observation points:
|
||||
`maybe_preempt`/`check!()` + the wakeup side of `park_current`/`yield_now`.
|
||||
Sentinel = `StopSentinel` (zero-size), caught in trampoline → `Outcome::Stopped`.
|
||||
`join()` on a stopped actor returns `Ok(())`.
|
||||
|
||||
Known gap (by design): a tight no-alloc loop without `check!()` cannot be
|
||||
stopped, same as preemption.
|
||||
|
||||
### 2. one_for_all / rest_for_one + ordered shutdown ✅ (`351dc9c`)
|
||||
`Strategy::{OneForOne,OneForAll,RestForOne}` via `.strategy()`. Triggering
|
||||
child's `Restart` policy decides whether anything restarts; strategy decides
|
||||
which siblings are cycled. Survivors stopped in reverse start order, awaited on
|
||||
the existing `supervisor_channel` funnel, restarted in start order.
|
||||
|
||||
`Signal::Stopped(pid)` + `DownReason::Stopped` added (distinct from Exit).
|
||||
|
||||
Also fixed a latent bug (`e80334b`): orphaned timers were gating shutdown.
|
||||
`live == 0` is now the shutdown condition; timer heap cleared on exit.
|
||||
|
||||
### 3. Links + trap_exit ✅ (`6581484`)
|
||||
`Slot.links: Vec<Pid>` (bidirectional). On finalize: abnormal death →
|
||||
`request_stop(peer)` unless peer traps, in which case deliver `ExitSignal`
|
||||
message. Normal exit does not propagate. `trap_exit()` returns a dedicated
|
||||
`Receiver<ExitSignal>` (distinct from monitor `Down` channel).
|
||||
|
||||
### 4. Selective receive ✅ (`03f3875`)
|
||||
`Receiver::recv_match(pred)` + `try_recv_match`. Scans `VecDeque` front-to-back,
|
||||
removes+returns first match, parks and re-scans on new arrivals. `pred` is
|
||||
`Fn(&T) -> bool` (not `FnMut`; runs under the channel lock — keep it cheap).
|
||||
|
||||
### 5. Grab-bag ✅
|
||||
- **demonitor** (`5181037`): `monitor()` → `Monitor { id, target, rx }`.
|
||||
`demonitor(&m) -> Option<MonitorId>`. Dropped `Sender` released post-lock
|
||||
(non-reentrant shared mutex discipline).
|
||||
- **gen_server** (`55221e9`): `GenServer` trait, `call`/`cast` over one
|
||||
`Envelope` inbox. `terminate` hook via drop guard. No `handle_info`, no call
|
||||
timeout (deferred — needs cross-channel mailbox merge or per-recv deadline).
|
||||
- **Named registry**: `register`/`whereis`/`send_by_name` — **not yet done**.
|
||||
HashMap in `SharedState`. No dependencies; can land any time.
|
||||
- **arm-port branch**: aarch64 context-switch backend extracted into
|
||||
`src/arch/`. Untested on hardware. Merge only after on-device validation.
|
||||
|
||||
---
|
||||
|
||||
## v0.5 — Runtime decomposition & run-path scaling ✅ DONE
|
||||
|
||||
Goal: dismantle the single `Mutex<SharedState>` along the run path so the
|
||||
runtime scales to dozens of cores, with crash isolation hardened so a torn
|
||||
write or a stray cancellation can never poison shared runtime state.
|
||||
|
||||
Guiding rules established this cycle:
|
||||
- **Lock count that matters is hot-path locks.** Cold lifecycle paths may keep a
|
||||
small per-slot lock; the run path targets zero locks.
|
||||
- **Lock ordering: io-before-shared, slot locks are leaves** (never hold two
|
||||
slot locks at once).
|
||||
- **No unwind inside a runtime critical section.** Stop sentinel only fires at
|
||||
lock-free observation points.
|
||||
- **Never hold a thread-local guard across a possible switch point.** An actor
|
||||
can be preempted at any allocation and resume on a different OS thread.
|
||||
`with_runtime`, `with_shared`, and every `RawMutex` guard disable preemption
|
||||
for this reason.
|
||||
|
||||
### Phase 1 — State decomposition ✅ (`3c7e26b`)
|
||||
`next_monitor_id` → `AtomicU64`. `timers` → own `Mutex<Timers>`. `io` → own
|
||||
`Mutex<Option<IoThread>>`. `pending_closures` → folded into `Slot::pending_closure`.
|
||||
`check_cancelled()` gated behind `PREEMPTION_ENABLED` (poison fix).
|
||||
|
||||
`SharedState` now holds only: `slots`, `free_list`, `run_queue`, `root_pid`.
|
||||
|
||||
### Phase 2 — Slot table split ✅ (`5e0c9d4`)
|
||||
Fixed slab `Box<[Slot]>`. Generation packed into state word: one
|
||||
`AtomicU64 = (gen << 32) | state` — gen check is atomic with every transition.
|
||||
Per-slot CAS state machine `Vacant/Queued/Running/RunningNotified/Parked/Done`.
|
||||
Per-slot `RawMutex` for cold collections. Free list → `RawMutex<Vec<u32>>`.
|
||||
`live_actors` atomic; termination = io_out==0 && queue empty && live==0.
|
||||
|
||||
### Phase 3 — Pluggable run queue ✅ (`1b3b618`)
|
||||
Compile-time selected via feature flags. Variants:
|
||||
- `rq-mutex` — `Mutex<VecDeque>`, baseline.
|
||||
- `rq-mpmc` — hand-rolled Vyukov bounded MPMC ring.
|
||||
- `rq-striped` — M Vyukov rings, fetch-add ticket distribution.
|
||||
|
||||
All variants compile in every build; feature only picks the alias. Bounded rings
|
||||
are sound via slab cap + at-most-once-enqueued invariant.
|
||||
|
||||
Queue ops require preemption disabled (a producer suspended mid-publish stalls
|
||||
every consumer behind its cell — livelock).
|
||||
|
||||
### Phase 4 — Bench harness ✅ (`6d9f369`)
|
||||
`benches/rq_micro.rs`: raw structure sweep. `benches/rq_runtime.rs`:
|
||||
yield-storm, ping-pong-pairs, spawn-storm. `scripts/bench_rq.sh`: rebuilds per
|
||||
feature, aggregates to `bench_results/summary.csv`.
|
||||
|
||||
### Phase 5 — Safety hardening & model checking ✅ (`039703d`)
|
||||
State machine extracted to `src/slot_state.rs` for loom checking. Loom on the
|
||||
rings via `src/sync_shim.rs`. `NoPreempt`/no-unwind/TLS-guard audit complete.
|
||||
Invariant-assertion sweep: every `StateWord` transition self-asserts its
|
||||
precondition.
|
||||
|
||||
---
|
||||
|
||||
## v0.6 — Fast follows & open tracks
|
||||
|
||||
Priority order within the cycle; each item is its own commit.
|
||||
|
||||
### 1. Channel mutex migration (first change)
|
||||
`channel::Inner<T>` holds a `std::sync::Mutex`. Phase 2 surfaced a sharp
|
||||
correctness issue: the `MutexGuard` is held with preemption **enabled**, so a
|
||||
timeslice switch inside a channel critical section migrates the actor and
|
||||
releases the pthread mutex from a different OS thread — UB (Linux futexes happen
|
||||
to tolerate it, but it's not guaranteed). The Phase-1 `check_cancelled` gating
|
||||
already removes the unwind source, so poison is safe today; this is about the
|
||||
cross-thread release.
|
||||
|
||||
Migrate to `RawMutex` (which disables preemption and is cross-thread-release
|
||||
sound by construction). `recv_match` runs a user predicate under this lock —
|
||||
keep it cheap/pure.
|
||||
|
||||
### 2. Named registry
|
||||
`register(name, pid)` / `whereis(name) -> Option<Pid>` / `send_by_name`.
|
||||
`HashMap<String, Pid>` in `SharedState`, guarded by the shared lock. No
|
||||
architecture changes; can land immediately after #1.
|
||||
|
||||
### 3. gen_server: handle_info + call timeout
|
||||
- `handle_info`: needs a cross-channel mailbox merge (call/cast inbox +
|
||||
out-of-band signals). The still-unmade decision: whether to introduce a
|
||||
unified per-actor mailbox or keep per-channel `recv_match` composition.
|
||||
- Call timeout: `call` with a deadline — monitor the server, wait
|
||||
reply-or-Down-or-deadline, then `demonitor`+drop so a timed-out call leaks
|
||||
no registration. Requires a per-`recv` deadline / `Signal::Timeout` primitive
|
||||
that doesn't exist yet.
|
||||
|
||||
These two can land independently; call timeout is smaller and more useful
|
||||
first.
|
||||
|
||||
### 4. Unbounded / configurable-bounded actor count
|
||||
v0.5 ships a fixed slab with a loud assert (`Config::max_actors(n)`, default
|
||||
16 384). Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling
|
||||
segment sizes, append-only) once the cap is actually hit or a workload demands
|
||||
it. Do not let the fixed cap calcify into an assumption.
|
||||
|
||||
### 5. Idle-wakeup eventcount
|
||||
Idle scheduler threads currently poll-sleep at 100 µs. A futex-based eventcount
|
||||
would reduce idle latency. Defer until benches show it matters.
|
||||
|
||||
### 6. IO fd hygiene on actor death
|
||||
Pre-existing v0.2 TODO in `io.rs`: fds registered with epoll for a dead actor
|
||||
are not cleaned up. Audit and fix.
|
||||
|
||||
### 7. arm-port validation & merge
|
||||
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
|
||||
hardware. Steps: build + run full test suite on an aarch64 device; check
|
||||
`chained_spawn` / `yield_many` bench medians for regressions vs x86 baseline;
|
||||
merge and update README module table to reflect multi-arch support.
|
||||
|
||||
---
|
||||
|
||||
## Invariants & gotchas (respect these across all cycles)
|
||||
|
||||
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
|
||||
`with_shared`. Never send on a channel while holding the shared lock. Pattern:
|
||||
`mem::take` data under the lock, send after releasing. See `finalize_actor`.
|
||||
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
|
||||
Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs →
|
||||
unpark joiners → reclaim slot if `outstanding_handles==0`. Death notifications
|
||||
always precede reclamation.
|
||||
- **Slot lifecycle reset in THREE places:** `Slot::vacant()`, `reclaim_slot()`
|
||||
(runtime.rs), slot-init block in `spawn_under` (scheduler.rs). Any new `Slot`
|
||||
field must be reset in all three.
|
||||
- **Pid = (index, generation).** Stale handles caught by generation mismatch in
|
||||
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
|
||||
- **No `select`, no unified per-process mailbox.** Why the supervisor uses the
|
||||
single `supervisor_channel` funnel. `recv_match` is per-channel only; a
|
||||
cross-channel merge is the still-unmade decision (see v0.6 #3).
|
||||
- **Cooperative-only.** Preemption and cancellation both depend on the actor
|
||||
reaching `check!()`/yield/alloc/blocking points.
|
||||
- **Queue ops require preemption disabled.** A producer suspended mid-publish
|
||||
stalls every consumer — livelock. `with_runtime`, `with_shared`, and
|
||||
`RawMutex` guards all disable preemption for their span.
|
||||
- **`run()` is single-thread** (`Config::exact(1)`); tests rely on deterministic
|
||||
single-thread ordering. Multi-thread via `runtime::init(Config…)`.
|
||||
-157
@@ -1,157 +0,0 @@
|
||||
# smarm v0.5 — Runtime decomposition & run-path scaling
|
||||
|
||||
Goal: dismantle the single `Mutex<SharedState>` along the run path so the
|
||||
runtime scales to dozens of cores, with crash isolation hardened so a torn
|
||||
write or a stray cancellation can never poison shared runtime state.
|
||||
|
||||
Guiding rules established this cycle:
|
||||
- **Lock count that matters is hot-path locks.** Cold lifecycle paths
|
||||
(spawn/join/monitor/link/finalize) may keep a small per-slot lock; the run
|
||||
path (yield/park/unpark/pop/resume) targets zero locks.
|
||||
- **Lock ordering: io-before-shared, and slot locks are leaves** (never hold
|
||||
two slot locks at once). Any new lock states its position in this order.
|
||||
- **No unwind inside a runtime critical section.** The stop sentinel only
|
||||
fires at lock-free observation points.
|
||||
- **Never hold a thread-local guard across a possible switch point.** An
|
||||
actor can be preempted at any allocation and resume on a different OS
|
||||
thread; a live `RefCell` borrow (or std `MutexGuard`) then pairs its
|
||||
acquire/release across two threads' locals — count underflow / UB.
|
||||
`with_runtime`, `with_shared`, and every `RawMutex` guard disable
|
||||
preemption for exactly this reason (phase 2 found this the hard way).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — State decomposition: easy peel-offs ✅ DONE
|
||||
Split independent concerns out of `SharedState` so the global lock guards less.
|
||||
- [x] `next_monitor_id` → `AtomicU64` on `RuntimeInner` (lock-free id minting).
|
||||
- [x] `timers` → own `Mutex<Timers>` (only drain-winner + blocking prims touch it).
|
||||
- [x] `io` → own `Mutex<Option<IoThread>>` (completion queue already self-locked).
|
||||
- [x] `pending_closures: Vec` → folded into `Slot::pending_closure` (per-actor data).
|
||||
- [x] Termination check split: read io liveness *before* `shared`; ordering
|
||||
argument documented in `schedule_loop`.
|
||||
- [x] **Poison fix:** gate `check_cancelled()` in `maybe_preempt` behind
|
||||
`PREEMPTION_ENABLED`, so a cancellation sentinel can never unwind while a
|
||||
`std::sync::Mutex` (shared/channel) is held. Regression test:
|
||||
`tests/poison_stop.rs`.
|
||||
|
||||
`SharedState` now holds only: `slots`, `free_list`, `run_queue`, `root_pid`.
|
||||
|
||||
## Phase 2 — Slot table split ✅ DONE
|
||||
Make slot lookup lock-free and per-slot state independently mutable.
|
||||
- [x] Fixed slab: `Box<[Slot]>`, slots never move → stable
|
||||
addresses, lock-free index. **Assert on exhaustion** with a panic message
|
||||
naming `Config::max_actors(n)` as the fix. Default `max_actors = 16_384`.
|
||||
(Mental note: must become unbounded or configurable-bounded later — see
|
||||
"Deferred" below. Do not let the fixed cap calcify into an assumption.)
|
||||
- [x] Generation packed INTO the state word (better than the planned separate
|
||||
`AtomicU32`): one `AtomicU64 = (gen << 32) | state`, so the gen check is
|
||||
atomic with every transition — no ABA, no spurious unparks, by CAS.
|
||||
- [x] Per-slot CAS state machine `Vacant/Queued/Running/RunningNotified/
|
||||
Parked/Done` replacing `state` + `pending_unpark`. Also fixed a latent
|
||||
lost wakeup in the Blocking-IO completion path (result set for a
|
||||
still-Running actor without a flag).
|
||||
- [x] `sp` → relaxed `AtomicUsize`; stop flag + first-resume closure as
|
||||
`AtomicPtr`s — resume path fully lock-free.
|
||||
- [x] Per-slot raw non-poisoning futex mutex (`src/raw_mutex.rs`) for the
|
||||
cold collections. Guard enters `NoPreempt`.
|
||||
- [x] Free list → `RawMutex<Vec<u32>>` (a leaf lock). Treiber/striped only if
|
||||
the phase-4 spawn-storm bench shows contention — revisit then.
|
||||
- [x] `finalize_actor` link cascade locks peers one at a time; acyclicity
|
||||
argument written at the site. `link()` registers target-first with the
|
||||
race argument at the site.
|
||||
- [x] `live_actors` atomic; termination = io_out == 0 (read pre-queue-lock)
|
||||
&& queue empty && live == 0; decrement-last ordering documented as the
|
||||
correctness crux at the site.
|
||||
|
||||
## Phase 3 — Pluggable run queue ✅ DONE (shootout = phase 4 harness)
|
||||
- [x] `RunQueue` alias in `src/run_queue.rs`, compile-time selected,
|
||||
`compile_error!` guards for zero / >1 features. No runtime dispatch.
|
||||
All variants compile in every build (unit tests always run); the
|
||||
feature only picks the alias. Non-default variants:
|
||||
`--no-default-features --features rq-…`.
|
||||
- `rq-mutex` — `Mutex<VecDeque>`, the control/baseline.
|
||||
- `rq-mpmc` — single hand-rolled Vyukov bounded MPMC ring
|
||||
(per-cell seq numbers). Strict FIFO; "one hot cache line".
|
||||
- `rq-striped` — M Vyukov rings, fetch-add ticket distribution. Relaxed
|
||||
FIFO, reordering bounded by ~M. Predicted winner @20c.
|
||||
- [x] Bounded rings sound via slab cap + at-most-once-enqueued (occupancy ≤
|
||||
`max_actors`); mpmc capacity = next_pow2(max_actors), striped
|
||||
Σcapacity ≈ 2×max_actors with probe-from-home-stripe. A full ring
|
||||
panics as an invariant violation (double enqueue), never spins.
|
||||
- [x] All hand-rolled, dependency-free.
|
||||
- [x] Two contracts surfaced by the extraction, documented + debug-asserted
|
||||
in `run_queue.rs`: queue ops require preemption disabled (a producer
|
||||
suspended mid-publish stalls every consumer behind its cell — livelock);
|
||||
and pop-None is a snapshot, not a fence — termination is counter-first
|
||||
(`live == 0` alone implies the queue holds nothing actionable; argument
|
||||
rewritten at the `schedule_loop` site).
|
||||
|
||||
## Phase 4 — Bench harness ✅ DONE (harness; real numbers from the 20-core box)
|
||||
- [x] `benches/rq_micro.rs`: raw structures, threads × p:c ratio sweep. One
|
||||
binary covers all three structures (types compile in every build).
|
||||
- [x] `benches/rq_runtime.rs`: yield-storm, ping-pong-pairs, spawn-storm,
|
||||
sweeping scheduler count; variant baked in by feature.
|
||||
- [x] `scripts/bench_rq.sh`: rebuilds per `rq-*` feature, aggregates RQCSV
|
||||
lines into `bench_results/summary.csv`. Knobs via SMARM_BENCH_* env;
|
||||
e.g. `SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh`.
|
||||
- [x] Harness validated end-to-end at smoke scale on the 1-core sandbox;
|
||||
contention curves and the actual variant decision come from the
|
||||
20-core box.
|
||||
|
||||
## Phase 5 — Safety hardening & model checking ✅ DONE
|
||||
- [x] State machine extracted to `src/slot_state.rs` (mechanism only; the
|
||||
protocol rationale stays in `runtime.rs`) so loom checks the PRODUCTION
|
||||
transitions. Models: lost-wakeup (park vs unpark), at-most-once
|
||||
(two unparkers), ABA (stale unpark vs reclaim+reuse), claim coalescing.
|
||||
- [x] Loom on the rings via `src/sync_shim.rs` (std normally, `loom::sync`
|
||||
under `--cfg loom`): exactly-once through lap wraparound, push/pop
|
||||
race, striped two-producer drain. `[target.'cfg(loom)'.dependencies]`;
|
||||
run with `RUSTFLAGS="--cfg loom" cargo test --lib --release`.
|
||||
RawMutex is deliberately NOT loom-modeled: futexes can't be, and it is
|
||||
Drepper's textbook mutex3 with stress + unwind tests.
|
||||
- [x] NoPreempt / no-unwind / TLS-guard audit: with_runtime + RawMutex guards
|
||||
+ run-queue ops + trace::record all gate preemption (and thereby the
|
||||
stop sentinel) for their span. Sole remaining exception: channel's std
|
||||
MutexGuard — the documented first fast-follow.
|
||||
- [x] Invariant-assertion sweep (the fast-follow, pulled forward): every
|
||||
StateWord transition self-asserts its precondition; enqueue asserts
|
||||
exactly (gen, Queued); live_actors underflow asserted; RawMutex
|
||||
enforces the leaf rule with a debug-build held-count; slab overflow and
|
||||
ring overflow stay loud panics. House style from here on.
|
||||
|
||||
---
|
||||
|
||||
## Fast follow (post-v0.5, written down so it isn't lost)
|
||||
- **Assert the invariants we lean on.** This cycle accumulated load-bearing
|
||||
invariants: at-most-once-enqueued, queue-ops-under-NoPreempt, never two
|
||||
cold locks, cold-path generation re-verify under the lock, finalize's
|
||||
decrement-last, pushes-pair-with-Queued-transitions, thread-local guards
|
||||
never crossing a switch point. Whenever code RELIES on one and a cheap
|
||||
check exists, assert it at the point of reliance — `debug_assert!` on hot
|
||||
paths, full `assert!`/loud panic on cold ones — so a violation fails at
|
||||
the breakage site, not three modules downstream (the slab-overflow panic
|
||||
and the queue-op preemption debug_assert are the pattern). Sweep the
|
||||
existing code for missed spots; new code adopts it as house style. The
|
||||
phase-5 audit is the natural vehicle for the sweep.
|
||||
- **Channel mutex migration.** `channel::Inner<T>` is `Arc<Mutex<_>>` of the
|
||||
same poison class as the old shared lock; `recv_match` even runs a user
|
||||
predicate under it. The Phase-1 `check_cancelled` gating already removes the
|
||||
unwind *source* globally, so channels are poison-safe today — but phase 2
|
||||
surfaced a second, sharper reason to migrate: the std `MutexGuard` is held
|
||||
with preemption ENABLED, so a timeslice switch inside a channel critical
|
||||
section migrates the actor and unlocks the pthread mutex from a different
|
||||
OS thread — technically UB (Linux futexes happen to tolerate it). The
|
||||
`RawMutex` guard disables preemption and is cross-thread-release sound by
|
||||
construction. Migrate as the first post-v0.5 change.
|
||||
|
||||
## Deferred / explicitly out of scope for v0.5
|
||||
- **Unbounded / configurable-bounded actor count.** v0.5 ships a fixed slab
|
||||
with a loud assert. Revisit with a segmented slab (array of
|
||||
`AtomicPtr<Segment>`, doubling segment sizes, append-only) once we actually
|
||||
hit the cap or a workload demands it.
|
||||
- **Idle-wakeup eventcount.** Idle scheduler threads keep the current 100µs
|
||||
poll-sleep. A futex-based eventcount is a later optimization if benches show
|
||||
idle latency matters.
|
||||
- **User-facing safe data structures** (ArcSwap-style cells, structurally-
|
||||
shared persistent maps). Context for the runtime work, not in scope here.
|
||||
- **IO fd hygiene on actor death** (pre-existing v0.2 TODO in `io.rs`).
|
||||
@@ -1,263 +0,0 @@
|
||||
# smarm — task.md (next steps)
|
||||
|
||||
Handoff for a future session reusing this sandbox. Read top to bottom once
|
||||
before starting; the gotchas section is hard-won and will save you a faceplant.
|
||||
|
||||
## Resume the environment
|
||||
|
||||
- Repo: `smarm`. Two branches (the old single `arm-port` stack was split):
|
||||
- `master` — the mainline, and HEAD. Carries roadmap #1–#5: cooperative
|
||||
cancellation, supervisor strategies (one_for_one/all, rest_for_one) + the
|
||||
orphaned-timer shutdown fix, links/trap_exit, selective receive, gen_server,
|
||||
and demonitor/`MonitorId`. Tagged `v0.4.0`. x86-64 Linux only.
|
||||
- `arm-port` — `master` plus a single commit: `feat(arch): aarch64 context
|
||||
switch + cycle counter`. Extracts the x86-64 context-switch / stack-init /
|
||||
cycle-counter out of `context.rs` into a `target_arch`-gated `src/arch/`
|
||||
(x86_64 + aarch64 backends) and adds an AAPCS64 backend. ⚠️ UNTESTED: never
|
||||
built or run on real ARM hardware. The x86-64 path is unchanged
|
||||
(`arch/x86_64.rs` is the old `context.rs` body verbatim), so the x86 suite
|
||||
passing says nothing about the aarch64 backend. Build + test on-device
|
||||
before trusting it.
|
||||
- Toolchain is installed but NOT on PATH in a fresh shell. First line of every
|
||||
session: `. "$HOME/.cargo/env"` (rustc/cargo 1.96).
|
||||
- Build `cargo build` · all tests `cargo test` · one suite `cargo test --test monitor`.
|
||||
- Bench probe `cargo bench --bench general` (custom print-only harness; compiles
|
||||
tokio in release the first time — slow — but `target/` persists across git
|
||||
checkouts so it's paid once).
|
||||
- Perf regression check: `git checkout <pre-change-sha>` →
|
||||
`cargo bench --bench general | tee before.txt` → `git checkout arm-port` →
|
||||
run again → diff the `smarm 1-thread` medians for `chained_spawn` and
|
||||
`yield_many` (those exercise spawn/finalize/scheduler). Numbers are noisy on
|
||||
this shared CPU; treat as "regression beyond noise?" not a precise delta.
|
||||
|
||||
## Roadmap (dependency order)
|
||||
|
||||
### 1. Cooperative cancellation — the keystone ✅ DONE (`a8ddb4a`)
|
||||
Everything below (one_for_all/rest_for_one, links) needs a *safe* way to stop a
|
||||
running peer. Forcible teardown of another green thread's stack is unsound here
|
||||
(shared heap + Drop). So: cooperative stop the actor observes and unwinds itself.
|
||||
|
||||
Shipped as designed (sentinel unwind, not Result-threading). Notes for what
|
||||
came next / future readers:
|
||||
- Stop flag lives on `Actor` behind `Arc<AtomicBool>` (fresh per spawn), NOT a
|
||||
`Slot` field — sidesteps the three-place reset, at the cost of one small
|
||||
alloc per spawn. The scheduler hands the resume path a raw `*const AtomicBool`
|
||||
(no per-resume refcount traffic); `yield_many` bench stayed at baseline,
|
||||
`chained_spawn` ~+6% from that alloc (left as-is; move to a `Slot` field if it
|
||||
ever matters).
|
||||
- Observation points: amortised `maybe_preempt`/`check!()` path + the wakeup
|
||||
side of `park_current`/`yield_now`. Sentinel = `StopSentinel` (zero-size),
|
||||
recognised in the trampoline → `Outcome::Stopped`. `join()` on a stopped actor
|
||||
returns `Ok(())` (no payload to propagate; reason is on the monitor channel).
|
||||
- Documented gaps confirmed by tests: no-observation-point loop can't be stopped
|
||||
(same as preemption); a user `catch_unwind` can swallow the sentinel but the
|
||||
flag stays set so the next yield re-raises.
|
||||
|
||||
Original plan, for reference:
|
||||
- Add a per-actor stop flag (Slot field + atomic, or check via shared state).
|
||||
- `request_stop(pid)`: set the flag, unpark if parked.
|
||||
- Realize the stop as a **controlled unwind**: when the scheduler resumes a
|
||||
stop-requested actor, inject a sentinel panic (dedicated payload type) so the
|
||||
existing `trampoline` `catch_unwind` tears the stack down and runs Drop. The
|
||||
trampoline recognizes the sentinel and reports a new `Outcome::Stopped`
|
||||
(distinct from a user `Panic`). This avoids changing every blocking-op
|
||||
signature.
|
||||
- Alternative considered: thread `Result<_, Cancelled>` through recv/sleep/
|
||||
lock/io. Rejected — large API churn. Go with the sentinel unwind.
|
||||
- Caveat to document: user code with its own `catch_unwind` can swallow the
|
||||
sentinel (cf. Erlang `catch`); re-check the flag at the next yield and/or
|
||||
re-raise. And a tight no-alloc loop without `check!()` can't be stopped —
|
||||
same inherent limitation as preemption.
|
||||
- Observation points: `maybe_preempt()`/`check!()` (cheap flag check) and the
|
||||
blocking parks (recv/sleep/mutex/io) on the stop-driven unpark.
|
||||
- Tests: looping actor on `check!()` gets stopped → `Outcome::Stopped`, Drop
|
||||
guards ran; parked-on-recv actor gets stopped; no-check loop documents the gap.
|
||||
|
||||
### 2. one_for_all / rest_for_one + ordered shutdown ✅ DONE (`351dc9c`)
|
||||
Shipped. What landed vs the plan:
|
||||
- `Strategy::{OneForOne,OneForAll,RestForOne}` selected via `.strategy()`,
|
||||
default `OneForOne`. The struct keeps the `OneForOne` name (compat; existing
|
||||
tests untouched) — a rename to `Supervisor` is a deferred refactor.
|
||||
- The *triggering* child's `Restart` policy decides whether anything restarts;
|
||||
the strategy decides which live siblings are cycled (all / index-> after the
|
||||
failed one). Survivors are `request_stop`'d in reverse start order, awaited on
|
||||
the existing `supervisor_channel` funnel (no new channel, no `select`),
|
||||
restarted in start order. One failure = one intensity tick regardless of group
|
||||
size. Out-of-band signals during an await are stashed and replayed.
|
||||
- `Signal::Stopped(pid)` + `DownReason::Stopped` added (kept distinct from Exit,
|
||||
as planned). A `Stopped` signal counts as abnormal for the restart decision.
|
||||
- Ordered shutdown: on cap-trip / mailbox-close-with-survivors, stop remaining
|
||||
children in reverse start order and await them (no-op on the normal exit).
|
||||
- ⚠️ Surfaced + fixed a latent keystone bug (`e80334b`): a cancelled
|
||||
sleeping/timeout actor orphans its timer entry, and the scheduler's shutdown
|
||||
check counted pending timers → `run()` hung until the dead actor's deadline
|
||||
fired (a `sleep(30s)` sleeper hung shutdown 30s). Fix: timers no longer gate
|
||||
shutdown (`live == 0` already implies nothing a timer could wake); heap is
|
||||
cleared on exit. Independent of the supervisor work.
|
||||
- Tests: all-restart (sibling cycled despite clean exit), suffix-restart
|
||||
(prefix child left alone), reverse-order teardown.
|
||||
|
||||
Original plan, for reference:
|
||||
- one_for_all: on any child failure, `request_stop` all siblings, await their
|
||||
termination signals, restart all per spec.
|
||||
- rest_for_one: stop+restart the failed child and those started after it.
|
||||
- Supervisor shutdown: stop children in reverse start order.
|
||||
- Decide signal surface: add `Signal::Stopped(pid)` + `DownReason::Stopped`
|
||||
rather than folding into Exit (clearer for the supervisor's await logic).
|
||||
- Tests: all-restart, suffix-restart, reverse-order shutdown.
|
||||
|
||||
### 3. Links + trap_exit ✅ DONE (`6581484`)
|
||||
- `Slot.links: Vec<Pid>` (bidirectional); `link`/`unlink`; `trap_exit()` flag
|
||||
lives on `Actor` (fresh per spawn → a restarted child starts un-trapped, and
|
||||
no fourth slot-reset site).
|
||||
- On finalize, reverse links are cleared under the lock (always — keeps the
|
||||
cascade acyclic), then for each linked peer: abnormal death (`Panic`/
|
||||
`Stopped`) → `request_stop(peer)` unless peer traps, in which case deliver an
|
||||
`ExitSignal` *message* instead. Normal exit does NOT propagate. Linking an
|
||||
already-dead pid delivers an immediate `NoProc` signal (message if trapping,
|
||||
else `request_stop(self)` — not a silent no-op).
|
||||
- Resolved: the trap inbox is a **dedicated** channel (`trap_exit() ->
|
||||
Receiver<ExitSignal>`), distinct from the monitor `Down` channel; `ExitSignal`
|
||||
reuses `DownReason` and carries no panic payload (joiner-only, as with
|
||||
monitors). `spawn_link` deferred to #5.
|
||||
- Tests (`tests/link.rs`): linked pair one panics → other stopped (+Drop ran);
|
||||
trap_exit → other gets a message and survives; normal exit doesn't propagate;
|
||||
dead-pid link stops a non-trapper / messages a trapper; `unlink` prevents
|
||||
propagation.
|
||||
|
||||
### 4. Selective receive (independent track) ✅ DONE (`03f3875`)
|
||||
Shipped. What landed vs the plan:
|
||||
- `Receiver::recv_match(pred) -> Result<T, RecvError>` scans the queued
|
||||
`VecDeque` front-to-back, removes+returns the first match, leaves the rest in
|
||||
arrival order; parks and re-scans when nothing matches. `try_recv_match` (the
|
||||
non-blocking variant, mirroring `try_recv`) rolled in same commit.
|
||||
- Wakeup turned out cheaper than feared: `Sender::send` *already* took
|
||||
`parked_receiver` on every push, so "wake on ANY send" needed no send-side
|
||||
change. The one real edit was relaxing `Sender::drop` to unpark the parked
|
||||
receiver on the last-sender drop regardless of queue emptiness — a selective
|
||||
receiver can park on a non-empty no-match queue and must wake to observe
|
||||
closure. No-op for plain `recv` (only ever parks on an empty queue); stress
|
||||
suite stays green.
|
||||
- `pred` is `Fn(&T) -> bool` (not `FnMut`) on purpose: it's re-run from scratch
|
||||
on every scan, so a stateful predicate would re-count surprisingly. It runs
|
||||
under the channel lock — keep it cheap/pure, don't re-enter the channel.
|
||||
- Close semantics: `recv_match` returns `Err(RecvError)` only when closed AND no
|
||||
queued message matches; a match is still returned on a closed channel.
|
||||
Non-matches are left for a later `recv`.
|
||||
- Tests (`tests/selective_recv.rs`): out-of-order match pulled first; non-matches
|
||||
remain in order; park-on-non-empty then wake on a match; closed-with-only-
|
||||
non-matches → Err; closed-but-match-present → match; `try_recv_match` states.
|
||||
|
||||
Original plan, for reference:
|
||||
- Add `Receiver::recv_match(pred) -> T`: scan the queued `VecDeque`, remove+return
|
||||
first match, leave the rest in order; park and re-scan on new arrivals.
|
||||
- This changes channel wakeup: a parked selective receiver must wake on ANY send
|
||||
(not just empty→nonempty) and re-scan. Touches `channel.rs` carefully — the
|
||||
stress tests guard lost-wakeup invariants; keep them green.
|
||||
- Tests: messages arrive out of interest-order; match pulled first; non-matches
|
||||
remain for a later `recv`.
|
||||
|
||||
### 5. Grab-bag (each its own small commit)
|
||||
- `spawn_link`: spawn-and-link atomically (deferred from #3); thin wrapper over
|
||||
`spawn_under` + `link`, but do it under one lock so there's no window where
|
||||
the child dies before the link is recorded.
|
||||
- `demonitor`: needs a per-monitor id to remove a specific sender. Decide the
|
||||
monitor API NOW before more code depends on it — likely return a
|
||||
`Monitor { id, rx }` instead of a bare `Receiver<Down>`.
|
||||
✅ DONE (this commit). What landed vs the plan:
|
||||
- `monitor()` now returns `Monitor { id, target, rx }` (added `target` over
|
||||
the sketched `{id, rx}` so `demonitor` jumps straight to the slot instead of
|
||||
scanning every slot for the id). `MonitorId(u64)` is opaque, from a
|
||||
monotonic `next_monitor_id` counter on `SharedState`, bumped under the
|
||||
shared lock in `monitor()` — no atomics, deterministic, never reused.
|
||||
- `Slot.monitors: Vec<(MonitorId, Sender<Down>)>`. The three slot-reset sites
|
||||
were untouched — they `.clear()`/`Vec::new()`, which is element-type-
|
||||
agnostic, so no new reset obligation. `finalize_actor` just destructures
|
||||
`(_, m)` and sends as before.
|
||||
- `demonitor(&Monitor) -> Option<MonitorId>`: `Some(id)` when a live
|
||||
registration was found+removed, `None` when it had already fired (drained on
|
||||
finalize), was `NoProc`, or the slot was reclaimed. Chose `Option<MonitorId>`
|
||||
over a bare bool — names which registration went. Generation half of the pid
|
||||
makes a stale demonitor a clean no-op: a recycled slot index fails
|
||||
`slot_mut`'s generation check, so it can never strip a different actor's
|
||||
monitor.
|
||||
- ⚠️ Reentrancy: the removed `Sender` is `remove`d out of the Vec under the
|
||||
lock but **dropped after the lock is released** — `Sender::drop` can unpark a
|
||||
parked receiver → `with_shared`, and the shared mutex is non-reentrant. Same
|
||||
discipline as `finalize_actor`.
|
||||
- "Flush" (discard a `Down` the target already queued) falls out of dropping
|
||||
the `Monitor`: `demonitor(&m); drop(m)`. That's the cleanup the still-to-come
|
||||
gen_server **call timeout** wants — monitor the server, wait reply-or-Down-
|
||||
or-deadline, then demonitor+drop so a timed-out call leaks no registration
|
||||
and no stale `Down`.
|
||||
- Perf: touches `Slot` + `finalize_actor`, but `chained_spawn`/`yield_many`
|
||||
register no monitors, so the Vec stays empty (take-empty is identical cost,
|
||||
finalize loop runs zero times). before/after `general` probe medians within
|
||||
noise. Tests (`tests/monitor.rs`): demonitor-stops-delivery, one-of-many
|
||||
(siblings untouched), after-fire-is-None.
|
||||
- Named registry: `register(name,pid)`/`whereis`/`send_by_name`; a
|
||||
`HashMap<String,Pid>` in `SharedState`.
|
||||
- gen_server-style call/cast: request-reply correlation as a thin layer over
|
||||
channels (`call` sends `{req, reply_tx}`, awaits `reply_rx`); no runtime change.
|
||||
✅ DONE (`a4fcf6c`). What landed vs the plan:
|
||||
- `GenServer` trait on the state value: assoc `Call`/`Reply`/`Cast` types,
|
||||
required `handle_call`/`handle_cast`, optional `init`/`terminate` hooks.
|
||||
`ServerRef<G>` is a clonable inbox sender + `pid()`; `start` / `start_under`.
|
||||
- One inbox, not two: a single `Envelope { Call(req, reply_tx) | Cast }`
|
||||
channel, dispatched by variant. Forced by no-`select`/no-unified-mailbox —
|
||||
a server can't wait on a call channel and a cast channel at once.
|
||||
- Server-down falls out of channel closure (no monitor needed): `send` fails
|
||||
if the inbox is gone; the reply sender drops on the server's unwind so a
|
||||
parked caller wakes to `Err`. Both → `Call/CastError::ServerDown`.
|
||||
- `terminate` runs via a drop guard → fires on *every* exit path (clean inbox
|
||||
close, handler panic, `request_stop`), not just the clean one. Caveat: it
|
||||
may run mid-unwind, so keep it non-blocking (a panic inside it during an
|
||||
unwind double-panics → abort).
|
||||
- No `handle_info`, no call timeout — both deferred to land with timeouts
|
||||
(`handle_info` needs the still-unmade cross-channel mailbox merge; a call
|
||||
timeout needs a per-`recv` deadline / `Signal::Timeout`).
|
||||
- Pure additive layer (no Slot/scheduler/spawn/finalize change) → no perf
|
||||
check run. Tests (`tests/gen_server.rs`): cast→call roundtrip, init/terminate
|
||||
ordering, both server-down paths.
|
||||
- Docs: README now points at the experimental, untested aarch64 port on the
|
||||
`arm-port` branch. The module table still calls `context` x86-64-only — true
|
||||
for `master`, since the `src/arch/` split rides on `arm-port`. Fold the arch/
|
||||
split and ARM64-supported wording into the README module table + build section
|
||||
once `arm-port` is validated on hardware and merged.
|
||||
|
||||
## Gotchas / invariants (respect these)
|
||||
|
||||
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
|
||||
`with_shared`. NEVER send on a channel while holding the shared lock. Pattern:
|
||||
`mem::take` the senders/data under the lock, send after releasing. See
|
||||
`finalize_actor` (supervisor signal + monitor Downs both sent post-lock).
|
||||
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
|
||||
Done/outcome → recycle stack (post-lock) → deliver supervisor Signal + monitor
|
||||
Downs (post-lock) → unpark joiners → reclaim slot iff `outstanding_handles==0`.
|
||||
Death notifications always precede reclamation, so a pid carried in a
|
||||
Signal/Down is still matchable even as its slot is about to be reused.
|
||||
- **Slot lifecycle is reset in THREE places** — `Slot::vacant()`,
|
||||
`reclaim_slot()` (runtime.rs), and the slot-init block in `spawn_under`
|
||||
(scheduler.rs). Any new Slot field must be reset in all three (monitors was).
|
||||
- **Pid = (index, generation);** stale handles caught by generation mismatch in
|
||||
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
|
||||
- **No `select`, no unified per-process mailbox.** Why the supervisor uses the
|
||||
single `supervisor_channel` funnel rather than N monitor channels. trap_exit
|
||||
resolved this by giving each trapping actor a dedicated `Receiver<ExitSignal>`
|
||||
inbox (see #3); selective receive (#4) stayed *per-channel* (`recv_match`
|
||||
scans one channel's queue) rather than introducing a cross-channel mailbox —
|
||||
if selective receive ever needs to span the monitor/trap inboxes too, that
|
||||
cross-channel merge is the still-unmade decision.
|
||||
- **Cooperative-only**: preemption and (future) cancellation both depend on the
|
||||
actor reaching `check!()`/yield/alloc/blocking points.
|
||||
- `run()` is single-thread (`Config::exact(1)`); tests rely on deterministic
|
||||
single-thread ordering (parent runs until it parks). Multi-thread via
|
||||
`runtime::init(Config…)`.
|
||||
|
||||
## Workflow expectations (from the human)
|
||||
|
||||
- TDD: write the failing test first, then implement.
|
||||
- Commit incrementally with conventional-commit messages; keep each commit a
|
||||
reviewable unit (they diff in their IDE and are the filter to the codebase).
|
||||
- Run the full suite before each commit; check perf when a change touches
|
||||
Slot/scheduler/spawn/finalize hot paths.
|
||||
Reference in New Issue
Block a user