Compare commits

..
38 Commits
Author SHA1 Message Date
Claude 8e5b754249 docs(roadmap): record the chunk-2 session candidates — no-io idle sleep gap, cross-thread unpark
The terminal-wake fix cannot reach the (deadline, no-io) idle branch:
thread::sleep, no pipe to write. Documented as a Later item with the
two candidate mechanisms (clamped sleep vs condvar/futex), plus the
cross-thread-unpark observation from the same session.
2026-06-11 21:15:45 +00:00
Claude 7bab4d23ea fix(scheduler): entry-side check_cancelled in park_current — stop against a QUEUED actor was lossy
request_stop sets the flag and issues a wildcard unpark, but unpark
no-ops on a Queued actor (the pending run "is" the wake). If the
actor's first action on resume is a blocking park — no allocation, no
check!() on the way — the wake-side-only check in park_current is
unreachable: the actor parks with the stop flag already set and no
wake is ever coming. The runtime then idles forever.

Fix: check_cancelled at the ENTRY of park_current, before the switch.
The remaining window (flag lands after the entry check, before the
park) is covered by the existing protocol: the stop's unpark finds
Running / the prep-to-park window, sets Notified, and the park-return
re-queues into the wake-side check. Unwinding from the entry is the
same unwind path as the wake side; leftover wait registrations are
stale-epoch / dead-generation and self-clean at their wakers' failed
CAS.

This is bug #1 of the urus chunk-2 session (bug #2, the terminal
wake, is the preceding commit). Repro: tests/cancel.rs
stop_flagged_while_queued_lands_at_first_park — deadlocks without the
fix (watchdog-bounded), passes in <50ms with it.
2026-06-11 21:15:03 +00:00
Claude eddf3fe929 fix(runtime): wake idle sibling schedulers at termination
An idle scheduler thread blocks in poll_wake on a snapshot of
peek_deadline / io_outstanding. Only the io threads ever write the wake
pipe (on completion push); enqueue does not. Mid-flight that is masked —
the thread that caused an enqueue is awake and processes it — but at
termination the snapshot can go terminally stale, two ways:

- Stale io_outstanding (infinite hang): an actor parked in wait_readable
  is request_stop'ped. Cancellation deregisters the waiter and produces
  NO completion, so the wake pipe is never written. A sibling blocked on
  io_outstanding > 0 with no timers pending sits in poll(-1) forever.

- Orphaned timer deadline (finite stall): an actor cancelled out of a
  long sleep leaves its timer entry behind. Clearing the wheel at
  AllDone doesn't help a sibling that already blocked on that deadline;
  it sleeps it out in full.

In both cases the remaining actors finish on the other scheduler thread,
which reaches the live==0 && io_out==0 verdict and returns without
waking the blocked one. Runtime::run then stalls or hangs in its worker
join. Found via urus's graceful-shutdown tests (~10% flake, one test per
variant); confirmed by gdb dumps of hung processes showing run() in
JoinHandle::join over a sibling in poll(wake_fd, 59750ms) resp. poll(-1).

Fix: Io::wake() writes the wake pipe directly; the AllDone arm calls it
after the timer clear. One byte wakes every poller; each re-runs the
verdict, independently reaches AllDone, and re-wakes — idempotent.

tests/terminal_wake.rs reproduces both variants deterministically: the
root busy-spins (no timer entries, occupies one scheduler thread) so the
sibling settles into the stale idle wait before the stop is issued. Both
hang without the fix and pass in <0.5s with it.

Known residual gap, deliberately unfixed: the timers-pending /
no-io-subsystem idle branch blocks in thread::sleep and has no wake
mechanism at all — the same stall exists for runtimes that never
initialize io. Roadmap candidate alongside cross-thread unpark and
entry-side check_cancelled in park_current (the lossy-QUEUED-stop bug).
2026-06-11 21:10:30 +00:00
smarm 51f1a61a40 perf: check in cleaner baseline 2026-06-11 23:07:23 +02:00
Claude 1ed179fc12 benches: stable medians via independent process sets in sweep.py
Replace single-process run_benches() with a two-layer approach:

- run_benches_once(): one cargo bench invocation with SMARM_BENCH_SETS=1,
  yielding one median per label from ITERS raw samples.
- run_benches(): calls run_benches_once() BENCH_SETS=5 times, accumulates
  the per-set medians as independent samples, and returns the median of
  those 5 values.

Each set is a fully isolated process (fresh PID, cold caches, independent
OS scheduler context, new warmup run), so the 5 samples are statistically
independent. Correlated noise within a single sustained process run — CPU
frequency scaling, sticky OS scheduler decisions, warm branch predictors —
no longer contaminates the final median.

The previous fix (SMARM_BENCH_SETS loop inside run_n) collected more
samples but within one process, giving correlated samples that were still
vulnerable to a noisy measurement window for any one label.
2026-06-11 21:01:28 +00:00
Claude f9f60a43d1 benches: add SMARM_BENCH_SETS (default 5) for stable medians
Each run_n call now collects ITERS × SETS raw samples across all sets
(one warmup before the first set, discarded), then takes a single global
median. Previously only ITERS=15 samples were taken per bench invocation,
which produced noisy results in sweep.py regress.

SMARM_BENCH_SETS is read from the environment; the default of 5 gives
75 samples per bench row (5×15), matching the stability of the multi-run
bash approach without requiring multiple cargo bench invocations.

Also moves scripts/bench_rq.sh into benches/ alongside sweep.py.
The cd "$(dirname "$0")/.." path logic is unchanged.
2026-06-11 20:50:29 +00:00
smarm 37d931968e fix(scheduler): align(64) on SchedulerStats to prevent false sharing
Adding slot_hits + slot_displacements in RFC 005 grew SchedulerStats
from 16 to 32 bytes — exactly 2 per cache line. run_queue_len (written
on every push/pop by thread N) then false-shared with fields of thread
N+1, causing ~45-60% yield-storm regression at t≥8 visible equally in
slot-on and slot-off paths.
2026-06-11 22:37:47 +02:00
Claude 2708042990 feat(scheduler): RFC 005 wake slot — per-scheduler capacity-one wake cache
A thread-local Cell<Option<Pid>> per scheduler, checked before the shared
run queue. Runtime-selected via Config { wake_slot: bool }, default OFF
until the slot shootout accepts it (one binary benches both arms).

Push policy: slot-eligible iff the wake originates from actor context
(current_pid().is_some()) — the slot push replaces run_queue.push at the
tail of the unpark protocol's Parked → Queued CAS, so at-most-once-enqueued
holds verbatim as (slot ⊕ shared queue). Scheduler-context wakes (timer/IO
drain) and spawns always go shared (spawns never reach unpark_inner at all).
Displacement is Go semantics: newest wake takes the slot, occupant pushed
shared — moved, never copied.

Pop order: slot, then shared. Slot-popped actors skip reset_timeslice() and
inherit the waker's remaining slice; a handoff chain is bounded by one
slice, after which the preempt-yield re-enqueue goes shared (a yield is not
a wake) — the one-slice starvation bound, zero new counters. Idle and
AllDone are only reachable with an empty local slot by pop order; an
occupied slot elsewhere holds a Queued (live) actor, so the counter-first
termination argument is untouched.

Observability: per-thread slot_hits / slot_displacements (reset at run()
start so post-run stats() reads are per-run), SlotPush/SlotPop trace events.

Bench plan (roadmap v0.9 item 2): rq_runtime gains the slot on/off
dimension (SMARM_BENCH_SLOT, default "0 1") — ping-pong-pairs is the
target metric, yield-storm the regression guard, spawn-storm the
neutrality check. RQCSV grows a slot column; RQSLOT lines carry the
counters; bench_rq.sh aggregates both. Tests pin the push policy through
the counters (actor-context hits, spawn/join bypass, displacement,
default-off, per-run reset).
2026-06-11 20:20:07 +02:00
smarm f09e992f32 commit new baseline before perf work 2026-06-11 20:18:51 +02:00
Claude 393cdd01f4 feat(select): fd arms in select + timed fd waits (RFC 008 phase 1)
FdArm composes fd readiness with channel arms on one wait epoch.
Selectable grows fallible sel_register and an eager-cleanup hook;
losing/stop-unwound/timed-out fd arms are unregistered (waiters entry
+ kernel ONESHOT) so the fd is never poisoned. try_select /
try_select_timeout surface registration errors (EBADF, EMFILE,
AlreadyExists) instead of RFC 008's permanently-ready lean, which
would busy-loop a healthy-but-unregistrable fd; select/select_timeout
stay infallible for channel-only arms. Adds wait_readable_timeout /
wait_writable_timeout as one-arm selects.

Known benign race (pre-existing, slightly widened): a queued FdReady
racing the cleanup DEL can spuriously wake a fresh waiter on that fd;
absorbed by select's defensive re-loop. Fixable by epoch-stamping
completions.
2026-06-11 11:28:58 +00:00
Claude 4e4b617559 docs(roadmap): restructure around the rq shootout verdict
Shootout (6d9f369 harness, 24-core sweep) landed in RFC 005's World 3:
queue variants indistinguishable in rq_runtime; per-wake latency is the
ceiling. Record the queue-topology closure (rq-mutex frozen as default,
benchmark-driven reopening only), define v0.9 as the wake-path latency
cycle (RFC 005 slot -> bench -> RFC 004 spinning workers -> bench +
interaction pass), and add per-switch cost to Later pending a profiling
spike + RFC.

Completed cycles compacted to the two trailing minors (v0.7, v0.8);
earlier history lives in git.
2026-06-10 21:58:40 +00:00
Claude 7a244d057c docs(deep-dive): update to v0.8 — slot slab, park-epoch, run queue variants, gen_server, OTP
- Replace v0.3 content throughout; bump version label to v0.8
- New SVG: Krebs-cycle scheduler loop (6 stations, actor handoff as
  energy exchange, finalize leaving the cycle)
- New SVG: gen_server dual-plane (model vs implementation mapping)
- New SVG: slot state machine with packed word, park-epoch and all
  transitions (loom theorems annotated)
- Updated SVG: module map (22 modules, 4 layers incl. slot_state,
  run_queue, raw_mutex, sync_shim, monitor, link, registry, trace)
- Updated SVG: dep graph (runtime hub, OTP layer on public surface)
- New sections: Slot State & Park-Epoch, Run Queue variants,
  GenServer, OTP (monitors/links/supervisors/registry)
- Updated: init (slab allocation, counter-based termination), spawn
  (closure in slot AtomicPtr, stack pool, live_actors ordering),
  preemption (stop sentinel + StopSentinel/Outcome::Stopped),
  IO (epoch-matched completions, fd-leak fix), gotchas (lost-wakeup
  and global-mutex flipped to green; 4 new cards)
- Yield-sources table grown to 9 rows (select, recv_timeout,
  call, request_stop)
- Fix: swap local css2.css font ref for Google Fonts CDN
2026-06-10 21:44:09 +00:00
Claude 0e3df6ec5b docs(roadmap,readme): v0.8 complete — record outcomes and deviations 2026-06-10 15:08:04 +00:00
Claude f6969e538b fix(io): deregister a stopped actor's fd wait on the unwind path
A stopped actor unwinding out of wait_fd's park leaked its waiters entry
and the kernel-side EPOLLONESHOT registration; the stale entry then failed
every future wait_*() on that fd with AlreadyExists (the defensive bare DEL
in epoll_register sits behind the contains_key check, so it never ran).

Fix where the invariant breaks: a drop guard in wait_fd, armed after a
successful register and forgotten on the normal wake (where FdReady already
removed + DEL'd). On unwind it cleans up iff the entry is still this wait's
(pid, epoch) — an entry consumed by a racing FdReady means the fd may carry
another actor's fresh registration, which must be left alone. Closes the
v0.2 fd-hygiene TODO.
2026-06-10 15:07:35 +00:00
Claude 24b95c99ae feat(gen_server): handle_down — runtime monitor forwarding via ServerCtx/Watcher
Monitors are created at runtime (you watch a worker you just spawned in a
handler), so down arms can't ride the static info list. init grows a
&ServerCtx parameter (breaking; no-op default) whose clonable Watcher hands
Monitors to the loop over a control arm; the loop selects the live down
arms ahead of everything else. Arm priority: downs → control → infos →
inbox. A delivered Down retires its arm (monitors are one-shot); a state
that never clones the Watcher closes the control arm after init and the
loop falls back to the plain-inbox park.
2026-06-10 14:59:44 +00:00
Claude e5d1b3b54b feat(gen_server): handle_info — static info arms selected ahead of the inbox
type Info + handle_info (no-op default) on GenServer; ServerBuilder
(with_info/under/start) so start variants don't multiply — start/start_under
stay as wrappers. The loop selects [infos.., inbox] in priority order when
info arms exist, and keeps the plain recv() park when none do. Closed info
arms are silently removed (closed-arm-is-ready-forever); a closed inbox
still means graceful shutdown.
2026-06-10 14:56:51 +00:00
Claude e545f818fa docs(roadmap): v0.8 plan — handle_info/handle_down on select; io fd hygiene 2026-06-10 14:53:10 +00:00
smarm a69b296190 docs(roadmap,readme): v0.7 complete — record outcomes and deviations
Gotchas now lead with the consuming-wake invariant ('the only wildcard wake
is request_stop') and the obligations it puts on future wakers. Deviations
recorded: the no-park exit's retire_wait/clear_notify (a protocol piece the
plan missed), Drop-guard deregistration superseded by own-pid assert
relaxation, and closed arms being ready forever.
2026-06-10 07:48:27 +00:00
smarm a0a93b61bb feat(channel): select_timeout — bounded select on a stateless stamped timer arm
The deadline is one more stamped waker on the select's wait epoch: a unit
TimerTarget whose on_timeout is a bare unpark_at. Nothing registers in any
arm for it, so there is nothing to cancel or leak — an arm winning leaves
the timer entry to die at its epoch CAS; the timer winning leaves the arms'
registrations to self-clean like any select loser's. Classification is pure
channel state (wakes are precise): some arm ready -> Some(first, priority
order); none -> the timer was the only remaining stamped waker -> None.
Message-first on a raced deadline, same as recv_timeout.

Registration pass factored into register_arms, which owns the retire_wait
obligation on the ready-now exit for both entry points.
2026-06-10 07:46:52 +00:00
smarm 00128f32a2 feat(channel): select — ready-index wait over multiple receivers
select(&[&dyn Selectable]) -> usize registers (pid, epoch) in every arm
under one wait epoch, parks once, and returns the first ready index in
documented priority order (BEAM-style; no fairness promise). A closed arm
counts as ready — and stays ready forever, so callers drop it from the set
once observed. Losing arms need no cancellation pass: the winning wake
consumed the epoch, so their registrations die at their wakers' failed CAS
or are overwritten by the receiver's next wait on that channel (the
single-receiver debug_asserts relax to 'none or own pid' accordingly).

The one genuinely new protocol piece is the no-park exit: returning with an
arm ready at registration time leaves earlier arms holding LIVE-epoch
registrations, whose wakes could fault a later one-shot park as a pending
notification. scheduler::retire_wait closes it — bump the epoch (in-flight
wakes die at their CAS), eat a notification that already landed
(StateWord::clear_notify), then re-observe the stop flag, in that order.
Proved by the new loom theorem retire_eats_late_arm_notification; the
integration probes in tests/select.rs fire stale loser-arm wakes and assert
a subsequent sleep's one-shot park holds its full duration.
2026-06-10 07:45:00 +00:00
smarm 400854ac5d refactor(wakes): epoch-stamp every registration-based wake; retire per-primitive wait seqs
The slot epoch is THE wait identity, so the hand-rolled per-primitive copies
go away: channel loses cur_wait/next_wait_seq/timed_out, mutex loses Wait.seq
and next_seq, TimerTarget::on_timeout takes the epoch. Registrations become
(pid, epoch) — channel parked_receiver, mutex waiters, io fd waiters,
Blocking io completions, sleep timers, joiner lists — and their wakers move
to unpark_at. begin_wait is lock-free, so each primitive opens the wait
inside the same critical section that publishes the registration.

recv_timeout's wake-classification loop collapses: wakes are precise, so
queue → Ok, senders==0 → Disconnected, else Timeout — the 'Defensive'
re-register branch is now unreachable by protocol, not by audit. Same for
Mutex::lock_timeout's one-shot park.

End-state invariant, auditable in one sentence: the only wildcard wake is
request_stop, which is terminal.
2026-06-10 07:15:29 +00:00
smarm 4913835c02 feat(slot_state): park-epoch in the slot word; consuming, epoch-matched unparks
Word repacks to (gen << 32) | (epoch << 8) | state. begin_wait opens a wait
identity; every successful wake consumes it, so at most one wake lands per
wait by construction. unpark grows an Option<epoch> match; RuntimeInner gains
unpark_at. Claim/yield/park transitions become epoch-preserving CAS loops.

Loom: all four prior theorems re-proved on the new word, plus
consumed_epoch_unpark_never_lands — a late waker stamped with a consumed
epoch can neither enqueue nor notify, in every interleaving (the property
select's loser arms and every one-shot park site lean on).

No callers stamped yet; behaviour identical. Two transient dead_code
warnings (begin_wait, unpark_at) are consumed by the next commit.
2026-06-10 07:07:44 +00:00
smarm aa295582c4 docs(roadmap): v0.7 plan — select via epoch-stamped consuming wakes 2026-06-10 07:02:03 +00:00
smarm 088280f3c9 docs(roadmap,readme): v0.6 complete — record outcomes and deviations
Marks the three v0.6 items done with their hashes and the load-bearing
deviations: two-class lock order replacing the strict leaf rule, the
registry as a bimap with name_of and without send_by_name (no per-pid
mailbox to send to), and call_timeout built on recv_timeout instead of
monitor machinery. Adds the Leaf -> Channel order to the standing
invariants; README module table picks up recv_timeout, call_timeout,
and the registry row.
2026-06-09 22:58:26 +00:00
smarm 90b7040504 feat(gen_server): call_timeout via recv_timeout on the reply channel
Deviates deliberately from the roadmap's monitor-based sketch
(monitor + wait reply-or-Down-or-deadline + demonitor): server death is
already observable on the reply channel itself - the reply sender drops
as the server unwinds, closing the channel and waking the parked caller.
So a bounded call is exactly recv_timeout on the reply channel, mapping
Disconnected -> ServerDown and Timeout -> Timeout. No registration
exists, so a timed-out call leaks nothing by construction - the
guarantee the monitor design had to engineer.

Semantics on timeout match Erlang: the request stays in the inbox and
is still handled; the late reply's send fails harmlessly against the
dropped reply receiver. CallTimeoutError keeps ServerDown and Timeout
distinguishable; the infallible call() is unchanged.

Tests: reply within deadline, timeout against a slow (parking) handler
with elapsed bounds, server survival across an abandoned call (late
reply discarded, bounded and unbounded calls keep working), and
ServerDown-not-Timeout for both death paths (mid-call panic and
already-gone inbox).
2026-06-09 22:57:31 +00:00
smarm 134ff52c8a feat(channel): recv_timeout - bounded receive on the WaitTimeout machinery
The per-recv deadline the roadmap deferred, built exactly the way
Mutex::lock_timeout already works: register the wait with a per-wait
seq, arm a timer::Reason::WaitTimeout with the channel inner (now a
TimerTarget) as the target, park. On expiry on_timeout cancels the wait
only if that same seq is still parked; satisfied or abandoned waits
leave a stale heap entry that no-ops on seq mismatch, per the
no-cancellation convention in timer.rs.

Race resolution is message-first: a send that lands by the time the
woken receiver runs is delivered even if the deadline had also passed.
Closure is reported as RecvTimeoutError::Disconnected, keeping timeout
and server/sender death distinguishable for callers (gen_server call
timeout builds on exactly this distinction next).

The timer is armed outside the channel critical section (the timers
lock must never nest under a Channel-class lock); the unpark race this
opens is absorbed by the RunningNotified protocol.

Tests: immediate delivery, actual timeout (with elapsed check), prompt
wake on send, Disconnected on close, zero-duration poll, post-timeout
seq isolation (stale entry must not cancel later waits), and a
4-scheduler mixed-outcome run (12 fed / 12 timed out).
2026-06-09 22:56:19 +00:00
smarm 2c7cf0b811 feat(registry): named pid registry as a bimap
register/whereis/unregister plus the inverse lookup name_of. Erlang
semantics: one name per pid, one pid per name, registering over a live
binding errors; same-binding re-register is an idempotent Ok.

Bimap = two HashMaps held in exact inverse under one Leaf RawMutex in
RuntimeInner; the invariant is debug-asserted on every mutation. The
inverse direction costs one extra String per binding and buys O(1)
pid->name for supervisors/tracing/diagnostics.

Cleanup is lazy: no finalize_actor hook, no Slot field (which would buy
into the reset-in-three-places invariant). Liveness rides the
generation-checked slot word - pids are never reused, so a stale binding
is detectable, never misdirected - and bindings to dead actors behave as
absent, pruned on contact by whereis/name_of/register/unregister.
Liveness checks under the registry lock read only the atomic slot word,
so the leaf rule holds trivially.

send_by_name is deliberately absent: smarm has no per-pid mailbox, so
there is nothing generic to send *to* - the registry yields a Pid,
usable with monitor/link/request_stop and as routing for user channels.

Tests: roundtrip both directions, idempotence, NameTaken on live holder,
one-name-per-pid, NoProc, death-evaporates-binding (via lookup pruning
and via register's own eviction path), unregister, cross-actor lookup
wired into request_stop, and a 4-scheduler name-contention churn test.
2026-06-09 22:53:43 +00:00
smarm 8ff6cf4afd feat(channel): migrate Inner to RawMutex; two-class lock order (Leaf -> Channel)
Phase 2 surfaced the hole this closes: channel guards were held with
preemption enabled, so a timeslice switch inside a channel critical
section could release the pthread mutex from a different OS thread (UB).
RawMutex disables preemption for the guard span and is
cross-thread-release sound by construction; poison goes away with it.

The strict single-class leaf rule cannot survive the migration: finalize
clones the supervisor/trap senders and monitor() clones the Down sender,
all under a cold lock, and those senders live in the slot - the nesting
is structural. The leaf check is therefore generalized to two classes:
Leaf (cold locks, free list, stack pool) and Channel. Order is
Leaf -> Channel, at most one of each; both directions of violation are
debug-asserted at the acquisition site. Channel critical sections call
only the lock-free unpark protocol, so the order is acyclic.

Tests: class-ordering unit tests in raw_mutex (allowed nesting + all
three rejected shapes), plus a multi-scheduler integration test driving
channels through monitor churn and actor death so any ordering
regression trips the debug assert instead of deadlocking.
2026-06-09 22:51:37 +00:00
smarm d789d301e0 docs(roadmap): split v0.6 into up-next / later; add RFC 004 summary
Up next: channel mutex migration, named registry, gen_server call timeout.
Later: RFC 004 (tunable scheduler idle policy), unbounded slab, handle_info,
IO fd hygiene, arm-port.
2026-06-09 21:58:42 +00:00
smarm 5428697f9e docs(roadmap): consolidate task.md + ROADMAP_v0.5.md into ROADMAP.md
Single source of truth: v0.4 history, v0.5 phases, v0.6 open tracks, and
invariants/gotchas. Old task.md and ROADMAP_v0.5.md removed.
2026-06-09 21:53:01 +00:00
Claude a7ca6646d7 docs(bench,test): READMEs for benches/ and tests/
benches/README.md: catalog of the two bench families (cross-runtime
comparisons + the v0.5 run-queue shootout), how to run the shootout
(scripts/bench_rq.sh, SMARM_BENCH_* knobs, the --no-default-features dance,
summary.csv/RQCSV format), honest-numbers caveats (core count is the
experiment; striped losing at low thread counts is expected), and the
conventions for adding a bench.

tests/README.md: catalog grouped by layer (low-level units / feature areas /
regression+stress), the run matrix (debug-first — that's where the invariant
asserts live — three queue variants, release, loom, trace), why loom tests
live in src/ rather than tests/, and the house conventions: one runtime per
test, oversubscription on purpose, regression tests validated against the
reintroduced bug, odds-stacking for stochastic tests, ordering-not-duration
time assertions, and new-invariant = assert + loom model.

Also fixes a stale header in tests/mutex.rs that claimed 'loom::Mutex' —
it tests smarm::Mutex, and the old name is actively confusing now that loom
is real in this repo.
2026-06-09 21:35:08 +00:00
Claude 039703dbeb feat(safety): phase 5 — loom model checking + invariant audit/assertion sweep
State machine extracted to src/slot_state.rs as a standalone unit (StateWord:
publish_queued / try_claim / yield_return / park_return / unpark / set_done /
reclaim, plus the Status view for cold paths). runtime.rs keeps the protocol
rationale and consumes the mechanism; every transition self-asserts its
precondition per the assert-the-invariants rule.

src/sync_shim.rs: std vs loom indirection (atomics + UnsafeCell with the
with/with_mut access API) for the two loom-modeled modules. Loom models run
the PRODUCTION code, not a replica:
- slot_state: no-lost-wakeup (park vs unpark), two-unparkers-one-enqueue,
  stale-unpark-never-hits-reused-slot (the ABA theorem), unpark-vs-claim.
- run_queue: mpmc exactly-once through a lap wraparound, push/pop race,
  striped two-producer drain.
RUSTFLAGS="--cfg loom" cargo test --lib --release — 7 models, all pass.
RawMutex deliberately not loom-modeled (futexes can't be; textbook mutex3
with stress + unwind coverage).

Assertion sweep (invariants now checked at the point of reliance):
- enqueue debug-asserts the word reads EXACTLY (gen, Queued) — the
  at-most-once-enqueued invariant the ring capacity proof leans on.
- RawMutex enforces the leaf rule mechanically: debug-build thread-local
  held-count, panics at the acquisition that violates it.
- live_actors underflow (double finalize) asserted.
- StateWord transition preconditions asserted (yield/park/done/reclaim/
  publish/claim).

Audit: with_runtime, RawMutex guards, run-queue ops, and trace::record all
gate preemption (and thereby the stop sentinel) for their span; trace was
already self-gating. Sole remaining exception is the channel std MutexGuard
— the documented first post-v0.5 fast-follow.

Review fixes to the branched-in work:
- slot_state::status_for mapped (matching gen, Vacant) to Stale instead of
  unreachable!: Pid::new is public, so a forged/never-issued pid (e.g.
  monitor(Pid::new(5,0)) on a fresh slab) could reach it — "no such actor"
  is the correct total answer; issued pids still can't get there.
- Tightened enqueue's assert from Status::Live to the exact (gen, Queued)
  word its own comment argues for.

Validated: 22 suites in debug (all asserts + leaf counter live) for
rq-mutex/rq-mpmc/rq-striped, release for rq-mutex, smarm-trace build +
stress, loom 7/7.
2026-06-09 21:27:45 +00:00
Claude 6d9f3698d4 feat(bench): phase 4 — run-queue bench harness + shootout driver
- benches/rq_micro.rs: raw-structure microbench, threads x producer:consumer
  ratio sweep. Benches all three queue types in one binary (they compile in
  every build; only the runtime alias is feature-selected), so no rebuild
  dance. Queues sized to the op count so the occupancy contract is met.
- benches/rq_runtime.rs: whole-runtime benches with the selected variant:
  yield-storm (pure queue churn), ping-pong-pairs (park/unpark latency),
  spawn-storm (slab + free list + queue under churn). Scheduler-count sweep.
- scripts/bench_rq.sh: rebuilds rq_runtime per rq-* feature, runs rq_micro
  once, aggregates RQCSV lines into bench_results/summary.csv.
- All knobs via SMARM_BENCH_* env vars; house table format + machine lines.
- run_queue module is now #[doc(hidden)] pub (types + push/pop/len +
  MpmcRing::with_capacity) solely so the external bench binary can drive the
  raw structures.

docs(roadmap): phase 4 ticked (harness done; numbers from the 20-core box).
New fast-follow per review: assert the invariants we lean on — debug_assert!
on hot paths, loud assert!/panic on cold ones, at the point of reliance;
sweep existing code during the phase-5 audit, adopt as house style.

Validated end-to-end at smoke scale on the 1-core sandbox: full driver run,
24-row summary.csv across micro (3 structures x sweeps) and runtime
(3 variants x 3 benches x thread sweep).
2026-06-09 20:44:10 +00:00
Claude 1b3b618aa7 feat(runtime): phase 3 — pluggable run queue (rq-mutex / rq-mpmc / rq-striped)
src/run_queue.rs: the run queue extracted behind a compile-time-selected
type alias; mutually-exclusive cargo features with compile_error! guards
(zero or >1 selected). No runtime dispatch. All variants compile in every
build so their unit tests always run; the feature only picks the alias.

- rq-mutex (default): Mutex<VecDeque>, the control/baseline.
- rq-mpmc: hand-rolled Vyukov bounded MPMC ring, per-cell sequence numbers,
  padded enqueue/dequeue counters. Strict FIFO, lock-free.
- rq-striped: M Vyukov rings (M = thread_count rounded up to pow2),
  fetch-add ticket distribution, probe-from-home on push. Relaxed FIFO with
  stripe-bounded skew; Σcapacity ≈ 2×max_actors so the probe terminates.

Capacity soundness: occupancy ≤ max_actors by the at-most-once-enqueued
invariant + slab cap, rings sized ≥ that bound, so push is infallible; a
full ring panics as a double-enqueue invariant violation rather than spin.

Two contracts the extraction made explicit (documented + debug-asserted):
- Queue ops require preemption disabled: a producer suspended between
  claiming a cell and publishing its sequence stalls every consumer behind
  it — livelock, since the suspended actor's own resume entry is behind the
  hole. Structurally guaranteed since the phase-2 with_runtime NoPreempt fix.
- pop()==None is a snapshot, not a fence. Termination is counter-first:
  every queue entry's target stays Queued (hence live) until that entry is
  popped, so live==0 alone implies nothing actionable is or can be queued;
  argument rewritten at the schedule_loop site. SharedState and with_shared
  are deleted — nothing global is mutex-guarded on the run path under the
  ring variants.

Validated: 22 suites green per variant (release for all three; debug with
live asserts for all three), ring unit tests (FIFO, lap wraparound,
4p/4c exactly-once, skewed drain) in every build, both compile_error!
guards verified to fire.
2026-06-09 20:34:32 +00:00
Claude a78e17e1eb docs(roadmap): phase 2 done; record the thread-local-guard rule; sharpen the channel fast-follow rationale (cross-thread std MutexGuard release is UB) 2026-06-09 20:09:09 +00:00
Claude 5e0c9d45da feat(runtime): phase 2 — fixed slab, per-slot packed-atomic state machine, raw cold locks
The slot table split (ROADMAP_v0.5 phase 2): slot lookup is lock-free, the
run path (yield/park/unpark/pop/resume) takes zero locks beyond the queue
mutex itself, and SharedState shrinks to { run_queue }.

Core pieces:

- src/raw_mutex.rs: hand-rolled 3-state futex mutex (Drepper mutex3),
  non-poisoning; the guard enters NoPreempt, which both bounds hold times and
  structurally closes the unwind-under-lock hole (the stop sentinel shares
  the gate). Used for per-slot cold data, the free list, and the stack pool.

- Fixed slab Box<[Slot]> (default max_actors = 16_384, ~4 MiB, align(128)
  against false sharing). Slots never move -> stable addresses, lock-free
  index. Exhaustion panics loudly, naming Config::max_actors(n) as the fix.
  Unbounded/segmented slab stays deferred (see ROADMAP).

- Per-slot AtomicU64 packing (generation << 32 | state), states
  Vacant/Queued/Running/RunningNotified/Parked/Done. Every transition CASes
  the packed word, so the generation check is atomic with the transition: no
  ABA, no spurious unparks on recycled slots. RunningNotified replaces the
  pending_unpark bool (the lost-wakeup window is a state, not a flag) and
  uniformly fixes a LATENT LOST WAKEUP in the old Blocking-IO completion
  path, which set the result for a still-Running actor without flagging it.

- Invariant: a pid is in the run queue at most once (pushes pair 1:1 with
  transitions into Queued; only the scheduler does Queued->Running). This is
  what makes phase 3's bounded rings sound.

- sp -> relaxed AtomicUsize; stop flag + first-resume closure as AtomicPtrs
  (closure double-boxed for a thin pointer, swap-to-take): the resume path is
  fully atomic.

- finalize_actor: Done published under the dying slot's cold lock (join's
  check-or-register is linearized by it); link cascade locks peers ONE AT A
  TIME (cold locks are leaves) with the acyclicity argument written at the
  site; link() registers on the target first, then self (stale self-entries
  are benign, every walk re-verifies the peer's word).

- Termination by counters: live_actors incremented in spawn pre-enqueue,
  decremented at the very END of finalize after all wakeup enqueues; exit on
  io_out == 0 (read before the queue lock, phase-1 ordering) && queue empty
  && live == 0. Soundness note at the site: any enqueue targets a live actor.

- spawn boxes the closure and acquires the stack BEFORE any runtime lock: no
  allocation ever happens under a global lock anymore.

- with_runtime/try_with_runtime now enter NoPreempt for their full span.
  This fixes a bug the rework exposed: install_actor allocated with
  preemption enabled while the RUNTIME RefCell borrow was live; a timeslice
  preemption there migrates the actor across OS threads and the borrow guard
  increments one thread's RefCell count and decrements another's — underflow
  to 'permanently mutably borrowed', cascading panics (caught by stress
  suite: deterministic non-unwinding-panic abort in lost_wakeup_many_pairs).
  The old code was safe only by accident; now it's structural.

- Behavior note: request_stop on a RUNNING target now marks it
  RunningNotified, so its next park returns immediately to an observation
  point — faster stop observation; parked/queued/done semantics unchanged.

Validated: 22 suites green in release (1/2/8-thread oversubscribed on the
1-core sandbox) and in debug with all debug_asserts live; stress suite x5.
2026-06-09 20:08:12 +00:00
Claude 453f6f491f fix(runtime): stale-PID pop retries immediately instead of taking the idle path
The phase-1 split filled Idle::next_deadline from the timers lock after the
shared lock was released, which turned the stale-PID branch (formerly a 100µs
poll) into a potential sleep-until-timer-deadline while runnable actors sat in
the queue — a hard stall on exact(1). Stale pops are now PopResult::Retry and
loop without sleeping.

test(poison): strengthen regression coverage. The stop-storm test never fired
the sentinel under a lock (its actors only allocate at lock-free points). Add
self_stop_during_spawn_does_not_poison_shared_mutex: a stop-flagged actor whose
next allocation is the Box::new(closure) inside spawn's with_shared. Validated
both ways: passes with the gate, SIGABRTs (unwind-in-allocator) with the gate
removed. Also alloc_interval(1) so every allocation is an observation point.
2026-06-09 19:08:25 +00:00
Claude 3c7e26bc98 feat(runtime): phase 1 — peel timers/io/monitor-id out of SharedState; gate stop sentinel behind PREEMPTION_ENABLED
- next_monitor_id -> AtomicU64 on RuntimeInner
- timers -> own Mutex<Timers>; io -> own Mutex<Option<IoThread>> (lock order: io-before-shared)
- pending_closures Vec folded into Slot::pending_closure
- termination check reads io liveness before shared; ordering argument documented
- poison fix: check_cancelled() no longer fires while preemption is disabled,
  so a cancellation unwind can never poison a runtime/channel mutex
- regression test: tests/poison_stop.rs
- ROADMAP_v0.5.md added
2026-06-09 18:56:17 +00:00
52 changed files with 7964 additions and 2046 deletions
-2
View File
@@ -1,2 +0,0 @@
[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
+1
View File
@@ -1,3 +1,4 @@
target
Cargo.lock
smarm_trace.json
/bench_results/
+20
View File
@@ -4,12 +4,24 @@ version = "0.4.0"
edition = "2021"
rust-version = "1.95"
[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ["cfg(loom)"] }
[features]
default = ["rq-mutex"]
smarm-trace = []
# Run-queue selection: exactly one, compile-time (see src/run_queue.rs).
# Non-default variants need --no-default-features (features are additive).
rq-mutex = []
rq-mpmc = []
rq-striped = []
[dependencies]
libc = "0.2"
[target.'cfg(loom)'.dependencies]
loom = "0.7"
[dev-dependencies]
libc = "0.2"
tokio = { version = "1", features = ["rt", "rt-multi-thread", "macros", "sync", "time"] }
@@ -41,3 +53,11 @@ harness = false
[[bench]]
name = "tokio_favored"
harness = false
[[bench]]
name = "rq_micro"
harness = false
[[bench]]
name = "rq_runtime"
harness = false
+3 -2
View File
@@ -21,14 +21,15 @@ convenience wrapper around `runtime::init(Config::exact(1)).run(f)`.
| `pid` | `(index, generation)` PIDs; stale handles are detectable, not silent |
| `actor` | Trampoline + `catch_unwind` boundary at the actor entry point |
| `scheduler` | Run queue, slot table, spawn/join, parking, idle path |
| `channel` | Unbounded MPSC channel; `recv` parks the actor |
| `channel` | Unbounded MPSC channel; `recv` parks the actor; `recv_timeout` bounds it; `select`/`select_timeout` park on many receivers at once (ready-index, priority order) |
| `mutex` | `Mutex<T>` with mandatory timeout; FIFO waiters; parks the green thread |
| `timer` | Min-heap of `(deadline, reason)`; `Sleep` and `WaitTimeout` reasons |
| `io` | `block_on_io` for blocking work; `wait_readable`/`wait_writable` + `read`/`write` via epoll |
| `supervisor` | `Signal::Exit`/`Panic`/`Stopped` funnelled to a parent; `OneForOne`/`OneForAll`/`RestForOne` strategies + restart-intensity cap |
| `monitor` | `monitor(pid)``Monitor { id, target, rx }`; one-shot `Down` via `rx`; `demonitor(&m)` tears one registration down; unidirectional death notice |
| `link` | bidirectional `link`/`unlink`; abnormal death propagates (cooperative stop, or an `ExitSignal` message under `trap_exit`) |
| `gen_server` | `call` (sync request-reply) / `cast` (async) over one inbox; `ServerRef` + `init`/`terminate` hooks; server-down via channel closure |
| `gen_server` | `call`/`call_timeout` (sync request-reply) / `cast` (async) over one inbox; `handle_info` over static info arms + `handle_down` via `Watcher`-fed monitors, selected ahead of the inbox; `ServerRef`/`ServerBuilder` + `init`/`terminate` hooks; server-down via channel closure |
| `registry` | `register`/`whereis`/`name_of`: name ↔ pid bimap; lazy generation-checked cleanup |
## Quick taste
+196
View File
@@ -0,0 +1,196 @@
# smarm — Roadmap
## Shipped (compacted — full cycle plans and deviation records live in git history)
Cycles before v0.7 (v0.4 actor primitives, v0.5 runtime decomposition &
pluggable run queue, v0.6 actor ergonomics): see `git log ROADMAP.md`.
### v0.7 — select, on epoch-stamped consuming wakes ✅
A 24-bit park-epoch packed into the slot word gives every wait an identity:
registrations carry `(pid, epoch)`, every successful wake **consumes** the
epoch, stale wakes die at one failed CAS. Subsumed the per-primitive wait
seqs (channel, mutex, timer); the only wildcard wake left is `request_stop`
(terminal). On top: `select`/`select_timeout` — ready-index wait over many
receivers, priority order, no cancellation pass. Loom theorems re-proved on
the new word. Notable deviations: `retire_wait` for the no-park exit path
(plan missed it); the Drop-guard deregistration was superseded by relaxed
single-receiver asserts; a closed arm is ready *forever* (documented gotcha).
Commits `4913835``00128f3`, `a0a93b6`.
### v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅
Spent `select` on the server loop: static info arms (`type Info`,
`ServerBuilder::with_info`) and dynamic monitor forwarding
(`ServerCtx`/`Watcher` + a control arm), priority downs → control → infos →
inbox. Closed the v0.2 fd hole: a drop guard in `wait_fd` DELs the kernel
registration on unwind (the leak was worse than documented — a stale waiters
entry permanently poisoned the fd). Deviation: the plain-inbox fast path
narrowed; servers holding a `Watcher` select forever.
Commits `e5d1b3b`, `24b95c9`, `f6969e5`.
---
## Decision record — queue topology 🔒 CLOSED (2026-06-10)
The run-queue shootout (harness `6d9f369`, 24-core sweep 124 schedulers,
report `bench_report_rq_shootout.html`) landed in **RFC 005's World 3**: the
three queue variants are within 1015% of each other in `rq_runtime` at every
scheduler count ≥ 4, on all three workloads. Striped's real micro-bench
dominance (24.8 M ops/s at 24t, 4.3× over mutex) does not propagate to the
runtime because the queue is not the hot path in any measured workload. USL
fits put σ at 5.57.6 (super-serial) for ping-pong: the ceiling is per-wake
latency through the park/unpark protocol, not queue push/pop.
Consequences:
- **`rq-mutex` stays the default** — simplest correct, no capacity
constraints, locking model already integrated with the preemption-disabled
queue-op invariant.
- **Feature plumbing stays as is.** All three variants keep compiling in
every build; `rq-mpmc`/`rq-striped` remain selectable for benching.
- **Reopening is benchmark-driven only.** The report documents the
conditional upgrade paths if a future workload qualifies: mpmc for
message-passing-dominant loads at N ≤ 8 (1.3× lower σ, 0.16 µs ping-pong
at 1t); striped for high-contention balanced push/pop at N ≥ 16 — neither
is a scheduler workload as measured.
- **Effort redirects to the wake path**: RFC 005 (billed as a latency patch,
per its own World 3 framing), RFC 004, and eventually per-switch cost.
---
## v0.9 — Wake-path latency
Goal: attack per-wake latency — the measured ceiling — with the two specified
mechanisms, each benched against a clean baseline before the next lands.
Order: RFC 005 first (the slot is measured against the just-benched idle
policy), then RFC 004 (measured against the slot-accepted baseline), then an
interaction pass.
Full specs: `rfc_005-wake-slot.md`, `rfc_004-tunable-scheduler-idle-policy.md`
(artefacts.kalsbeek.dev).
### 1. Wake slot (RFC 005)
Per-scheduler, thread-local, capacity-one wake cache, checked before the
shared queue. Runtime-selected via `Config { wake_slot: bool }`, default off
until accepted (one binary benches both arms).
- **Push policy:** slot-eligible iff the wake originates from actor context
(`current_pid().is_some()`). Scheduler-context wakes (timer/IO drain) and
spawns always go shared.
- **Displacement:** newest wake takes the slot, occupant pushed shared (Go
semantics); the branch swap is the fallback if benches look pathological.
- **Pop order:** slot, then shared. Slot-popped actors inherit the waker's
remaining timeslice — a handoff chain is bounded by one slice, so the
shared queue is consulted at least once per slice per scheduler (the
starvation bound, zero new counters).
- **Invariant care:** the slot push replaces `run_queue.push` at the tail of
the `Parked → Queued` CAS; at-most-once-enqueued holds verbatim (pid in
slot ⊕ shared queue). Stall blast radius grows by exactly one actor.
- Counters: `slot_hits`, `slot_displacements`.
### 2. Slot shootout
Extend `rq_runtime`/`bench_rq.sh` with the slot on/off dimension (cheap —
it's a Config knob, not a feature rebuild). Same sweep as the rq shootout.
- **ping-pong-pairs** — target metric; expect the win, single- and
multi-scheduler.
- **yield-storm** — regression guard; yields never touch the slot, any delta
is pop-path overhead.
- **spawn-storm** — neutrality check; spawns bypass the slot by policy.
Acceptance flips the default on and re-baselines for RFC 004.
### 3. Spinning workers (RFC 004)
Bounded spin-before-park for idle schedulers, killing the ~100 µs
`thread::sleep` worst case on cross-thread handoff. Two Config knobs:
`spin_budget_cycles` (0 recovers today's behaviour) and `max_spinners`
(default N/2); one new atomic `n_spinning`. No run-queue changes — lands on
the frozen `rq-mutex` substrate, independent of the slot mechanically.
### 4. RFC 004 bench + interaction pass
Re-run the sweep with spinning enabled, slot on and off. The known
interaction: once idle pickup latency ≪ slice, the slot's latency trade
(occupant waits out the waker's slice while other schedulers idle) may stop
paying — RFC 005's documented escape hatch is Go's "spinners exist → push
shared instead" check. In scope only if the data demands it.
---
## Later
### Unwakeable idle sleep when io is absent (terminal-wake residual)
The `(Some(deadline), None)` idle branch — timers pending, io subsystem never
initialized — blocks in `thread::sleep` with no wake mechanism at all. The
terminal wake (writes the wake pipe at AllDone) cannot reach it: no io, no
pipe. Same stall as the fixed bug, in any no-io runtime: a sibling that
blocked on an orphaned deadline sleeps it out in full after everything else
finished. Candidates, mutually exclusive: (a) clamp the sleep (cheap, but
turns idle into periodic wakeups), or (b) park the branch on a condvar/futex
the AllDone path signals — and at that point consider making the condvar the
idle primitive for the no-io runtime generally (a cross-thread unpark could
signal it too, see below). Decide before any no-io deployment.
### Cross-thread unpark
`RuntimeInner::enqueue` does not wake idle sibling schedulers — only io
completions write the wake pipe. Mid-flight this is masked (the enqueuing
thread is awake and eats the work itself), but it costs parallelism: work
enqueued by a busy thread waits until the sibling's idle poll times out.
Candidate from the urus chunk-2 session; needs bench evidence (does the
shared-queue handoff latency actually show up?) before a mechanism is picked.
### Per-switch cost (context shims, epoch protocol)
The shootout's residual: per-wake latency is 0.160.18 µs at N=1 and
0.81.2 µs at N=8+, dominated by the context-switch shims and the epoch
protocol, not the queue. On current evidence this is the larger constant —
"the whole game" alongside the v0.9 work — but there is no spec yet. Needs a
profiling spike (where do the cycles actually go per park/unpark round-trip)
and then an RFC before it can be scheduled.
### Unbounded / configurable-bounded actor count
Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384).
Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling segment
sizes, append-only) once the cap is actually hit. Do not let it calcify.
### arm-port validation & merge
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
hardware. Build + run full test suite on an aarch64 device; check
`chained_spawn` / `yield_many` bench medians; merge and update README.
---
## Invariants & gotchas (respect these across all cycles)
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark`
`with_shared`. Never send on a channel while holding the shared lock. Pattern:
`mem::take` data under the lock, send after releasing. See `finalize_actor`.
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs →
unpark joiners → reclaim slot if `outstanding_handles==0`. Death notifications
always precede reclamation.
- **Slot lifecycle reset in THREE places:** `Slot::vacant()`, `reclaim_slot()`
(runtime.rs), slot-init block in `spawn_under` (scheduler.rs). Any new `Slot`
field must be reset in all three.
- **Pid = (index, generation).** Stale handles caught by generation mismatch in
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
- **The only wildcard wake is `request_stop`, and it is terminal.** Every
registration-based waker (channel sends, mutex grants, wait-timers, io
completions, joiner wakes, `select` arms) carries the wait's park-epoch
and wakes through `unpark_at`; every successful wake consumes the epoch.
Wakes are therefore *meaningful*: one-shot park sites interpret them
without loops, and `select` needs no cancellation pass. When adding a new
waker, decide which form it is — if its registration handle can outlive
the wait it was created for, it MUST be epoch-stamped; a wait that can
exit without parking MUST `retire_wait` first (see slot_state.rs).
- **`select` exists; a unified per-process mailbox still does not.** The
supervisor keeps its single `supervisor_channel` funnel; `recv_match`
stays per-channel. `select` composes channels at the wait, not into one
queue — gen_server's `handle_info`/`handle_down` (v0.8) are built on
exactly that composition, with documented arm priority (downs → control
→ infos → inbox) instead of mailbox FIFO. A hot higher-priority arm
starves lower ones by design; that's the contract.
- **Cooperative-only.** Preemption and cancellation both depend on the actor
reaching `check!()`/yield/alloc/blocking points.
- **Lock order is Leaf → Channel, one of each at most** (debug-asserted in
`raw_mutex.rs`). Leaf = cold locks / free list / stack pool / registry,
mutual leaves. A channel lock may be taken under a Leaf (finalize/monitor
clone senders living in slots); nothing may be locked under a channel lock.
- **Queue ops require preemption disabled.** A producer suspended mid-publish
stalls every consumer — livelock. `with_runtime`, `with_shared`, and
`RawMutex` guards all disable preemption for their span.
- **`run()` is single-thread** (`Config::exact(1)`); tests rely on deterministic
single-thread ordering. Multi-thread via `runtime::init(Config…)`.
+89
View File
@@ -0,0 +1,89 @@
# Benches
Two families live here: **comparison benches** (smarm vs tokio, predating
v0.5) and the **run-queue shootout** (v0.5 phase 4). All are plain binaries
(`harness = false` in `Cargo.toml`), so `cargo bench` just builds in release
and runs `main()` — no criterion, no magic.
```
cargo bench --bench <name> # one bench
cargo bench # all of them (slow; rarely what you want)
```
## Catalog
| file | what it measures |
|---|---|
| `primes.rs` | Compute fan-out/fan-in: counts primes across W workers. Pure compute throughput + spawn/join/channel cost. |
| `multi_scheduler.rs` | The original cross-runtime matrix: smarm (1 thread / N threads) vs tokio (current_thread / multi_thread) on compute, ping-pong, and spawn throughput. |
| `general.rs` | Workloads where neither runtime has a structural edge. Large gaps here mean real per-task/per-yield overhead differences — watch these for regressions. |
| `smarm_favored.rs` | Workloads the stackful green-thread model is built for. Single-thread numbers isolate per-switch cost from contention. |
| `tokio_favored.rs` | Workloads tokio's model is built for. Expect to lose; the value is knowing *by how much* and catching the gap widening. |
| `rq_micro.rs` | Run-queue **structures** in isolation (no runtime, no actors): push/pop throughput sweeping thread count × producer:consumer ratio. Covers all three queue types in one binary — the types compile in every build; only the runtime's alias is feature-selected. |
| `rq_runtime.rs` | The **whole scheduler** with the compile-time-selected queue: yield-storm (pure queue churn), ping-pong-pairs (park/unpark latency), spawn-storm (slab + free list + queue churn), sweeping scheduler count. Comparing variants requires rebuilding per `rq-*` feature. |
## The run-queue shootout
One command; it rebuilds `rq_runtime` once per queue variant, runs `rq_micro`
once, and aggregates:
```
./scripts/bench_rq.sh
# on a big box:
SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh
```
Outputs land in `bench_results/` (gitignored): one full log per run, plus
`summary.csv` assembled from the machine-readable `RQCSV,...` lines every
config prints alongside the human table.
Manual single-variant runs need the feature dance (features are additive, so
the default `rq-mutex` must be switched off):
```
cargo bench --bench rq_runtime --no-default-features --features rq-striped
```
### Knobs (env vars, all optional)
| var | default | used by |
|---|---|---|
| `SMARM_BENCH_THREADS` | `"1 2 4"` | both — space-separated sweep |
| `SMARM_BENCH_RUNS` | `5` | both — repetitions; the **median** is reported |
| `SMARM_BENCH_ITEMS` | `200000` | `rq_micro` — items per measurement |
| `SMARM_BENCH_YIELD_ACTORS` / `_YIELDS` | `200` / `500` | `rq_runtime` yield-storm |
| `SMARM_BENCH_PAIRS` / `_ROUNDTRIPS` | `32` / `1000` | `rq_runtime` ping-pong |
| `SMARM_BENCH_SPAWNS` | `5000` | `rq_runtime` spawn-storm |
## Reading the numbers honestly
- **Core count is the experiment.** On a 1-core machine (CI, sandboxes) the
sweep only validates the harness and catches gross pathologies —
oversubscribed schedulers measure context-switch noise, not contention.
Variant decisions come from a many-core box.
- The striped queue *should lose* at low thread counts (ticket overhead with
no contention to amortize) — that's expected, not a bug.
- Medians over `SMARM_BENCH_RUNS` absorb scheduling noise but not thermal /
turbo drift; for publishable numbers, pin the CPU governor and run a warmup
pass first.
- `spawn-storm` batches joins (1024 at a time) to stay well under the slab
cap; if you raise `SMARM_BENCH_SPAWNS` massively, that batching is why it
still works.
## Adding a bench
1. `benches/<name>.rs` with a plain `main()`; print the house table (see any
existing bench) and, if it belongs to a sweep, a greppable CSV line with a
distinctive prefix (`RQCSV,` for the shootout family).
2. Register it in `Cargo.toml`:
```toml
[[bench]]
name = "<name>"
harness = false
```
3. Take parameters from `SMARM_BENCH_*` env vars with modest defaults — the
defaults must finish in seconds on one core, the env scales them up on
real hardware.
4. Report **medians**, and keep one measurement = one fresh runtime
(`init(Config::exact(t))` inside the measured closure constructor, the
`run()` inside the timed region) so runs don't contaminate each other.
+142 -142
View File
@@ -2,313 +2,313 @@
"chained_spawn": {
"smarm 1-thread": {
"result": 1000,
"median": 266,
"min": 242,
"max": 351
"median": 413,
"min": 410,
"max": 439
},
"smarm 24-thread": {
"result": 1000,
"median": 742,
"min": 696,
"max": 860
"median": 909,
"min": 888,
"max": 951
},
"tokio current_thread": {
"result": 1000,
"median": 62,
"min": 61,
"max": 68
"max": 62
},
"tokio multi-thread": {
"result": 1000,
"median": 190,
"min": 169,
"max": 207
"median": 197,
"min": 194,
"max": 210
}
},
"yield_many": {
"smarm 1-thread": {
"result": 200000,
"median": 19071,
"min": 18776,
"max": 19396
"median": 16475,
"min": 16393,
"max": 16732
},
"smarm 24-thread": {
"result": 200000,
"median": 172454,
"min": 166246,
"max": 174230
"median": 148708,
"min": 111213,
"max": 156462
},
"tokio current_thread": {
"result": 200000,
"median": 4737,
"min": 4644,
"max": 5065
"median": 4751,
"min": 4740,
"max": 5259
},
"tokio multi-thread": {
"result": 200000,
"median": 8738,
"min": 7852,
"max": 9770
"median": 8320,
"min": 7862,
"max": 8882
}
},
"fan_out_compute": {
"smarm 1-thread": {
"result": 33860,
"median": 13234,
"min": 13196,
"max": 13390
"median": 13453,
"min": 13305,
"max": 15077
},
"smarm 24-thread": {
"result": 33860,
"median": 2244,
"min": 2162,
"max": 2380
"median": 2451,
"min": 2330,
"max": 2520
},
"tokio current_thread": {
"result": 33860,
"median": 14049,
"min": 14035,
"max": 14300
"median": 14019,
"min": 12339,
"max": 14045
},
"tokio multi-thread": {
"result": 33860,
"median": 1474,
"min": 1285,
"max": 1823
"median": 1500,
"min": 1426,
"max": 1600
}
},
"ping_pong_oneshot": {
"smarm 1-thread": {
"result": 1000,
"median": 751,
"min": 727,
"max": 913
"median": 898,
"min": 782,
"max": 920
},
"smarm 24-thread": {
"result": 1000,
"median": 1308,
"min": 1227,
"max": 1396
"median": 1494,
"min": 1489,
"max": 1546
},
"tokio current_thread": {
"result": 1000,
"median": 407,
"min": 400,
"max": 444
"median": 396,
"min": 389,
"max": 409
},
"tokio multi-thread": {
"result": 1000,
"median": 10869,
"min": 8683,
"max": 11688
"median": 10382,
"min": 9559,
"max": 10924
}
},
"spawn_storm_busy": {
"smarm 1-thread": {
"result": 10000,
"median": 112045,
"min": 99936,
"max": 117329
"median": 106232,
"min": 105627,
"max": 107693
},
"smarm 24-thread": {
"result": 10000,
"median": 137105,
"min": 130852,
"max": 147707
"median": 49198,
"min": 48894,
"max": 52606
},
"tokio current_thread": {
"result": 10000,
"median": 1128,
"min": 1123,
"max": 1435
"median": 1140,
"min": 1018,
"max": 1169
},
"tokio multi-thread": {
"result": 10000,
"median": 19674,
"min": 16013,
"max": 27234
"median": 18650,
"min": 16486,
"max": 19396
}
},
"mpsc_contention": {
"smarm 1-thread": {
"result": 320000,
"median": 3667,
"min": 3608,
"max": 4126
"median": 6173,
"min": 5446,
"max": 6226
},
"smarm 24-thread": {
"result": 320000,
"median": 45681,
"min": 31908,
"max": 51287
"median": 36329,
"min": 35121,
"max": 37590
},
"tokio current_thread": {
"result": 320000,
"median": 6228,
"min": 6210,
"max": 6514
"median": 5563,
"min": 5527,
"max": 6239
},
"tokio multi-thread": {
"result": 320000,
"median": 66173,
"min": 42208,
"max": 83255
"median": 63966,
"min": 59972,
"max": 67534
}
},
"many_timers": {
"smarm 1-thread": {
"result": 10000,
"median": 119988,
"min": 107308,
"max": 123557
"median": 107826,
"min": 107093,
"max": 119034
},
"smarm 24-thread": {
"result": 10000,
"median": 218842,
"min": 182009,
"max": 256988
"median": 96539,
"min": 95413,
"max": 97804
},
"tokio current_thread": {
"result": 10000,
"median": 12432,
"min": 12308,
"max": 13468
"median": 12584,
"min": 12539,
"max": 12627
},
"tokio multi-thread": {
"result": 10000,
"median": 16311,
"min": 15026,
"max": 16897
"median": 16183,
"min": 16024,
"max": 16541
}
},
"multi_thread_scaling": {
"smarm 1-thread": {
"result": 33860,
"median": 14908,
"min": 14857,
"max": 15218
"median": 15083,
"min": 15071,
"max": 15283
},
"smarm 2-thread": {
"result": 33860,
"median": 7834,
"min": 7717,
"max": 8033
"median": 8070,
"min": 8003,
"max": 8096
},
"smarm 4-thread": {
"result": 33860,
"median": 4393,
"min": 4326,
"max": 4435
"median": 4460,
"min": 4454,
"max": 4516
},
"smarm 24-thread": {
"result": 33860,
"median": 2173,
"min": 2068,
"max": 2405
"median": 2333,
"min": 2294,
"max": 2348
},
"tokio multi 1-thread": {
"result": 33860,
"median": 14432,
"min": 14219,
"max": 14763
"median": 14504,
"min": 14193,
"max": 14562
},
"tokio multi 2-thread": {
"result": 33860,
"median": 7333,
"min": 7222,
"max": 7477
"median": 7297,
"min": 7289,
"max": 7398
},
"tokio multi 4-thread": {
"result": 33860,
"median": 3741,
"min": 3681,
"max": 3876
"median": 3795,
"min": 3756,
"max": 3799
},
"tokio multi 24-thread": {
"result": 33860,
"median": 1513,
"min": 1375,
"max": 1979
"median": 1544,
"min": 1520,
"max": 1610
}
},
"deep_recursion": {
"smarm 1-thread": {
"result": 1,
"median": 102,
"min": 96,
"max": 123
"median": 226,
"min": 222,
"max": 243
},
"smarm 24-thread": {
"result": 1,
"median": 597,
"min": 576,
"max": 682
"median": 745,
"min": 744,
"max": 776
},
"tokio current_thread": {
"result": 1,
"median": 13,
"min": 11,
"max": 35
"median": 11,
"min": 10,
"max": 13
},
"tokio multi-thread": {
"result": 1,
"median": 56,
"min": 46,
"max": 65
"median": 53,
"min": 53,
"max": 57
}
},
"yield_in_hot_loop": {
"smarm 1-thread": {
"result": 1000000,
"median": 80680,
"min": 80308,
"max": 81845
"median": 64849,
"min": 64396,
"max": 65283
},
"tokio current_thread": {
"result": 1000000,
"median": 72606,
"min": 72154,
"max": 77206
"median": 68507,
"min": 62018,
"max": 72341
}
},
"uncontended_channel": {
"smarm 1-thread": {
"result": 1000000,
"median": 9257,
"min": 9223,
"max": 12049
"median": 11949,
"min": 11928,
"max": 13596
},
"tokio current_thread": {
"result": 1000000,
"median": 16925,
"min": 16848,
"max": 17019
"median": 15083,
"min": 15038,
"max": 16994
}
},
"catch_unwind_panics": {
"smarm 1-thread": {
"result": 10000,
"median": 116821,
"min": 111345,
"max": 128261
"median": 110932,
"min": 110182,
"max": 124147
},
"smarm 24-thread": {
"result": 10000,
"median": 117487,
"min": 107011,
"max": 129307
"median": 13665,
"min": 13172,
"max": 13784
},
"tokio current_thread": {
"result": 10000,
"median": 10425,
"min": 10141,
"max": 10604
"median": 10431,
"min": 9346,
"max": 10915
},
"tokio multi-thread": {
"result": 10000,
"median": 6418,
"min": 3715,
"max": 7144
"median": 6171,
"min": 5626,
"max": 6555
}
}
}
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Run-queue shootout driver (ROADMAP_v0.5 phase 4; RFC 005 slot dimension
# added for the v0.9 slot shootout).
#
# Rebuilds the runtime bench once per rq-* feature and runs the raw-structure
# microbench once (it covers all structures in a single binary). The RFC 005
# wake slot is a runtime Config knob, NOT a feature — each rq_runtime binary
# sweeps slot off/on internally (SMARM_BENCH_SLOT, default "0 1"). Results
# land in bench_results/ as full logs; the RQCSV lines are aggregated into
# bench_results/summary.csv and the RQSLOT counter lines (slot hits /
# displacements, slot-on configs only) into bench_results/slot_counters.csv.
#
# Tune the sweep for the box, e.g. on the 20-core machine:
# SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh
# Slot-only re-run against the frozen rq-mutex substrate:
# SMARM_BENCH_SLOT="0 1" cargo bench --bench rq_runtime
set -euo pipefail
cd "$(dirname "$0")/.."
OUT=bench_results
mkdir -p "$OUT"
: "${SMARM_BENCH_THREADS:=1 2 4}"
: "${SMARM_BENCH_SLOT:=0 1}"
export SMARM_BENCH_THREADS SMARM_BENCH_SLOT
echo "== raw structures (one binary, all variants) =="
cargo bench --bench rq_micro 2>&1 | tee "$OUT/micro.txt"
for v in rq-mutex rq-mpmc rq-striped; do
echo "== runtime benches: $v (slot sweep: $SMARM_BENCH_SLOT) =="
cargo bench --bench rq_runtime --no-default-features --features "$v" \
2>&1 | tee "$OUT/runtime-$v.txt"
done
# runtime rows: kind,variant,slot,bench,threads,work,median_us,ops_per_s
# micro rows: kind,structure,threads,p:c,items,median_us,items_per_s (one
# column narrower, as before — split on kind when plotting)
echo "kind,a,b,c,d,e,median_us,ops_per_s" > "$OUT/summary.csv"
grep -h '^RQCSV,' "$OUT"/*.txt | sed 's/^RQCSV,//' >> "$OUT/summary.csv"
echo "variant,bench,threads,slot_hits,slot_displacements" > "$OUT/slot_counters.csv"
grep -h '^RQSLOT,' "$OUT"/*.txt | sed 's/^RQSLOT,//' >> "$OUT/slot_counters.csv" || true
echo
echo "Summary: $OUT/summary.csv ($(($(wc -l < "$OUT/summary.csv") - 1)) rows)"
echo "Slot counters: $OUT/slot_counters.csv ($(($(wc -l < "$OUT/slot_counters.csv") - 1)) rows)"
+18 -7
View File
@@ -29,6 +29,13 @@ fn available_threads() -> usize {
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
}
fn env_sets() -> u32 {
std::env::var("SMARM_BENCH_SETS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(5)
}
fn print_header(title: &str) {
println!("\n{}", "=".repeat(80));
println!(" {title}");
@@ -41,14 +48,17 @@ fn print_header(title: &str) {
}
fn run_n<F: FnMut() -> (u64, u128)>(name: &str, n: u32, mut f: F) {
let mut times = Vec::new();
let sets = env_sets();
let mut times = Vec::with_capacity((n * sets) as usize);
let mut last = 0u64;
// One warmup iteration, discarded.
// One warmup before all sets, discarded.
let _ = f();
for _ in 0..n {
let (v, t) = f();
times.push(t);
last = v;
for _ in 0..sets {
for _ in 0..n {
let (v, t) = f();
times.push(t);
last = v;
}
}
times.sort_unstable();
let median = times[times.len() / 2];
@@ -406,7 +416,8 @@ fn main() {
let n = available_threads();
println!("smarm general benchmarks");
println!("available parallelism: {n} threads");
println!("ITERS={ITERS} (+1 warmup, discarded)");
let sets = env_sets();
println!("ITERS={ITERS}×{sets} sets = {} samples (+1 warmup, discarded)", ITERS * sets);
println!(
"CHAIN_DEPTH={CHAIN_DEPTH}, YIELD_TASKS={YIELD_TASKS}×{YIELD_ROUNDS}, \
PRIME_N={PRIME_N}/{PRIME_WORKERS} workers, PP_ROUNDS={PP_ROUNDS}"
+186
View File
@@ -0,0 +1,186 @@
//! Raw run-queue microbench (ROADMAP_v0.5 phase 4).
//!
//! Benches the three queue STRUCTURES directly — no runtime, no actors — to
//! isolate the data structure under contention. All three types compile in
//! every build, so this binary covers the whole matrix in one run; it does
//! NOT need the rq-* feature rebuild dance (that's `rq_runtime`).
//!
//! Sweeps thread count × producer:consumer ratio. Queues are sized to the
//! item count, so the occupancy contract holds trivially and producers never
//! block on capacity.
//!
//! Knobs (env):
//! SMARM_BENCH_THREADS space-separated sweep, default "1 2 4"
//! SMARM_BENCH_ITEMS items per measurement, default 200_000
//! SMARM_BENCH_RUNS repetitions per config (median reported), default 5
//!
//! Output: the house table, plus one machine-readable line per config:
//! RQCSV,micro,<structure>,<threads>,<p:c>,<items>,<median_us>,<items_per_s>
//!
//! NOTE: numbers from a 1-core sandbox only validate the harness; real
//! contention curves come from the many-core box (scripts/bench_rq.sh).
use smarm::pid::Pid;
use smarm::run_queue::{MpmcRing, MutexQueue, StripedRing};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Instant;
fn env_usize(key: &str, default: usize) -> usize {
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}
fn env_threads() -> Vec<usize> {
std::env::var("SMARM_BENCH_THREADS")
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
.unwrap_or_else(|_| vec![1, 2, 4])
}
/// Generic driver: `producers` threads push `items` total, `consumers`
/// threads pop until everything is accounted for. Returns elapsed µs.
fn drive<Q: Send + Sync + 'static>(
q: Arc<Q>,
push: fn(&Q, Pid),
pop: fn(&Q) -> Option<Pid>,
producers: usize,
consumers: usize,
items: usize,
) -> u128 {
let remaining = Arc::new(AtomicUsize::new(items));
let start = Instant::now();
let mut hs = Vec::new();
let per = items / producers;
for p in 0..producers {
let q = q.clone();
// Give the last producer the remainder.
let n = if p == producers - 1 { items - per * (producers - 1) } else { per };
hs.push(std::thread::spawn(move || {
let pid = Pid::new(p as u32, 0);
for _ in 0..n {
push(&q, pid);
}
}));
}
for _ in 0..consumers {
let q = q.clone();
let remaining = remaining.clone();
hs.push(std::thread::spawn(move || loop {
// Claim-then-pop so consumers exit promptly when the budget hits
// zero; the claim is backed out on a miss.
let r = remaining.load(Ordering::Relaxed);
if r == 0 {
return;
}
if pop(&q).is_some() {
remaining.fetch_sub(1, Ordering::Relaxed);
} else {
std::hint::spin_loop();
}
}));
}
for h in hs {
h.join().unwrap();
}
start.elapsed().as_micros()
}
/// Single-thread alternating push/pop (the T = 1 case).
fn drive_single<Q>(q: &Q, push: fn(&Q, Pid), pop: fn(&Q) -> Option<Pid>, items: usize) -> u128 {
let pid = Pid::new(0, 0);
let start = Instant::now();
for _ in 0..items {
push(q, pid);
assert!(pop(q).is_some());
}
start.elapsed().as_micros()
}
struct Case {
structure: &'static str,
threads: usize,
producers: usize,
consumers: usize,
}
fn ratios_for(threads: usize) -> Vec<(usize, usize)> {
if threads < 2 {
return vec![(1, 1)]; // label only; T=1 runs the alternating driver
}
let mut v = vec![(threads / 2, threads - threads / 2)]; // balanced
if threads >= 4 {
v.push((3 * threads / 4, threads - 3 * threads / 4)); // producer-heavy
v.push((threads / 4, threads - threads / 4)); // consumer-heavy
}
v
}
fn main() {
let threads_sweep = env_threads();
let items = env_usize("SMARM_BENCH_ITEMS", 200_000);
let runs = env_usize("SMARM_BENCH_RUNS", 5);
println!("\n{}", "=".repeat(86));
println!(" run-queue raw structures — items={items}, runs={runs} (median)");
println!("{}", "=".repeat(86));
println!(
"{:>10} | {:>7} | {:>7} | {:>10} | {:>14}",
"structure", "threads", "p:c", "median µs", "items/s"
);
println!("{}", "-".repeat(86));
let mut cases = Vec::new();
for &t in &threads_sweep {
for (p, c) in ratios_for(t) {
for s in ["mutex", "mpmc", "striped"] {
cases.push(Case { structure: s, threads: t, producers: p, consumers: c });
}
}
}
for case in cases {
let mut times: Vec<u128> = (0..runs)
.map(|_| {
// Fresh queue per run; capacity = items so pushes never stall.
match case.structure {
"mutex" => {
let q = Arc::new(MutexQueue::new(case.threads, items));
if case.threads < 2 {
drive_single(&*q, MutexQueue::push, MutexQueue::pop, items)
} else {
drive(q, MutexQueue::push, MutexQueue::pop, case.producers, case.consumers, items)
}
}
"mpmc" => {
let q = Arc::new(MpmcRing::with_capacity(items));
if case.threads < 2 {
drive_single(&*q, MpmcRing::push, MpmcRing::pop, items)
} else {
drive(q, MpmcRing::push, MpmcRing::pop, case.producers, case.consumers, items)
}
}
"striped" => {
let q = Arc::new(StripedRing::new(case.threads.max(1), items));
if case.threads < 2 {
drive_single(&*q, StripedRing::push, StripedRing::pop, items)
} else {
drive(q, StripedRing::push, StripedRing::pop, case.producers, case.consumers, items)
}
}
_ => unreachable!(),
}
})
.collect();
times.sort_unstable();
let median = times[times.len() / 2];
let per_s = (items as f64 / (median as f64 / 1e6)) as u64;
let ratio = format!("{}:{}", case.producers, case.consumers);
println!(
"{:>10} | {:>7} | {:>7} | {:>10} | {:>14}",
case.structure, case.threads, ratio, median, per_s
);
println!(
"RQCSV,micro,{},{},{},{},{},{}",
case.structure, case.threads, ratio, items, median, per_s
);
}
}
+252
View File
@@ -0,0 +1,252 @@
//! Runtime-level run-queue benches (ROADMAP_v0.5 phase 4; slot dimension
//! added for the v0.9 slot shootout, RFC 005).
//!
//! These exercise the WHOLE scheduler with the compile-time-selected queue,
//! so comparing variants means rebuilding per rq-* feature — that's what
//! scripts/bench_rq.sh does. The RFC 005 wake slot is a *runtime* Config
//! knob, so one binary benches both arms; the slot on/off sweep happens
//! inside this binary. Workloads:
//!
//! yield-storm — N actors yield K times each. Pure queue churn:
//! every yield is a push + pop with nothing in between.
//! Slot role: REGRESSION GUARD — yields never touch the
//! slot, any slot-on delta is pop-path overhead.
//! ping-pong-pairs — P channel pairs, M roundtrips each. Park/unpark
//! latency through the queue.
//! Slot role: TARGET METRIC — every send-wake is an
//! actor-context unpark, the slot's home pattern.
//! spawn-storm — S spawn+join of trivial actors. Slab + queue + free
//! list under churn.
//! Slot role: NEUTRALITY CHECK — spawns bypass the slot
//! by policy; join wakes fire from finalize (scheduler
//! context), also shared.
//!
//! Knobs (env):
//! SMARM_BENCH_THREADS scheduler-count sweep, default "1 2 4"
//! SMARM_BENCH_SLOT wake-slot sweep, default "0 1" (off then on)
//! SMARM_BENCH_RUNS repetitions per config (median), default 5
//! SMARM_BENCH_YIELD_ACTORS / _YIELDS default 200 / 500
//! SMARM_BENCH_PAIRS / _ROUNDTRIPS default 32 / 1000
//! SMARM_BENCH_SPAWNS default 5000
//!
//! Output: house table + one line per config:
//! RQCSV,runtime,<variant>,<slot>,<bench>,<threads>,<work>,<median_us>,<ops_per_s>
//! plus, for slot-on configs, the RFC 005 observability counters:
//! RQSLOT,<variant>,<bench>,<threads>,<slot_hits>,<slot_displacements>
//! (hits/displacements are taken from the same run as the median time).
//!
//! NOTE: a 1-core sandbox validates the harness, not the scaling story;
//! real curves come from the many-core box.
use smarm::runtime::{init, Config};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Instant;
fn variant() -> &'static str {
if cfg!(feature = "rq-mpmc") {
"rq-mpmc"
} else if cfg!(feature = "rq-striped") {
"rq-striped"
} else {
"rq-mutex"
}
}
fn env_usize(key: &str, default: usize) -> usize {
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}
fn env_threads() -> Vec<usize> {
std::env::var("SMARM_BENCH_THREADS")
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
.unwrap_or_else(|_| vec![1, 2, 4])
}
fn env_slots() -> Vec<bool> {
std::env::var("SMARM_BENCH_SLOT")
.map(|v| {
v.split_whitespace()
.filter_map(|t| match t {
"0" | "off" | "false" => Some(false),
"1" | "on" | "true" => Some(true),
_ => None,
})
.collect()
})
.unwrap_or_else(|_| vec![false, true])
}
/// One measured run: (total_ops, elapsed_µs, slot_hits, slot_displacements).
struct Sample {
ops: u64,
us: u128,
hits: u64,
displacements: u64,
}
fn yield_storm(threads: usize, slot: bool, actors: usize, yields: usize) -> Sample {
let rt = init(Config::exact(threads).wake_slot(slot));
let start = Instant::now();
rt.run(move || {
let handles: Vec<_> = (0..actors)
.map(|_| {
smarm::spawn(move || {
for _ in 0..yields {
smarm::yield_now();
}
})
})
.collect();
for h in handles {
let _ = h.join();
}
});
let us = start.elapsed().as_micros();
let stats = rt.stats();
Sample {
ops: (actors * yields) as u64,
us,
hits: stats.slot_hits(),
displacements: stats.slot_displacements(),
}
}
fn ping_pong_pairs(threads: usize, slot: bool, pairs: usize, roundtrips: usize) -> Sample {
let rt = init(Config::exact(threads).wake_slot(slot));
let total = Arc::new(AtomicU64::new(0));
let t2 = total.clone();
let start = Instant::now();
rt.run(move || {
let handles: Vec<_> = (0..pairs)
.map(|_| {
let total = t2.clone();
smarm::spawn(move || {
let (tx_ab, rx_ab) = smarm::channel::channel::<u64>();
let (tx_ba, rx_ba) = smarm::channel::channel::<u64>();
let n = roundtrips as u64;
let echo = smarm::spawn(move || {
for _ in 0..n {
let v = rx_ab.recv().expect("echo recv");
tx_ba.send(v + 1).expect("echo send");
}
});
for i in 0..n {
tx_ab.send(i).expect("ping send");
let v = rx_ba.recv().expect("ping recv");
assert_eq!(v, i + 1);
}
let _ = echo.join();
total.fetch_add(n, Ordering::Relaxed);
})
})
.collect();
for h in handles {
let _ = h.join();
}
});
let us = start.elapsed().as_micros();
let stats = rt.stats();
Sample {
ops: total.load(Ordering::Relaxed),
us,
hits: stats.slot_hits(),
displacements: stats.slot_displacements(),
}
}
fn spawn_storm(threads: usize, slot: bool, spawns: usize) -> Sample {
let rt = init(Config::exact(threads).wake_slot(slot));
let start = Instant::now();
rt.run(move || {
// Batches bound simultaneous liveness well below the slab cap.
const BATCH: usize = 1024;
let mut left = spawns;
while left > 0 {
let n = left.min(BATCH);
let handles: Vec<_> = (0..n).map(|_| smarm::spawn(|| {})).collect();
for h in handles {
let _ = h.join();
}
left -= n;
}
});
let us = start.elapsed().as_micros();
let stats = rt.stats();
Sample {
ops: spawns as u64,
us,
hits: stats.slot_hits(),
displacements: stats.slot_displacements(),
}
}
fn main() {
let threads_sweep = env_threads();
let slot_sweep = env_slots();
let runs = env_usize("SMARM_BENCH_RUNS", 5);
let ya = env_usize("SMARM_BENCH_YIELD_ACTORS", 200);
let yy = env_usize("SMARM_BENCH_YIELDS", 500);
let pp = env_usize("SMARM_BENCH_PAIRS", 32);
let pr = env_usize("SMARM_BENCH_ROUNDTRIPS", 1000);
let ss = env_usize("SMARM_BENCH_SPAWNS", 5000);
println!("\n{}", "=".repeat(106));
println!(
" runtime benches — variant={}, runs={runs} (median)",
variant()
);
println!("{}", "=".repeat(106));
println!(
"{:>16} | {:>7} | {:>4} | {:>16} | {:>10} | {:>14} | {:>10} | {:>9}",
"bench", "threads", "slot", "work", "median µs", "ops/s", "slot hits", "displaced"
);
println!("{}", "-".repeat(106));
type Bench = (&'static str, String, Box<dyn Fn(usize, bool) -> Sample>);
let benches: Vec<Bench> = vec![
(
"yield-storm",
format!("{ya}x{yy}"),
Box::new(move |t, s| yield_storm(t, s, ya, yy)),
),
(
"ping-pong-pairs",
format!("{pp}x{pr}"),
Box::new(move |t, s| ping_pong_pairs(t, s, pp, pr)),
),
(
"spawn-storm",
format!("{ss}"),
Box::new(move |t, s| spawn_storm(t, s, ss)),
),
];
for (name, work, f) in &benches {
for &t in &threads_sweep {
for &slot in &slot_sweep {
let mut samples: Vec<Sample> = (0..runs).map(|_| f(t, slot)).collect();
// Median by elapsed time; report the counters from that
// same run so hits/time stay paired.
samples.sort_unstable_by_key(|s| s.us);
let mid = &samples[samples.len() / 2];
let per_s = (mid.ops as f64 / (mid.us as f64 / 1e6)) as u64;
let slot_str = if slot { "on" } else { "off" };
println!(
"{:>16} | {:>7} | {:>4} | {:>16} | {:>10} | {:>14} | {:>10} | {:>9}",
name, t, slot_str, work, mid.us, per_s, mid.hits, mid.displacements
);
println!(
"RQCSV,runtime,{},{},{},{},{},{},{}",
variant(), slot_str, name, t, work, mid.us, per_s
);
if slot {
println!(
"RQSLOT,{},{},{},{},{}",
variant(), name, t, mid.hits, mid.displacements
);
}
}
}
}
}
+19 -7
View File
@@ -40,6 +40,13 @@ fn available_threads() -> usize {
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
}
fn env_sets() -> u32 {
std::env::var("SMARM_BENCH_SETS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(5)
}
fn print_header(title: &str) {
println!("\n{}", "=".repeat(80));
println!(" {title}");
@@ -52,13 +59,17 @@ fn print_header(title: &str) {
}
fn run_n<F: FnMut() -> (u64, u128)>(name: &str, n: u32, mut f: F) {
let mut times = Vec::new();
let sets = env_sets();
let mut times = Vec::with_capacity((n * sets) as usize);
let mut last = 0u64;
let _ = f(); // warmup
for _ in 0..n {
let (v, t) = f();
times.push(t);
last = v;
// One warmup before all sets, discarded.
let _ = f();
for _ in 0..sets {
for _ in 0..n {
let (v, t) = f();
times.push(t);
last = v;
}
}
times.sort_unstable();
let median = times[times.len() / 2];
@@ -385,7 +396,8 @@ fn main() {
let n = available_threads();
println!("smarm smarm-favored benchmarks");
println!("available parallelism: {n} threads");
println!("ITERS={ITERS} (+1 warmup, discarded)");
let sets = env_sets();
println!("ITERS={ITERS}×{sets} sets = {} samples (+1 warmup, discarded)", ITERS * sets);
println!(
"RECURSE_DEPTH={RECURSE_DEPTH}, HOT_YIELDS={HOT_YIELDS}×2, \
UNCONT_MSGS={UNCONT_MSGS}, PANIC_TASKS={PANIC_TASKS}"
+52 -5
View File
@@ -50,6 +50,11 @@ SWEEP_GRID = [
(128, 1_200_000),
]
# Number of independent cargo bench processes per measurement point.
# Each process is a fully isolated run (fresh warmup, cold caches, new PID),
# so the final median is a median of independent samples — robust to OS noise.
BENCH_SETS = 5
# Regression threshold: warn if median is more than this % worse than baseline.
REGRESSION_THRESHOLD_PCT = 10
@@ -103,9 +108,12 @@ def parse_output(text: str) -> dict[str, dict[str, dict]]:
# Running
# ---------------------------------------------------------------------------
def run_benches(env_extra: dict[str, str] | None = None) -> dict[str, dict[str, dict]]:
"""Run all BENCHES and return merged parsed results."""
def run_benches_once(env_extra: dict[str, str] | None = None) -> dict[str, dict[str, dict]]:
"""Run all BENCHES once and return merged parsed results."""
env = os.environ.copy()
# Each process does exactly one set of ITERS samples — no within-process
# accumulation; the caller handles multi-set aggregation.
env["SMARM_BENCH_SETS"] = "1"
if env_extra:
env.update(env_extra)
@@ -129,6 +137,45 @@ def run_benches(env_extra: dict[str, str] | None = None) -> dict[str, dict[str,
return all_results
def run_benches(env_extra: dict[str, str] | None = None, sets: int = BENCH_SETS) -> dict[str, dict[str, dict]]:
"""Run BENCH_SETS independent processes and return median-of-medians per label.
Each set is a separate cargo bench invocation with its own warmup and OS
context, so samples are statistically independent. The final median and
min/max are computed over the per-set medians.
"""
# Accumulate per-set medians: {bench: {label: [median_set1, median_set2, ...]}}
accumulated: dict[str, dict[str, list[int]]] = {}
last_result: dict[str, dict[str, int]] = {}
for i in range(sets):
print(f" set {i + 1}/{sets}", flush=True)
set_results = run_benches_once(env_extra)
for bench, labels in set_results.items():
accumulated.setdefault(bench, {})
last_result.setdefault(bench, {})
for label, data in labels.items():
accumulated[bench].setdefault(label, [])
accumulated[bench][label].append(data["median"])
last_result[bench][label] = data["result"]
# Collapse to final stats.
final: dict[str, dict[str, dict]] = {}
for bench, labels in accumulated.items():
final[bench] = {}
for label, medians in labels.items():
medians.sort()
mid = medians[len(medians) // 2]
final[bench][label] = {
"result": last_result[bench][label],
"median": mid,
"min": medians[0],
"max": medians[-1],
}
return final
# ---------------------------------------------------------------------------
# Baseline JSON
# ---------------------------------------------------------------------------
@@ -252,7 +299,7 @@ def cmd_run(args) -> None:
["cargo", "build", "--release", "--benches"],
cwd=REPO, check=True, capture_output=True,
)
print("Running benches…")
print(f"Running benches ({BENCH_SETS} independent sets)")
results = run_benches()
print_results(results, "Results (default knobs)")
if args.save_baseline:
@@ -266,7 +313,7 @@ def cmd_regress(args) -> None:
["cargo", "build", "--release", "--benches"],
cwd=REPO, check=True, capture_output=True,
)
print("Running benches…")
print(f"Running benches ({BENCH_SETS} independent sets)")
current = run_benches()
print_results(current, "Current results")
print(f"\nRegression check (threshold: >{REGRESSION_THRESHOLD_PCT}% slower than baseline)")
@@ -288,7 +335,7 @@ def cmd_sweep(args) -> None:
for interval, cycles in SWEEP_GRID:
tag = f"alloc_interval={interval}, timeslice_cycles={cycles}"
print(f" Running: {tag}", flush=True)
print(f" Running: {tag} ({BENCH_SETS} sets)", flush=True)
env_extra = {
"SMARM_ALLOC_INTERVAL": str(interval),
"SMARM_TIMESLICE_CYCLES": str(cycles),
+19 -7
View File
@@ -39,6 +39,13 @@ fn available_threads() -> usize {
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
}
fn env_sets() -> u32 {
std::env::var("SMARM_BENCH_SETS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(5)
}
fn print_header(title: &str) {
println!("\n{}", "=".repeat(80));
println!(" {title}");
@@ -51,13 +58,17 @@ fn print_header(title: &str) {
}
fn run_n<F: FnMut() -> (u64, u128)>(name: &str, n: u32, mut f: F) {
let mut times = Vec::new();
let sets = env_sets();
let mut times = Vec::with_capacity((n * sets) as usize);
let mut last = 0u64;
let _ = f(); // warmup
for _ in 0..n {
let (v, t) = f();
times.push(t);
last = v;
// One warmup before all sets, discarded.
let _ = f();
for _ in 0..sets {
for _ in 0..n {
let (v, t) = f();
times.push(t);
last = v;
}
}
times.sort_unstable();
let median = times[times.len() / 2];
@@ -434,7 +445,8 @@ fn main() {
let n = available_threads();
println!("smarm tokio-favored benchmarks");
println!("available parallelism: {n} threads");
println!("ITERS={ITERS} (+1 warmup, discarded)");
let sets = env_sets();
println!("ITERS={ITERS}×{sets} sets = {} samples (+1 warmup, discarded)", ITERS * sets);
println!(
"STORM_BACKGROUND={STORM_BACKGROUND}, STORM_SPAWN={STORM_SPAWN}, \
MPSC={MPSC_PRODUCERS}×{MPSC_PER_PRODUCER}, \
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -122,9 +122,9 @@ pub struct Actor {
/// The PID this actor was assigned at spawn time.
pub pid: Pid,
/// The stack the actor runs on. Dropped (munmap'd) when the actor dies.
/// (The saved stack pointer lives on the `Slot` as an atomic, not here:
/// it is hot scheduling state, read/written without the cold lock.)
pub stack: Stack,
/// The saved stack pointer. Updated on every yield.
pub sp: usize,
/// The PID of this actor's supervisor. Used to deliver `Signal` on death.
pub supervisor: Pid,
/// Cooperative-cancellation flag. `request_stop` sets it (and unparks a
-165
View File
@@ -1,165 +0,0 @@
//! Cooperative context switching and cycle counter, aarch64 (AAPCS64).
//!
//! The aarch64 mirror of the x86-64 backend. The protocol is identical — save
//! callee-saved state, swap stack pointers via the shared `*_SP` thread-locals,
//! restore, return — but the mechanics differ from x86 in two ways worth
//! stating up front, because they drive the stack layout:
//!
//! 1. **Return is via the link register, not the stack.** x86 `ret` pops the
//! return address off the stack; aarch64 `ret` jumps to whatever is in
//! `x30` (lr). So the entry point is parked in the saved-`x30` slot, and
//! the shim restores `x30` and `ret`s to it. There is no "ret target word"
//! sitting on the stack the way there is on x86.
//!
//! 2. **sp must be 16-byte aligned at all times**, not just at call entry.
//! We save 12 registers (x19x28, x29, x30) = 96 bytes, already a multiple
//! of 16, so the frame is aligned by construction.
//!
//! FP/SIMD registers (v8v15, whose low 64 bits are callee-saved under AAPCS64)
//! are NOT saved, for the same reason the x86 backend skips XMM: every yield
//! goes through a Rust call boundary, so the compiler has already spilled any
//! live vector state. If we ever yield from a non-call-boundary, this breaks.
use super::{get_actor_sp, get_scheduler_sp, set_actor_sp, set_scheduler_sp};
// ---------------------------------------------------------------------------
// Initial stack layout
//
// Twelve 8-byte slots, all zero except the x30 slot which holds `entry`. The
// first `switch_to_actor` loads x19x28/x29/x30 back and `ret`s to x30,
// landing at the top of `entry` with sp 16-aligned (the AAPCS64 requirement).
//
// Layout (high → low), relative to aligned_top = top & ~15:
//
// aligned_top - 8 : x30 = entry ← `ret` jumps here.
// aligned_top - 16 : x29 = 0 (fp)
// aligned_top - 24 : x28 = 0
// aligned_top - 32 : x27 = 0
// aligned_top - 40 : x26 = 0
// aligned_top - 48 : x25 = 0
// aligned_top - 56 : x24 = 0
// aligned_top - 64 : x23 = 0
// aligned_top - 72 : x22 = 0
// aligned_top - 80 : x21 = 0
// aligned_top - 88 : x20 = 0
// aligned_top - 96 : x19 = 0 ← initial sp (16-aligned)
//
// The restore order in the shim (ldp pairs, low→high) must match this exactly.
// ---------------------------------------------------------------------------
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
unsafe {
let aligned_top = top as usize & !15;
let mut sp = aligned_top;
sp -= 8; (sp as *mut usize).write(entry as usize); // x30 (lr) → entry
sp -= 8; (sp as *mut usize).write(0); // x29 (fp)
sp -= 8; (sp as *mut usize).write(0); // x28
sp -= 8; (sp as *mut usize).write(0); // x27
sp -= 8; (sp as *mut usize).write(0); // x26
sp -= 8; (sp as *mut usize).write(0); // x25
sp -= 8; (sp as *mut usize).write(0); // x24
sp -= 8; (sp as *mut usize).write(0); // x23
sp -= 8; (sp as *mut usize).write(0); // x22
sp -= 8; (sp as *mut usize).write(0); // x21
sp -= 8; (sp as *mut usize).write(0); // x20
sp -= 8; (sp as *mut usize).write(0); // x19 ← initial sp
sp
}
}
// ---------------------------------------------------------------------------
// Context switch shims
//
// Each shim:
// 1. Pushes x19x28, x29, x30 as six 16-byte stp pairs (sp pre-decrement).
// 2. Moves sp into x0 and calls the Rust helper that stores it.
// 3. Calls the Rust helper that returns the *other* side's saved sp.
// 4. Moves that into sp.
// 5. Restores the twelve registers and rets (to the restored x30).
//
// The helper calls clobber x30 themselves, but that's fine: the outgoing x30
// was already saved to the stack in step 1, and step 5 restores the incoming
// side's x30 before `ret`. The push/pop register order is paired so that the
// `ldp` sequence reverses the `stp` sequence exactly.
// ---------------------------------------------------------------------------
#[unsafe(naked)]
unsafe extern "C" fn switch_to_actor_asm() {
core::arch::naked_asm!(
"stp x19, x20, [sp, #-96]!",
"stp x21, x22, [sp, #16]",
"stp x23, x24, [sp, #32]",
"stp x25, x26, [sp, #48]",
"stp x27, x28, [sp, #64]",
"stp x29, x30, [sp, #80]",
"mov x0, sp",
"bl {set_sched_sp}",
"bl {get_actor_sp}",
"mov sp, x0",
"ldp x21, x22, [sp, #16]",
"ldp x23, x24, [sp, #32]",
"ldp x25, x26, [sp, #48]",
"ldp x27, x28, [sp, #64]",
"ldp x29, x30, [sp, #80]",
"ldp x19, x20, [sp], #96",
"ret",
set_sched_sp = sym set_scheduler_sp,
get_actor_sp = sym get_actor_sp,
);
}
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
pub unsafe fn switch_to_actor() {
unsafe { switch_to_actor_asm() };
}
#[unsafe(naked)]
pub unsafe extern "C" fn switch_to_scheduler() {
core::arch::naked_asm!(
"stp x19, x20, [sp, #-96]!",
"stp x21, x22, [sp, #16]",
"stp x23, x24, [sp, #32]",
"stp x25, x26, [sp, #48]",
"stp x27, x28, [sp, #64]",
"stp x29, x30, [sp, #80]",
"mov x0, sp",
"bl {set_actor_sp}",
"bl {get_sched_sp}",
"mov sp, x0",
"ldp x21, x22, [sp, #16]",
"ldp x23, x24, [sp, #32]",
"ldp x25, x26, [sp, #48]",
"ldp x27, x28, [sp, #64]",
"ldp x29, x30, [sp, #80]",
"ldp x19, x20, [sp], #96",
"ret",
set_actor_sp = sym set_actor_sp,
get_sched_sp = sym get_scheduler_sp,
);
}
// ---------------------------------------------------------------------------
// Cycle counter
//
// CNTVCT_EL0 is the virtual count register, readable from EL0 on Linux. `isb`
// serialises so we don't sample the counter before prior instructions retire
// (the aarch64 analogue of x86's `lfence` before rdtsc).
//
// Unit: CNTVCT ticks, whose frequency is CNTFRQ_EL0 — typically 150 MHz,
// NOT the CPU clock. This is a different and much coarser unit than the x86
// TSC, which is why `TIMESLICE_CYCLES` must be tuned separately for aarch64.
// ---------------------------------------------------------------------------
#[inline(always)]
pub fn read_cycle_counter() -> u64 {
let cnt: u64;
unsafe {
core::arch::asm!(
"isb",
"mrs {cnt}, cntvct_el0",
cnt = out(reg) cnt,
options(nostack, nomem),
);
}
cnt
}
-54
View File
@@ -1,54 +0,0 @@
//! Architecture abstraction layer.
//!
//! Everything that depends on the target ISA lives behind this module: the
//! cooperative context-switch shims, the initial-stack layout, and the
//! cycle counter used by the preemption timeslice. The rest of the runtime
//! talks only to the API re-exported here and never names a register.
//!
//! Each backend (`x86_64`, `aarch64`) exposes the same four items:
//!
//! - `init_actor_stack(top, entry) -> usize`
//! Build a fresh actor stack so the first `switch_to_actor` lands
//! inside `entry` with the ABI-correct stack alignment.
//! - `switch_to_actor()`
//! Resume the actor whose sp is in `ACTOR_SP`; returns when it yields.
//! - `switch_to_scheduler()`
//! Yield from the current actor back to its scheduler thread.
//! - `read_cycle_counter() -> u64`
//! Monotonic per-core cycle counter for the timeslice clock. The unit
//! is ISA-defined (x86 TSC ticks vs. aarch64 virtual-counter ticks),
//! so `TIMESLICE_CYCLES` must be tuned per-arch — see `preempt`.
//!
//! The `SCHEDULER_SP` / `ACTOR_SP` thread-locals are shared across backends
//! and live here, since the saved-stack-pointer protocol is identical
//! regardless of which registers the shim happens to push.
use std::cell::Cell;
thread_local! {
static SCHEDULER_SP: Cell<usize> = const { Cell::new(0) };
static ACTOR_SP: Cell<usize> = const { Cell::new(0) };
}
// Used by the naked shims (via `sym`) and by the scheduler/tests through the
// re-exports below. `pub` because the asm references them as symbols.
pub(crate) fn get_scheduler_sp() -> usize { SCHEDULER_SP.with(|c| c.get()) }
pub(crate) fn set_scheduler_sp(v: usize) { SCHEDULER_SP.with(|c| c.set(v)) }
pub fn get_actor_sp() -> usize { ACTOR_SP.with(|c| c.get()) }
pub fn set_actor_sp(v: usize) { ACTOR_SP.with(|c| c.set(v)) }
#[cfg(target_arch = "x86_64")]
mod x86_64;
#[cfg(target_arch = "x86_64")]
pub use x86_64::{init_actor_stack, read_cycle_counter, switch_to_actor, switch_to_scheduler};
#[cfg(target_arch = "aarch64")]
mod aarch64;
#[cfg(target_arch = "aarch64")]
pub use aarch64::{init_actor_stack, read_cycle_counter, switch_to_actor, switch_to_scheduler};
#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
compile_error!(
"smarm's context-switch and cycle-counter layer supports only x86_64 and \
aarch64. Add a backend in src/arch/ for this target."
);
-118
View File
@@ -1,118 +0,0 @@
//! Cooperative context switching and cycle counter, x86-64 (SysV AMD64).
//!
//! Two naked-asm functions move execution between a scheduler thread and an
//! actor running on its own mmap'd stack. The compiler cannot do this; the
//! whole point of `#[unsafe(naked)]` is that we control every instruction.
//!
//! `init_actor_stack` builds the initial stack so that the first
//! `switch_to_actor` lands inside the entry function with `rsp % 16 == 8`
//! (the x86-64 ABI requirement at function entry).
use super::{get_actor_sp, get_scheduler_sp, set_actor_sp, set_scheduler_sp};
// ---------------------------------------------------------------------------
// Initial stack layout
//
// We start from aligned_top = top & ~15, then bias by 8 and push seven 8-byte
// slots (downward). The first `switch_to_actor` pops r15..rbx and `ret`s —
// landing in `entry` with rsp % 16 == 8 (the x86-64 ABI state at the point
// just after a `call`, which is what `entry` is compiled to expect).
//
// Layout (high → low), relative to aligned_top = top & ~15. Note the entry
// slot is at aligned_top - 16, NOT aligned_top - 8: the function does
// `(top & ~15) - 8` and *then* a `-= 8` before the first write, so the first
// stored word lands at aligned_top - 16. Verified by single-stepping the
// `ret` under llmdbg: entry sits at an address with %16 == 0, so the post-ret
// rsp is %16 == 8.
//
// aligned_top - 8 : (unused padding; keeps entry's slot %16 == 0)
// aligned_top - 16 : entry ptr ← `ret` target. Post-ret: rsp % 16 == 8.
// aligned_top - 24 : rbx = 0
// aligned_top - 32 : rbp = 0
// aligned_top - 40 : r12 = 0
// aligned_top - 48 : r13 = 0
// aligned_top - 56 : r14 = 0
// aligned_top - 64 : r15 = 0 ← initial rsp
// ---------------------------------------------------------------------------
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
unsafe {
let mut sp = (top as usize & !15) - 8;
sp -= 8; (sp as *mut usize).write(entry as usize); // ret target
sp -= 8; (sp as *mut usize).write(0); // rbx
sp -= 8; (sp as *mut usize).write(0); // rbp
sp -= 8; (sp as *mut usize).write(0); // r12
sp -= 8; (sp as *mut usize).write(0); // r13
sp -= 8; (sp as *mut usize).write(0); // r14
sp -= 8; (sp as *mut usize).write(0); // r15
sp
}
}
// ---------------------------------------------------------------------------
// Context switch shims
//
// Each shim:
// 1. Pushes the six callee-saved integer registers.
// 2. Snaps rsp into rdi and calls the Rust helper that stores it.
// 3. Calls the Rust helper that returns the *other* side's saved rsp.
// 4. Moves that into rsp.
// 5. Pops the six registers and rets.
//
// XMM registers are NOT saved here. We rely on every yield happening through
// a Rust call site, which means the compiler has spilled any live XMM state
// to the stack before we get here. (This is the same argument the compiler
// uses internally — callee-saved regs are what survive a `call`, and the
// SysV AMD64 ABI says XMM015 are all caller-saved.) If we ever yield from
// a place that isn't a Rust call boundary, this assumption breaks.
// ---------------------------------------------------------------------------
#[unsafe(naked)]
unsafe extern "C" fn switch_to_actor_asm() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_sched_sp}",
"call {get_actor_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_sched_sp = sym set_scheduler_sp,
get_actor_sp = sym get_actor_sp,
);
}
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
pub unsafe fn switch_to_actor() {
unsafe { switch_to_actor_asm() };
}
#[unsafe(naked)]
pub unsafe extern "C" fn switch_to_scheduler() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_actor_sp}",
"call {get_sched_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_actor_sp = sym set_actor_sp,
get_sched_sp = sym get_scheduler_sp,
);
}
// ---------------------------------------------------------------------------
// Cycle counter
//
// `lfence` serialises the instruction stream so we don't measure time before
// prior instructions retire. Unit: TSC ticks (≈ CPU base clock).
// ---------------------------------------------------------------------------
#[inline(always)]
pub fn read_cycle_counter() -> u64 {
unsafe {
core::arch::asm!("lfence", options(nostack, nomem, preserves_flags));
core::arch::x86_64::_rdtsc()
}
}
+466 -22
View File
@@ -1,9 +1,23 @@
//! Unbounded MPSC channels.
//!
//! Inner state is `Arc<Mutex<Inner<T>>>` so channels can be sent across OS
//! Inner state is `Arc<RawMutex<Inner<T>>>` so channels can be sent across OS
//! threads (required for the multi-scheduler runtime where a sender and
//! receiver may run on different scheduler threads simultaneously).
//!
//! ## Why `RawMutex` (Channel class), not `std::sync::Mutex`
//!
//! An actor holding a guard with preemption *enabled* can be timesliced
//! inside the critical section and resume on a different OS thread — the
//! pthread mutex would then be released from a thread that didn't lock it,
//! which is UB (Linux futexes happen to tolerate it, but it's not
//! guaranteed). `RawMutex` disables preemption for the guard's span and is
//! cross-thread-release sound by construction, closing the hole. It also
//! cannot poison. Channel locks form their own [`LockClass::Channel`]
//! (raw_mutex.rs): they may be taken under a cold (Leaf) lock — finalize and
//! `monitor()` clone senders that live in slots — but nothing may be locked
//! under them, which the debug build enforces. `recv_match` runs its user
//! predicate under this lock: keep it cheap, pure, and channel-free.
//!
//! Semantics:
//! - Senders are clonable; the last sender drop closes the channel.
//! - `Receiver::recv` on an empty open channel parks the receiver.
@@ -15,11 +29,12 @@
//! parked, the receiver is unparked.
use crate::pid::Pid;
use crate::raw_mutex::RawMutex;
use std::collections::VecDeque;
use std::sync::{Arc, Mutex};
use std::sync::Arc;
pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
let inner = Arc::new(Mutex::new(Inner {
let inner = Arc::new(RawMutex::new_channel(Inner {
queue: VecDeque::new(),
parked_receiver: None,
senders: 1,
@@ -30,17 +45,24 @@ pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
struct Inner<T> {
queue: VecDeque<T>,
parked_receiver: Option<Pid>,
/// The parked receiver's `(pid, park-epoch)`. The epoch is the slot
/// word's runtime-wide wait identity (see slot_state.rs): wakers call
/// `unpark_at(pid, epoch)`, so an entry left over from an already-woken
/// wait — a `select` loser arm, a satisfied `recv_timeout`'s timer — is
/// inert: the wake fails the word's epoch CAS and no-ops. This replaces
/// the old per-channel `cur_wait`/`next_wait_seq`/`timed_out` trio: wait
/// identity now exists exactly once, in the slot word.
parked_receiver: Option<(Pid, u32)>,
senders: usize,
receiver_alive: bool,
}
pub struct Sender<T> {
inner: Arc<Mutex<Inner<T>>>,
inner: Arc<RawMutex<Inner<T>>>,
}
pub struct Receiver<T> {
inner: Arc<Mutex<Inner<T>>>,
inner: Arc<RawMutex<Inner<T>>>,
}
#[derive(Debug, PartialEq, Eq)]
@@ -57,9 +79,30 @@ impl std::fmt::Display for RecvError {
impl std::error::Error for RecvError {}
/// Returned by [`Receiver::recv_timeout`].
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum RecvTimeoutError {
/// The deadline passed with no message available.
Timeout,
/// All senders dropped with no message available — the bounded analogue
/// of [`RecvError`].
Disconnected,
}
impl std::fmt::Display for RecvTimeoutError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
RecvTimeoutError::Timeout => write!(f, "recv timed out"),
RecvTimeoutError::Disconnected => write!(f, "channel closed"),
}
}
}
impl std::error::Error for RecvTimeoutError {}
impl<T> Clone for Sender<T> {
fn clone(&self) -> Self {
self.inner.lock().unwrap().senders += 1;
self.inner.lock().senders += 1;
Sender { inner: self.inner.clone() }
}
}
@@ -67,7 +110,7 @@ impl<T> Clone for Sender<T> {
impl<T> Drop for Sender<T> {
fn drop(&mut self) {
let unpark = {
let mut g = self.inner.lock().unwrap();
let mut g = self.inner.lock();
g.senders -= 1;
// Wake the parked receiver on the last sender drop regardless of
// whether the queue is empty. A plain `recv` only ever parks on an
@@ -81,31 +124,31 @@ impl<T> Drop for Sender<T> {
None
}
};
if let Some(pid) = unpark {
crate::scheduler::unpark(pid);
if let Some((pid, epoch)) = unpark {
crate::scheduler::unpark_at(pid, epoch);
}
}
}
impl<T> Drop for Receiver<T> {
fn drop(&mut self) {
self.inner.lock().unwrap().receiver_alive = false;
self.inner.lock().receiver_alive = false;
}
}
impl<T> Sender<T> {
pub fn send(&self, value: T) -> Result<(), SendError<T>> {
let unpark = {
let mut g = self.inner.lock().unwrap();
let mut g = self.inner.lock();
if !g.receiver_alive {
return Err(SendError(value));
}
g.queue.push_back(value);
g.parked_receiver.take()
};
if let Some(pid) = unpark {
if let Some((pid, epoch)) = unpark {
crate::te!(crate::trace::Event::Send { sender: crate::actor::current_pid().unwrap_or(crate::pid::Pid::new(u32::MAX, u32::MAX)), receiver: Some(pid) });
crate::scheduler::unpark(pid);
crate::scheduler::unpark_at(pid, epoch);
} else {
crate::te!(crate::trace::Event::Send { sender: crate::actor::current_pid().unwrap_or(crate::pid::Pid::new(u32::MAX, u32::MAX)), receiver: None });
}
@@ -117,7 +160,7 @@ impl<T> Receiver<T> {
pub fn recv(&self) -> Result<T, RecvError> {
loop {
{
let mut g = self.inner.lock().unwrap();
let mut g = self.inner.lock();
if let Some(v) = g.queue.pop_front() {
return Ok(v);
}
@@ -127,10 +170,13 @@ impl<T> Receiver<T> {
let me = crate::actor::current_pid()
.expect("recv() called outside an actor");
debug_assert!(
g.parked_receiver.is_none(),
g.parked_receiver.is_none_or(|(p, _)| p == me),
"channel has more than one receiver"
);
g.parked_receiver = Some(me);
// begin_wait is lock-free — legal under the Channel lock;
// registering in the same critical section makes the epoch
// atomic with the senders' view of the registration.
g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
crate::te!(crate::trace::Event::RecvPark(me));
}
// Release the lock before parking — the unparker will need it.
@@ -140,6 +186,75 @@ impl<T> Receiver<T> {
}
}
/// Bounded receive: like [`recv`](Self::recv), but gives up once
/// `timeout` has elapsed, returning [`RecvTimeoutError::Timeout`].
///
/// Built on the same timer machinery as `Mutex::lock_timeout`: the wait
/// registers a `WaitTimeout` entry stamped with the wait's park-epoch;
/// on expiry the channel (as the
/// [`TimerTarget`](crate::timer::TimerTarget)) checks whether *this*
/// wait is still parked and, only then, cancels it. A wake that races
/// the deadline resolves message-first: if a message is available when
/// the receiver runs, it is delivered even if the timer had already
/// fired. A satisfied or abandoned wait leaves its timer entry to expire
/// as a no-op (registration gone; epoch consumed), per the
/// no-cancellation convention in `timer.rs`.
///
/// The wake is classified from state alone — wakes are precise (the only
/// stamped wakers of this wait are a send, the last-sender drop, and the
/// timer; a stop wake unwinds out of `park_current` and never reaches
/// the classification), so: message queued → `Ok`; `senders == 0` →
/// `Disconnected`; neither → it was the timer → `Timeout`.
///
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-
/// due timer is drained, then reports `Timeout` unless a message was
/// already queued.
pub fn recv_timeout(&self, timeout: std::time::Duration) -> Result<T, RecvTimeoutError>
where
T: Send + 'static,
{
let me = crate::actor::current_pid()
.expect("recv_timeout() called outside an actor");
// Fast path + wait registration, one critical section.
let epoch;
{
let mut g = self.inner.lock();
if let Some(v) = g.queue.pop_front() {
return Ok(v);
}
if g.senders == 0 {
return Err(RecvTimeoutError::Disconnected);
}
debug_assert!(
g.parked_receiver.is_none_or(|(p, _)| p == me),
"channel has more than one receiver"
);
epoch = crate::scheduler::begin_wait();
g.parked_receiver = Some((me, epoch));
crate::te!(crate::trace::Event::RecvPark(me));
}
// Arm the timer after releasing the channel lock (insert takes the
// timers lock; never nest under a Channel lock). A send or even the
// timer itself may unpark us before we park — the RunningNotified
// protocol makes the park below return immediately in that case.
let deadline = crate::timer::deadline_from_now(timeout);
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = self.inner.clone();
crate::scheduler::insert_wait_timer(deadline, me, target, epoch);
crate::scheduler::park_current();
crate::te!(crate::trace::Event::RecvWake(crate::actor::current_pid().unwrap()));
let mut g = self.inner.lock();
if let Some(v) = g.queue.pop_front() {
return Ok(v);
}
if g.senders == 0 {
return Err(RecvTimeoutError::Disconnected);
}
Err(RecvTimeoutError::Timeout)
}
/// Selective receive: remove and return the first queued message for which
/// `pred` holds, leaving the rest in arrival order. If no queued message
/// matches, parks and re-scans on every send (a selective receiver may park
@@ -156,7 +271,7 @@ impl<T> Receiver<T> {
{
loop {
{
let mut g = self.inner.lock().unwrap();
let mut g = self.inner.lock();
if let Some(i) = g.queue.iter().position(|v| pred(v)) {
// position() found it, so remove() returns Some.
return Ok(g.queue.remove(i).unwrap());
@@ -168,10 +283,10 @@ impl<T> Receiver<T> {
let me = crate::actor::current_pid()
.expect("recv_match() called outside an actor");
debug_assert!(
g.parked_receiver.is_none(),
g.parked_receiver.is_none_or(|(p, _)| p == me),
"channel has more than one receiver"
);
g.parked_receiver = Some(me);
g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
crate::te!(crate::trace::Event::RecvPark(me));
}
// Release the lock before parking — the unparker will need it.
@@ -188,7 +303,7 @@ impl<T> Receiver<T> {
where
F: Fn(&T) -> bool,
{
let mut g = self.inner.lock().unwrap();
let mut g = self.inner.lock();
if let Some(i) = g.queue.iter().position(|v| pred(v)) {
return Ok(Some(g.queue.remove(i).unwrap()));
}
@@ -201,7 +316,7 @@ impl<T> Receiver<T> {
/// Non-blocking. `Ok(Some(v))` if a message was available, `Ok(None)` if
/// the channel is empty but open, `Err(RecvError)` if closed and drained.
pub fn try_recv(&self) -> Result<Option<T>, RecvError> {
let mut g = self.inner.lock().unwrap();
let mut g = self.inner.lock();
if let Some(v) = g.queue.pop_front() {
return Ok(Some(v));
}
@@ -211,3 +326,332 @@ impl<T> Receiver<T> {
Ok(None)
}
}
// ---------------------------------------------------------------------------
// TimerTarget — the expiry half of recv_timeout
// ---------------------------------------------------------------------------
impl<T: Send + 'static> crate::timer::TimerTarget for RawMutex<Inner<T>> {
fn on_timeout(&self, pid: Pid, epoch: u32) {
// Cancel the wait only if THIS wait (epoch match) is still
// registered. If a sender already took `parked_receiver`, the
// receiver is waking with a message — message wins, the timer
// no-ops. If a later wait by the same receiver is registered, the
// epoch mismatches — stale entry, no-op. (The unpark_at would fail
// its word CAS in either case anyway; checking under the lock keeps
// the registration bookkeeping exact.)
let unpark = {
let mut g = self.lock();
if g.parked_receiver == Some((pid, epoch)) {
g.parked_receiver = None;
true
} else {
false
}
};
// Unpark outside the channel lock — it may take the run-queue lock;
// legal under a Channel lock, but pointless to nest.
if unpark {
crate::scheduler::unpark_at(pid, epoch);
}
}
}
// ---------------------------------------------------------------------------
// select — ready-index wait over multiple receivers
// ---------------------------------------------------------------------------
pub(crate) mod sealed {
pub trait Sealed {}
}
impl<T> sealed::Sealed for Receiver<T> {}
/// An arm of a [`select`]. Implemented by [`Receiver`]; sealed, because the
/// registration contract below is part of the runtime's wake protocol.
///
/// Contract (all under the arm's own lock): `sel_register` checks-or-
/// registers atomically — if the arm is ready it does NOT register and
/// returns `Ok(false)`; otherwise it publishes `(pid, epoch)` where its
/// wakers will find it and returns `Ok(true)`. "Ready" means a receive
/// would not park: a message is queued, or the arm is closed. `Err` means
/// the arm could not register at all (only fd arms can fail; channel
/// registration is infallible) — the wait must be retired and earlier
/// eager-cleanup arms unregistered.
pub trait Selectable: sealed::Sealed {
#[doc(hidden)]
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool>;
#[doc(hidden)]
fn sel_ready(&self) -> bool;
/// Remove this arm's `(pid, epoch)` registration if — and only if — it
/// is still in place. Default no-op: a losing channel arm's stale
/// registration is inert (its wakers die at the epoch CAS; the next
/// wait overwrites the slot). Fd arms override this: their staleness
/// poisons the fd (waiters entry + kernel-side ONESHOT registration)
/// and needs an eager cleanup pass.
#[doc(hidden)]
fn sel_unregister(&self, _pid: Pid, _epoch: u32) {}
/// Whether this arm requires the eager cleanup pass at all. Gates the
/// post-wake `sel_unregister` sweep so channel-only selects keep
/// today's zero-cancellation hot path.
#[doc(hidden)]
fn sel_eager_cleanup(&self) -> bool {
false
}
}
impl<T> Selectable for Receiver<T> {
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool> {
let mut g = self.inner.lock();
if !g.queue.is_empty() || g.senders == 0 {
return Ok(false);
}
debug_assert!(
g.parked_receiver.is_none_or(|(p, _)| p == pid),
"channel has more than one receiver"
);
g.parked_receiver = Some((pid, epoch));
Ok(true)
}
fn sel_ready(&self) -> bool {
let g = self.inner.lock();
!g.queue.is_empty() || g.senders == 0
}
}
/// Park on every arm at once; return the index of the first ready one.
///
/// "Ready" means a receive on that arm would not park: a message is queued,
/// or the arm is **closed** (so the caller's `try_recv` observes the
/// disconnect — a dead arm is an event, not a hang). The caller consumes the
/// arm itself, typically via [`Receiver::try_recv`]; single-receiver
/// channels guarantee nothing can steal the message in between.
///
/// A closed arm stays ready *forever*: once its disconnect has been
/// observed, drop it from the arm set — under priority order it would
/// otherwise win every subsequent call and starve every higher-indexed arm.
///
/// Arms are scanned **in order**: index 0 is the highest priority, both on
/// the immediate-ready path and after a wake. This is a documented
/// guarantee (compose like BEAM receive clauses: put control channels
/// first), not an accident — and therefore there is NO fairness promise; a
/// saturated arm 0 starves arm 1 by design.
///
/// One actor may select on a channel and later `recv` on it (or select on
/// overlapping sets) freely. What stays illegal is what was always illegal:
/// two *different* actors receiving on one channel.
///
/// Built on the consuming-wake protocol (see slot_state.rs): all arms are
/// registered under one wait epoch; the winning wake consumes it, so losing
/// arms' registrations are inert and need no cancellation pass — they
/// self-clean at their wakers' failed CAS, or get overwritten by this
/// receiver's next wait on that channel.
///
/// Panics if `arms` is empty, when called outside an actor, or if an fd
/// arm fails to register (EBADF, EMFILE, a second waiter on one fd —
/// see [`try_select`] for the fallible form; channel-only selects cannot
/// fail).
pub fn select(arms: &[&dyn Selectable]) -> usize {
try_select(arms).expect("select(): fd arm failed to register (use try_select)")
}
/// [`select`], fallible: `Err` when an arm fails to register (only fd
/// arms can — EBADF, EMFILE on the epoll set, or a second waiter on an
/// fd that already has one). On `Err` the wait is fully retired and no
/// registration is left behind: every arm registered before the failing
/// one has been unregistered.
pub fn try_select(arms: &[&dyn Selectable]) -> std::io::Result<usize> {
assert!(!arms.is_empty(), "select() on an empty arm list");
let me = crate::actor::current_pid().expect("select() called outside an actor");
loop {
let epoch = crate::scheduler::begin_wait();
if let Some(i) = register_arms(me, epoch, arms)? {
return Ok(i);
}
// Stale fd registrations are not harmless (a losing fd arm's
// waiters entry poisons the fd with AlreadyExists and its
// kernel-side ONESHOT registration can fire arbitrarily late), so
// selects containing fd arms run an eager cleanup pass after the
// park — including when a terminal stop unwinds out of it, via
// the guard. Channel-only selects skip all of it: `eager` is
// false, the guard is disarmed, and the loser-arm self-cleaning
// story is unchanged.
let eager = arms.iter().any(|a| a.sel_eager_cleanup());
let mut guard = UnregisterGuard { arms, me, epoch, armed: eager };
crate::scheduler::park_current();
if eager {
unregister_arms(arms, me, epoch);
}
guard.armed = false;
drop(guard);
// Woken precisely: an arm's send (message) or last-sender drop
// (closure) consumed our epoch, and both leave their arm ready —
// return the first one, in priority order (which may be a
// different, higher-priority arm than the one that woke us; its
// message stays queued and re-reports ready on the next call).
// Fd arms classify by a fresh zero-timeout poll, so they too are
// a pure function of state — independent of the registration the
// cleanup pass just removed.
for (i, arm) in arms.iter().enumerate() {
if arm.sel_ready() {
return Ok(i);
}
}
// Unreachable by protocol (a stop wake unwinds out of
// park_current). Defensive: re-open the wait and re-register —
// stale own-registrations are overwritten (channels) or were
// removed by the cleanup pass above (fds).
}
}
/// Eager-cleanup sweep: remove every fd arm's registration that is still
/// ours. No-op per channel arm (one virtual call); one io-lock visit per
/// fd arm.
fn unregister_arms(arms: &[&dyn Selectable], me: Pid, epoch: u32) {
for arm in arms {
if arm.sel_eager_cleanup() {
arm.sel_unregister(me, epoch);
}
}
}
/// Stop-unwind twin of the explicit cleanup pass: a terminal stop unwinds
/// out of `park_current`, and a registered fd arm must not outlive its
/// actor (the generalization of `wait_fd`'s `Dereg`). Disarmed on the
/// normal path after the explicit pass runs; never armed when no fd arm
/// registered, keeping the channel-only path guard-free in effect.
struct UnregisterGuard<'a> {
arms: &'a [&'a dyn Selectable],
me: Pid,
epoch: u32,
armed: bool,
}
impl Drop for UnregisterGuard<'_> {
fn drop(&mut self) {
if self.armed {
unregister_arms(self.arms, self.me, self.epoch);
}
}
}
/// The registration pass shared by [`select`] and [`select_timeout`]:
/// check-or-register each arm, in priority order, each atomically under its
/// own lock. Cross-arm atomicity is unnecessary: an arm becoming ready
/// right after its registration wakes the caller through the protocol (the
/// prep-to-park window is closed by RunningNotified).
///
/// `Ok(Some(i))` = arm `i` was ready, the pass stopped, and the wait has
/// been RETIRED (no park may follow): earlier arms hold live-epoch
/// registrations, so earlier *fd* arms are unregistered eagerly, then the
/// epoch is bumped, a landed notification eaten, and a pending stop
/// re-observed — without which a stale arm wake could fault a later
/// one-shot park. `Err` = an arm failed to register; identical unwind
/// (earlier fd arms unregistered, wait retired). `Ok(None)` = every arm
/// registered; the caller parks.
fn register_arms(
me: Pid,
epoch: u32,
arms: &[&dyn Selectable],
) -> std::io::Result<Option<usize>> {
for (i, arm) in arms.iter().enumerate() {
let registered = match arm.sel_register(me, epoch) {
Ok(r) => r,
Err(e) => {
unregister_arms(&arms[..i], me, epoch);
crate::scheduler::retire_wait();
return Err(e);
}
};
if !registered {
unregister_arms(&arms[..i], me, epoch);
crate::scheduler::retire_wait();
return Ok(Some(i));
}
}
Ok(None)
}
/// The [`select_timeout`] timer target: stateless, because precise wakes
/// make classification a pure function of channel state. The entry is
/// stamped with the select's epoch; if an arm already won, this unpark dies
/// at the word's epoch CAS (the no-cancellation convention in `timer.rs`).
struct SelectTimeout;
impl crate::timer::TimerTarget for SelectTimeout {
fn on_timeout(&self, pid: Pid, epoch: u32) {
crate::scheduler::unpark_at(pid, epoch);
}
}
/// [`select`] with a deadline: returns `Some(index)` like `select`, or
/// `None` once `timeout` elapses with no arm ready.
///
/// All of `select`'s semantics carry over (priority order, closed arms
/// permanently ready, no fairness promise). The timeout is one more stamped
/// waker on the same wait epoch — nothing is registered in any arm for it,
/// so there is nothing to cancel or leak: an arm winning leaves the timer
/// entry to expire as a stale-epoch no-op; the timer winning leaves the
/// arms' registrations to self-clean exactly as a `select` loser's would.
///
/// The wake is classified from state alone (wakes are precise): some arm
/// ready → `Some` of the first, in priority order; none ready → the timer
/// was the only remaining stamped waker → `None`. A message that races the
/// deadline resolves message-first, as `recv_timeout` does.
///
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-due
/// timer is drained, then reports `None` unless an arm was already ready.
///
/// Panics if `arms` is empty, when called outside an actor, or if an fd
/// arm fails to register (see [`try_select_timeout`] for the fallible
/// form; channel-only selects cannot fail).
pub fn select_timeout(
arms: &[&dyn Selectable],
timeout: std::time::Duration,
) -> Option<usize> {
try_select_timeout(arms, timeout)
.expect("select_timeout(): fd arm failed to register (use try_select_timeout)")
}
/// [`select_timeout`], fallible: `Err` when an arm fails to register
/// (only fd arms can). On `Err` the wait is fully retired and no
/// registration — arm-side or kernel-side — is left behind.
pub fn try_select_timeout(
arms: &[&dyn Selectable],
timeout: std::time::Duration,
) -> std::io::Result<Option<usize>> {
assert!(!arms.is_empty(), "select_timeout() on an empty arm list");
let me = crate::actor::current_pid().expect("select_timeout() called outside an actor");
let epoch = crate::scheduler::begin_wait();
if let Some(i) = register_arms(me, epoch, arms)? {
return Ok(Some(i)); // ready now: the timer was never armed
}
// Arm the timer after the registration pass, outside every Channel
// lock (insert takes the timers lock).
let deadline = crate::timer::deadline_from_now(timeout);
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = std::sync::Arc::new(SelectTimeout);
crate::scheduler::insert_wait_timer(deadline, me, target, epoch);
// Same eager-cleanup story as `try_select`: the timer arm needs none
// (stateless, stale entries die at the epoch CAS), channel arms need
// none, fd arms do — and a timer win in particular leaves every fd
// arm's registration behind, which without this pass would poison
// those fds until a kernel event happened to fire.
let eager = arms.iter().any(|a| a.sel_eager_cleanup());
let mut guard = UnregisterGuard { arms, me, epoch, armed: eager };
crate::scheduler::park_current();
if eager {
unregister_arms(arms, me, epoch);
}
guard.armed = false;
drop(guard);
// Woken precisely: an arm (ready below) or the timer (nothing ready).
Ok(arms.iter().position(|arm| arm.sel_ready()))
}
+111 -11
View File
@@ -1,14 +1,114 @@
//! Cooperative context switching — public façade over the arch backend.
//! Cooperative context switching, x86-64.
//!
//! The real implementation lives in `crate::arch`, selected per `target_arch`.
//! This module re-exports the stable surface (`init_actor_stack`, the two
//! `switch_to_*` shims, and the `*_actor_sp` accessors) so the scheduler,
//! `preempt`, and the `tests/context.rs` integration tests keep importing
//! `smarm::context::{...}` unchanged regardless of which ISA is built.
//! Two naked-asm functions move execution between a scheduler thread and an
//! actor running on its own mmap'd stack. The compiler cannot do this; the
//! whole point of `#[unsafe(naked)]` is that we control every instruction.
//!
//! See `crate::arch` for the contract every backend implements, and
//! `crate::arch::{x86_64, aarch64}` for the per-ISA register choreography.
//! `SCHEDULER_SP` and `ACTOR_SP` are thread-locals holding each side's saved
//! stack pointer. `init_actor_stack` builds the initial stack so that the
//! first `switch_to_actor` lands inside the entry function with `rsp % 16 == 8`
//! (the x86-64 ABI requirement at function entry).
pub use crate::arch::{
get_actor_sp, init_actor_stack, set_actor_sp, switch_to_actor, switch_to_scheduler,
};
use std::cell::Cell;
thread_local! {
static SCHEDULER_SP: Cell<usize> = const { Cell::new(0) };
static ACTOR_SP: Cell<usize> = const { Cell::new(0) };
}
fn get_scheduler_sp() -> usize { SCHEDULER_SP.with(|c| c.get()) }
fn set_scheduler_sp(v: usize) { SCHEDULER_SP.with(|c| c.set(v)) }
pub fn get_actor_sp() -> usize { ACTOR_SP.with(|c| c.get()) }
pub fn set_actor_sp(v: usize) { ACTOR_SP.with(|c| c.set(v)) }
// ---------------------------------------------------------------------------
// Initial stack layout
//
// We start from aligned_top = top & ~15, then bias by 8 and push seven 8-byte
// slots (downward). The first `switch_to_actor` pops r15..rbx and `ret`s —
// landing in `entry` with rsp % 16 == 8 (the x86-64 ABI state at the point
// just after a `call`, which is what `entry` is compiled to expect).
//
// Layout (high → low), relative to aligned_top = top & ~15. Note the entry
// slot is at aligned_top - 16, NOT aligned_top - 8: the function does
// `(top & ~15) - 8` and *then* a `-= 8` before the first write, so the first
// stored word lands at aligned_top - 16. Verified by single-stepping the
// `ret` under llmdbg: entry sits at an address with %16 == 0, so the post-ret
// rsp is %16 == 8.
//
// aligned_top - 8 : (unused padding; keeps entry's slot %16 == 0)
// aligned_top - 16 : entry ptr ← `ret` target. Post-ret: rsp % 16 == 8.
// aligned_top - 24 : rbx = 0
// aligned_top - 32 : rbp = 0
// aligned_top - 40 : r12 = 0
// aligned_top - 48 : r13 = 0
// aligned_top - 56 : r14 = 0
// aligned_top - 64 : r15 = 0 ← initial rsp
// ---------------------------------------------------------------------------
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
unsafe {
let mut sp = (top as usize & !15) - 8;
sp -= 8; (sp as *mut usize).write(entry as usize); // ret target
sp -= 8; (sp as *mut usize).write(0); // rbx
sp -= 8; (sp as *mut usize).write(0); // rbp
sp -= 8; (sp as *mut usize).write(0); // r12
sp -= 8; (sp as *mut usize).write(0); // r13
sp -= 8; (sp as *mut usize).write(0); // r14
sp -= 8; (sp as *mut usize).write(0); // r15
sp
}
}
// ---------------------------------------------------------------------------
// Context switch shims
//
// Each shim:
// 1. Pushes the six callee-saved integer registers.
// 2. Snaps rsp into rdi and calls the Rust helper that stores it.
// 3. Calls the Rust helper that returns the *other* side's saved rsp.
// 4. Moves that into rsp.
// 5. Pops the six registers and rets.
//
// XMM registers are NOT saved here. We rely on every yield happening through
// a Rust call site, which means the compiler has spilled any live XMM state
// to the stack before we get here. (This is the same argument the compiler
// uses internally — callee-saved regs are what survive a `call`, and the
// SysV AMD64 ABI says XMM015 are all caller-saved.) If we ever yield from
// a place that isn't a Rust call boundary, this assumption breaks.
// ---------------------------------------------------------------------------
#[unsafe(naked)]
unsafe extern "C" fn switch_to_actor_asm() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_sched_sp}",
"call {get_actor_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_sched_sp = sym set_scheduler_sp,
get_actor_sp = sym get_actor_sp,
);
}
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
pub unsafe fn switch_to_actor() {
unsafe { switch_to_actor_asm() };
}
#[unsafe(naked)]
pub unsafe extern "C" fn switch_to_scheduler() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_actor_sp}",
"call {get_sched_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_actor_sp = sym set_actor_sp,
get_sched_sp = sym get_scheduler_sp,
);
}
+258 -35
View File
@@ -6,14 +6,23 @@
//! [`call`](ServerRef::call) (synchronous, returns a reply) or
//! [`cast`](ServerRef::cast) (fire-and-forget).
//!
//! ## Why a single inbox
//! ## One inbox, many arms
//!
//! smarm has no `select` and no unified per-process mailbox (see the runtime
//! gotchas in `task.md`), so a server cannot wait on a "call channel" and a
//! "cast channel" simultaneously. Instead every message — call or cast —
//! travels the *same* inbox channel as an [`Envelope`], and the server loop
//! dispatches by variant. A `call` carries a freshly made one-shot reply
//! channel; the server sends the reply straight back down it.
//! Every call and cast travels the *same* inbox channel as an [`Envelope`]
//! (calls and casts are ordered relative to each other, like Erlang); the
//! server loop dispatches by variant. A `call` carries a freshly made
//! one-shot reply channel; the server sends the reply straight back down it.
//!
//! Out-of-band messages ride *separate* channels composed at the wait via
//! [`select`](crate::channel::select): info channels handed over at start
//! ([`ServerBuilder::with_info`]) are dispatched to
//! [`handle_info`](GenServer::handle_info). Arm priority is
//! **infos before inbox**, in declaration order — a hot inbox cannot starve
//! an out-of-band message; conversely a hot info channel CAN starve the
//! inbox, deliberately (system-message semantics). An info channel whose
//! senders are all gone is silently dropped from the arm set (per the
//! closed-arm-is-ready-forever rule on `select`); a closed *inbox* still
//! means graceful shutdown.
//!
//! ## Server death
//!
@@ -36,13 +45,12 @@
//!
//! ## Not here (yet)
//!
//! No `handle_info` and no call timeout. Both need machinery smarm currently
//! defers: a call timeout needs a per-`recv` deadline (cf. the deferred
//! `Signal::Timeout`), and `handle_info` needs the cross-channel mailbox merge
//! that selective receive deliberately left unmade. They are slated to land
//! together with timeouts.
//! No dynamic info subscription: the info-channel set is fixed at start.
//! Revisit against a real consumer. (Monitor `Down` forwarding is dynamic —
//! see `handle_down` — because monitors are inherently created at runtime.)
use crate::channel::{channel, Receiver, Sender};
use crate::channel::{channel, select, Receiver, Selectable, Sender};
use crate::monitor::{Down, Monitor};
use crate::pid::Pid;
use crate::scheduler::{spawn, spawn_under};
@@ -58,9 +66,17 @@ pub trait GenServer: Send + 'static {
type Reply: Send + 'static;
/// Asynchronous request type (carried by [`ServerRef::cast`]).
type Cast: Send + 'static;
/// Out-of-band message type, delivered to [`handle_info`](Self::handle_info)
/// from the info channels registered at start
/// ([`ServerBuilder::with_info`]). Servers with several out-of-band
/// sources enum them up into one `Info`. Use `()` if unused.
type Info: Send + 'static;
/// Runs once inside the server actor before any message is handled.
fn init(&mut self) {}
/// Runs once inside the server actor before any message is handled. The
/// [`ServerCtx`] is the loop's one runtime hook: clone its [`Watcher`]
/// into the state here to be able to [`watch`](Watcher::watch) monitors
/// from any later handler.
fn init(&mut self, _ctx: &ServerCtx) {}
/// Handle a synchronous call and produce the reply sent back to the caller.
fn handle_call(&mut self, request: Self::Call) -> Self::Reply;
@@ -68,6 +84,14 @@ pub trait GenServer: Send + 'static {
/// Handle a fire-and-forget cast.
fn handle_cast(&mut self, request: Self::Cast);
/// Handle an out-of-band message from one of the info channels. Default:
/// drop it.
fn handle_info(&mut self, _info: Self::Info) {}
/// Handle a [`Down`] from a monitor handed to the loop via
/// [`Watcher::watch`]. Default: drop it.
fn handle_down(&mut self, _down: Down) {}
/// Runs as the server actor exits, on any exit path (see module docs).
fn terminate(&mut self) {}
}
@@ -100,6 +124,19 @@ pub enum CallError {
ServerDown,
}
/// Returned by [`ServerRef::call_timeout`].
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum CallTimeoutError {
/// The server was already gone, or died before replying.
ServerDown,
/// The deadline passed before a reply arrived. The request stays in the
/// server's inbox: it will still be *handled*, but the reply is discarded
/// (the abandoned reply channel's receiver is dropped, so the server's
/// reply send fails harmlessly). Erlang behaves the same way; design
/// idempotent calls accordingly.
Timeout,
}
/// Returned by [`ServerRef::cast`] when the server is no longer reachable.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum CastError {
@@ -124,6 +161,33 @@ impl<G: GenServer> ServerRef<G> {
reply_rx.recv().map_err(|_| CallError::ServerDown)
}
/// Bounded synchronous request-reply: like [`call`](Self::call), but
/// gives up after `timeout`, returning [`CallTimeoutError::Timeout`].
///
/// The roadmap sketched this as monitor + wait-reply-or-Down + demonitor;
/// that machinery is unnecessary here because server death is already
/// observable on the reply channel itself — the reply sender is dropped
/// as the server's stack unwinds, closing the channel and waking the
/// parked caller (see the module docs). So a bounded call is exactly
/// [`Receiver::recv_timeout`] on the reply channel: `Ok` on a reply,
/// `Disconnected` → [`CallTimeoutError::ServerDown`], `Timeout` →
/// [`CallTimeoutError::Timeout`]. Nothing is registered, so nothing can
/// leak on the timeout path by construction.
pub fn call_timeout(
&self,
request: G::Call,
timeout: std::time::Duration,
) -> Result<G::Reply, CallTimeoutError> {
let (reply_tx, reply_rx) = channel::<G::Reply>();
self.tx
.send(Envelope::Call(request, reply_tx))
.map_err(|_| CallTimeoutError::ServerDown)?;
reply_rx.recv_timeout(timeout).map_err(|e| match e {
crate::channel::RecvTimeoutError::Disconnected => CallTimeoutError::ServerDown,
crate::channel::RecvTimeoutError::Timeout => CallTimeoutError::Timeout,
})
}
/// Fire-and-forget request. Returns once the message is enqueued; does not
/// wait for the server to handle it. [`CastError::ServerDown`] if the inbox
/// is already closed.
@@ -134,24 +198,109 @@ impl<G: GenServer> ServerRef<G> {
}
}
/// The server loop's runtime hook, passed to [`GenServer::init`]. Currently
/// carries only the [`Watcher`]; opaque so fields can grow without breaking.
pub struct ServerCtx {
watcher: Watcher,
}
impl ServerCtx {
/// A clonable handle to the loop's monitor intake. Store it in the state
/// during `init` to watch monitors from later handlers.
pub fn watcher(&self) -> Watcher {
self.watcher.clone()
}
/// Shorthand for `ctx.watcher().watch(m)` when watching during `init`.
pub fn watch(&self, m: Monitor) {
self.watcher.watch(m)
}
}
/// Hands [`Monitor`]s to a server loop; their [`Down`]s are dispatched to
/// [`GenServer::handle_down`]. Down arms outrank every other arm (a death
/// notice cannot be starved), and a delivered `Down` retires its arm —
/// monitors are one-shot.
#[derive(Clone)]
pub struct Watcher {
tx: Sender<Monitor>,
}
impl Watcher {
/// Transfer `m` to the server loop. If the server is already gone the
/// monitor is silently dropped (its queued `Down`, if any, with it) —
/// there is no loop left to care.
pub fn watch(&self, m: Monitor) {
let _ = self.tx.send(m);
}
}
/// Configure-then-start construction for servers that need more than a bare
/// [`start`]: info channels, a supervisor. Consumed by [`start`](Self::start).
///
/// ```ignore
/// let server = ServerBuilder::new(state)
/// .with_info(events_rx)
/// .under(supervisor_pid)
/// .start();
/// ```
pub struct ServerBuilder<G: GenServer> {
state: G,
infos: Vec<Receiver<G::Info>>,
supervisor: Option<Pid>,
}
impl<G: GenServer> ServerBuilder<G> {
pub fn new(state: G) -> Self {
ServerBuilder { state, infos: Vec::new(), supervisor: None }
}
/// Add an out-of-band channel; messages arriving on it are dispatched to
/// [`GenServer::handle_info`]. Repeatable; arm priority is call order
/// (earlier = higher), and every info channel outranks the inbox.
pub fn with_info(mut self, rx: Receiver<G::Info>) -> Self {
self.infos.push(rx);
self
}
/// Spawn the server under an explicit supervisor pid (via [`spawn_under`])
/// so it slots into the supervision tree.
pub fn under(mut self, supervisor: Pid) -> Self {
self.supervisor = Some(supervisor);
self
}
/// Spawn the server actor and hand back its [`ServerRef`]. The server's
/// lifetime is governed by its refs, not by joining, so the backing join
/// handle is dropped.
pub fn start(self) -> ServerRef<G> {
let (tx, rx) = channel::<Envelope<G>>();
let ServerBuilder { state, infos, supervisor } = self;
let handle = match supervisor {
Some(sup) => spawn_under(sup, move || server_loop::<G>(rx, state, infos)),
None => spawn(move || server_loop::<G>(rx, state, infos)),
};
ServerRef { tx, pid: handle.pid() }
}
}
/// Spawn `state` as a server under the current actor (via [`spawn`]). Returns a
/// [`ServerRef`]; the server's lifetime is governed by its refs, not by
/// joining, so the backing join handle is dropped.
/// [`ServerRef`]. Shorthand for `ServerBuilder::new(state).start()`.
pub fn start<G: GenServer>(state: G) -> ServerRef<G> {
let (tx, rx) = channel::<Envelope<G>>();
let handle = spawn(move || server_loop::<G>(rx, state));
ServerRef { tx, pid: handle.pid() }
ServerBuilder::new(state).start()
}
/// Like [`start`], but spawns the server under an explicit supervisor pid (via
/// [`spawn_under`]) so it slots into the supervision tree.
pub fn start_under<G: GenServer>(supervisor: Pid, state: G) -> ServerRef<G> {
let (tx, rx) = channel::<Envelope<G>>();
let handle = spawn_under(supervisor, move || server_loop::<G>(rx, state));
ServerRef { tx, pid: handle.pid() }
ServerBuilder::new(state).under(supervisor).start()
}
fn server_loop<G: GenServer>(rx: Receiver<Envelope<G>>, state: G) {
fn server_loop<G: GenServer>(
rx: Receiver<Envelope<G>>,
state: G,
mut infos: Vec<Receiver<G::Info>>,
) {
// terminate() must run on every exit path (clean close, panic, stop), so it
// lives in this guard's Drop rather than after the loop.
struct Terminate<G: GenServer>(G);
@@ -161,22 +310,96 @@ fn server_loop<G: GenServer>(rx: Receiver<Envelope<G>>, state: G) {
}
}
let mut guard = Terminate(state);
guard.0.init();
loop {
match rx.recv() {
Ok(Envelope::Call(request, reply_tx)) => {
let reply = guard.0.handle_call(request);
fn dispatch<G: GenServer>(state: &mut G, env: Envelope<G>) {
match env {
Envelope::Call(request, reply_tx) => {
let reply = state.handle_call(request);
// The caller may have gone away (e.g. cancelled while parked);
// a failed reply send is not the server's problem.
let _ = reply_tx.send(reply);
}
Ok(Envelope::Cast(request)) => guard.0.handle_cast(request),
// All ServerRefs dropped → inbox closed → graceful shutdown.
Err(_) => break,
Envelope::Cast(request) => state.handle_cast(request),
}
// Observation point so a server whose inbox is never empty stays
}
let mut guard = Terminate(state);
// The control arm: Watchers feed Monitors to the loop through it. The
// ctx (and with it the loop's own sender) drops right after init — a
// state that didn't clone the Watcher closes the arm, the first select
// observes the closure, and the loop falls back to the plain-inbox park.
let (watch_tx, watch_rx) = channel::<Monitor>();
guard.0.init(&ServerCtx { watcher: Watcher { tx: watch_tx } });
let mut monitors: Vec<Monitor> = Vec::new();
let mut watch_open = true;
loop {
if monitors.is_empty() && !watch_open && infos.is_empty() {
// Nothing to select over: park on the inbox alone, exactly the
// pre-v0.8 loop.
match rx.recv() {
Ok(env) => dispatch(&mut guard.0, env),
// All ServerRefs dropped → inbox closed → graceful shutdown.
Err(_) => break,
}
} else {
// Arm priority: downs, then the control arm, then infos (each in
// declaration order), then the inbox. The arm slice is rebuilt
// per iteration because every set but the inbox shrinks or grows.
let nd = monitors.len();
let nw = watch_open as usize;
let i = {
let mut arms: Vec<&dyn Selectable> =
Vec::with_capacity(nd + nw + infos.len() + 1);
for m in &monitors {
arms.push(&m.rx);
}
if watch_open {
arms.push(&watch_rx);
}
for r in &infos {
arms.push(r);
}
arms.push(&rx);
select(&arms)
};
if i < nd {
// A Down retires its arm either way: delivered (one-shot) or
// closed without delivering (defensive; shouldn't happen).
let m = monitors.remove(i);
if let Ok(Some(down)) = m.rx.try_recv() {
guard.0.handle_down(down);
}
} else if i < nd + nw {
match watch_rx.try_recv() {
Ok(Some(m)) => monitors.push(m),
// Single-receiver: nothing can drain the arm between
// select's ready and our try_recv.
Ok(None) => debug_assert!(false, "ready control arm was empty"),
// Every Watcher gone: stop selecting on the arm.
Err(_) => watch_open = false,
}
} else if i < nd + nw + infos.len() {
let j = i - nd - nw;
match infos[j].try_recv() {
Ok(Some(info)) => guard.0.handle_info(info),
Ok(None) => debug_assert!(false, "ready info arm was empty"),
// Senders all gone: drop the arm, keep serving. `remove`
// (not swap_remove) — order is priority.
Err(_) => {
infos.remove(j);
}
}
} else {
match rx.try_recv() {
Ok(Some(env)) => dispatch(&mut guard.0, env),
Ok(None) => debug_assert!(false, "ready inbox was empty"),
Err(_) => break,
}
}
}
// Observation point so a server whose arms are never empty stays
// preemptible and cancellable.
crate::check!();
}
+31 -16
View File
@@ -44,11 +44,13 @@
//!
//! Fd hygiene
//! ==========
//! If an actor dies while waiting on an fd, the registration is leaked
//! (the fd stays in the epollfd, armed). EPOLLONESHOT bounds the damage:
//! at most one stale wakeup, after which the kernel disarms. The stale
//! wakeup hits a dead pid in `waiters` and is dropped. Acceptable for v0.2;
//! a future pass should DEL on actor death.
//! An actor stopped while waiting on an fd unwinds out of `wait_fd`'s park;
//! a drop guard there (armed after a successful register, forgotten on a
//! normal wake) removes the `waiters` entry iff it is still that wait's
//! `(pid, epoch)` and only then `EPOLL_CTL_DEL`s the fd — an entry already
//! consumed by a racing `FdReady` means the fd may carry someone else's
//! fresh registration, which must be left alone. `epoll_register` keeps a
//! defensive bare DEL before ADD as belt-and-braces.
//!
//! Buffers used with `read`/`write` should be on fds opened with
//! `O_NONBLOCK`. If they aren't, the syscall may block the scheduler
@@ -84,6 +86,9 @@ use std::thread::JoinHandle as OsJoinHandle;
pub type IoResult = Result<Box<dyn Any + Send>, Box<dyn Any + Send>>;
struct Request {
/// The submitter's park-epoch — carried through to the `Blocking`
/// completion so the wake is epoch-matched.
epoch: u32,
pid: Pid,
/// The work to perform. Returns the wire-form result directly.
work: Box<dyn FnOnce() -> IoResult + Send>,
@@ -93,7 +98,7 @@ struct Request {
pub enum Completion {
/// A `block_on_io` closure has finished (Ok = return value, Err = panic
/// payload).
Blocking { pid: Pid, result: IoResult },
Blocking { pid: Pid, epoch: u32, result: IoResult },
/// An fd registered via `wait_readable`/`wait_writable` is ready. The
/// scheduler looks up the parked pid in `waiters`, unparks it, and
/// removes the entry. `pid` isn't in this variant because the epoll
@@ -131,7 +136,7 @@ pub struct IoThread {
/// One parked actor per registered fd. Populated by `wait_readable` /
/// `wait_writable` and drained by the scheduler when a `FdReady`
/// completion is processed.
pub waiters: HashMap<RawFd, Pid>,
pub waiters: HashMap<RawFd, (Pid, u32)>,
// ----- Threads -----
@@ -231,12 +236,12 @@ impl IoThread {
}
/// Hand a request to the pool. Increments `outstanding`.
pub fn submit(&mut self, pid: Pid, work: Box<dyn FnOnce() -> IoResult + Send>) {
pub fn submit(&mut self, pid: Pid, epoch: u32, work: Box<dyn FnOnce() -> IoResult + Send>) {
self.outstanding += 1;
// Send can only fail if the pool has hung up, which only happens
// on shutdown. submit during shutdown is a bug.
self.tx
.send(Request { pid, work })
.send(Request { pid, epoch, work })
.expect("io pool hung up unexpectedly");
}
@@ -255,6 +260,15 @@ impl IoThread {
self.wake_read
}
/// Write the wake pipe directly: rouse every scheduler thread blocked in
/// its idle `poll_wake`. Used by the terminal (AllDone) path — an idle
/// sibling may be blocked on a snapshot that nothing will ever refresh
/// (an orphaned timer deadline, or `io_outstanding` from a waiter that
/// was stop-cancelled and so never produces a completion).
pub fn wake(&self) {
wake_scheduler(self.wake_write);
}
/// Register interest in `fd` becoming readable/writable; record `pid`
/// as the parked waiter. The epoll thread will push a `FdReady`
/// completion when the kernel signals.
@@ -265,6 +279,7 @@ impl IoThread {
&mut self,
fd: RawFd,
pid: Pid,
epoch: u32,
readable: bool,
writable: bool,
) -> io::Result<()> {
@@ -278,10 +293,10 @@ impl IoThread {
));
}
// Defensive cleanup: if a previous actor died while waiting on this
// fd, the kernel-side registration was leaked (we don't walk all
// waiters on actor death). A bare DEL is harmless if the fd isn't
// registered (ENOENT), and removes any leak.
// Belt-and-braces: the unwind guard in `wait_fd` is responsible for
// cleaning up a stopped waiter's registration, but a bare DEL is
// harmless if the fd isn't registered (ENOENT) and removes any leak
// a path we haven't thought of might leave behind.
unsafe {
libc::epoll_ctl(self.epollfd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut());
}
@@ -303,7 +318,7 @@ impl IoThread {
if r < 0 {
return Err(io::Error::last_os_error());
}
self.waiters.insert(fd, pid);
self.waiters.insert(fd, (pid, epoch));
Ok(())
}
@@ -369,7 +384,7 @@ fn pool_loop(
completions: Arc<Mutex<VecDeque<Completion>>>,
wake_write: RawFd,
) {
while let Ok(Request { pid, work }) = rx.recv() {
while let Ok(Request { pid, epoch, work }) = rx.recv() {
let result: IoResult = match panic::catch_unwind(panic::AssertUnwindSafe(work)) {
Ok(r) => r,
Err(payload) => Err(payload),
@@ -377,7 +392,7 @@ fn pool_loop(
completions
.lock()
.unwrap()
.push_back(Completion::Blocking { pid, result });
.push_back(Completion::Blocking { pid, epoch, result });
wake_scheduler(wake_write);
}
}
+14 -4
View File
@@ -11,7 +11,6 @@
//!
//! See `LOOM.md` for the design intent and the deferred-for-later list.
pub mod arch;
pub mod stack;
pub mod context;
pub mod preempt;
@@ -24,9 +23,15 @@ pub mod timer;
pub mod io;
pub mod mutex;
pub mod monitor;
pub mod registry;
pub mod link;
pub mod gen_server;
pub mod runtime;
pub(crate) mod raw_mutex;
pub(crate) mod slot_state;
pub(crate) mod sync_shim;
#[doc(hidden)] // pub only so benches/rq_micro.rs can drive the raw structures
pub mod run_queue;
pub mod trace;
// ---------------------------------------------------------------------------
@@ -40,16 +45,21 @@ static ALLOCATOR: preempt::PreemptingAllocator = preempt::PreemptingAllocator;
// Public API re-exports
// ---------------------------------------------------------------------------
pub use channel::{channel, Receiver, RecvError, Sender};
pub use gen_server::{CallError, CastError, GenServer, ServerRef};
pub use channel::{
channel, select, select_timeout, try_select, try_select_timeout, Receiver, RecvError,
RecvTimeoutError, Selectable, Sender,
};
pub use gen_server::{CallError, CallTimeoutError, CastError, GenServer, ServerBuilder, ServerCtx, ServerRef, Watcher};
pub use link::{link, trap_exit, unlink, ExitSignal};
pub use monitor::{demonitor, monitor, Down, DownReason, Monitor, MonitorId};
pub use mutex::{LockTimeout, Mutex, MutexGuard};
pub use pid::Pid;
pub use registry::{name_of, register, unregister, whereis, RegisterError};
pub use runtime::{init, Config, Runtime};
pub use scheduler::{
block_on_io, request_stop, run, self_pid, sleep, spawn, spawn_under, wait_readable,
wait_writable, yield_now, JoinError, JoinHandle,
wait_readable_timeout, wait_writable, wait_writable_timeout, yield_now, FdArm, JoinError,
JoinHandle,
};
pub use supervisor::{ChildSpec, OneForOne, Restart, Signal, Strategy};
+66 -44
View File
@@ -49,10 +49,9 @@
//! [`Down`]: crate::monitor::Down
//! [`request_stop`]: crate::scheduler::request_stop
use crate::channel::{channel, Receiver, Sender};
use crate::channel::{channel, Receiver};
use crate::monitor::DownReason;
use crate::pid::Pid;
use crate::runtime::State;
use crate::scheduler::{request_stop, self_pid, with_runtime};
/// A linked peer's death, delivered to a trapping actor's inbox.
@@ -78,11 +77,13 @@ pub fn trap_exit() -> Receiver<ExitSignal> {
let (tx, rx) = channel::<ExitSignal>();
let me = self_pid();
with_runtime(|inner| {
inner.with_shared(|s| {
if let Some(actor) = s.slot_mut(me).and_then(|slot| slot.actor.as_mut()) {
if let Some(slot) = inner.slot_at(me) {
let mut cold = slot.cold.lock();
// Own slot: generation is necessarily current (we're running).
if let Some(actor) = cold.actor.as_mut() {
actor.trap = Some(tx);
}
})
}
});
rx
}
@@ -100,45 +101,61 @@ pub fn link(target: Pid) {
return;
}
// Under the lock: if the target is live, record the link both ways and
// return `None`. If it is gone, return the caller's trap sender (if any)
// so we can deliver the NoProc signal after releasing the lock.
let dead_action: Option<Option<Sender<ExitSignal>>> = with_runtime(|inner| {
inner.with_shared(|s| {
let target_live = matches!(
s.slot(target),
Some(slot) if slot.actor.is_some() && !matches!(slot.state, State::Done)
);
if target_live {
if let Some(slot) = s.slot_mut(me) {
if !slot.links.contains(&target) {
slot.links.push(target);
}
// Cold locks are leaves: never hold two at once. The link is recorded one
// side at a time, TARGET FIRST — that ordering is what makes the race
// window sound:
//
// - Once `me` is in `target.links`, the target's death always reaches us
// (its finalize cascade walks that list). So after step 1 succeeds, the
// link semantics are already live.
// - If the target dies between step 1 and step 2, its cascade removes
// `target` from OUR links (a no-op, we haven't added it yet) and
// delivers the exit signal — correct, the link was established. Our
// subsequent step-2 insert leaves a stale `target` entry in `me.links`;
// stale entries are benign by construction (every cascade walk
// re-verifies the peer's word; `unlink` removes them like any other).
//
// The reverse order would be unsound: target dying in the window would
// walk its links WITHOUT us — a silently dead link that we believe is live.
let registered_on_target = with_runtime(|inner| match inner.slot_at(target) {
Some(slot) => {
let mut cold = slot.cold.lock();
if slot.is_live_for(target) && cold.actor.is_some() {
if !cold.links.contains(&me) {
cold.links.push(me);
}
if let Some(slot) = s.slot_mut(target) {
if !slot.links.contains(&me) {
slot.links.push(me);
}
}
None
true
} else {
// Grab our own trap sender so the NoProc delivery (below)
// doesn't need a second lock acquisition.
Some(
s.slot(me)
.and_then(|slot| slot.actor.as_ref())
.and_then(|a| a.trap.clone()),
)
false
}
})
}
None => false,
});
match dead_action {
None => {} // linked successfully
Some(Some(tx)) => {
if registered_on_target {
with_runtime(|inner| {
let slot = inner.slot_at(me).expect("link: own slot vanished");
let mut cold = slot.cold.lock();
if !cold.links.contains(&target) {
cold.links.push(target);
}
});
return;
}
// Target already gone: deliver NoProc to ourselves — as a message if
// trapping, otherwise as a cooperative stop.
let my_trap = with_runtime(|inner| {
inner.slot_at(me).and_then(|slot| {
let cold = slot.cold.lock();
cold.actor.as_ref().and_then(|a| a.trap.clone())
})
});
match my_trap {
Some(tx) => {
let _ = tx.send(ExitSignal { from: target, reason: DownReason::NoProc });
}
Some(None) => request_stop(me),
None => request_stop(me),
}
}
@@ -152,13 +169,18 @@ pub fn unlink(target: Pid) {
return;
}
with_runtime(|inner| {
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(me) {
slot.links.retain(|p| *p != target);
// One cold lock at a time (leaf rule). Order is immaterial here:
// a half-removed link is just a stale entry on one side, and stale
// entries are benign (re-verified on every cascade walk).
if let Some(slot) = inner.slot_at(me) {
let mut cold = slot.cold.lock();
cold.links.retain(|p| *p != target);
}
if let Some(slot) = inner.slot_at(target) {
let mut cold = slot.cold.lock();
if slot.generation() == target.generation() {
cold.links.retain(|p| *p != me);
}
if let Some(slot) = s.slot_mut(target) {
slot.links.retain(|p| *p != me);
}
})
}
});
}
+28 -25
View File
@@ -47,7 +47,6 @@
use crate::channel::{channel, Receiver, Sender};
use crate::pid::Pid;
use crate::runtime::State;
use crate::scheduler::with_runtime;
/// Why a monitored actor went down.
@@ -110,24 +109,26 @@ pub struct Monitor {
pub fn monitor(target: Pid) -> Monitor {
let (tx, rx) = channel::<Down>();
// Register under the shared lock. We allocate the id and (if the target is
// live) clone the sender into its monitor list, keeping the original `tx`
// for the NoProc fallback. `tx.clone()` only touches the channel's own
// mutex, never the shared runtime mutex, so it is safe under the lock — but
// we must not *send* here, as `Sender::send` can call back in to unpark a
// parked receiver and the shared mutex is not reentrant.
// Register under the target's cold lock. `tx.clone()` takes the channel's
// own lock — a Channel-class RawMutex, explicitly permitted *under* a Leaf
// (cold) lock by the lock order (see raw_mutex.rs). We must still not
// *send* under the lock, as `Sender::send` can unpark a parked receiver,
// and there's no reason to nest that.
let (id, registered) = with_runtime(|inner| {
inner.with_shared(|s| {
let id = s.alloc_monitor_id();
let registered = match s.slot_mut(target) {
Some(slot) if !matches!(slot.state, State::Done) => {
slot.monitors.push((id, tx.clone()));
let id = inner.alloc_monitor_id();
let registered = match inner.slot_at(target) {
Some(slot) => {
let mut cold = slot.cold.lock();
if slot.is_live_for(target) {
cold.monitors.push((id, tx.clone()));
true
} else {
false
}
_ => false,
};
(id, registered)
})
}
None => false,
};
(id, registered)
});
if !registered {
@@ -147,16 +148,18 @@ pub fn monitor(target: Pid) -> Monitor {
/// dropping the [`Monitor`] closes its receiver and the queued notice goes with
/// it — the analogue of Erlang's `demonitor(Ref, [flush])`.
pub fn demonitor(m: &Monitor) -> Option<MonitorId> {
// Remove the registration under the lock, but move the `Sender` *out* and
// let it drop only after the lock is released: dropping the last sender
// runs `Sender::drop`, which may unpark a parked receiver → `with_shared`,
// and the shared mutex is not reentrant.
// Remove the registration under the target's cold lock, but move the
// `Sender` *out* and let it drop only after the lock is released:
// dropping the last sender runs `Sender::drop`, which may unpark a parked
// receiver — legal under a cold lock, but pointless to nest.
let removed: Option<(MonitorId, Sender<Down>)> = with_runtime(|inner| {
inner.with_shared(|s| {
let slot = s.slot_mut(m.target)?;
let pos = slot.monitors.iter().position(|(mid, _)| *mid == m.id)?;
Some(slot.monitors.remove(pos))
})
let slot = inner.slot_at(m.target)?;
let mut cold = slot.cold.lock();
if slot.generation() != m.target.generation() {
return None; // slot reused; the Down already fired
}
let pos = cold.monitors.iter().position(|(mid, _)| *mid == m.id)?;
Some(cold.monitors.remove(pos))
});
// `removed`'s sender drops here, outside the lock.
removed.map(|(id, _sender)| id)
+24 -18
View File
@@ -33,13 +33,15 @@ impl std::error::Error for LockTimeout {}
struct Wait {
pid: Pid,
seq: u64,
/// The wait's park-epoch (slot-word wait identity, see slot_state.rs).
/// Grants and timeouts wake via `unpark_at(pid, epoch)`; a stale entry
/// can neither be granted by mistake nor wake the wrong wait.
epoch: u32,
}
struct MutexState {
holder: Option<Pid>,
waiters: VecDeque<Wait>,
next_seq: u64,
default_timeout: Duration,
}
@@ -53,7 +55,6 @@ impl MutexCore {
state: StdMutex::new(MutexState {
holder: None,
waiters: VecDeque::new(),
next_seq: 0,
default_timeout,
}),
}
@@ -61,17 +62,17 @@ impl MutexCore {
}
impl TimerTarget for MutexCore {
fn on_timeout(&self, pid: Pid, wait_seq: u64) {
fn on_timeout(&self, pid: Pid, epoch: u32) {
let unpark = {
let mut st = self.state.lock().unwrap();
// Remove from waiters only if still there with matching seq.
// Remove from waiters only if still there with matching epoch.
// If the lock was already granted (holder == Some(pid)), the
// timer fired after the grant — treat as no-op; the actor
// will see `is_holder == true` and return Ok.
if st.holder == Some(pid) {
return;
}
let pos = st.waiters.iter().position(|w| w.pid == pid && w.seq == wait_seq);
let pos = st.waiters.iter().position(|w| w.pid == pid && w.epoch == epoch);
if pos.is_some() {
st.waiters.remove(pos.unwrap());
true
@@ -80,7 +81,7 @@ impl TimerTarget for MutexCore {
}
};
if unpark {
scheduler::unpark(pid);
scheduler::unpark_at(pid, epoch);
}
}
}
@@ -133,20 +134,25 @@ impl<T> Mutex<T> {
// Slow path: register as a waiter, set timeout, park.
let _np = scheduler::NoPreempt::enter();
let seq = {
let epoch = {
let mut st = self.core.state.lock().unwrap();
let seq = st.next_seq;
st.next_seq = st.next_seq.wrapping_add(1);
st.waiters.push_back(Wait { pid: me, seq });
seq
// begin_wait is lock-free — legal under the state lock; this
// makes the epoch atomic with the registration's visibility to
// grants and timeouts.
let epoch = scheduler::begin_wait();
st.waiters.push_back(Wait { pid: me, epoch });
epoch
};
let target: Arc<dyn TimerTarget> = self.core.clone();
let deadline = timer::deadline_from_now(timeout);
scheduler::insert_wait_timer(deadline, me, target, seq);
scheduler::insert_wait_timer(deadline, me, target, epoch);
scheduler::park_current();
// Resumed. Are we the holder?
// Resumed — precisely: only our grant or our timer can wake this
// wait (both epoch-stamped; a stop wake unwinds out of
// park_current). The one-shot interpretation below is therefore
// exhaustive. Are we the holder?
let is_holder = self.core.state.lock().unwrap().holder == Some(me);
if is_holder {
let value = self.value.lock().unwrap().take()
@@ -228,12 +234,12 @@ impl<T> Drop for MutexGuard<'_, T> {
let v = self.value.take().expect("MutexGuard: double drop");
*self.mutex.value.lock().unwrap() = Some(v);
let next_pid = {
let next = {
let mut st = self.mutex.core.state.lock().unwrap();
match st.waiters.pop_front() {
Some(w) => {
st.holder = Some(w.pid);
Some(w.pid)
Some((w.pid, w.epoch))
}
None => {
st.holder = None;
@@ -241,8 +247,8 @@ impl<T> Drop for MutexGuard<'_, T> {
}
}
};
if let Some(pid) = next_pid {
scheduler::unpark(pid);
if let Some((pid, epoch)) = next {
scheduler::unpark_at(pid, epoch);
}
}
}
+21 -18
View File
@@ -30,17 +30,7 @@ use std::cell::Cell;
use std::sync::atomic::{AtomicBool, Ordering};
pub const DEFAULT_ALLOC_INTERVAL: u32 = 128;
// The timeslice is measured in `rdtsc()` ticks, and the tick *unit* differs by
// ISA (see `crate::arch::read_cycle_counter`). Both constants target ~100µs.
// x86-64 : TSC ≈ CPU base clock; 300_000 ticks ≈ 100µs on a 3 GHz core.
// aarch64: CNTVCT runs at CNTFRQ_EL0, commonly ~24 MHz; 2_400 ticks ≈ 100µs.
// A real implementation would read the frequency at startup and compute this;
// for now the constant is calibrated per-arch. Tune on-device if needed.
#[cfg(target_arch = "x86_64")]
pub const DEFAULT_TIMESLICE_CYCLES: u64 = 300_000; // ≈ 100µs on a 3 GHz CPU
#[cfg(target_arch = "aarch64")]
pub const DEFAULT_TIMESLICE_CYCLES: u64 = 2_400; // ≈ 100µs at a 24 MHz CNTVCT
thread_local! {
/// While `false`, the allocator hook is a no-op.
@@ -126,12 +116,14 @@ pub fn reset_timeslice() {
TIMESLICE_START.with(|c| c.set(rdtsc()));
}
/// Per-core cycle counter for the timeslice clock. Delegates to the active
/// arch backend (`rdtsc` on x86-64, `CNTVCT_EL0` on aarch64). The tick unit
/// is ISA-defined, so `DEFAULT_TIMESLICE_CYCLES` is calibrated per-arch below.
#[inline(always)]
pub fn rdtsc() -> u64 {
crate::arch::read_cycle_counter()
unsafe {
// SAFETY: x86-64 only. `lfence` serialises the instruction stream so
// we don't measure time before prior instructions retire.
core::arch::asm!("lfence", options(nostack, nomem, preserves_flags));
core::arch::x86_64::_rdtsc()
}
}
pub struct PreemptingAllocator;
@@ -179,11 +171,22 @@ pub fn maybe_preempt() {
let n = c.get();
if n == 0 {
c.set(CONFIGURED_ALLOC_INTERVAL.with(|i| i.get()));
// Cooperative cancellation shares the amortised cadence with the
// timeslice check. Observe a pending stop first: if we are being
// cancelled there is no point yielding, we unwind instead.
check_cancelled();
if PREEMPTION_ENABLED.with(|e| e.get()) {
// Cooperative cancellation shares the amortised cadence with
// the timeslice check, and shares its gate: while preemption
// is disabled (`NoPreempt`, `with_shared`, channel critical
// sections) the stop sentinel must NOT be raised, because an
// allocation-triggered unwind inside a region holding a
// `std::sync::Mutex` would poison it — one `request_stop` at
// the wrong moment would then cascade `lock().unwrap()`
// panics through every later user of that lock. Observation
// is merely deferred to the next enabled allocation or the
// wakeup side of the next park/yield, both of which are
// lock-free points by construction.
//
// Observe a pending stop first: if we are being cancelled
// there is no point yielding, we unwind instead.
check_cancelled();
let start = TIMESLICE_START.with(|s| s.get());
if rdtsc().saturating_sub(start) > CONFIGURED_TIMESLICE_CYCLES.with(|t| t.get()) {
// SAFETY: reachable only inside an actor (the scheduler
+360
View File
@@ -0,0 +1,360 @@
//! A minimal futex-based mutex that cannot poison.
//!
//! `std::sync::Mutex` poisons on unwind, turning one panic into a cascade of
//! `lock().unwrap()` panics in every later user. The runtime's internal
//! critical sections must never unwind anyway (the stop sentinel is gated
//! behind `PREEMPTION_ENABLED`, and the guard below disables preemption), so
//! poisoning buys nothing and costs a failure mode. This mutex has no poison
//! state by construction.
//!
//! Two further properties the runtime wants:
//!
//! - **The guard enters `NoPreempt`.** A timeslice switch while holding an OS
//! mutex would suspend the actor with the lock held, stalling every other
//! OS thread that touches it until the actor is resumed. Disabling
//! preemption for the (short) critical section keeps lock hold times
//! bounded. It also closes the unwind hole structurally: with
//! `PREEMPTION_ENABLED` false, `maybe_preempt` neither yields nor raises
//! the stop sentinel, so no allocation inside the critical section can
//! unwind it.
//! - **No std machinery.** One `AtomicU32` and two futex syscalls; friendlier
//! to an eventual embedded port than `std::sync::Mutex` (swap the futex for
//! a spin or WFE backend).
//!
//! Algorithm: the classic three-state futex mutex (Drepper, "Futexes Are
//! Tricky", mutex3). 0 = unlocked, 1 = locked, 2 = locked with (possible)
//! waiters. Uncontended lock/unlock is one CAS / one swap, no syscall.
//!
//! Lock-order position — two classes (see [`LockClass`]):
//!
//! - **Leaf**: slot cold locks, the free list, the stack pool, the name
//! registry. Mutual leaves — never hold two at once.
//! - **Channel**: a channel's internal lock. May be acquired *under* a Leaf
//! (finalize clones the supervisor/trap senders, and `monitor()` clones the
//! Down sender, all under a cold lock — structural, the sender lives in the
//! slot), but nothing may be acquired under a Channel lock: channel
//! critical sections call only the lock-free unpark protocol.
//!
//! So the total order is Leaf → Channel, one of each at most. Holding either
//! while pushing to the run queue is permitted (unpark from inside a cold or
//! channel section); the reverse — taking any `RawMutex` from inside a
//! run-queue op — cannot arise (queue ops call nothing).
use std::cell::UnsafeCell;
use std::ops::{Deref, DerefMut};
use std::sync::atomic::{AtomicU32, Ordering};
const UNLOCKED: u32 = 0;
const LOCKED: u32 = 1;
const CONTENDED: u32 = 2;
/// How many `pause` spins to burn before falling back to the futex. Critical
/// sections under this lock are tens of nanoseconds (push to a Vec, clone a
/// sender), so a short spin almost always avoids the syscall.
const SPIN_LIMIT: u32 = 64;
/// Which rung of the two-rung lock order a `RawMutex` occupies. Debug builds
/// enforce the order mechanically (see the module docs); release builds carry
/// no state and no checks.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum LockClass {
/// Runtime cold data: slot cold locks, free list, stack pool, registry.
/// Mutual leaves among themselves; a Channel lock may be taken under one.
Leaf,
/// A channel's internal lock. One at a time, nothing acquired under it;
/// may itself be acquired under a Leaf.
Channel,
}
// The ordering rules, mechanically enforced (debug builds): a deadlock from a
// violated order is a hang waiting for the right interleaving, so fail at the
// acquisition that violates it, not in the eventual hang.
#[cfg(debug_assertions)]
thread_local! {
static LEAVES_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
static CHANNELS_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
}
#[inline]
fn order_check_acquire(class: LockClass) {
#[cfg(debug_assertions)]
match class {
LockClass::Leaf => LEAVES_HELD.with(|l| {
debug_assert_eq!(
l.get(),
0,
"lock order violated: acquiring a Leaf RawMutex while already \
holding one (cold locks / free list / stack pool / registry \
are mutual leaves)"
);
CHANNELS_HELD.with(|c| {
debug_assert_eq!(
c.get(),
0,
"lock order violated: acquiring a Leaf RawMutex under a \
channel lock (order is Leaf -> Channel, never the reverse)"
);
});
l.set(l.get() + 1);
}),
LockClass::Channel => CHANNELS_HELD.with(|c| {
debug_assert_eq!(
c.get(),
0,
"lock order violated: acquiring a channel lock while already \
holding one (channel locks are mutual leaves)"
);
c.set(c.get() + 1);
}),
}
#[cfg(not(debug_assertions))]
let _ = class;
}
#[inline]
fn order_check_release(class: LockClass) {
#[cfg(debug_assertions)]
match class {
LockClass::Leaf => LEAVES_HELD.with(|c| c.set(c.get() - 1)),
LockClass::Channel => CHANNELS_HELD.with(|c| c.set(c.get() - 1)),
}
#[cfg(not(debug_assertions))]
let _ = class;
}
pub(crate) struct RawMutex<T> {
state: AtomicU32,
class: LockClass,
data: UnsafeCell<T>,
}
// SAFETY: standard mutex argument — exclusive access to `data` is mediated by
// `state`; `T: Send` suffices for both because `&RawMutex` only ever hands out
// access to one thread at a time.
unsafe impl<T: Send> Send for RawMutex<T> {}
unsafe impl<T: Send> Sync for RawMutex<T> {}
impl<T> RawMutex<T> {
/// A Leaf-class mutex — the default for runtime cold data.
pub(crate) const fn new(data: T) -> Self {
Self::with_class(data, LockClass::Leaf)
}
/// A Channel-class mutex — for channel internals only.
pub(crate) const fn new_channel(data: T) -> Self {
Self::with_class(data, LockClass::Channel)
}
pub(crate) const fn with_class(data: T, class: LockClass) -> Self {
Self {
state: AtomicU32::new(UNLOCKED),
class,
data: UnsafeCell::new(data),
}
}
#[inline]
pub(crate) fn lock(&self) -> RawMutexGuard<'_, T> {
// Enter NoPreempt *before* acquiring, so a preemption can't fire
// between acquisition and guard construction.
let prev_preempt = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
order_check_acquire(self.class);
if self
.state
.compare_exchange(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_err()
{
self.lock_slow();
}
RawMutexGuard { m: self, prev_preempt }
}
#[cold]
fn lock_slow(&self) {
// Bounded spin first: the expected hold time is far below the cost of
// a futex round trip.
let mut spins = 0;
loop {
let s = self.state.load(Ordering::Relaxed);
if s == UNLOCKED
&& self
.state
.compare_exchange_weak(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_ok()
{
return;
}
spins += 1;
if spins >= SPIN_LIMIT {
break;
}
std::hint::spin_loop();
}
// Futex path. Mark contended and sleep until woken; on wake, retake
// by swapping to CONTENDED (we cannot know whether other waiters
// remain, so we must conservatively keep the contended marker).
while self.state.swap(CONTENDED, Ordering::Acquire) != UNLOCKED {
futex_wait(&self.state, CONTENDED);
}
}
#[inline]
fn unlock(&self) {
if self.state.swap(UNLOCKED, Ordering::Release) == CONTENDED {
futex_wake(&self.state, 1);
}
}
}
pub(crate) struct RawMutexGuard<'a, T> {
m: &'a RawMutex<T>,
prev_preempt: bool,
}
impl<T> Deref for RawMutexGuard<'_, T> {
type Target = T;
#[inline]
fn deref(&self) -> &T {
// SAFETY: guard existence implies exclusive ownership of the lock.
unsafe { &*self.m.data.get() }
}
}
impl<T> DerefMut for RawMutexGuard<'_, T> {
#[inline]
fn deref_mut(&mut self) -> &mut T {
// SAFETY: as above, plus &mut self.
unsafe { &mut *self.m.data.get() }
}
}
impl<T> Drop for RawMutexGuard<'_, T> {
#[inline]
fn drop(&mut self) {
self.m.unlock();
order_check_release(self.m.class);
// Restore preemption only after the lock is released.
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(self.prev_preempt));
}
}
// ---------------------------------------------------------------------------
// futex (x86-64 Linux; master is x86-only, see arm-port branch)
// ---------------------------------------------------------------------------
fn futex_wait(state: &AtomicU32, expected: u32) {
// SAFETY: `state` is a valid, aligned u32 for the duration of the call.
// Spurious wakeups and EAGAIN (value already changed) are both handled by
// the caller's retry loop.
unsafe {
libc::syscall(
libc::SYS_futex,
state.as_ptr(),
libc::FUTEX_WAIT | libc::FUTEX_PRIVATE_FLAG,
expected,
std::ptr::null::<libc::timespec>(),
);
}
}
fn futex_wake(state: &AtomicU32, n: i32) {
// SAFETY: as above.
unsafe {
libc::syscall(
libc::SYS_futex,
state.as_ptr(),
libc::FUTEX_WAKE | libc::FUTEX_PRIVATE_FLAG,
n,
);
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[test]
fn uncontended_lock_unlock() {
let m = RawMutex::new(0u64);
for _ in 0..1000 {
*m.lock() += 1;
}
assert_eq!(*m.lock(), 1000);
}
#[test]
fn contended_counter_is_exact() {
const THREADS: usize = 8;
const PER: u64 = 50_000;
let m = Arc::new(RawMutex::new(0u64));
let hs: Vec<_> = (0..THREADS)
.map(|_| {
let m = m.clone();
std::thread::spawn(move || {
for _ in 0..PER {
*m.lock() += 1;
}
})
})
.collect();
for h in hs {
h.join().unwrap();
}
assert_eq!(*m.lock(), THREADS as u64 * PER);
}
#[test]
fn no_poison_on_unwind() {
let m = Arc::new(RawMutex::new(0u64));
let m2 = m.clone();
let _ = std::thread::spawn(move || {
let _g = m2.lock();
panic!("unwind while holding");
})
.join();
// A std Mutex would now be poisoned; this one just works.
*m.lock() += 1;
assert_eq!(*m.lock(), 1);
}
#[test]
fn channel_lock_nests_under_leaf() {
// The permitted ordering: Leaf -> Channel (finalize/monitor clone a
// sender under a cold lock). Must not trip the order check.
let leaf = RawMutex::new(0u64);
let chan = RawMutex::new_channel(0u64);
let _l = leaf.lock();
let _c = chan.lock();
}
#[cfg(debug_assertions)]
#[test]
#[should_panic(expected = "lock order violated")]
fn leaf_under_channel_is_rejected() {
let leaf = RawMutex::new(0u64);
let chan = RawMutex::new_channel(0u64);
let _c = chan.lock();
let _l = leaf.lock(); // Channel -> Leaf: forbidden
}
#[cfg(debug_assertions)]
#[test]
#[should_panic(expected = "lock order violated")]
fn two_leaves_are_rejected() {
let a = RawMutex::new(0u64);
let b = RawMutex::new(0u64);
let _ga = a.lock();
let _gb = b.lock(); // leaves are mutual: forbidden
}
#[cfg(debug_assertions)]
#[test]
#[should_panic(expected = "lock order violated")]
fn two_channel_locks_are_rejected() {
let a = RawMutex::new_channel(0u64);
let b = RawMutex::new_channel(0u64);
let _ga = a.lock();
let _gb = b.lock(); // channel locks are mutual leaves: forbidden
}
}
+169
View File
@@ -0,0 +1,169 @@
//! Named pid registry.
//!
//! `register(name, pid)` binds a name to a live actor; `whereis(name)` looks
//! it up; `name_of(pid)` is the inverse. Erlang semantics: at most one name
//! per pid, at most one pid per name, registering over a live binding is an
//! error. The registry is a *bimap* (two `HashMap`s kept in exact inverse
//! under one lock) so both lookup directions are O(1) — `name_of` exists so
//! supervisors, tracing, and diagnostics can label a pid without scanning.
//!
//! ## Cleanup is lazy
//!
//! There is no hook in `finalize_actor` and no name field in `Slot` (which
//! would buy into the reset-in-three-places slot invariant). Instead every
//! operation that touches a binding checks the bound pid's liveness via the
//! generation-checked slot word — a `Pid` is `(index, generation)` and a
//! generation is never reused, so a stale binding is *detectable*, never
//! misdirected. Bindings to dead actors behave as absent and are pruned on
//! contact: `whereis`/`name_of` return `None`, `register` treats the name as
//! free. The cost is that a dead binding lingers until something looks at it;
//! the payoff is zero coupling to the actor lifecycle.
//!
//! ## Locking
//!
//! One `RawMutex` (Leaf class) in `RuntimeInner`. Liveness checks under it
//! read only the atomic slot state word — no second lock is ever taken, so
//! the leaf rule holds trivially.
use crate::pid::Pid;
use crate::scheduler::with_runtime;
use std::collections::HashMap;
/// Why a [`register`] call was rejected.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RegisterError {
/// The name is bound to a different, still-live actor.
NameTaken { holder: Pid },
/// The pid already has a (live) registered name; one name per pid.
PidAlreadyRegistered { name: String },
/// The pid does not refer to a live actor.
NoProc,
}
impl std::fmt::Display for RegisterError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
RegisterError::NameTaken { holder } => {
write!(f, "name is already registered to live actor {holder}")
}
RegisterError::PidAlreadyRegistered { name } => {
write!(f, "pid is already registered as {name:?}")
}
RegisterError::NoProc => write!(f, "pid does not refer to a live actor"),
}
}
}
impl std::error::Error for RegisterError {}
/// The bimap. Invariant (held under the registry lock): `by_name` and
/// `by_pid` are exact inverses of each other at all times.
pub(crate) struct Registry {
by_name: HashMap<String, Pid>,
by_pid: HashMap<Pid, String>,
}
impl Registry {
pub(crate) fn new() -> Self {
Self { by_name: HashMap::new(), by_pid: HashMap::new() }
}
fn remove_binding(&mut self, name: &str, pid: Pid) {
self.by_name.remove(name);
self.by_pid.remove(&pid);
debug_assert_eq!(self.by_name.len(), self.by_pid.len(), "bimap out of sync");
}
fn insert_binding(&mut self, name: String, pid: Pid) {
self.by_pid.insert(pid, name.clone());
self.by_name.insert(name, pid);
debug_assert_eq!(self.by_name.len(), self.by_pid.len(), "bimap out of sync");
}
}
/// Is `pid` a live actor right now? Atomic slot-word read; no lock.
fn live(inner: &crate::runtime::RuntimeInner, pid: Pid) -> bool {
inner.slot_at(pid).is_some_and(|s| s.is_live_for(pid))
}
/// Bind `name` to the live actor `pid`.
///
/// Fails with [`RegisterError::NoProc`] if `pid` is not live,
/// [`RegisterError::NameTaken`] if the name is bound to a *live* actor
/// (a binding to a dead actor is evicted and the name treated as free), and
/// [`RegisterError::PidAlreadyRegistered`] if `pid` already has a name.
///
/// Panics if called outside `Runtime::run()`.
pub fn register(name: impl Into<String>, pid: Pid) -> Result<(), RegisterError> {
let name = name.into();
with_runtime(|inner| {
let mut reg = inner.registry.lock();
if !live(inner, pid) {
return Err(RegisterError::NoProc);
}
let prior = reg.by_name.get(&name).copied();
if let Some(holder) = prior {
if holder == pid {
return Ok(()); // already exactly this binding: idempotent
}
if live(inner, holder) {
return Err(RegisterError::NameTaken { holder });
}
// Stale binding: the holder died. Evict and treat the name as free.
reg.remove_binding(&name, holder);
}
if let Some(existing) = reg.by_pid.get(&pid) {
// `pid` is live (checked above) and pids are never reused, so this
// binding is necessarily current — one name per pid.
return Err(RegisterError::PidAlreadyRegistered { name: existing.clone() });
}
reg.insert_binding(name, pid);
Ok(())
})
}
/// Look up the pid bound to `name`. `None` if unbound, or if the bound actor
/// is no longer live (the stale binding is pruned on the way out).
pub fn whereis(name: &str) -> Option<Pid> {
with_runtime(|inner| {
let mut reg = inner.registry.lock();
let pid = *reg.by_name.get(name)?;
if live(inner, pid) {
Some(pid)
} else {
reg.remove_binding(name, pid);
None
}
})
}
/// The inverse lookup: the name `pid` is registered under, if any. `None` if
/// unregistered or no longer live (pruning the stale binding).
pub fn name_of(pid: Pid) -> Option<String> {
with_runtime(|inner| {
let mut reg = inner.registry.lock();
let name = reg.by_pid.get(&pid)?.clone();
if live(inner, pid) {
Some(name)
} else {
reg.remove_binding(&name, pid);
None
}
})
}
/// Remove the binding for `name`, returning the pid it was bound to if that
/// actor is still live. A binding to a dead actor is pruned and reported as
/// `None`, consistent with [`whereis`].
pub fn unregister(name: &str) -> Option<Pid> {
with_runtime(|inner| {
let mut reg = inner.registry.lock();
let pid = *reg.by_name.get(name)?;
reg.remove_binding(name, pid);
if live(inner, pid) {
Some(pid)
} else {
None
}
})
}
+562
View File
@@ -0,0 +1,562 @@
//! The run queue, selected at COMPILE TIME by mutually-exclusive cargo
//! features (no runtime dispatch — the scheduler's pop loop is the hottest
//! code in the runtime):
//!
//! - `rq-mutex` (default) — `Mutex<VecDeque>`. The control/baseline:
//! strictly FIFO, trivially correct, one global lock.
//! - `rq-mpmc` — a single hand-rolled Vyukov bounded MPMC ring (per-cell
//! sequence numbers). Strict FIFO, lock-free, one hot
//! enqueue/dequeue cache-line pair.
//! - `rq-striped` — M Vyukov rings with fetch-add ticket distribution.
//! *Relaxed* FIFO: ordering across stripes is bounded-skewed
//! (≈ one ring's worth of reordering per stripe), in exchange
//! for spreading the hot line M ways. Predicted winner at
//! high core counts; phase 4's shootout decides.
//!
//! Select non-default variants with `--no-default-features --features rq-…`
//! (cargo features are additive, so the default must be switched off).
//!
//! All variants are compiled unconditionally (so every build runs every
//! variant's unit tests); the feature only picks which one the runtime uses
//! via the [`RunQueue`] alias.
//!
//! # Contract (shared by all variants)
//!
//! - **Occupancy is bounded by `max_actors`.** A pid is in the queue at most
//! once (pushes pair 1:1 with transitions into `Queued`; only the
//! scheduler transitions `Queued → Running` — see the state-machine docs
//! in `runtime.rs`), and at most `max_actors` actors exist. With the RFC
//! 005 wake slot enabled the invariant reads "in (slot ⊕ shared queue) at
//! most once" — a slot push *replaces* the queue push at the same
//! protocol point, and a displacement moves the occupant, never copies
//! it — so the bound holds verbatim. The bounded rings are sized ≥
//! `max_actors`, so **`push` is infallible**; a full
//! ring is an invariant violation and panics loudly rather than spinning.
//! - **Preemption must be disabled around every push/pop** (debug-asserted).
//! For the mutex variant this is the usual no-switch/no-unwind-under-lock
//! rule. For the rings it is *load-bearing in a sharper way*: a producer
//! suspended between claiming a cell and publishing its sequence number
//! stalls every consumer behind that cell — on a busy runtime that is a
//! livelock, since the suspended actor's own resume entry sits behind the
//! hole. Callers get this for free: every queue op happens inside
//! `with_runtime`/`try_with_runtime` (NoPreempt for their span since
//! phase 2) or on a scheduler thread between resumes (preemption off).
//! - **`pop() == None` is a snapshot, not a fence.** A push that is mid-
//! publish (or in a stripe the probe already passed) may be missed; the
//! caller's idle path sleeps ≤ 100µs and retries, so the cost is a bounded
//! latency blip, never a lost entry. Termination does not lean on this:
//! the all-clear is `live_actors == 0` (+ io quiescent), and `live == 0`
//! already implies the queue holds nothing actionable — see the argument
//! in `schedule_loop`.
//! - `len()` is approximate (stats only).
use crate::pid::Pid;
use crate::sync_shim::{AtomicUsize, Ordering, UnsafeCell};
use std::mem::MaybeUninit;
// ---------------------------------------------------------------------------
// Feature selection
// ---------------------------------------------------------------------------
#[cfg(not(any(feature = "rq-mutex", feature = "rq-mpmc", feature = "rq-striped")))]
compile_error!(
"smarm: no run queue selected. Enable exactly one of the features \
`rq-mutex` (default), `rq-mpmc`, `rq-striped`."
);
#[cfg(all(feature = "rq-mutex", feature = "rq-mpmc"))]
compile_error!(
"smarm: features `rq-mutex` and `rq-mpmc` are mutually exclusive \
(use --no-default-features to drop the default `rq-mutex`)."
);
#[cfg(all(feature = "rq-mutex", feature = "rq-striped"))]
compile_error!(
"smarm: features `rq-mutex` and `rq-striped` are mutually exclusive \
(use --no-default-features to drop the default `rq-mutex`)."
);
#[cfg(all(feature = "rq-mpmc", feature = "rq-striped"))]
compile_error!("smarm: features `rq-mpmc` and `rq-striped` are mutually exclusive.");
#[cfg(feature = "rq-mutex")]
pub(crate) type RunQueue = MutexQueue;
#[cfg(feature = "rq-mpmc")]
pub(crate) type RunQueue = MpmcRing;
#[cfg(feature = "rq-striped")]
pub(crate) type RunQueue = StripedRing;
#[inline]
fn assert_no_preempt() {
debug_assert!(
!crate::preempt::PREEMPTION_ENABLED.with(|c| c.get()),
"run-queue op with preemption enabled — a switch mid-op stalls or \
corrupts the queue; route through with_runtime or scheduler context"
);
}
// ---------------------------------------------------------------------------
// rq-mutex — the baseline
// ---------------------------------------------------------------------------
#[allow(dead_code)]
pub struct MutexQueue {
q: std::sync::Mutex<std::collections::VecDeque<Pid>>,
}
#[allow(dead_code)]
impl MutexQueue {
pub fn new(_threads: usize, max_actors: usize) -> Self {
Self {
// Pre-size: the queue can never outgrow the slab, and one
// allocation at init beats reallocating under the lock later.
q: std::sync::Mutex::new(std::collections::VecDeque::with_capacity(max_actors)),
}
}
pub fn push(&self, pid: Pid) {
assert_no_preempt();
self.q.lock().unwrap().push_back(pid);
}
pub fn pop(&self) -> Option<Pid> {
assert_no_preempt();
self.q.lock().unwrap().pop_front()
}
pub fn len(&self) -> u64 {
self.q.lock().unwrap().len() as u64
}
}
// ---------------------------------------------------------------------------
// rq-mpmc — Vyukov bounded MPMC ring
// ---------------------------------------------------------------------------
//
// Dmitry Vyukov's bounded MPMC queue: each cell carries a sequence number.
// A producer may write cell `i` when `seq == pos` (its turn); it publishes
// with `seq = pos + 1`. A consumer may read when `seq == pos + 1`; it
// releases the cell to the next lap with `seq = pos + capacity`. Producers
// and consumers each contend on one counter; cell handoff is a per-cell
// Acquire/Release pair, so unrelated push/pop pairs don't serialize.
/// Pad to a cache-line pair so the producer and consumer counters (and the
/// cells) don't false-share.
#[repr(align(128))]
struct CachePadded<T>(T);
struct Cell {
seq: AtomicUsize,
pid: UnsafeCell<MaybeUninit<Pid>>,
}
#[allow(dead_code)]
pub struct MpmcRing {
buf: Box<[Cell]>,
mask: usize,
enqueue_pos: CachePadded<AtomicUsize>,
dequeue_pos: CachePadded<AtomicUsize>,
}
// SAFETY: cells are handed off between threads via the per-cell seq
// (Release on publish, Acquire on claim); Pid is Copy + Send.
unsafe impl Send for MpmcRing {}
unsafe impl Sync for MpmcRing {}
#[allow(dead_code)]
impl MpmcRing {
pub fn new(_threads: usize, max_actors: usize) -> Self {
Self::with_capacity(max_actors)
}
/// Pub for the raw-structure microbench (sized to the op count so the
/// occupancy contract is trivially met there). Runtime code uses `new`.
pub fn with_capacity(min_cap: usize) -> Self {
// Occupancy ≤ max_actors (queue contract), so capacity = the next
// power of two ≥ max_actors can never overflow. (≥ 2 so mask works.)
let cap = min_cap.next_power_of_two().max(2);
let buf: Box<[Cell]> = (0..cap)
.map(|i| Cell {
seq: AtomicUsize::new(i),
pid: UnsafeCell::new(MaybeUninit::uninit()),
})
.collect();
Self {
buf,
mask: cap - 1,
enqueue_pos: CachePadded(AtomicUsize::new(0)),
dequeue_pos: CachePadded(AtomicUsize::new(0)),
}
}
pub fn push(&self, pid: Pid) {
assert_no_preempt();
assert!(
self.try_push(pid),
"smarm: run queue overflow — occupancy exceeded the slab bound, \
which the at-most-once-enqueued invariant forbids. This is a \
runtime bug (double enqueue), not a capacity tuning problem."
);
}
/// One full claim attempt; `false` only if the ring is full.
fn try_push(&self, pid: Pid) -> bool {
let mut pos = self.enqueue_pos.0.load(Ordering::Relaxed);
loop {
let cell = &self.buf[pos & self.mask];
let seq = cell.seq.load(Ordering::Acquire);
let diff = seq as isize - pos as isize;
if diff == 0 {
// Our turn: claim the position.
match self.enqueue_pos.0.compare_exchange_weak(
pos, pos + 1, Ordering::Relaxed, Ordering::Relaxed,
) {
Ok(_) => {
// SAFETY: the claim gives us exclusive write access
// to this cell until we publish below.
cell.pid.with_mut(|p| unsafe { (*p).write(pid) });
cell.seq.store(pos + 1, Ordering::Release);
return true;
}
Err(actual) => pos = actual,
}
} else if diff < 0 {
return false; // full — a whole lap behind
} else {
pos = self.enqueue_pos.0.load(Ordering::Relaxed);
}
}
}
pub fn pop(&self) -> Option<Pid> {
assert_no_preempt();
let mut pos = self.dequeue_pos.0.load(Ordering::Relaxed);
loop {
let cell = &self.buf[pos & self.mask];
let seq = cell.seq.load(Ordering::Acquire);
let diff = seq as isize - (pos + 1) as isize;
if diff == 0 {
match self.dequeue_pos.0.compare_exchange_weak(
pos, pos + 1, Ordering::Relaxed, Ordering::Relaxed,
) {
Ok(_) => {
// SAFETY: the claim gives us exclusive read access;
// the producer's Release publish made `pid` visible
// to our Acquire load of `seq`.
let pid = cell.pid.with(|p| unsafe { (*p).assume_init_read() });
// Release the cell for the next lap.
cell.seq.store(pos + self.mask + 1, Ordering::Release);
return Some(pid);
}
Err(actual) => pos = actual,
}
} else if diff < 0 {
// Empty (or the producer at this cell hasn't published yet —
// a snapshot miss the caller's idle-retry loop absorbs).
return None;
} else {
pos = self.dequeue_pos.0.load(Ordering::Relaxed);
}
}
}
pub fn len(&self) -> u64 {
let e = self.enqueue_pos.0.load(Ordering::Relaxed);
let d = self.dequeue_pos.0.load(Ordering::Relaxed);
e.saturating_sub(d) as u64
}
}
// ---------------------------------------------------------------------------
// rq-striped — M Vyukov rings, ticket-distributed
// ---------------------------------------------------------------------------
//
// Producers fetch-add a ticket and start probing at stripe `ticket % M`;
// consumers do the same with their own ticket. Under symmetric load the
// tickets spread producers and consumers uniformly, so each stripe sees
// ~1/M of the traffic and the single hot cache-line pair becomes M cooler
// ones. FIFO is relaxed: two pushes that land in different stripes can be
// popped in either order, with skew bounded by stripe occupancy imbalance.
//
// Push probes forward from its home stripe until a `try_push` succeeds.
// Σ stripe capacity ≥ 2 × max_actors while occupancy ≤ max_actors, so at
// every instant at least half the total capacity is free and the probe
// terminates (in practice on the first stripe).
#[allow(dead_code)]
pub struct StripedRing {
stripes: Box<[MpmcRing]>,
/// Stripe count minus one (count is a power of two).
stripe_mask: usize,
push_ticket: CachePadded<AtomicUsize>,
pop_ticket: CachePadded<AtomicUsize>,
}
#[allow(dead_code)]
impl StripedRing {
pub fn new(threads: usize, max_actors: usize) -> Self {
// One stripe per scheduler thread, rounded up to a power of two —
// more stripes than threads buys nothing (at most `threads` ops are
// in flight) and costs pop-probe latency when mostly empty.
let n = threads.max(1).next_power_of_two();
// Per-stripe capacity: 2 × max_actors / n in total, and never below
// a floor that keeps degenerate configs (tiny slab, many threads)
// trivially correct.
let per = ((2 * max_actors) / n).next_power_of_two().max(8);
let stripes: Box<[MpmcRing]> = (0..n).map(|_| MpmcRing::with_capacity(per)).collect();
Self {
stripes,
stripe_mask: n - 1,
push_ticket: CachePadded(AtomicUsize::new(0)),
pop_ticket: CachePadded(AtomicUsize::new(0)),
}
}
pub fn push(&self, pid: Pid) {
assert_no_preempt();
let home = self.push_ticket.0.fetch_add(1, Ordering::Relaxed);
// Probe from the home stripe; capacity headroom (Σ ≥ 2×occupancy)
// guarantees a free stripe exists, so the outer loop terminates.
// The retry-from-home lap handles the racy case where every stripe
// momentarily refused us.
loop {
for i in 0..=self.stripe_mask {
let s = &self.stripes[(home + i) & self.stripe_mask];
if s.try_push(pid) {
return;
}
}
std::hint::spin_loop();
}
}
pub fn pop(&self) -> Option<Pid> {
assert_no_preempt();
let home = self.pop_ticket.0.fetch_add(1, Ordering::Relaxed);
for i in 0..=self.stripe_mask {
if let Some(pid) = self.stripes[(home + i) & self.stripe_mask].pop() {
return Some(pid);
}
}
None // snapshot miss possible across stripes; idle-retry absorbs it
}
pub fn len(&self) -> u64 {
self.stripes.iter().map(|s| s.len()).sum()
}
}
// ---------------------------------------------------------------------------
// Tests — all variants, in every build (the feature only picks the alias)
// ---------------------------------------------------------------------------
#[cfg(all(test, not(loom)))]
mod tests {
use super::*;
use std::collections::HashSet;
use std::sync::Arc;
fn pid(i: u32) -> Pid {
Pid::new(i, 0)
}
fn fifo_smoke<Q>(q: &Q, push: impl Fn(&Q, Pid), pop: impl Fn(&Q) -> Option<Pid>) {
for i in 0..100 {
push(q, pid(i));
}
for i in 0..100 {
assert_eq!(pop(q), Some(pid(i)));
}
assert_eq!(pop(q), None);
}
#[test]
fn mutex_fifo() {
let q = MutexQueue::new(1, 1024);
fifo_smoke(&q, |q, p| q.push(p), |q| q.pop());
}
#[test]
fn mpmc_fifo_single_thread() {
let q = MpmcRing::new(1, 1024);
fifo_smoke(&q, |q, p| q.push(p), |q| q.pop());
}
#[test]
fn mpmc_wraps_many_laps() {
let q = MpmcRing::with_capacity(8);
for lap in 0..1000u32 {
for i in 0..8 {
q.push(pid(lap * 8 + i));
}
for i in 0..8 {
assert_eq!(q.pop(), Some(pid(lap * 8 + i)));
}
}
assert_eq!(q.pop(), None);
}
/// N producers, M consumers, every element exactly once. Run on plain OS
/// threads (PREEMPTION_ENABLED defaults false, satisfying the contract).
fn exactly_once<Q: Send + Sync + 'static>(
q: Q,
push: fn(&Q, Pid),
pop: fn(&Q) -> Option<Pid>,
producers: u32,
consumers: u32,
per_producer: u32,
) {
let q = Arc::new(q);
let total = (producers * per_producer) as usize;
let popped = Arc::new(std::sync::Mutex::new(Vec::with_capacity(total)));
let remaining = Arc::new(AtomicUsize::new(total));
let mut hs = Vec::new();
for p in 0..producers {
let q = q.clone();
hs.push(std::thread::spawn(move || {
for i in 0..per_producer {
push(&q, pid(p * per_producer + i));
}
}));
}
for _ in 0..consumers {
let q = q.clone();
let popped = popped.clone();
let remaining = remaining.clone();
hs.push(std::thread::spawn(move || {
let mut local = Vec::new();
while remaining.load(Ordering::Relaxed) > 0 {
if let Some(pid) = pop(&q) {
remaining.fetch_sub(1, Ordering::Relaxed);
local.push(pid);
} else {
std::hint::spin_loop();
}
}
popped.lock().unwrap().extend(local);
}));
}
for h in hs {
h.join().unwrap();
}
let popped = popped.lock().unwrap();
assert_eq!(popped.len(), total, "count mismatch");
let set: HashSet<u64> = popped.iter().map(|p| ((p.index() as u64) << 32) | p.generation() as u64).collect();
assert_eq!(set.len(), total, "duplicate or lost element");
assert_eq!(pop(&q), None);
}
#[test]
fn mpmc_exactly_once_contended() {
exactly_once(MpmcRing::new(8, 4096), |q, p| q.push(p), |q| q.pop(), 4, 4, 1000);
}
#[test]
fn striped_exactly_once_contended() {
exactly_once(StripedRing::new(8, 4096), |q, p| q.push(p), |q| q.pop(), 4, 4, 1000);
}
#[test]
fn striped_drains_after_skewed_load() {
// Hammer pushes from one thread (all tickets walk the stripes in
// order) and verify a single consumer sees every element.
let q = StripedRing::new(4, 64);
let mut seen = HashSet::new();
for i in 0..64 {
q.push(pid(i));
}
while let Some(p) = q.pop() {
assert!(seen.insert(p.index()));
}
assert_eq!(seen.len(), 64);
}
}
// ---------------------------------------------------------------------------
// loom model tests — RUSTFLAGS="--cfg loom" cargo test --lib --release
// ---------------------------------------------------------------------------
#[cfg(all(test, loom))]
mod loom_tests {
use super::*;
use loom::sync::Arc;
use loom::thread;
fn pid(i: u32) -> Pid {
Pid::new(i, 0)
}
/// Two producers, main-thread consumer: both elements arrive exactly
/// once, across every interleaving — including through a lap wraparound
/// (capacity 2 forces cell reuse).
#[test]
fn mpmc_two_producers_exactly_once() {
loom::model(|| {
let q = Arc::new(MpmcRing::with_capacity(2));
let mut hs = Vec::new();
for i in 0..2u32 {
let q = q.clone();
hs.push(thread::spawn(move || q.push(pid(i))));
}
let mut got = Vec::new();
while got.len() < 2 {
match q.pop() {
Some(p) => got.push(p.index()),
None => thread::yield_now(),
}
}
for h in hs {
h.join().unwrap();
}
got.sort_unstable();
assert_eq!(got, vec![0, 1]);
assert!(q.pop().is_none());
});
}
/// Producer races a consumer on a single element: the consumer either
/// gets it or sees a clean None — never a torn/duplicated element.
#[test]
fn mpmc_push_pop_race() {
loom::model(|| {
let q = Arc::new(MpmcRing::with_capacity(2));
let q2 = q.clone();
let prod = thread::spawn(move || q2.push(pid(7)));
let seen = q.pop();
prod.join().unwrap();
match seen {
Some(p) => {
assert_eq!(p.index(), 7);
assert!(q.pop().is_none());
}
None => assert_eq!(q.pop().map(|p| p.index()), Some(7)),
}
});
}
/// Striped: two producers landing in (potentially) different stripes,
/// main-thread consumer drains both exactly once.
#[test]
fn striped_two_producers_exactly_once() {
loom::model(|| {
let q = Arc::new(StripedRing::new(2, 4));
let mut hs = Vec::new();
for i in 0..2u32 {
let q = q.clone();
hs.push(thread::spawn(move || q.push(pid(i))));
}
let mut got = Vec::new();
while got.len() < 2 {
match q.pop() {
Some(p) => got.push(p.index()),
None => thread::yield_now(),
}
}
for h in hs {
h.join().unwrap();
}
got.sort_unstable();
assert_eq!(got, vec![0, 1]);
assert!(q.pop().is_none());
});
}
}
+834 -438
View File
File diff suppressed because it is too large Load Diff
+341 -134
View File
@@ -21,18 +21,39 @@ use std::sync::Arc;
// ---------------------------------------------------------------------------
/// Borrow the current runtime. Panics if called outside `Runtime::run()`.
///
/// The whole span runs with preemption disabled. Two reasons, both load-
/// bearing:
///
/// - The `RUNTIME` thread-local borrow is live across `f`. A preemption-
/// driven context switch inside `f` can resume the actor on a DIFFERENT OS
/// thread; the borrow guard would then increment this thread's RefCell but
/// decrement the other's — a count underflow that leaves that thread's
/// RefCell permanently "mutably borrowed". Holding a thread-local guard
/// across a potential switch point is the one unforgivable sin of green
/// threads; disabling preemption makes "no switch inside `f`" structural
/// instead of an accident of which bodies happen to allocate.
/// - `f` is runtime bookkeeping. Suspending an actor halfway through it (or
/// unwinding via the stop sentinel, which shares the gate) is never wanted.
pub(crate) fn with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> R {
RUNTIME.with(|r| {
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| {
let b = r.borrow();
let inner = b.as_ref().expect("smarm: not inside Runtime::run()");
f(inner)
})
});
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
result
}
/// Borrow the runtime if present; returns `None` otherwise.
/// Used on cleanup paths (channel Drop during teardown).
/// Same preemption gate as [`with_runtime`]; same reasons.
pub(crate) fn try_with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> Option<R> {
RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)))
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)));
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
result
}
// ---------------------------------------------------------------------------
@@ -54,22 +75,34 @@ impl JoinHandle {
pub fn join(mut self) -> Result<(), JoinError> {
use crate::actor::Outcome;
use crate::runtime::State; // need State visibility
let me = current_pid().expect("join() called outside an actor");
loop {
// Check-Done-or-register-waiter is atomic under the target's cold
// lock; finalize publishes Done and takes the waiter list under
// the same lock, so we either see the outcome or are woken.
let outcome = with_runtime(|inner| {
inner.with_shared(|s| {
let slot = s.slot_mut(self.pid)
.expect("join: target slot has been reused");
if matches!(slot.state, State::Done) {
Some(slot.outcome.take().expect("Done slot must have outcome"))
} else {
slot.waiters.push(me);
let slot = inner.slot_at(self.pid)
.expect("join: pid index out of range");
let mut cold = slot.cold.lock();
match slot.status_for(self.pid) {
// Our outstanding handle pins the slot: it cannot be
// reclaimed (generation cannot change) while we hold it.
crate::slot_state::Status::Stale => {
panic!("join: target slot has been reused")
}
crate::slot_state::Status::Done => {
Some(cold.outcome.take().expect("Done slot must have outcome"))
}
crate::slot_state::Status::Live => {
// begin_wait is lock-free, legal under the cold lock;
// registering under it makes the epoch atomic with
// the check-Done-or-register linearization point.
cold.waiters.push((me, begin_wait()));
None
}
})
}
});
match outcome {
@@ -95,20 +128,25 @@ impl JoinHandle {
fn decrement_handle_count(&mut self) {
with_runtime(|inner| {
inner.with_shared(|s| {
let should_reclaim = match s.slot_mut(self.pid) {
Some(slot) => {
slot.outstanding_handles =
slot.outstanding_handles.saturating_sub(1);
matches!(slot.state, crate::runtime::State::Done)
&& slot.outstanding_handles == 0
let should_reclaim = match inner.slot_at(self.pid) {
Some(slot) => {
let mut cold = slot.cold.lock();
match slot.status_for(self.pid) {
crate::slot_state::Status::Stale => false,
status => {
cold.outstanding_handles =
cold.outstanding_handles.saturating_sub(1);
cold.outstanding_handles == 0
&& status == crate::slot_state::Status::Done
}
}
None => false,
};
if should_reclaim {
crate::runtime::reclaim_slot(s, self.pid);
}
})
None => false,
};
if should_reclaim {
// Re-verified inside; benign if finalize's reclaim won a race.
crate::runtime::reclaim_slot(inner, self.pid);
}
});
}
}
@@ -129,56 +167,28 @@ impl Drop for JoinHandle {
// ---------------------------------------------------------------------------
pub fn spawn(f: impl FnOnce() + Send + 'static) -> JoinHandle {
let parent = current_pid()
.or_else(|| with_runtime(|inner| inner.with_shared(|s| s.root_pid)))
.expect("spawn() before run()");
let parent = current_pid().unwrap_or_else(|| {
// Outside an actor but inside run(): the initial spawn. with_runtime
// panics with "not inside Runtime::run()" if there's no runtime at all.
with_runtime(|_| crate::runtime::ROOT_PID)
});
spawn_under(parent, f)
}
pub fn spawn_under(supervisor: Pid, f: impl FnOnce() + Send + 'static) -> JoinHandle {
// Try to reuse a stack from the pool; fall back to a fresh mmap if empty.
// Allocation happens before taking the shared lock so any syscall doesn't
// stall other scheduler threads.
let stack = with_runtime(|inner| inner.stack_pool.lock().unwrap().pop())
// Stack + closure boxing happen before ANY runtime lock is taken: no
// syscall and no allocation ever stalls another scheduler thread.
let stack = with_runtime(|inner| inner.stack_pool.lock().pop())
.unwrap_or_else(|| {
crate::stack::Stack::new(crate::runtime::ACTOR_STACK_SIZE)
.expect("stack allocation failed")
});
let sp = init_actor_stack(stack.top(), crate::actor::trampoline);
let closure: crate::runtime::Closure = Box::new(f);
let pid = with_runtime(|inner| {
inner.with_shared(|s| {
let (idx, gen) = s.allocate_slot();
let pid = Pid::new(idx, gen);
let slot = &mut s.slots[idx as usize];
slot.actor = Some(crate::actor::Actor {
pid,
stack,
sp,
supervisor,
stop: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)),
trap: None,
});
slot.state = crate::runtime::State::Runnable;
slot.outstanding_handles = 1;
slot.outcome = None;
slot.waiters.clear();
slot.supervisor_channel = None;
slot.monitors.clear();
slot.links.clear();
slot.pending_unpark = false;
slot.pending_io_result = None;
s.run_queue.push_back(pid);
// Grow the closures vec to cover this slot index, then store.
let idx = idx as usize;
if s.pending_closures.len() <= idx {
s.pending_closures.resize_with(idx + 1, || None);
}
s.pending_closures[idx] = Some(Box::new(f) as crate::runtime::Closure);
crate::te!(crate::trace::Event::Spawn { parent: supervisor, child: pid });
crate::te!(crate::trace::Event::Enqueue(pid));
pid
})
let idx = inner.allocate_slot(); // panics loudly on slab exhaustion
crate::runtime::install_actor(inner, idx, sp, stack, supervisor, closure)
});
JoinHandle { pid, consumed: false }
@@ -202,6 +212,18 @@ pub fn yield_now() {
}
pub fn park_current() {
// Entry-side observation point: a stop flagged while we were QUEUED is
// otherwise lost — the stop's wildcard unpark no-ops on a Queued actor
// (the pending run "is" the wake), so if our first action on resume is
// this park, no wake is ever coming and the wake-side check below is
// unreachable. Checking here closes that hole; a flag that lands after
// this check is covered by the existing protocol (the stop's unpark
// finds Running / the prep-to-park window, sets Notified, and the
// park-return re-queues us into the wake-side check). Unwinding from
// here is the same unwind path as the wake side: leftover wait
// registrations are stale-epoch / dead-generation and self-clean at
// their wakers' failed CAS.
crate::preempt::check_cancelled();
runtime::set_yield_intent(YieldIntent::Park);
unsafe { crate::context::switch_to_scheduler() };
// Observation point on the wakeup side of every blocking primitive
@@ -212,39 +234,54 @@ pub fn park_current() {
}
pub fn unpark(pid: Pid) {
let result = try_with_runtime(|inner| {
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
match slot.state {
crate::runtime::State::Parked => {
// Actor is suspended — safe to re-queue immediately.
slot.state = crate::runtime::State::Runnable;
s.run_queue.push_back(pid);
crate::te!(crate::trace::Event::UnparkDirect(pid));
crate::te!(crate::trace::Event::Enqueue(pid));
}
crate::runtime::State::Runnable => {
// Actor is still running (between registering its
// parked_receiver and calling park_current). Set the
// flag; the scheduler will re-queue after the Park
// yield instead of sleeping.
slot.pending_unpark = true;
crate::te!(crate::trace::Event::UnparkDeferred(pid));
}
crate::runtime::State::Done => {}
}
}
})
});
let _ = result;
// The whole protocol lives on the slot's packed word (gen + epoch +
// state in one CAS) — see slot_state.rs. No runtime lock unless we
// enqueue. WILDCARD form: reserved for terminal wakes (request_stop);
// every registration-based waker must use `unpark_at`.
let _ = try_with_runtime(|inner| inner.unpark(pid));
}
/// Epoch-matched unpark: wake `pid` only if its current wait is still the
/// one this waker registered for. The form every registration-based waker
/// (channel senders, mutex grants, wait-timers, io completions, joiner
/// wakes) must use — see slot_state.rs for the consuming-wake rules.
pub(crate) fn unpark_at(pid: Pid, epoch: u32) {
let _ = try_with_runtime(|inner| inner.unpark_at(pid, epoch));
}
/// Open a new wait for the CURRENT actor: bump its park-epoch and return
/// it. Call once per wait, before registering `(pid, epoch)` with any
/// waker. Lock-free (one CAS on the own slot word), so it is legal under
/// any lock, including a Channel-class lock.
pub(crate) fn begin_wait() -> u32 {
let me = current_pid().expect("begin_wait() called outside an actor");
with_runtime(|inner| inner.begin_wait(me))
}
/// Close the current actor's wait WITHOUT parking on it. For the no-park
/// exit of multi-registration waits (`select` finding an arm ready at
/// registration time): bumps the epoch so in-flight wakes die at their CAS,
/// eats a notification that already landed, then observes a pending stop —
/// in that order. After this, leftover registrations are stale-epoch and
/// self-clean at their wakers' failed CAS; nothing can fault the actor's
/// next one-shot park.
pub(crate) fn retire_wait() {
let me = current_pid().expect("retire_wait() called outside an actor");
with_runtime(|inner| inner.retire_wait(me));
// A request_stop that fired before the clear had its notification
// eaten, but it set the stop flag first — observe it here and unwind.
// One that fires after re-notifies a Running word as usual.
crate::preempt::check_cancelled();
}
/// Request cooperative cancellation of `pid`.
///
/// Sets the actor's stop flag and, if it is parked, wakes it so it observes
/// the flag promptly. The actor realises the stop as a controlled unwind at
/// its next observation point (`check!()`/allocation, or the wakeup side of a
/// blocking park), terminating with `Outcome::Stopped`.
/// Sets the actor's stop flag and wakes it so it observes the flag promptly:
/// a parked target is re-queued; a running target is marked notified, so its
/// next park returns immediately to an observation point. The actor realises
/// the stop as a controlled unwind at its next observation point
/// (`check!()`/allocation, or the wakeup side of a blocking park),
/// terminating with `Outcome::Stopped`.
///
/// This is best-effort and cooperative: an actor that never reaches an
/// observation point — a tight loop with no `check!()`, no allocation, and no
@@ -252,21 +289,19 @@ pub fn unpark(pid: Pid) {
/// if `pid` is already gone.
pub fn request_stop(pid: Pid) {
let _ = try_with_runtime(|inner| {
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
if let Some(actor) = slot.actor.as_ref() {
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
}
// Wake a parked target so it reaches its post-park observation
// point now. A Runnable target will observe on its next yield;
// a Done target has nothing to stop.
if matches!(slot.state, crate::runtime::State::Parked) {
slot.state = crate::runtime::State::Runnable;
s.run_queue.push_back(pid);
crate::te!(crate::trace::Event::Enqueue(pid));
if let Some(slot) = inner.slot_at(pid) {
{
let cold = slot.cold.lock();
// Verify under the cold lock: generation can't change while
// we hold it (reclaim takes the same lock).
if slot.generation() == pid.generation() {
if let Some(actor) = cold.actor.as_ref() {
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
}
}
}
})
inner.unpark(pid);
}
});
}
@@ -296,8 +331,9 @@ impl Drop for NoPreempt {
pub fn sleep(duration: std::time::Duration) {
let me = current_pid().expect("sleep() called outside an actor");
let _np = NoPreempt::enter();
let epoch = begin_wait();
let deadline = crate::timer::deadline_from_now(duration);
with_runtime(|inner| inner.with_shared(|s| s.timers.insert_sleep(deadline, me)));
with_runtime(|inner| inner.timers.lock().unwrap().insert_sleep(deadline, me, epoch));
park_current();
}
@@ -305,16 +341,14 @@ pub fn insert_wait_timer(
deadline: std::time::Instant,
pid: Pid,
target: std::sync::Arc<dyn crate::timer::TimerTarget>,
wait_seq: u64,
epoch: u32,
) {
with_runtime(|inner| {
inner.with_shared(|s| {
s.timers.insert(
deadline,
pid,
crate::timer::Reason::WaitTimeout { target, wait_seq },
);
})
inner.timers.lock().unwrap().insert(
deadline,
pid,
crate::timer::Reason::WaitTimeout { target, epoch },
);
});
}
@@ -334,19 +368,24 @@ where
});
{
let _np = NoPreempt::enter();
with_runtime(|inner| inner.with_shared(|s| {
let io = s.io.as_mut().expect("io thread not started");
io.submit(me, work);
}));
let epoch = begin_wait();
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
io.as_mut().expect("io thread not started").submit(me, epoch, work);
});
park_current();
}
let result = with_runtime(|inner| inner.with_shared(|s| {
s.slot_mut(me)
.expect("block_on_io: own slot vanished")
.pending_io_result
let result = with_runtime(|inner| {
let slot = inner.slot_at(me).expect("block_on_io: own slot vanished");
let mut cold = slot.cold.lock();
debug_assert_eq!(
slot.generation(), me.generation(),
"block_on_io: own slot reused mid-park"
);
cold.pending_io_result
.take()
.expect("block_on_io: resumed without a result")
}));
});
match result {
Ok(any) => *any.downcast::<T>().expect("block_on_io: type mismatch"),
Err(payload) => std::panic::resume_unwind(payload),
@@ -364,14 +403,179 @@ pub fn wait_writable(fd: std::os::fd::RawFd) -> std::io::Result<()> {
fn wait_fd(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::Result<()> {
let me = current_pid().expect("wait_*() called outside an actor");
let _np = NoPreempt::enter();
with_runtime(|inner| inner.with_shared(|s| {
let io = s.io.as_mut().expect("io thread not started");
io.epoll_register(fd, me, readable, writable)
}))?;
let epoch = begin_wait();
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
io.as_mut().expect("io thread not started").epoll_register(fd, me, epoch, readable, writable)
})?;
// If a terminal stop unwinds us out of the park below, the registration
// must not outlive us: a stale `waiters` entry fails every future
// `wait_*` on this fd with AlreadyExists, and the kernel-side ADD leaks
// until the fd happens to be reused. Clean up iff the entry is still
// ours — a `FdReady` racing the stop may have consumed it already (it
// removes + DELs under the io lock), after which the fd may even carry
// ANOTHER actor's fresh registration; in that case touch nothing.
struct Dereg {
fd: std::os::fd::RawFd,
me: Pid,
epoch: u32,
}
impl Drop for Dereg {
fn drop(&mut self) {
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
if let Some(io) = io.as_mut() {
if io.waiters.get(&self.fd) == Some(&(self.me, self.epoch)) {
io.waiters.remove(&self.fd);
io.epoll_deregister(self.fd);
}
}
});
}
}
let guard = Dereg { fd, me, epoch };
park_current();
// Normal wake: the FdReady path removed the entry and DEL'd the fd
// before the unpark, so the guard's check would be a guaranteed no-op —
// skip the io lock on the hot path. (Dereg owns nothing; forget leaks
// no resource.)
std::mem::forget(guard);
Ok(())
}
// ---------------------------------------------------------------------------
// FdArm — fd readiness as a select arm (RFC 008)
// ---------------------------------------------------------------------------
/// An fd-readiness arm for [`crate::select`] / [`crate::select_timeout`]:
/// ready when the fd is readable (resp. writable), composable with channel
/// receivers on one wait epoch. Phase-1 rules apply: one waiter per fd at a
/// time, one direction per arm (duplex on a single fd needs `dup`; epoll
/// registrations key on the open file description, so dup'd fds register
/// independently).
pub struct FdArm {
fd: std::os::fd::RawFd,
readable: bool,
writable: bool,
}
impl FdArm {
pub fn readable(fd: std::os::fd::RawFd) -> Self {
FdArm { fd, readable: true, writable: false }
}
pub fn writable(fd: std::os::fd::RawFd) -> Self {
FdArm { fd, readable: false, writable: true }
}
}
impl crate::channel::sealed::Sealed for FdArm {}
impl crate::channel::Selectable for FdArm {
/// Ready-now check is a zero-timeout `poll(2)`; if the requested events
/// are pending the wait is retired without registering (`Ok(false)`,
/// the channel-arm contract). Otherwise register with the io thread —
/// every failure surfaces as `Err` (EBADF including a closed-fd
/// POLLNVAL, EMFILE on the epoll set, AlreadyExists for a second
/// waiter on the fd): the fallible-out, nothing-left-behind rule, in
/// deviation from RFC 008's permanently-ready lean, which would spin a
/// consumer whose fd is healthy but unregistrable (EMFILE).
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool> {
if poll_events(self.fd, self.readable, self.writable)? {
return Ok(false);
}
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
io.as_mut()
.expect("io thread not started")
.epoll_register(self.fd, pid, epoch, self.readable, self.writable)
})?;
Ok(true)
}
/// Classification is the same zero-timeout poll: a pure function of fd
/// state, independent of the registration the cleanup pass removed. An
/// error here (EBADF: fd closed mid-wait) reports READY — the
/// consumer's read/write surfaces the errno; a dead arm is an event,
/// not a hang.
fn sel_ready(&self) -> bool {
poll_events(self.fd, self.readable, self.writable).unwrap_or(true)
}
/// `wait_fd`'s `Dereg` compare, verbatim: remove the waiters entry and
/// kernel-side registration iff the entry is still `(pid, epoch)`-ours.
/// A `FdReady` racing the wake may have consumed it (it removes + DELs
/// under the io lock), after which the fd may even carry ANOTHER
/// actor's fresh registration; in that case touch nothing.
fn sel_unregister(&self, pid: Pid, epoch: u32) {
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
if let Some(io) = io.as_mut() {
if io.waiters.get(&self.fd) == Some(&(pid, epoch)) {
io.waiters.remove(&self.fd);
io.epoll_deregister(self.fd);
}
}
});
}
fn sel_eager_cleanup(&self) -> bool {
true
}
}
/// Zero-timeout `poll(2)`: are any of the requested events (or ERR/HUP,
/// which make the consumer's read/write fail loudly rather than park
/// forever) pending on `fd`? POLLNVAL maps to `Err(EBADF)`.
fn poll_events(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::Result<bool> {
let mut events: libc::c_short = 0;
if readable {
events |= libc::POLLIN;
}
if writable {
events |= libc::POLLOUT;
}
let mut pfd = libc::pollfd { fd, events, revents: 0 };
loop {
let r = unsafe { libc::poll(&mut pfd, 1, 0) };
if r < 0 {
let e = std::io::Error::last_os_error();
if e.kind() == std::io::ErrorKind::Interrupted {
continue;
}
return Err(e);
}
if r == 0 {
return Ok(false);
}
if pfd.revents & libc::POLLNVAL != 0 {
return Err(std::io::Error::from_raw_os_error(libc::EBADF));
}
return Ok(pfd.revents & (events | libc::POLLERR | libc::POLLHUP) != 0);
}
}
/// Wait until `fd` is readable or `timeout` elapses: `Ok(true)` = ready,
/// `Ok(false)` = timed out. A one-arm [`crate::try_select_timeout`].
pub fn wait_readable_timeout(
fd: std::os::fd::RawFd,
timeout: std::time::Duration,
) -> std::io::Result<bool> {
let arm = FdArm::readable(fd);
Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some())
}
/// Wait until `fd` is writable or `timeout` elapses: `Ok(true)` = ready,
/// `Ok(false)` = timed out.
pub fn wait_writable_timeout(
fd: std::os::fd::RawFd,
timeout: std::time::Duration,
) -> std::io::Result<bool> {
let arm = FdArm::writable(fd);
Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some())
}
pub fn read(fd: std::os::fd::RawFd, buf: &mut [u8]) -> std::io::Result<usize> {
wait_readable(fd)?;
let n = unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) };
@@ -389,13 +593,16 @@ pub fn write(fd: std::os::fd::RawFd, buf: &[u8]) -> std::io::Result<usize> {
// ---------------------------------------------------------------------------
pub fn register_supervisor_channel(pid: Pid, sender: Sender<Signal>) {
with_runtime(|inner| inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
slot.supervisor_channel = Some(sender);
} else {
panic!("register_supervisor_channel: pid {:?} not found", pid);
}
}));
with_runtime(|inner| {
let slot = inner.slot_at(pid)
.unwrap_or_else(|| panic!("register_supervisor_channel: pid {:?} not found", pid));
let mut cold = slot.cold.lock();
assert_eq!(
slot.generation(), pid.generation(),
"register_supervisor_channel: pid {:?} not found", pid
);
cold.supervisor_channel = Some(sender);
});
}
// ---------------------------------------------------------------------------
+624
View File
@@ -0,0 +1,624 @@
//! The per-slot scheduling state machine, as a standalone unit.
//!
//! One atomic word packs `(generation << 32) | (epoch << 8) | state`; every
//! transition is a CAS on the packed word, so the generation check is atomic
//! with the transition — no ABA, no acting on a recycled slot. The diagram
//! and the full protocol rationale live in `runtime.rs`; this module is the
//! mechanism, factored out so that:
//!
//! - loom can model-check the production transitions directly (see the
//! `loom_tests` module; built with `RUSTFLAGS="--cfg loom"`), and
//! - every method asserts the precondition it relies on (`debug_assert!` —
//! these are hot paths), per the assert-the-invariants house rule.
//!
//! ## The park-epoch (wait identity)
//!
//! The middle 24 bits carry the slot's *park-epoch*: the identity of the
//! actor's current (or most recent) wait. The rules:
//!
//! - [`begin_wait`](StateWord::begin_wait) bumps the epoch and returns it;
//! the actor calls it once per wait, *before* registering itself with any
//! waker. Registrations carry `(pid, epoch)`.
//! - A wake may be **epoch-matched** (`unpark(gen, Some(epoch))`): it lands
//! only if the word still carries that epoch. Wakers whose registration
//! handle can outlive the wait it was created for (channel senders, mutex
//! grants, wait-timers) MUST use this form.
//! - Every successful wake **consumes** the epoch — `Parked(e) → Queued(e+1)`,
//! `Running(e) → RunningNotified(e+1)` — so at most one wake can ever land
//! per wait, by construction. A loser in a multi-waker race (e.g. the
//! non-winning arms of a `select`) fails the epoch check and no-ops; it can
//! neither steal a future wait's wake nor leave a pending notification that
//! would fault a later one-shot park (`Mutex::lock_timeout`, `sleep`,
//! `block_on_io`, `wait_fd` all rely on wakes being *meaningful*).
//! - The wildcard form (`unpark(gen, None)`) also consumes, and is reserved
//! for terminal wakes — `request_stop` — which never return control to the
//! code that parked.
//!
//! Epoch wrap (24 bits = 16.7M waits) is harmless: a collision would require
//! a taken registration to stay in flight across a full wrap of the *same
//! actor's* waits, and registrations are consumed at take-time under their
//! primitive's lock — the exposure is the taker's instruction window.
//!
//! Atomics come from `sync_shim` (std normally, `loom::sync` under
//! `cfg(loom)`).
use crate::sync_shim::{AtomicU64, Ordering};
pub(crate) const ST_VACANT: u64 = 0;
pub(crate) const ST_QUEUED: u64 = 1;
pub(crate) const ST_RUNNING: u64 = 2;
pub(crate) const ST_RUNNING_NOTIFIED: u64 = 3;
pub(crate) const ST_PARKED: u64 = 4;
pub(crate) const ST_DONE: u64 = 5;
/// Park-epoch width: 24 bits, packed at word bits 8..32.
pub(crate) const EPOCH_MASK: u32 = 0x00FF_FFFF;
#[inline]
pub(crate) const fn pack(gen: u32, epoch: u32, st: u64) -> u64 {
debug_assert!(epoch & !EPOCH_MASK == 0);
((gen as u64) << 32) | ((epoch as u64) << 8) | st
}
#[inline]
pub(crate) const fn word_gen(w: u64) -> u32 {
(w >> 32) as u32
}
#[inline]
pub(crate) const fn word_epoch(w: u64) -> u32 {
((w >> 8) as u32) & EPOCH_MASK
}
#[inline]
pub(crate) const fn word_state(w: u64) -> u64 {
w & 0xFF
}
/// What an unpark amounted to. The caller owns the side effects (enqueue,
/// trace events) — this module is pure state.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub(crate) enum Unpark {
/// Parked → Queued: the caller must enqueue the pid.
Enqueue,
/// Running → RunningNotified: the scheduler's park-return will re-queue.
Notified,
/// Stale generation, stale epoch, already queued/notified, done, or
/// vacant.
Noop,
}
/// A pid's-eye view of the slot, for cold paths that hold the slot lock.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub(crate) enum Status {
/// The generation no longer matches: the slot was reclaimed (and possibly
/// reused) — the pid is stale.
Stale,
/// The actor terminated; its outcome is (or was) in the slot.
Done,
/// Alive in some scheduling state (Queued / Running / Notified / Parked).
Live,
}
pub(crate) struct StateWord(AtomicU64);
impl StateWord {
pub(crate) fn new() -> Self {
Self(AtomicU64::new(pack(0, 0, ST_VACANT)))
}
#[inline]
pub(crate) fn load(&self) -> u64 {
self.0.load(Ordering::Acquire)
}
#[inline]
pub(crate) fn generation(&self) -> u32 {
word_gen(self.load())
}
#[inline]
pub(crate) fn status_for(&self, gen: u32) -> Status {
let w = self.load();
if word_gen(w) != gen {
return Status::Stale;
}
match word_state(w) {
ST_DONE => Status::Done,
// A matching generation on a Vacant slot is unreachable for any
// ISSUED pid — reclaim bumps the generation in the very store
// that vacates, and install publishes Queued before the pid
// escapes. But `Pid::new` is public, so a forged / never-issued
// pid (e.g. `Pid::new(5, 0)` against a fresh slab) can land
// here; for those, "no such actor" is the correct total answer.
ST_VACANT => Status::Stale,
_ => Status::Live,
}
}
/// Spawn-side publish: Vacant → Queued. The caller owns the vacant slot
/// exclusively (it popped the index from the free list), so this is a
/// plain Release store; it is the moment the actor becomes visible to
/// pops, unparks, and stops. The epoch starts at 0 for each occupancy
/// (`set_done` zeroes it; wait identity never crosses a lifetime).
pub(crate) fn publish_queued(&self, gen: u32) {
debug_assert_eq!(
self.load(),
pack(gen, 0, ST_VACANT),
"publish over a non-vacant slot"
);
self.0.store(pack(gen, 0, ST_QUEUED), Ordering::Release);
}
/// Scheduler pop-side claim: Queued → Running, epoch preserved. `false`
/// means the popped pid is stale — by the at-most-once-enqueued
/// invariant, a generation mismatch is the only possible failure
/// (asserted). Nothing can move a matching-gen word off Queued (wakes
/// no-op on Queued), so the CAS loop is single-shot in practice.
#[must_use]
pub(crate) fn try_claim(&self, gen: u32) -> bool {
loop {
let w = self.load();
if word_gen(w) != gen {
return false;
}
debug_assert_eq!(
word_state(w),
ST_QUEUED,
"queued pid found in unexpected state {} — double enqueue?",
word_state(w)
);
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), ST_RUNNING),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return true;
}
}
}
/// Yield return path: Running | RunningNotified → Queued, epoch
/// preserved. A notification that arrived mid-run coalesces into the
/// re-queue. Caller must enqueue. CAS loop because a notify can bump the
/// epoch between the read and the exchange.
pub(crate) fn yield_return(&self, gen: u32) {
loop {
let w = self.load();
debug_assert!(
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(w) == gen,
"yield return from invalid word {w:#x}"
);
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), ST_QUEUED),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return;
}
}
}
/// Park return path. `true` = actually parked. `false` = an unpark landed
/// in the prep-to-park window (RunningNotified); the word is already back
/// to Queued and the caller must enqueue — the lost-wakeup window,
/// closed. Epoch preserved on both paths (the notify already consumed
/// it).
#[must_use]
pub(crate) fn park_return(&self, gen: u32) -> bool {
loop {
let w = self.load();
debug_assert_eq!(word_gen(w), gen, "park return with stale gen");
let target = match word_state(w) {
ST_RUNNING => ST_PARKED,
ST_RUNNING_NOTIFIED => ST_QUEUED,
st => unreachable!("park return from invalid state {st}"),
};
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), target),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return target == ST_PARKED;
}
}
}
/// Open a new wait: bump the park-epoch and return it. Called by the
/// waiting actor itself (so the state is Running, or RunningNotified if
/// a terminal wake is already pending — the bump preserves the pending
/// notification), once per wait, BEFORE registering `(pid, epoch)` with
/// any waker.
#[must_use]
pub(crate) fn begin_wait(&self, gen: u32) -> u32 {
loop {
let w = self.load();
debug_assert!(
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(w) == gen,
"begin_wait from invalid word {w:#x}"
);
let next = word_epoch(w).wrapping_add(1) & EPOCH_MASK;
if self
.0
.compare_exchange(
w,
pack(gen, next, word_state(w)),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return next;
}
}
}
/// The unpark protocol — the one way anything outside the scheduler makes
/// an actor runnable. See [`Unpark`] for the caller's obligations.
///
/// `want = Some(epoch)` is the epoch-matched form: lands only if the word
/// still carries that epoch (i.e. the wait it was registered for is still
/// the current, un-woken wait). `want = None` is the wildcard, reserved
/// for terminal wakes. Both forms CONSUME the epoch on success.
#[must_use]
pub(crate) fn unpark(&self, gen: u32, want: Option<u32>) -> Unpark {
loop {
let w = self.load();
if word_gen(w) != gen {
return Unpark::Noop;
}
if let Some(e) = want {
if word_epoch(w) != e {
return Unpark::Noop;
}
}
let bumped = word_epoch(w).wrapping_add(1) & EPOCH_MASK;
match word_state(w) {
ST_PARKED => {
if self
.0
.compare_exchange(
w,
pack(gen, bumped, ST_QUEUED),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return Unpark::Enqueue;
}
}
ST_RUNNING => {
if self
.0
.compare_exchange(
w,
pack(gen, bumped, ST_RUNNING_NOTIFIED),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return Unpark::Notified;
}
}
_ => return Unpark::Noop, // Queued | Notified | Done | Vacant
}
}
}
/// Eat a pending notification: RunningNotified → Running, epoch
/// preserved; no-op on Running. Called by the RUNNING actor itself, on
/// the no-park exit of a wait it registered for but never parked on
/// (`select` returning a ready arm at registration time), AFTER bumping
/// the epoch and BEFORE re-checking its stop flag:
///
/// - post-bump, the only wakers that can have set RunningNotified are
/// ones stamped with the just-retired epoch (a select arm) or a
/// terminal wildcard (`request_stop`);
/// - the caller's stop-flag check AFTER the clear catches the terminal
/// case (the flag is set before the wake fires), so eating its
/// notification loses nothing — and a stop arriving later re-notifies
/// a Running word as usual;
/// - what remains eaten is exactly the stale arm wake that would
/// otherwise fault the actor's next one-shot park.
///
/// Returns whether a notification was eaten.
pub(crate) fn clear_notify(&self, gen: u32) -> bool {
loop {
let w = self.load();
debug_assert!(
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(w) == gen,
"clear_notify from invalid word {w:#x}"
);
if word_state(w) != ST_RUNNING_NOTIFIED {
return false;
}
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), ST_RUNNING),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return true;
}
}
}
/// Finalize: Running | RunningNotified → Done, epoch zeroed (wait
/// identity never crosses an occupancy). Called by the scheduler that
/// just ran the actor to completion (so those are the only legal prior
/// states), under the slot's cold lock so join's check-or-register is
/// linearized against it.
pub(crate) fn set_done(&self, gen: u32) {
let prev = self.0.swap(pack(gen, 0, ST_DONE), Ordering::AcqRel);
debug_assert!(
matches!(word_state(prev), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(prev) == gen,
"finalize from invalid word {prev:#x}"
);
}
/// Reclaim: Done → Vacant(gen + 1). The generation bump IS the reclaim:
/// every stale pid is dead from this store onwards. Caller holds the cold
/// lock and has verified eligibility (asserted).
pub(crate) fn reclaim(&self, gen: u32) {
debug_assert_eq!(
self.load(),
pack(gen, 0, ST_DONE),
"reclaim of a non-Done slot"
);
self.0
.store(pack(gen.wrapping_add(1), 0, ST_VACANT), Ordering::Release);
}
}
// ---------------------------------------------------------------------------
// loom model tests — RUSTFLAGS="--cfg loom" cargo test --lib --release
// ---------------------------------------------------------------------------
#[cfg(all(test, loom))]
mod loom_tests {
use super::*;
use loom::sync::atomic::{AtomicBool, AtomicUsize};
use loom::sync::Arc;
use loom::thread;
use std::sync::atomic::Ordering as O;
/// THE lost-wakeup theorem. A waiter registers a condition check then
/// parks (as every parking site does); a waker sets the condition then
/// unparks. In every interleaving the waiter must end up runnable —
/// parked-forever-with-condition-set must be unreachable.
#[test]
fn no_lost_wakeup_park_vs_unpark() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0)); // scheduler claimed: actor Running
let epoch = word.begin_wait(0); // actor opens the wait
let ready = Arc::new(AtomicBool::new(false));
let enqueues = Arc::new(AtomicUsize::new(0));
// Waker: make the condition true, then wake the registered wait.
let w = word.clone();
let r = ready.clone();
let e = enqueues.clone();
let waker = thread::spawn(move || {
r.store(true, O::SeqCst);
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
e.fetch_add(1, O::SeqCst);
}
});
// Waiter (as the scheduler executes it): re-check the condition,
// park only if still false; a Notified park-return re-queues.
let parked = if ready.load(O::SeqCst) {
false // condition already visible: doesn't park at all
} else if word.park_return(0) {
true
} else {
enqueues.fetch_add(1, O::SeqCst); // notified → re-queued
false
};
waker.join().unwrap();
let w = word.load();
if parked {
// Parked is only a FINAL state if the waker's unpark moved it
// back to Queued (+ one enqueue). Parked-and-stays-parked
// would be the lost wakeup.
assert_eq!(word_state(w), ST_QUEUED, "lost wakeup: parked forever");
assert_eq!(enqueues.load(O::SeqCst), 1);
} else {
// Never more than one enqueue (at-most-once-enqueued).
assert!(enqueues.load(O::SeqCst) <= 1);
}
});
}
/// Two concurrent unparkers, one parked actor: exactly one wins the
/// enqueue (at-most-once), regardless of interleaving. Both stamped with
/// the live epoch — the consuming bump is what serializes them.
#[test]
fn two_unparkers_one_enqueue() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0));
let epoch = word.begin_wait(0);
assert!(word.park_return(0)); // actor parked
let enqueues = Arc::new(AtomicUsize::new(0));
let mut hs = Vec::new();
for _ in 0..2 {
let w = word.clone();
let e = enqueues.clone();
hs.push(thread::spawn(move || {
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
e.fetch_add(1, O::SeqCst);
}
}));
}
for h in hs {
h.join().unwrap();
}
assert_eq!(enqueues.load(O::SeqCst), 1);
assert_eq!(word_state(word.load()), ST_QUEUED);
});
}
/// The stale-epoch theorem — what `select`'s loser arms lean on. An
/// actor opens a wait, two registered wakers race it (against the park
/// itself, covering the prep-to-park window); afterwards the actor is
/// runnable exactly once, and a LATE waker still stamped with the
/// consumed epoch can neither enqueue nor notify — in every
/// interleaving. (Under wildcard semantics the late waker would corrupt
/// the actor's NEXT one-shot park; this is the theorem that buys
/// `Mutex::lock_timeout`/`sleep`/`block_on_io` their unchanged code.)
#[test]
fn consumed_epoch_unpark_never_lands() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0));
let epoch = word.begin_wait(0);
// Two arms race the wake, concurrent with the park itself.
let enqueues = Arc::new(AtomicUsize::new(0));
let mut hs = Vec::new();
for _ in 0..2 {
let w = word.clone();
let e = enqueues.clone();
hs.push(thread::spawn(move || {
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
e.fetch_add(1, O::SeqCst);
}
}));
}
let mut runnable_via_notify = false;
if !word.park_return(0) {
runnable_via_notify = true; // notified in prep-to-park
}
for h in hs {
h.join().unwrap();
}
// Exactly one path made the actor runnable.
let direct = enqueues.load(O::SeqCst);
if runnable_via_notify {
assert_eq!(direct, 0, "woken twice: notify AND enqueue");
} else {
assert_eq!(direct, 1, "parked forever, or woken twice");
}
assert_eq!(word_state(word.load()), ST_QUEUED);
// The actor runs again. A waker still holding the OLD epoch —
// a select loser arm firing later — must be a strict no-op,
// not a pending notification.
assert!(word.try_claim(0));
assert_eq!(word.unpark(0, Some(epoch)), Unpark::Noop);
assert_eq!(word_state(word.load()), ST_RUNNING, "stale epoch notified a live run");
});
}
/// The retire theorem — `select`'s no-park exit. An actor opens a wait
/// and registers, then finds an arm ready and returns WITHOUT parking;
/// a loser arm's waker fires concurrently, stamped with the live epoch.
/// The exit retires the wait (bump, then eat): in every interleaving
/// the run ends on a clean Running word — no pending notification
/// survives to fault the actor's next one-shot park — and the waker
/// never enqueues.
#[test]
fn retire_eats_late_arm_notification() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0));
let epoch = word.begin_wait(0); // select opens + registers
let w = word.clone();
let waker = thread::spawn(move || w.unpark(0, Some(epoch)));
// No-park exit: bump (invalidates in-flight wakes), then eat
// (consumes one that already landed).
let _ = word.begin_wait(0);
word.clear_notify(0);
assert_ne!(waker.join().unwrap(), Unpark::Enqueue);
assert_eq!(
word_state(word.load()),
ST_RUNNING,
"stale arm wake survived the retire"
);
});
}
/// The ABA theorem: a stale-generation unpark racing reclaim + reuse can
/// never touch the slot's new occupant.
#[test]
fn stale_unpark_never_hits_reused_slot() {
loom::model(|| {
let word = Arc::new(StateWord::new());
// Gen-0 actor runs to completion.
word.publish_queued(0);
assert!(word.try_claim(0));
let w = word.clone();
let stale = thread::spawn(move || w.unpark(0, None));
// Scheduler: finalize, reclaim, and a new spawn reuses the slot.
word.set_done(0);
word.reclaim(0);
word.publish_queued(1);
// The stale unpark may have squeezed in only while gen 0 was
// still Running (→ Notified) — in which case set_done's swap
// absorbed it — or it observed Done/Vacant/gen-1 and no-op'd.
// Either way it must never claim an enqueue.
assert_ne!(stale.join().unwrap(), Unpark::Enqueue);
// And the new occupant is exactly where its spawn put it.
assert_eq!(word.load(), pack(1, 0, ST_QUEUED));
});
}
/// Unpark racing the claim itself: whatever the interleaving, the actor
/// is Running or RunningNotified afterwards and nobody enqueued (it was
/// never parked).
#[test]
fn unpark_vs_claim_coalesces() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
let w = word.clone();
let unparker = thread::spawn(move || w.unpark(0, None));
assert!(word.try_claim(0)); // the entry is ours; claim must win
let r = unparker.join().unwrap();
assert_ne!(r, Unpark::Enqueue);
let st = word_state(word.load());
assert!(matches!(st, ST_RUNNING | ST_RUNNING_NOTIFIED));
});
}
}
+37
View File
@@ -0,0 +1,37 @@
//! std vs loom indirection for the modules that loom model-checks
//! (`slot_state`, `run_queue`). Everything else uses std paths directly —
//! the full runtime (context switches, futexes, real TLS) is not loom-able
//! and is never executed under `cfg(loom)`.
//!
//! Build the loom models with: `RUSTFLAGS="--cfg loom" cargo test --lib --release`
#[cfg(loom)]
pub(crate) use loom::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
#[cfg(not(loom))]
pub(crate) use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
/// `UnsafeCell` with loom's `with`/`with_mut` access API; pass-through cost
/// is zero in normal builds (`#[inline]`, newtype over std's cell).
#[cfg(loom)]
pub(crate) use loom::cell::UnsafeCell;
#[cfg(not(loom))]
pub(crate) struct UnsafeCell<T>(std::cell::UnsafeCell<T>);
#[cfg(not(loom))]
impl<T> UnsafeCell<T> {
pub(crate) fn new(v: T) -> Self {
Self(std::cell::UnsafeCell::new(v))
}
#[inline]
pub(crate) fn with<R>(&self, f: impl FnOnce(*const T) -> R) -> R {
f(self.0.get())
}
#[inline]
pub(crate) fn with_mut<R>(&self, f: impl FnOnce(*mut T) -> R) -> R {
f(self.0.get())
}
}
+19 -16
View File
@@ -18,9 +18,9 @@
//! No cancellation. When a non-timer wakeup happens (e.g. lock granted
//! before timeout), the timer entry is left in the heap. It will be popped
//! eventually and the dispatch will observe "actor is no longer parked /
//! wait_seq is stale" and no-op. Cost is ~32 bytes per stale entry plus a
//! few cycles on pop; acceptable given the upper bound is "one entry per
//! parked actor".
//! the wait's epoch was consumed" and no-op. Cost is ~32 bytes per stale
//! entry plus a few cycles on pop; acceptable given the upper bound is "one
//! entry per parked actor".
//!
//! Stale pids (slot reused since the timer was inserted) are filtered on
//! pop by the scheduler — same convention as the run queue.
@@ -35,18 +35,21 @@ use std::time::{Duration, Instant};
///
/// Held inside `Entry`, dispatched by the scheduler in `pop_due`.
pub enum Reason {
/// `loom::sleep(d)`. Unpark `pid` unconditionally (modulo the usual
/// "still parked?" check the scheduler applies).
Sleep,
/// A bounded wait — currently only `Mutex::lock_timeout`. On expiry the
/// scheduler calls `target.on_timeout(pid, wait_seq)`. The target then
/// decides whether `pid` was actually still waiting, and if so unparks
/// it with whatever error the wait was bounded for. `wait_seq` lets the
/// target tell apart "this wait" from "a later wait by the same actor
/// on the same target".
/// `sleep(d)`. Wake `pid` via the epoch-matched unpark: if anything
/// else (necessarily a terminal wake) already consumed the wait, the
/// entry is stale and no-ops at the CAS.
Sleep { epoch: u32 },
/// A bounded wait (`Mutex::lock_timeout`, `Receiver::recv_timeout`,
/// `select_timeout`). On expiry the scheduler calls
/// `target.on_timeout(pid, epoch)`. The target then decides whether
/// `pid` was actually still waiting (registration still present under
/// its lock), and if so takes the registration and unparks via
/// `unpark_at`. The epoch is the slot-word park-epoch — the runtime-wide
/// wait identity — so a stale entry is doubly inert: the registration
/// check misses, and even a racing unpark fails the word's epoch CAS.
WaitTimeout {
target: Arc<dyn TimerTarget>,
wait_seq: u64,
epoch: u32,
},
}
@@ -55,7 +58,7 @@ pub enum Reason {
/// Implementors: do not touch `SchedulerState` other than via the public
/// `unpark` / channel APIs. The scheduler is mid-iteration when this fires.
pub trait TimerTarget: Send + Sync {
fn on_timeout(&self, pid: Pid, wait_seq: u64);
fn on_timeout(&self, pid: Pid, epoch: u32);
}
pub struct Entry {
@@ -104,8 +107,8 @@ impl Timers {
}
/// Insert a `Sleep` timer. Convenience for the common case.
pub fn insert_sleep(&mut self, deadline: Instant, pid: Pid) {
self.insert(deadline, pid, Reason::Sleep);
pub fn insert_sleep(&mut self, deadline: Instant, pid: Pid, epoch: u32) {
self.insert(deadline, pid, Reason::Sleep { epoch });
}
/// Insert an arbitrary timer entry.
+5
View File
@@ -58,6 +58,9 @@ mod inner {
// Queue
Enqueue(Pid),
Dequeue(Pid),
// RFC 005 wake slot
SlotPush(Pid), // actor-context wake parked in the waking thread's slot
SlotPop(Pid), // scheduler resumed a pid from its own slot
}
// -----------------------------------------------------------------------
@@ -237,6 +240,8 @@ mod inner {
Event::RecvWake(p) => ("recv_wake".into(), p.index()),
Event::Enqueue(p) => ("enqueue".into(), p.index()),
Event::Dequeue(p) => ("dequeue".into(), p.index()),
Event::SlotPush(p) => ("slot_push".into(), p.index()),
Event::SlotPop(p) => ("slot_pop".into(), p.index()),
}
}
-263
View File
@@ -1,263 +0,0 @@
# smarm — task.md (next steps)
Handoff for a future session reusing this sandbox. Read top to bottom once
before starting; the gotchas section is hard-won and will save you a faceplant.
## Resume the environment
- Repo: `smarm`. Two branches (the old single `arm-port` stack was split):
- `master` — the mainline, and HEAD. Carries roadmap #1#5: cooperative
cancellation, supervisor strategies (one_for_one/all, rest_for_one) + the
orphaned-timer shutdown fix, links/trap_exit, selective receive, gen_server,
and demonitor/`MonitorId`. Tagged `v0.4.0`. x86-64 Linux only.
- `arm-port``master` plus a single commit: `feat(arch): aarch64 context
switch + cycle counter`. Extracts the x86-64 context-switch / stack-init /
cycle-counter out of `context.rs` into a `target_arch`-gated `src/arch/`
(x86_64 + aarch64 backends) and adds an AAPCS64 backend. ⚠️ UNTESTED: never
built or run on real ARM hardware. The x86-64 path is unchanged
(`arch/x86_64.rs` is the old `context.rs` body verbatim), so the x86 suite
passing says nothing about the aarch64 backend. Build + test on-device
before trusting it.
- Toolchain is installed but NOT on PATH in a fresh shell. First line of every
session: `. "$HOME/.cargo/env"` (rustc/cargo 1.96).
- Build `cargo build` · all tests `cargo test` · one suite `cargo test --test monitor`.
- Bench probe `cargo bench --bench general` (custom print-only harness; compiles
tokio in release the first time — slow — but `target/` persists across git
checkouts so it's paid once).
- Perf regression check: `git checkout <pre-change-sha>` →
`cargo bench --bench general | tee before.txt` → `git checkout arm-port` →
run again → diff the `smarm 1-thread` medians for `chained_spawn` and
`yield_many` (those exercise spawn/finalize/scheduler). Numbers are noisy on
this shared CPU; treat as "regression beyond noise?" not a precise delta.
## Roadmap (dependency order)
### 1. Cooperative cancellation — the keystone ✅ DONE (`a8ddb4a`)
Everything below (one_for_all/rest_for_one, links) needs a *safe* way to stop a
running peer. Forcible teardown of another green thread's stack is unsound here
(shared heap + Drop). So: cooperative stop the actor observes and unwinds itself.
Shipped as designed (sentinel unwind, not Result-threading). Notes for what
came next / future readers:
- Stop flag lives on `Actor` behind `Arc<AtomicBool>` (fresh per spawn), NOT a
`Slot` field — sidesteps the three-place reset, at the cost of one small
alloc per spawn. The scheduler hands the resume path a raw `*const AtomicBool`
(no per-resume refcount traffic); `yield_many` bench stayed at baseline,
`chained_spawn` ~+6% from that alloc (left as-is; move to a `Slot` field if it
ever matters).
- Observation points: amortised `maybe_preempt`/`check!()` path + the wakeup
side of `park_current`/`yield_now`. Sentinel = `StopSentinel` (zero-size),
recognised in the trampoline → `Outcome::Stopped`. `join()` on a stopped actor
returns `Ok(())` (no payload to propagate; reason is on the monitor channel).
- Documented gaps confirmed by tests: no-observation-point loop can't be stopped
(same as preemption); a user `catch_unwind` can swallow the sentinel but the
flag stays set so the next yield re-raises.
Original plan, for reference:
- Add a per-actor stop flag (Slot field + atomic, or check via shared state).
- `request_stop(pid)`: set the flag, unpark if parked.
- Realize the stop as a **controlled unwind**: when the scheduler resumes a
stop-requested actor, inject a sentinel panic (dedicated payload type) so the
existing `trampoline` `catch_unwind` tears the stack down and runs Drop. The
trampoline recognizes the sentinel and reports a new `Outcome::Stopped`
(distinct from a user `Panic`). This avoids changing every blocking-op
signature.
- Alternative considered: thread `Result<_, Cancelled>` through recv/sleep/
lock/io. Rejected — large API churn. Go with the sentinel unwind.
- Caveat to document: user code with its own `catch_unwind` can swallow the
sentinel (cf. Erlang `catch`); re-check the flag at the next yield and/or
re-raise. And a tight no-alloc loop without `check!()` can't be stopped —
same inherent limitation as preemption.
- Observation points: `maybe_preempt()`/`check!()` (cheap flag check) and the
blocking parks (recv/sleep/mutex/io) on the stop-driven unpark.
- Tests: looping actor on `check!()` gets stopped → `Outcome::Stopped`, Drop
guards ran; parked-on-recv actor gets stopped; no-check loop documents the gap.
### 2. one_for_all / rest_for_one + ordered shutdown ✅ DONE (`351dc9c`)
Shipped. What landed vs the plan:
- `Strategy::{OneForOne,OneForAll,RestForOne}` selected via `.strategy()`,
default `OneForOne`. The struct keeps the `OneForOne` name (compat; existing
tests untouched) — a rename to `Supervisor` is a deferred refactor.
- The *triggering* child's `Restart` policy decides whether anything restarts;
the strategy decides which live siblings are cycled (all / index-> after the
failed one). Survivors are `request_stop`'d in reverse start order, awaited on
the existing `supervisor_channel` funnel (no new channel, no `select`),
restarted in start order. One failure = one intensity tick regardless of group
size. Out-of-band signals during an await are stashed and replayed.
- `Signal::Stopped(pid)` + `DownReason::Stopped` added (kept distinct from Exit,
as planned). A `Stopped` signal counts as abnormal for the restart decision.
- Ordered shutdown: on cap-trip / mailbox-close-with-survivors, stop remaining
children in reverse start order and await them (no-op on the normal exit).
- ⚠️ Surfaced + fixed a latent keystone bug (`e80334b`): a cancelled
sleeping/timeout actor orphans its timer entry, and the scheduler's shutdown
check counted pending timers → `run()` hung until the dead actor's deadline
fired (a `sleep(30s)` sleeper hung shutdown 30s). Fix: timers no longer gate
shutdown (`live == 0` already implies nothing a timer could wake); heap is
cleared on exit. Independent of the supervisor work.
- Tests: all-restart (sibling cycled despite clean exit), suffix-restart
(prefix child left alone), reverse-order teardown.
Original plan, for reference:
- one_for_all: on any child failure, `request_stop` all siblings, await their
termination signals, restart all per spec.
- rest_for_one: stop+restart the failed child and those started after it.
- Supervisor shutdown: stop children in reverse start order.
- Decide signal surface: add `Signal::Stopped(pid)` + `DownReason::Stopped`
rather than folding into Exit (clearer for the supervisor's await logic).
- Tests: all-restart, suffix-restart, reverse-order shutdown.
### 3. Links + trap_exit ✅ DONE (`6581484`)
- `Slot.links: Vec<Pid>` (bidirectional); `link`/`unlink`; `trap_exit()` flag
lives on `Actor` (fresh per spawn → a restarted child starts un-trapped, and
no fourth slot-reset site).
- On finalize, reverse links are cleared under the lock (always — keeps the
cascade acyclic), then for each linked peer: abnormal death (`Panic`/
`Stopped`) → `request_stop(peer)` unless peer traps, in which case deliver an
`ExitSignal` *message* instead. Normal exit does NOT propagate. Linking an
already-dead pid delivers an immediate `NoProc` signal (message if trapping,
else `request_stop(self)` — not a silent no-op).
- Resolved: the trap inbox is a **dedicated** channel (`trap_exit() ->
Receiver<ExitSignal>`), distinct from the monitor `Down` channel; `ExitSignal`
reuses `DownReason` and carries no panic payload (joiner-only, as with
monitors). `spawn_link` deferred to #5.
- Tests (`tests/link.rs`): linked pair one panics → other stopped (+Drop ran);
trap_exit → other gets a message and survives; normal exit doesn't propagate;
dead-pid link stops a non-trapper / messages a trapper; `unlink` prevents
propagation.
### 4. Selective receive (independent track) ✅ DONE (`03f3875`)
Shipped. What landed vs the plan:
- `Receiver::recv_match(pred) -> Result<T, RecvError>` scans the queued
`VecDeque` front-to-back, removes+returns the first match, leaves the rest in
arrival order; parks and re-scans when nothing matches. `try_recv_match` (the
non-blocking variant, mirroring `try_recv`) rolled in same commit.
- Wakeup turned out cheaper than feared: `Sender::send` *already* took
`parked_receiver` on every push, so "wake on ANY send" needed no send-side
change. The one real edit was relaxing `Sender::drop` to unpark the parked
receiver on the last-sender drop regardless of queue emptiness — a selective
receiver can park on a non-empty no-match queue and must wake to observe
closure. No-op for plain `recv` (only ever parks on an empty queue); stress
suite stays green.
- `pred` is `Fn(&T) -> bool` (not `FnMut`) on purpose: it's re-run from scratch
on every scan, so a stateful predicate would re-count surprisingly. It runs
under the channel lock — keep it cheap/pure, don't re-enter the channel.
- Close semantics: `recv_match` returns `Err(RecvError)` only when closed AND no
queued message matches; a match is still returned on a closed channel.
Non-matches are left for a later `recv`.
- Tests (`tests/selective_recv.rs`): out-of-order match pulled first; non-matches
remain in order; park-on-non-empty then wake on a match; closed-with-only-
non-matches → Err; closed-but-match-present → match; `try_recv_match` states.
Original plan, for reference:
- Add `Receiver::recv_match(pred) -> T`: scan the queued `VecDeque`, remove+return
first match, leave the rest in order; park and re-scan on new arrivals.
- This changes channel wakeup: a parked selective receiver must wake on ANY send
(not just empty→nonempty) and re-scan. Touches `channel.rs` carefully — the
stress tests guard lost-wakeup invariants; keep them green.
- Tests: messages arrive out of interest-order; match pulled first; non-matches
remain for a later `recv`.
### 5. Grab-bag (each its own small commit)
- `spawn_link`: spawn-and-link atomically (deferred from #3); thin wrapper over
`spawn_under` + `link`, but do it under one lock so there's no window where
the child dies before the link is recorded.
- `demonitor`: needs a per-monitor id to remove a specific sender. Decide the
monitor API NOW before more code depends on it — likely return a
`Monitor { id, rx }` instead of a bare `Receiver<Down>`.
✅ DONE (this commit). What landed vs the plan:
- `monitor()` now returns `Monitor { id, target, rx }` (added `target` over
the sketched `{id, rx}` so `demonitor` jumps straight to the slot instead of
scanning every slot for the id). `MonitorId(u64)` is opaque, from a
monotonic `next_monitor_id` counter on `SharedState`, bumped under the
shared lock in `monitor()` — no atomics, deterministic, never reused.
- `Slot.monitors: Vec<(MonitorId, Sender<Down>)>`. The three slot-reset sites
were untouched — they `.clear()`/`Vec::new()`, which is element-type-
agnostic, so no new reset obligation. `finalize_actor` just destructures
`(_, m)` and sends as before.
- `demonitor(&Monitor) -> Option<MonitorId>`: `Some(id)` when a live
registration was found+removed, `None` when it had already fired (drained on
finalize), was `NoProc`, or the slot was reclaimed. Chose `Option<MonitorId>`
over a bare bool — names which registration went. Generation half of the pid
makes a stale demonitor a clean no-op: a recycled slot index fails
`slot_mut`'s generation check, so it can never strip a different actor's
monitor.
- ⚠️ Reentrancy: the removed `Sender` is `remove`d out of the Vec under the
lock but **dropped after the lock is released** — `Sender::drop` can unpark a
parked receiver → `with_shared`, and the shared mutex is non-reentrant. Same
discipline as `finalize_actor`.
- "Flush" (discard a `Down` the target already queued) falls out of dropping
the `Monitor`: `demonitor(&m); drop(m)`. That's the cleanup the still-to-come
gen_server **call timeout** wants — monitor the server, wait reply-or-Down-
or-deadline, then demonitor+drop so a timed-out call leaks no registration
and no stale `Down`.
- Perf: touches `Slot` + `finalize_actor`, but `chained_spawn`/`yield_many`
register no monitors, so the Vec stays empty (take-empty is identical cost,
finalize loop runs zero times). before/after `general` probe medians within
noise. Tests (`tests/monitor.rs`): demonitor-stops-delivery, one-of-many
(siblings untouched), after-fire-is-None.
- Named registry: `register(name,pid)`/`whereis`/`send_by_name`; a
`HashMap<String,Pid>` in `SharedState`.
- gen_server-style call/cast: request-reply correlation as a thin layer over
channels (`call` sends `{req, reply_tx}`, awaits `reply_rx`); no runtime change.
✅ DONE (`a4fcf6c`). What landed vs the plan:
- `GenServer` trait on the state value: assoc `Call`/`Reply`/`Cast` types,
required `handle_call`/`handle_cast`, optional `init`/`terminate` hooks.
`ServerRef<G>` is a clonable inbox sender + `pid()`; `start` / `start_under`.
- One inbox, not two: a single `Envelope { Call(req, reply_tx) | Cast }`
channel, dispatched by variant. Forced by no-`select`/no-unified-mailbox —
a server can't wait on a call channel and a cast channel at once.
- Server-down falls out of channel closure (no monitor needed): `send` fails
if the inbox is gone; the reply sender drops on the server's unwind so a
parked caller wakes to `Err`. Both → `Call/CastError::ServerDown`.
- `terminate` runs via a drop guard → fires on *every* exit path (clean inbox
close, handler panic, `request_stop`), not just the clean one. Caveat: it
may run mid-unwind, so keep it non-blocking (a panic inside it during an
unwind double-panics → abort).
- No `handle_info`, no call timeout — both deferred to land with timeouts
(`handle_info` needs the still-unmade cross-channel mailbox merge; a call
timeout needs a per-`recv` deadline / `Signal::Timeout`).
- Pure additive layer (no Slot/scheduler/spawn/finalize change) → no perf
check run. Tests (`tests/gen_server.rs`): cast→call roundtrip, init/terminate
ordering, both server-down paths.
- Docs: README now points at the experimental, untested aarch64 port on the
`arm-port` branch. The module table still calls `context` x86-64-only — true
for `master`, since the `src/arch/` split rides on `arm-port`. Fold the arch/
split and ARM64-supported wording into the README module table + build section
once `arm-port` is validated on hardware and merged.
## Gotchas / invariants (respect these)
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
`with_shared`. NEVER send on a channel while holding the shared lock. Pattern:
`mem::take` the senders/data under the lock, send after releasing. See
`finalize_actor` (supervisor signal + monitor Downs both sent post-lock).
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
Done/outcome → recycle stack (post-lock) → deliver supervisor Signal + monitor
Downs (post-lock) → unpark joiners → reclaim slot iff `outstanding_handles==0`.
Death notifications always precede reclamation, so a pid carried in a
Signal/Down is still matchable even as its slot is about to be reused.
- **Slot lifecycle is reset in THREE places** — `Slot::vacant()`,
`reclaim_slot()` (runtime.rs), and the slot-init block in `spawn_under`
(scheduler.rs). Any new Slot field must be reset in all three (monitors was).
- **Pid = (index, generation);** stale handles caught by generation mismatch in
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
- **No `select`, no unified per-process mailbox.** Why the supervisor uses the
single `supervisor_channel` funnel rather than N monitor channels. trap_exit
resolved this by giving each trapping actor a dedicated `Receiver<ExitSignal>`
inbox (see #3); selective receive (#4) stayed *per-channel* (`recv_match`
scans one channel's queue) rather than introducing a cross-channel mailbox —
if selective receive ever needs to span the monitor/trap inboxes too, that
cross-channel merge is the still-unmade decision.
- **Cooperative-only**: preemption and (future) cancellation both depend on the
actor reaching `check!()`/yield/alloc/blocking points.
- `run()` is single-thread (`Config::exact(1)`); tests rely on deterministic
single-thread ordering (parent runs until it parks). Multi-thread via
`runtime::init(Config…)`.
## Workflow expectations (from the human)
- TDD: write the failing test first, then implement.
- Commit incrementally with conventional-commit messages; keep each commit a
reviewable unit (they diff in their IDE and are the filter to the codebase).
- Run the full suite before each commit; check perf when a change touches
Slot/scheduler/spawn/finalize hot paths.
+120
View File
@@ -0,0 +1,120 @@
# Tests
Integration tests for the runtime. Each file owns one feature area or one
class of bug. Everything here runs under plain `cargo test`; the loom model
tests are the exception — they live **in the library** (`src/slot_state.rs`,
`src/run_queue.rs`), not in this directory, because loom must compile the
production code with shimmed atomics (see "Loom" below).
## Running
```
cargo test # debug build — RUN THIS ONE: all invariant asserts live
cargo test --release # what users actually execute (LTO, no debug_asserts)
```
Debug builds are not just "slower tests": the runtime self-checks its
invariants only there — every `StateWord` transition asserts its
precondition, `enqueue` asserts the exact `(gen, Queued)` word, `RawMutex`
enforces the never-two-cold-locks leaf rule with a per-thread held-count,
`live_actors` checks for double-finalize underflow. A green release run with
a red debug run means an invariant broke without (yet) corrupting behavior —
treat it as a real failure.
### The queue-variant matrix
The run queue is compile-time selected; the suite must pass under all three
(features are additive, so drop the default first):
```
cargo test # rq-mutex (default)
cargo test --no-default-features --features rq-mpmc
cargo test --no-default-features --features rq-striped
```
### Loom (model checking)
```
RUSTFLAGS="--cfg loom" cargo test --lib --release
```
Exhaustively explores interleavings of the slot state machine
(`src/slot_state.rs`: lost-wakeup, at-most-once-enqueue, the stale-pid ABA
theorem, unpark-vs-claim) and the ring queues (`src/run_queue.rs`:
exactly-once through lap wraparound, push/pop races). Models run the
production transitions through `src/sync_shim.rs` — std atomics normally,
`loom::sync` under `--cfg loom`. `RawMutex` is deliberately not modeled:
futexes can't be, and it's the textbook Drepper mutex3 with stress and
unwind-safety tests of its own.
### Trace feature
`cargo test --features smarm-trace` exists mainly to catch bit-rot in the
`te!()` call sites; run it after touching scheduler paths.
### Before a runtime-core PR
The full matrix, in rough order of bug-finding power per minute:
1. `cargo test` (debug, default variant)
2. debug under `rq-mpmc` and `rq-striped`
3. `cargo test --release`
4. loom
5. `cargo build --features smarm-trace`
## Catalog
**Low-level units (no scheduler)**
| file | covers |
|---|---|
| `context.rs` | `init_actor_stack` + the naked-asm context-switch shims, poked directly |
| `stack.rs` | the mmap'd stack allocator |
| `pid.rs` | pid packing/equality |
**Feature areas (run under a real runtime)**
| file | covers |
|---|---|
| `runtime.rs` | `Config`, `Runtime::run`, re-running a runtime, correctness under genuine parallelism |
| `scheduler.rs` | spawn / join / panic delivery / `yield_now` / `self_pid` |
| `channel.rs` | send/recv (recv parks, so these need the runtime) |
| `selective_recv.rs` | `recv_match` / `try_recv_match` |
| `mutex.rs` | the actor-blocking `Mutex<T>` (lock parks) |
| `timer.rs` | `sleep` ordering — time-sensitive, generous tolerances by design |
| `io.rs` | `block_on_io`: blocking closures on the pool while the actor parks |
| `io_epoll.rs` | `wait_readable` / `wait_writable` + the `read`/`write` sugar |
| `preempt.rs` | explicit preemption via `smarm::check!()` |
| `cancel.rs` | cooperative cancellation (`request_stop`) — the keystone semantics |
| `monitor.rs` | `monitor` delivers exactly one `Down`; `demonitor` |
| `link.rs` | bidirectional links + `trap_exit` |
| `supervisor.rs` | one-for-one supervision |
| `gen_server.rs` | call/cast round-trips, lifecycle callbacks, server-down detection |
**Regression & stress**
| file | covers |
|---|---|
| `stress.rs` | lost wakeups, pid-table pressure, thundering herds, panic isolation under concurrency. Where the phase-2 RefCell-migration bug was caught. |
| `poison_stop.rs` | `request_stop` racing an alloc-under-lock must not poison/abort. See its header for the full story. |
| `many_timers_multi_thread.rs` | multi-thread sleep-timer lost-wakeup regression |
## Conventions
- **Each test owns its runtime.** `init(Config::exact(N))` + `rt.run(...)`;
never share a `Runtime` between tests. Oversubscription (`exact(4)` on one
core) is deliberate — forced interleaving at yield points is how
single-core CI finds races at all.
- **Regression tests must be validated against the bug.** A regression test
that passes with the bug reintroduced is documentation, not a test.
Reintroduce the fix's inverse locally and watch it fail before trusting it
(`poison_stop.rs` went through exactly this: its first version never fired
the sentinel under a lock, and was rewritten until it SIGABRT'd pre-fix).
- **Stochastic tests get the odds stacked.** Use `Config::alloc_interval(1)`
to make every allocation an observation point, many actors, and both
phases of any every-other-allocation cadence (see
`poison_stop::self_stop_during_spawn...`).
- **Time-based assertions use ordering, not durations.** Assert
"didn't return instantly" / "A woke before B", with generous tolerances;
CI machines are slow and noisy.
- New invariants added to the runtime should come with the assert at the
point of reliance (debug_assert on hot paths) *and*, where the invariant is
a protocol, a loom model in the owning module — that combination is what
made phases 25 land without a single post-merge race so far.
+58
View File
@@ -130,3 +130,61 @@ fn join_on_stopped_actor_returns_ok() {
assert!(h.join().is_ok(), "join on a stopped actor returns Ok(())");
});
}
/// Regression: `request_stop` against a QUEUED actor must not be lossy.
///
/// The stop flag is set, but the wildcard unpark no-ops on a Queued actor
/// (the pending run "is" the wake). If the actor's first action on resume
/// is a blocking park — no allocation, no `check!()` on the way — a
/// wake-side-only check in `park_current` never runs: the actor parks with
/// the stop flag already set, and nothing will ever wake it. The runtime
/// then idles forever (the root is parked on the monitor channel).
///
/// Fix: an entry-side `check_cancelled` in `park_current`. The remaining
/// window (flag set after the entry check, before the park lands) is closed
/// by the existing protocol: the stop's unpark then finds Running /
/// the prep-to-park window, sets Notified, and the park-return re-queues
/// into the wake-side check.
///
/// Watchdog harness: without the fix this deadlocks, so the runtime runs on
/// a side thread and the test fails on a timeout instead of hanging cargo.
#[test]
fn stop_flagged_while_queued_lands_at_first_park() {
use std::sync::mpsc;
use std::time::Duration;
let dropped = Arc::new(AtomicBool::new(false));
let saw_stopped = Arc::new(AtomicBool::new(false));
let (d, s) = (dropped.clone(), saw_stopped.clone());
let (done_tx, done_rx) = mpsc::channel::<()>();
std::thread::spawn(move || {
let rt = smarm::init(smarm::Config::exact(1));
rt.run(move || {
let h = spawn(move || {
let _g = DropFlag(d);
let (tx, rx) = channel::<u8>();
let _keep = tx; // keep the channel open: recv() parks
let _ = rx.recv(); // first observation point is this park
});
let pid = h.pid();
let down = monitor(pid);
// Stop while the child is still QUEUED — before it ever runs.
// The unpark no-ops; only the flag is left behind.
request_stop(pid);
let dn = down.rx.recv().expect("monitor channel closed before Down");
assert_eq!(dn.pid, pid);
if matches!(dn.reason, DownReason::Stopped) {
s.store(true, Ordering::SeqCst);
}
let _ = h.join();
});
let _ = done_tx.send(());
});
done_rx
.recv_timeout(Duration::from_secs(10))
.expect("runtime deadlocked: stop against a QUEUED actor was lost at its first park");
assert!(saw_stopped.load(Ordering::SeqCst), "expected DownReason::Stopped");
assert!(dropped.load(Ordering::SeqCst), "Drop guard must run during the cancellation unwind");
}
+186
View File
@@ -108,3 +108,189 @@ fn recv_returns_err_when_all_senders_dropped() {
assert!(saw_err.load(std::sync::atomic::Ordering::SeqCst));
}
#[test]
fn channel_ops_interleaved_with_monitor_churn_multi_thread() {
// Regression for the RawMutex migration: monitor registration clones the
// Down sender under the target's cold (Leaf) lock, which now nests a
// Channel-class lock under it. Debug builds enforce the Leaf -> Channel
// ordering on every acquisition, so driving channels, monitors, and actor
// death concurrently across schedulers makes any ordering regression
// panic here rather than deadlock in the field.
use std::sync::atomic::{AtomicI64, Ordering};
use std::sync::Arc;
let total = Arc::new(AtomicI64::new(0));
let total2 = total.clone();
smarm::init(smarm::Config::exact(4)).run(move || {
let (tx, rx) = channel::<i64>();
let consumer = spawn(move || {
let mut sum = 0;
while let Ok(v) = rx.recv() {
sum += v;
}
OUT.with(|c| c.set(sum)); // not asserted cross-thread; see total
total2.fetch_add(sum, Ordering::Relaxed);
});
let mut handles = Vec::new();
for i in 0..32i64 {
let tx = tx.clone();
handles.push(spawn(move || {
// Short-lived target whose death fires the monitor below.
let t = spawn(move || {
tx.send(i).unwrap();
});
let m = smarm::monitor(t.pid());
t.join().unwrap();
// Down delivery exercises send-from-finalize.
let d = m.rx.recv().unwrap();
assert_eq!(d.reason, smarm::DownReason::Exit);
}));
}
drop(tx);
for h in handles {
h.join().unwrap();
}
consumer.join().unwrap();
});
assert_eq!(total.load(std::sync::atomic::Ordering::Relaxed), (0..32).sum::<i64>());
}
// ---------------------------------------------------------------------------
// recv_timeout
// ---------------------------------------------------------------------------
use smarm::RecvTimeoutError;
use std::time::{Duration, Instant};
#[test]
fn recv_timeout_returns_queued_message_immediately() {
run(|| {
let (tx, rx) = channel::<i64>();
tx.send(5).unwrap();
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(5));
});
}
#[test]
fn recv_timeout_times_out_on_silent_channel() {
run(|| {
let (_tx, rx) = channel::<i64>();
let start = Instant::now();
let r = rx.recv_timeout(Duration::from_millis(50));
assert_eq!(r, Err(RecvTimeoutError::Timeout));
assert!(start.elapsed() >= Duration::from_millis(50));
});
}
#[test]
fn recv_timeout_wakes_promptly_on_send() {
run(|| {
let (tx, rx) = channel::<i64>();
let h = spawn(move || {
let start = Instant::now();
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(9));
// Far below the timeout: the send woke us, not the deadline.
assert!(start.elapsed() < Duration::from_secs(1));
});
smarm::yield_now();
tx.send(9).unwrap();
h.join().unwrap();
});
}
#[test]
fn recv_timeout_reports_disconnected_on_close() {
run(|| {
let (tx, rx) = channel::<i64>();
let h = spawn(move || {
assert_eq!(
rx.recv_timeout(Duration::from_secs(10)),
Err(RecvTimeoutError::Disconnected)
);
});
smarm::yield_now();
drop(tx);
h.join().unwrap();
});
}
#[test]
fn recv_timeout_zero_duration_is_a_bounded_poll() {
run(|| {
let (_tx, rx) = channel::<i64>();
assert_eq!(rx.recv_timeout(Duration::ZERO), Err(RecvTimeoutError::Timeout));
});
}
#[test]
fn channel_remains_usable_after_a_timeout() {
// The stale timer entry from the first (timed-out) wait must not cancel
// or corrupt later waits — seq isolation.
run(|| {
let (tx, rx) = channel::<i64>();
assert_eq!(
rx.recv_timeout(Duration::from_millis(10)),
Err(RecvTimeoutError::Timeout)
);
// Plain recv still works...
tx.send(1).unwrap();
assert_eq!(rx.recv(), Ok(1));
// ...and so does a second bounded wait, woken by a send.
let h = spawn(move || {
tx.send(2).unwrap();
});
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(2));
h.join().unwrap();
});
}
#[test]
fn recv_timeout_many_waiters_multi_thread() {
// Mixed outcomes under real parallelism: half the channels get fed,
// half time out; every actor must resolve correctly.
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;
let got = Arc::new(AtomicU32::new(0));
let timed_out = Arc::new(AtomicU32::new(0));
let (got2, timed_out2) = (got.clone(), timed_out.clone());
smarm::init(smarm::Config::exact(4)).run(move || {
let mut handles = Vec::new();
for i in 0..24i64 {
let (tx, rx) = channel::<i64>();
let got = got2.clone();
let timed_out = timed_out2.clone();
handles.push(spawn(move || match rx.recv_timeout(Duration::from_millis(100)) {
Ok(v) => {
assert_eq!(v, i);
got.fetch_add(1, Ordering::Relaxed);
}
Err(RecvTimeoutError::Timeout) => {
timed_out.fetch_add(1, Ordering::Relaxed);
}
Err(e) => panic!("unexpected: {e}"),
}));
if i % 2 == 0 {
handles.push(spawn(move || {
tx.send(i).unwrap();
}));
}
// odd i: tx drops here -> Disconnected, not Timeout! Keep it alive
// instead by leaking the sender into a holder actor that outlives
// the deadline.
else {
handles.push(spawn(move || {
smarm::sleep(Duration::from_millis(200));
drop(tx);
}));
}
}
for h in handles {
h.join().unwrap();
}
});
assert_eq!(got.load(std::sync::atomic::Ordering::Relaxed), 12);
assert_eq!(timed_out.load(std::sync::atomic::Ordering::Relaxed), 12);
}
-34
View File
@@ -58,7 +58,6 @@ use std::sync::OnceLock;
static REG_BEFORE: OnceLock<[u64; 4]> = OnceLock::new();
static REG_AFTER: OnceLock<[u64; 4]> = OnceLock::new();
#[cfg(target_arch = "x86_64")]
extern "C-unwind" fn actor_reg_check() {
unsafe {
let s0: u64 = 0xAAAA_BBBB_0000_0001;
@@ -84,39 +83,6 @@ extern "C-unwind" fn actor_reg_check() {
}
}
// aarch64 sibling: same intent, AAPCS64 callee-saved integer registers.
// LLVM reserves x19 (and uses x29/x30 as fp/lr), so those can't be named as
// asm operands. We use x23x26 instead — callee-saved and bindable — moving
// known values into them, yielding, then reading them back, exactly mirroring
// the x86 test's move-into-r12..r15 pattern. The shim saves x23x26 in its
// `stp x23,x24` / `stp x25,x26` pairs, so a clean round-trip proves they
// survive the switch.
#[cfg(target_arch = "aarch64")]
extern "C-unwind" fn actor_reg_check() {
unsafe {
let s0: u64 = 0xAAAA_BBBB_0000_0001;
let s1: u64 = 0xCCCC_DDDD_0000_0002;
let s2: u64 = 0xEEEE_FFFF_0000_0003;
let s3: u64 = 0x1111_2222_0000_0004;
core::arch::asm!(
"mov x23, {s0}", "mov x24, {s1}", "mov x25, {s2}", "mov x26, {s3}",
s0 = in(reg) s0, s1 = in(reg) s1, s2 = in(reg) s2, s3 = in(reg) s3,
out("x23") _, out("x24") _, out("x25") _, out("x26") _,
);
REG_BEFORE.set([s0, s1, s2, s3]).ok();
switch_to_scheduler();
let a0: u64; let a1: u64; let a2: u64; let a3: u64;
core::arch::asm!(
"mov {a0}, x23", "mov {a1}, x24", "mov {a2}, x25", "mov {a3}, x26",
a0 = out(reg) a0, a1 = out(reg) a1, a2 = out(reg) a2, a3 = out(reg) a3,
);
REG_AFTER.set([a0, a1, a2, a3]).ok();
switch_to_scheduler();
}
}
#[test]
fn callee_saved_registers_survive_yield() {
let stack = Stack::new(64 * 1024).unwrap();
+403
View File
@@ -0,0 +1,403 @@
//! RFC 008 — fd arms in select. Beyond the functional cases, the
//! *_stays_usable tests are the soundness probes for the one asymmetry the
//! RFC must close: a losing CHANNEL arm's stale registration is inert, but
//! a losing FD arm's registration (waiters entry + kernel ONESHOT) poisons
//! the fd with AlreadyExists until the eager cleanup pass removes it. Every
//! "loser" scenario therefore re-waits on the same fd afterwards and must
//! succeed — pre-cleanup, each of those re-waits errors or hangs.
//!
//! House pattern: actor panics are trampoline-caught and `run` returns
//! normally, so every test funnels its result into an outcome flag asserted
//! OUTSIDE `run` — an in-actor assertion alone passes vacuously.
use smarm::{
channel, run, select, select_timeout, spawn, try_select, wait_readable,
wait_readable_timeout, wait_writable_timeout, yield_now, FdArm,
};
use std::os::fd::RawFd;
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
// ---------------------------------------------------------------------------
// Pipe helper (as in io_epoll.rs)
// ---------------------------------------------------------------------------
struct Pipe {
read: RawFd,
write: RawFd,
}
impl Pipe {
fn new() -> Self {
let mut fds: [libc::c_int; 2] = [0; 2];
let r = unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_CLOEXEC | libc::O_NONBLOCK) };
assert_eq!(r, 0, "pipe2 failed");
Pipe { read: fds[0], write: fds[1] }
}
}
impl Drop for Pipe {
fn drop(&mut self) {
unsafe {
libc::close(self.read);
libc::close(self.write);
}
}
}
fn raw_write(fd: RawFd, buf: &[u8]) -> isize {
unsafe { libc::write(fd, buf.as_ptr() as *const _, buf.len()) }
}
fn raw_read(fd: RawFd, buf: &mut [u8]) -> isize {
unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) }
}
fn flag() -> (Arc<AtomicBool>, Arc<AtomicBool>) {
let f = Arc::new(AtomicBool::new(false));
(f.clone(), f)
}
// ---------------------------------------------------------------------------
// Ready-now: data already pending retires the wait without parking.
// ---------------------------------------------------------------------------
#[test]
fn fd_arm_ready_now_returns_without_parking() {
let (ok, ok2) = flag();
run(move || {
let p = Pipe::new();
assert_eq!(raw_write(p.write, b"x"), 1);
let (_tx, rx) = channel::<i64>();
let fd_arm = FdArm::readable(p.read);
// fd arm at index 1: the ready-now path must also work for a
// non-first arm (and clean nothing — channel arms are inert).
let i = select(&[&rx, &fd_arm]);
assert_eq!(i, 1);
let mut buf = [0u8; 1];
assert_eq!(raw_read(p.read, &mut buf), 1);
ok2.store(true, Ordering::SeqCst);
});
assert!(ok.load(Ordering::SeqCst));
}
// ---------------------------------------------------------------------------
// Park-then-wake: fd arm wins against an idle channel arm.
// ---------------------------------------------------------------------------
#[test]
fn fd_arm_parks_until_data_then_wins() {
let got = Arc::new(AtomicU32::new(0));
let got2 = got.clone();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let (_tx_keepalive, rx) = channel::<i64>();
let h = spawn(move || {
let fd_arm = FdArm::readable(rfd);
let i = select(&[&fd_arm, &rx]);
assert_eq!(i, 0);
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
got2.store(buf[0] as u32, Ordering::SeqCst);
});
yield_now(); // let it park
assert_eq!(raw_write(wfd, b"y"), 1);
let _ = h.join();
});
assert_eq!(got.load(Ordering::SeqCst), b'y' as u32);
}
// ---------------------------------------------------------------------------
// THE asymmetry probe: channel arm wins, losing fd arm must be cleaned —
// the same actor (and the io thread) must be able to wait that fd again.
// ---------------------------------------------------------------------------
#[test]
fn losing_fd_arm_is_cleaned_up_and_fd_stays_usable() {
let got = Arc::new(AtomicU32::new(0));
let got2 = got.clone();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let (tx, rx) = channel::<i64>();
let h = spawn(move || {
let fd_arm = FdArm::readable(rfd);
// Channel wins; the fd arm registered and lost.
let i = select(&[&fd_arm, &rx]);
assert_eq!(i, 1);
assert_eq!(rx.try_recv().unwrap(), Some(7));
// Pre-cleanup this wait_readable fails AlreadyExists (the
// waiters entry is stale-ours) — the cleanup pass must have
// removed it.
wait_readable(rfd).unwrap();
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
got2.store(buf[0] as u32, Ordering::SeqCst);
});
yield_now(); // let it park in the select
tx.send(7).unwrap();
yield_now(); // let it reach the second wait
assert_eq!(raw_write(wfd, b"z"), 1);
let _ = h.join();
});
assert_eq!(got.load(Ordering::SeqCst), b'z' as u32);
}
// ---------------------------------------------------------------------------
// Ready-now on a LATER arm must unregister an earlier fd arm (the
// register_arms prefix-cleanup path: no park ever happens).
// ---------------------------------------------------------------------------
#[test]
fn ready_now_later_arm_cleans_earlier_fd_arm() {
let (ok, ok2) = flag();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let (tx, rx) = channel::<i64>();
tx.send(1).unwrap(); // arm 1 ready before the select
let fd_arm = FdArm::readable(rfd);
let i = select(&[&fd_arm, &rx]);
assert_eq!(i, 1);
assert_eq!(rx.try_recv().unwrap(), Some(1));
// The fd arm registered (idle pipe), then arm 1 retired the wait.
// Its registration must have been removed in the same pass.
assert_eq!(raw_write(wfd, b"a"), 1);
wait_readable(rfd).unwrap();
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
ok2.store(true, Ordering::SeqCst);
});
assert!(ok.load(Ordering::SeqCst));
}
// ---------------------------------------------------------------------------
// Two fd arms in one select (phase-1: distinct fds, nothing special).
// ---------------------------------------------------------------------------
#[test]
fn two_fd_arms_second_fires_first_stays_usable() {
let (ok, ok2) = flag();
run(move || {
let pa = Pipe::new();
let pb = Pipe::new();
let (rfd_a, wfd_a) = (pa.read, pa.write);
let (rfd_b, wfd_b) = (pb.read, pb.write);
let h = spawn(move || {
let a = FdArm::readable(rfd_a);
let b = FdArm::readable(rfd_b);
let i = select(&[&a, &b]);
assert_eq!(i, 1);
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd_b, &mut buf), 1);
// Arm a lost; its fd must be immediately re-waitable.
assert_eq!(raw_write(wfd_a, b"q"), 1);
wait_readable(rfd_a).unwrap();
assert_eq!(raw_read(rfd_a, &mut buf), 1);
assert_eq!(buf[0], b'q');
ok2.store(true, Ordering::SeqCst);
});
yield_now();
assert_eq!(raw_write(wfd_b, b"b"), 1);
let _ = h.join();
});
assert!(ok.load(Ordering::SeqCst));
}
// ---------------------------------------------------------------------------
// select_timeout: timer wins over an idle fd arm; the fd is left clean.
// ---------------------------------------------------------------------------
#[test]
fn select_timeout_timer_beats_idle_fd_arm_and_fd_stays_usable() {
let (ok, ok2) = flag();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let fd_arm = FdArm::readable(rfd);
let start = Instant::now();
let r = select_timeout(&[&fd_arm], Duration::from_millis(30));
assert!(r.is_none(), "idle fd must time out");
assert!(start.elapsed() >= Duration::from_millis(30));
// Timer win is exactly the case where the fd arm's registration is
// left behind without an eager pass.
assert_eq!(raw_write(wfd, b"c"), 1);
wait_readable(rfd).unwrap();
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
ok2.store(true, Ordering::SeqCst);
});
assert!(ok.load(Ordering::SeqCst));
}
// ---------------------------------------------------------------------------
// Derived wrappers.
// ---------------------------------------------------------------------------
#[test]
fn wait_readable_timeout_times_out_then_succeeds_with_data() {
let (ok, ok2) = flag();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let start = Instant::now();
assert_eq!(wait_readable_timeout(rfd, Duration::from_millis(30)).unwrap(), false);
assert!(start.elapsed() >= Duration::from_millis(30));
// Timed-out wait must leave the fd clean; ready path returns true.
assert_eq!(raw_write(wfd, b"d"), 1);
assert_eq!(wait_readable_timeout(rfd, Duration::from_secs(5)).unwrap(), true);
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
ok2.store(true, Ordering::SeqCst);
});
assert!(ok.load(Ordering::SeqCst));
}
#[test]
fn wait_readable_timeout_wakes_on_late_data() {
let got = Arc::new(AtomicU32::new(0));
let got2 = got.clone();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let h = spawn(move || {
assert_eq!(wait_readable_timeout(rfd, Duration::from_secs(5)).unwrap(), true);
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
got2.store(buf[0] as u32, Ordering::SeqCst);
});
yield_now();
assert_eq!(raw_write(wfd, b"e"), 1);
let _ = h.join();
});
assert_eq!(got.load(Ordering::SeqCst), b'e' as u32);
}
#[test]
fn wait_writable_timeout_ready_now_on_empty_pipe() {
let (ok, ok2) = flag();
run(move || {
let p = Pipe::new();
// An empty pipe's write end is writable: ready-now path, no park.
assert_eq!(wait_writable_timeout(p.write, Duration::from_secs(5)).unwrap(), true);
ok2.store(true, Ordering::SeqCst);
});
assert!(ok.load(Ordering::SeqCst));
}
// ---------------------------------------------------------------------------
// Error surface: registration failure is an Err from try_select, with the
// wait retired (the actor can immediately wait on something else).
// ---------------------------------------------------------------------------
#[test]
fn try_select_surfaces_registration_error_and_retires_the_wait() {
let (ok, ok2) = flag();
run(move || {
let bad: RawFd = {
let p = Pipe::new();
p.read
}; // both ends closed by Drop: EBADF on registration
let fd_arm = FdArm::readable(bad);
let err = try_select(&[&fd_arm]).unwrap_err();
// EBADF, whether the pre-poll or epoll_ctl ADD reports it.
assert_eq!(err.raw_os_error(), Some(libc::EBADF));
// The wait was retired: a normal select right after works.
let (tx, rx) = channel::<i64>();
tx.send(9).unwrap();
let i = select(&[&rx]);
assert_eq!(i, 0);
assert_eq!(rx.try_recv().unwrap(), Some(9));
ok2.store(true, Ordering::SeqCst);
});
assert!(ok.load(Ordering::SeqCst));
}
// ---------------------------------------------------------------------------
// Stop-unwind: an actor stopped while parked in an fd-arm select must not
// poison the fd (the UnregisterGuard generalization of wait_fd's Dereg).
// Mirrors io_epoll.rs::stopped_waiter_does_not_poison_the_fd.
// ---------------------------------------------------------------------------
#[test]
fn stopped_selector_does_not_poison_the_fd() {
let seen = Arc::new(AtomicU32::new(0));
let seen_outer = seen.clone();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let (_tx_keepalive, rx) = channel::<i64>();
let h = spawn(move || {
let fd_arm = FdArm::readable(rfd);
select(&[&fd_arm, &rx]);
unreachable!("neither arm ever fires while this actor lives");
});
yield_now(); // let it reach the park
smarm::request_stop(h.pid());
let _ = h.join(); // Ok(()): stopped, not panicked
// Second waiter on the SAME fd must register and be woken.
let seen2 = seen.clone();
let h2 = spawn(move || {
wait_readable(rfd).unwrap();
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
seen2.store(buf[0] as u32, Ordering::SeqCst);
});
yield_now();
assert_eq!(raw_write(wfd, b"x"), 1);
let _ = h2.join();
});
assert_eq!(seen_outer.load(Ordering::SeqCst), b'x' as u32);
}
// ---------------------------------------------------------------------------
// Phase-1 misuse: a second waiter on an fd that already has one is an Err
// (AlreadyExists), not a hang — and does not disturb the first waiter.
// ---------------------------------------------------------------------------
#[test]
fn second_waiter_on_same_fd_errs_without_disturbing_the_first() {
let got = Arc::new(AtomicU32::new(0));
let got2 = got.clone();
let (ok, ok2) = flag();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
let h = spawn(move || {
wait_readable(rfd).unwrap();
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
got2.store(buf[0] as u32, Ordering::SeqCst);
});
yield_now(); // first waiter parked
let fd_arm = FdArm::readable(rfd);
let err = try_select(&[&fd_arm]).unwrap_err();
assert_eq!(err.kind(), std::io::ErrorKind::AlreadyExists);
// First waiter still wakes normally.
assert_eq!(raw_write(wfd, b"w"), 1);
let _ = h.join();
ok2.store(true, Ordering::SeqCst);
});
assert_eq!(got.load(Ordering::SeqCst), b'w' as u32);
assert!(ok.load(Ordering::SeqCst));
}
+304 -2
View File
@@ -1,7 +1,7 @@
//! gen_server tests: call round-trip, cast, lifecycle callbacks, and the two
//! server-down detection paths (reply-channel close vs. inbox-send failure).
use smarm::gen_server::{start, CallError, GenServer};
use smarm::gen_server::{start, CallError, GenServer, ServerBuilder};
use smarm::run;
use std::sync::{Arc, Mutex};
@@ -26,6 +26,7 @@ impl GenServer for Counter {
type Call = Req;
type Reply = i64;
type Cast = Op;
type Info = ();
fn handle_call(&mut self, req: Req) -> i64 {
match req {
@@ -68,8 +69,9 @@ impl GenServer for Lifecycle {
type Call = ();
type Reply = ();
type Cast = ();
type Info = ();
fn init(&mut self) {
fn init(&mut self, _ctx: &smarm::gen_server::ServerCtx) {
self.log.lock().unwrap().push("init");
}
@@ -135,3 +137,303 @@ fn call_after_server_gone_is_server_down() {
});
assert_eq!(*got.lock().unwrap(), Some(Err(CallError::ServerDown)));
}
// ---------------------------------------------------------------------------
// call_timeout
// ---------------------------------------------------------------------------
use smarm::gen_server::CallTimeoutError;
use std::time::{Duration, Instant};
/// Replies after sleeping `delay_ms` (parking the server actor, not the OS
/// thread), so callers can race a deadline against the reply.
struct Slow;
impl GenServer for Slow {
type Call = u64; // delay in ms
type Reply = u64;
type Cast = ();
type Info = ();
fn handle_call(&mut self, delay_ms: u64) -> u64 {
if delay_ms > 0 {
smarm::sleep(Duration::from_millis(delay_ms));
}
delay_ms
}
fn handle_cast(&mut self, _: ()) {}
}
#[test]
fn call_timeout_returns_reply_within_deadline() {
run(|| {
let srv = start(Slow);
assert_eq!(srv.call_timeout(0, Duration::from_secs(10)), Ok(0));
});
}
#[test]
fn call_timeout_times_out_on_slow_handler() {
run(|| {
let srv = start(Slow);
let start_t = Instant::now();
let r = srv.call_timeout(500, Duration::from_millis(50));
assert_eq!(r, Err(CallTimeoutError::Timeout));
let elapsed = start_t.elapsed();
// Gave up at the deadline, not at the reply.
assert!(elapsed >= Duration::from_millis(50));
assert!(elapsed < Duration::from_millis(500));
});
}
#[test]
fn server_survives_an_abandoned_call_and_late_reply_is_discarded() {
run(|| {
let srv = start(Slow);
assert_eq!(
srv.call_timeout(100, Duration::from_millis(20)),
Err(CallTimeoutError::Timeout)
);
// The timed-out request is still handled; its reply send fails
// harmlessly (receiver dropped). The server must keep serving, and
// the late reply must not leak into THIS call's reply channel.
assert_eq!(srv.call_timeout(0, Duration::from_secs(10)), Ok(0));
// Plain unbounded call still fine too.
assert_eq!(srv.call(0), Ok(0));
});
}
#[test]
fn call_timeout_to_dead_server_is_server_down_not_timeout() {
struct Bomb;
impl GenServer for Bomb {
type Call = ();
type Info = ();
type Reply = ();
type Cast = ();
fn handle_call(&mut self, _: ()) {
panic!("kaboom");
}
fn handle_cast(&mut self, _: ()) {}
}
run(|| {
let srv = start(Bomb);
// Dies mid-call: reply channel closes -> ServerDown (even though the
// generous deadline never fires).
assert_eq!(
srv.call_timeout((), Duration::from_secs(10)),
Err(CallTimeoutError::ServerDown)
);
// Already gone: inbox send fails -> ServerDown.
assert_eq!(
srv.call_timeout((), Duration::from_secs(10)),
Err(CallTimeoutError::ServerDown)
);
});
}
// ---------------------------------------------------------------------------
// handle_info: out-of-band channels selected alongside the inbox (v0.8)
// ---------------------------------------------------------------------------
/// Logs every message it handles, in order; a call reads the log back.
struct Logger {
log: Vec<&'static str>,
}
impl GenServer for Logger {
type Call = ();
type Reply = Vec<&'static str>;
type Cast = ();
type Info = &'static str;
fn handle_call(&mut self, _: ()) -> Vec<&'static str> {
self.log.clone()
}
fn handle_cast(&mut self, _: ()) {
self.log.push("cast");
}
fn handle_info(&mut self, info: &'static str) {
self.log.push(info);
}
}
// An info message is dispatched to handle_info, interleaved with normal
// service.
#[test]
fn info_is_dispatched() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (info_tx, info_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(info_rx)
.start();
info_tx.send("info").unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["info"]);
}
// Arm priority: with a cast AND an info both queued before the server first
// runs, the info is handled first — info arms outrank the inbox. Relies on
// run()'s deterministic single-thread ordering.
#[test]
fn info_outranks_inbox() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (info_tx, info_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(info_rx)
.start();
// The server actor hasn't run yet: both messages are queued before
// its first select. Inbox first in *send* order, info first in *arm*
// order — arm order must win.
server.cast(()).unwrap();
info_tx.send("info").unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["info", "cast"]);
}
// Two info channels: declaration order is priority order.
#[test]
fn info_arms_keep_declaration_priority() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (hi_tx, hi_rx) = smarm::channel::<&'static str>();
let (lo_tx, lo_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(hi_rx)
.with_info(lo_rx)
.start();
// Sent low-priority first; handled high-priority first.
lo_tx.send("lo").unwrap();
hi_tx.send("hi").unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["hi", "lo"]);
}
// A closed info arm is silently dropped and the server keeps serving; the
// closure does NOT reach handle_info and does NOT starve the inbox (the
// closed-arm-is-ready-forever gotcha).
#[test]
fn closed_info_arm_is_dropped_silently() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (info_tx, info_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(info_rx)
.start();
drop(info_tx); // closed before the server's first select
server.cast(()).unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["cast"]);
}
// ---------------------------------------------------------------------------
// handle_down: monitors handed to the loop via Watcher (v0.8)
// ---------------------------------------------------------------------------
use smarm::gen_server::Watcher;
use smarm::{monitor, spawn, DownReason, Pid};
/// The motivating pattern: a server that spawns workers from a handler,
/// watches them, and logs their deaths.
struct Pool {
watcher: Option<Watcher>,
log: Vec<DownReason>,
}
enum PoolCast {
SpawnDoomedWorker,
Watch(Pid),
}
impl GenServer for Pool {
type Call = ();
type Reply = Vec<DownReason>;
type Cast = PoolCast;
type Info = ();
fn init(&mut self, ctx: &smarm::gen_server::ServerCtx) {
self.watcher = Some(ctx.watcher());
}
fn handle_call(&mut self, _: ()) -> Vec<DownReason> {
self.log.clone()
}
fn handle_cast(&mut self, cast: PoolCast) {
let watcher = self.watcher.as_ref().expect("init ran first");
match cast {
PoolCast::SpawnDoomedWorker => {
let h = spawn(|| panic!("worker died"));
watcher.watch(monitor(h.pid()));
}
PoolCast::Watch(pid) => watcher.watch(monitor(pid)),
}
}
fn handle_down(&mut self, down: smarm::Down) {
self.log.push(down.reason);
}
}
// A worker spawned and watched from inside a handler delivers its Down to
// handle_down. Down arms outrank the inbox, so the death is in the log by
// the time the follow-up call is answered.
#[test]
fn worker_pool_down_reaches_handle_down() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let server = start(Pool { watcher: None, log: Vec::new() });
server.cast(PoolCast::SpawnDoomedWorker).unwrap();
let _ = server.call(()).unwrap(); // sync point: cast handled, worker live
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec![DownReason::Panic]);
}
// Watching an already-dead pid yields an immediate NoProc Down, and the down
// arm outranks the inbox: the call cast *after* the watch still observes it.
#[test]
fn watch_dead_pid_is_noproc_down() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let h = spawn(|| {});
let dead = h.pid();
h.join().unwrap();
let server = start(Pool { watcher: None, log: Vec::new() });
server.cast(PoolCast::Watch(dead)).unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec![DownReason::NoProc]);
}
// A state that never clones the Watcher closes the control arm; the loop
// falls back to the plain-inbox park and keeps serving. (Every pre-v0.8 test
// in this file also exercises this path.)
#[test]
fn unused_ctx_closes_control_arm_silently() {
let got = Arc::new(Mutex::new(0i64));
let got2 = got.clone();
run(move || {
let server = start(Counter { n: 0 });
server.cast(Op::Add(2)).unwrap();
server.cast(Op::Add(40)).unwrap();
*got2.lock().unwrap() = server.call(Req::Get).unwrap();
});
assert_eq!(*got.lock().unwrap(), 42);
}
+43
View File
@@ -322,3 +322,46 @@ fn wait_writable_on_empty_pipe_returns_quickly() {
elapsed
);
}
// ---------------------------------------------------------------------------
// Fd hygiene on actor death (v0.8)
// ---------------------------------------------------------------------------
// An actor stopped while parked on an fd must not leak its `waiters` entry:
// before the unwind-path guard in wait_fd, the stale entry made every
// future wait_*() on that fd fail with AlreadyExists, forever.
#[test]
fn stopped_waiter_does_not_poison_the_fd() {
let outcome = Arc::new(StdMutex::new(None::<u8>));
let outcome2 = outcome.clone();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
// First waiter parks on the (empty) pipe and is stopped in place.
let h = smarm::spawn(move || {
wait_readable(rfd).unwrap();
unreachable!("the pipe is never written while this actor lives");
});
yield_now(); // let it reach the park
smarm::request_stop(h.pid());
h.join().unwrap(); // Ok(()): stopped, not panicked
// Second waiter on the SAME fd must be able to register...
let seen = Arc::new(AtomicU32::new(0));
let seen2 = seen.clone();
let h2 = smarm::spawn(move || {
wait_readable(rfd).unwrap();
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
seen2.store(buf[0] as u32, Ordering::SeqCst);
});
yield_now(); // ...and park (a failed register would panic the unwrap)
// ...and actually be woken by readiness.
assert_eq!(raw_write(wfd, b"x"), 1);
h2.join().unwrap();
*outcome2.lock().unwrap() = Some(seen.load(Ordering::SeqCst) as u8);
});
assert_eq!(*outcome.lock().unwrap(), Some(b'x'));
}
+2 -1
View File
@@ -1,4 +1,5 @@
//! `loom::Mutex<T>` tests. All run under the scheduler because `lock()`
//! `smarm::Mutex<T>` (the actor-blocking mutex) tests. All run under the
//! scheduler because `lock()`
//! needs to be able to park.
use smarm::{run, spawn, yield_now, LockTimeout, Mutex};
+89
View File
@@ -0,0 +1,89 @@
//! Regression: request_stop racing an alloc-under-lock must not poison any
//! runtime mutex. Before the fix, check_cancelled() fired regardless of
//! PREEMPTION_ENABLED, so a sentinel unwind could trigger inside with_shared
//! (which allocates: run_queue.push_back, waiters.push, etc), poisoning the
//! shared mutex and cascading lock().unwrap() panics.
use smarm::runtime::{init, Config};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
#[test]
fn stop_storm_does_not_poison_runtime() {
// alloc_interval(1): every allocation is an observation point, so pre-fix
// the sentinel fires inside with_shared's alloc sites (run_queue.push_back,
// waiters.push, ...) with near-certainty rather than 1-in-128.
let rt = init(Config::exact(4).alloc_interval(1));
let completed = Arc::new(AtomicUsize::new(0));
let c = completed.clone();
rt.run(move || {
// Spawn many short actors that allocate + yield heavily (hammering
// with_shared), and request_stop each one mid-flight from siblings.
let mut handles = Vec::new();
for _ in 0..200 {
let h = smarm::spawn(|| {
for _ in 0..50 {
let _v: Vec<u8> = Vec::with_capacity(64); // alloc → maybe_preempt
smarm::yield_now();
}
});
handles.push(h);
}
// Cancel half of them; the others run to completion. If any lock got
// poisoned the runtime would panic on a subsequent lock().unwrap().
for (i, h) in handles.iter().enumerate() {
if i % 2 == 0 {
smarm::request_stop(h.pid());
}
}
for h in handles {
let _ = h.join();
}
c.fetch_add(1, Ordering::SeqCst);
});
assert_eq!(completed.load(Ordering::SeqCst), 1, "root completed cleanly");
}
/// The sharper repro: a stop-flagged actor whose *next allocation* is the
/// `Box::new(closure)` inside `spawn`'s `with_shared` critical section.
/// Pre-fix, the ungated `check_cancelled` in `maybe_preempt` raises the
/// sentinel right there, unwinding while the global shared mutex is held →
/// poisoned → every later `with_shared` panics on every scheduler thread and
/// the whole runtime collapses. Post-fix, observation is deferred to the next
/// lock-free point and each actor dies a clean `Outcome::Stopped`.
///
/// `request_stop(self_pid())` sets the flag without parking (we're running),
/// and the capturing closure makes `Box::new(f)` a real allocation. With
/// `alloc_interval(1)` the check fires every other allocation, so each actor
/// has ~50% odds of the check landing on the under-lock alloc; 64 actors make
/// a pre-fix escape astronomically unlikely.
#[test]
fn self_stop_during_spawn_does_not_poison_shared_mutex() {
let rt = init(Config::exact(4).alloc_interval(1));
let completed = Arc::new(AtomicUsize::new(0));
let c = completed.clone();
rt.run(move || {
let mut handles = Vec::new();
for i in 0..64usize {
let h = smarm::spawn(move || {
// Vary pre-stop allocation count to cover both phases of the
// every-other-allocation check cadence.
for _ in 0..(i % 2) {
let _phase: Vec<u8> = Vec::with_capacity(8);
}
let payload = vec![0u8; 64]; // captured → Box::new(f) allocates
smarm::request_stop(smarm::self_pid());
let grandchild = smarm::spawn(move || drop(payload));
let _ = grandchild.join();
});
handles.push(h);
}
for h in handles {
// Stopped reports Ok from join; the point is that join itself
// (with_shared) doesn't panic on a poisoned mutex.
let _ = h.join();
}
c.fetch_add(1, Ordering::SeqCst);
});
assert_eq!(completed.load(Ordering::SeqCst), 1, "root completed cleanly");
}
+221
View File
@@ -0,0 +1,221 @@
//! Named registry tests. Run under the scheduler: registration requires a
//! live runtime and live actors.
use smarm::{channel, name_of, register, run, spawn, unregister, whereis, RegisterError};
#[test]
fn register_whereis_roundtrip() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("worker", h.pid()).unwrap();
assert_eq!(whereis("worker"), Some(h.pid()));
assert_eq!(name_of(h.pid()).as_deref(), Some("worker"));
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn register_is_idempotent_for_same_binding() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("svc", h.pid()).unwrap();
// Same name, same pid: a no-op Ok, not NameTaken.
register("svc", h.pid()).unwrap();
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn duplicate_name_on_live_holder_is_rejected() {
run(|| {
let (tx, rx) = channel::<()>();
let (tx2, rx2) = channel::<()>();
let a = spawn(move || {
rx.recv().unwrap();
});
let b = spawn(move || {
rx2.recv().unwrap();
});
register("svc", a.pid()).unwrap();
assert_eq!(
register("svc", b.pid()),
Err(RegisterError::NameTaken { holder: a.pid() })
);
tx.send(()).unwrap();
tx2.send(()).unwrap();
a.join().unwrap();
b.join().unwrap();
});
}
#[test]
fn one_name_per_pid() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("first", h.pid()).unwrap();
assert_eq!(
register("second", h.pid()),
Err(RegisterError::PidAlreadyRegistered { name: "first".into() })
);
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn registering_a_dead_pid_is_noproc() {
run(|| {
let h = spawn(|| {});
let pid = h.pid();
h.join().unwrap();
assert_eq!(register("ghost", pid), Err(RegisterError::NoProc));
});
}
#[test]
fn binding_evaporates_on_death_and_name_is_reusable() {
run(|| {
let a = spawn(|| {});
let pid_a = a.pid();
register("svc", pid_a).unwrap();
a.join().unwrap();
// Dead holder: both lookup directions report unbound.
assert_eq!(whereis("svc"), None);
assert_eq!(name_of(pid_a), None);
// And the name is free for a successor.
let (tx, rx) = channel::<()>();
let b = spawn(move || {
rx.recv().unwrap();
});
register("svc", b.pid()).unwrap();
assert_eq!(whereis("svc"), Some(b.pid()));
tx.send(()).unwrap();
b.join().unwrap();
});
}
#[test]
fn register_over_a_dead_holder_succeeds_without_lookup_in_between() {
// The eviction path inside register() itself (not via whereis pruning).
run(|| {
let a = spawn(|| {});
let pid_a = a.pid();
register("svc", pid_a).unwrap();
a.join().unwrap();
let (tx, rx) = channel::<()>();
let b = spawn(move || {
rx.recv().unwrap();
});
register("svc", b.pid()).unwrap();
assert_eq!(whereis("svc"), Some(b.pid()));
assert_eq!(name_of(pid_a), None);
tx.send(()).unwrap();
b.join().unwrap();
});
}
#[test]
fn unregister_frees_both_directions() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("svc", h.pid()).unwrap();
assert_eq!(unregister("svc"), Some(h.pid()));
assert_eq!(whereis("svc"), None);
assert_eq!(name_of(h.pid()), None);
assert_eq!(unregister("svc"), None);
// The pid may take a new name afterwards.
register("svc2", h.pid()).unwrap();
assert_eq!(name_of(h.pid()).as_deref(), Some("svc2"));
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn whereis_from_another_actor_and_usable_with_runtime_apis() {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
let stopped = Arc::new(AtomicBool::new(false));
let stopped2 = stopped.clone();
run(move || {
let (tx, rx) = channel::<()>();
let svc = spawn(move || {
// Parks forever; only a request_stop ends it.
let _ = rx.recv();
});
register("stoppable", svc.pid()).unwrap();
let stopped3 = stopped2.clone();
let client = spawn(move || {
let pid = whereis("stoppable").expect("name must resolve cross-actor");
// The registry's pids plug into the rest of the runtime API.
smarm::request_stop(pid);
stopped3.store(true, Ordering::Relaxed);
});
client.join().unwrap();
svc.join().unwrap(); // a stopped actor joins Ok
drop(tx);
});
assert!(stopped.load(Ordering::Relaxed));
}
#[test]
fn registry_under_concurrent_churn_multi_thread() {
// Many actors racing to claim the same names while holders die: the
// bimap invariant (debug-asserted internally) and Erlang semantics must
// hold under real parallelism.
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;
let wins = Arc::new(AtomicU32::new(0));
let wins2 = wins.clone();
smarm::init(smarm::Config::exact(4)).run(move || {
let mut handles = Vec::new();
for round in 0..8 {
let name = format!("contested-{}", round % 2);
for _ in 0..8 {
let name = name.clone();
let wins = wins2.clone();
handles.push(spawn(move || {
let me = smarm::self_pid();
match register(&name, me) {
Ok(()) => {
wins.fetch_add(1, Ordering::Relaxed);
smarm::yield_now();
// May have been pruned-by-contact never; we are
// alive, so our binding must still resolve to us.
assert_eq!(whereis(&name), Some(me));
assert_eq!(unregister(&name), Some(me));
}
Err(RegisterError::NameTaken { .. }) => {}
Err(e) => panic!("unexpected register error: {e}"),
}
}));
}
}
for h in handles {
h.join().unwrap();
}
});
// At least one registration per name must have succeeded.
assert!(wins.load(Ordering::Relaxed) >= 2);
}
+340
View File
@@ -0,0 +1,340 @@
//! select tests. Beyond the functional cases, the *_stays_precise tests are
//! soundness probes for the consuming-wake protocol: they fire stale loser-
//! arm wakes at an actor and then run a one-shot park (`sleep`, whose early
//! return would be observable as a short elapsed time) to prove the stale
//! wake died at its epoch CAS instead of corrupting the next wait.
use smarm::{channel, run, select, spawn};
use std::sync::atomic::{AtomicI64, Ordering};
use std::time::{Duration, Instant};
static OUT: AtomicI64 = AtomicI64::new(0);
#[test]
fn ready_arm_returns_immediately_without_parking() {
OUT.store(0, Ordering::SeqCst);
run(|| {
let (txa, rxa) = channel::<i64>();
let (_txb, rxb) = channel::<i64>();
txa.send(42).unwrap();
let i = select(&[&rxb, &rxa]);
assert_eq!(i, 1);
OUT.store(rxa.try_recv().unwrap().expect("ready arm must hold a message"), Ordering::SeqCst);
});
assert_eq!(OUT.load(Ordering::SeqCst), 42);
}
#[test]
fn lower_index_wins_when_several_arms_are_ready() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
txa.send(1).unwrap();
txb.send(2).unwrap();
// Documented priority order: index 0 first, regardless of send order.
assert_eq!(select(&[&rxa, &rxb]), 0);
assert_eq!(select(&[&rxb, &rxa]), 0);
});
}
#[test]
fn parks_until_any_arm_fires() {
OUT.store(0, Ordering::SeqCst);
run(|| {
let (txa, _rxa_keepalive) = (channel::<i64>().0, ());
let _hold = txa; // arm a: sender alive, never sends
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let _keep_a_open = txa;
let h = spawn(move || {
smarm::sleep(Duration::from_millis(10));
txb.send(7).unwrap();
});
let i = select(&[&rxa, &rxb]);
assert_eq!(i, 1);
OUT.store(rxb.try_recv().unwrap().unwrap(), Ordering::SeqCst);
h.join().unwrap();
});
assert_eq!(OUT.load(Ordering::SeqCst), 7);
}
#[test]
fn closed_arm_counts_as_ready() {
run(|| {
let (_keep, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
drop(txb);
let i = select(&[&rxa, &rxb]);
assert_eq!(i, 1);
// The caller observes the closure on the arm itself.
assert!(rxb.try_recv().is_err());
});
}
#[test]
fn closure_while_parked_wakes_the_select() {
run(|| {
let (_keep, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(10));
drop(txb);
});
assert_eq!(select(&[&rxa, &rxb]), 1);
assert!(rxb.try_recv().is_err());
h.join().unwrap();
});
}
#[test]
fn loser_arm_wake_after_parked_select_stays_precise() {
// Both arms registered; arm 0 wins the wake; arm 1's sender then fires
// a wake stamped with the consumed epoch. The subsequent sleep is a
// one-shot park: an early return would mean the stale wake landed.
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(10));
txa.send(1).unwrap(); // wins
txb.send(2).unwrap(); // loser arm: stale-epoch wake
});
assert_eq!(select(&[&rxa, &rxb]), 0);
assert_eq!(rxa.try_recv().unwrap(), Some(1));
let t0 = Instant::now();
smarm::sleep(Duration::from_millis(40));
assert!(
t0.elapsed() >= Duration::from_millis(40),
"one-shot park returned early: a stale loser-arm wake landed"
);
// The loser's message was never lost.
assert_eq!(select(&[&rxa, &rxb]), 1);
assert_eq!(rxb.try_recv().unwrap(), Some(2));
h.join().unwrap();
});
}
#[test]
fn no_park_exit_retires_the_wait() {
// Arm 1 is ready at registration time, so select returns WITHOUT
// parking while arm 0 holds a live-epoch registration. Arm 0's sender
// then fires. retire_wait must have invalidated/eaten that wake;
// the sleep proves it.
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
txb.send(2).unwrap();
assert_eq!(select(&[&rxa, &rxb]), 1);
assert_eq!(rxb.try_recv().unwrap(), Some(2));
let h = spawn(move || {
txa.send(1).unwrap(); // fires at the (retired) select epoch
});
h.join().unwrap();
let t0 = Instant::now();
smarm::sleep(Duration::from_millis(40));
assert!(
t0.elapsed() >= Duration::from_millis(40),
"one-shot park returned early: the no-park exit leaked a wake"
);
assert_eq!(rxa.try_recv().unwrap(), Some(1));
});
}
#[test]
fn select_then_plain_recv_on_a_loser_arm() {
// A leftover own-registration from a select must not trip the next
// direct wait on that channel (it is overwritten, not asserted away).
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
txa.send(1).unwrap();
assert_eq!(select(&[&rxa, &rxb]), 0); // rxb may keep a registration
assert_eq!(rxa.try_recv().unwrap(), Some(1));
let h = spawn(move || {
smarm::sleep(Duration::from_millis(5));
txb.send(9).unwrap();
});
assert_eq!(rxb.recv().unwrap(), 9);
h.join().unwrap();
});
}
#[test]
fn select_loop_drains_two_producers_completely() {
const N: i64 = 200;
OUT.store(0, Ordering::SeqCst);
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let ha = spawn(move || {
for v in 0..N {
txa.send(v).unwrap();
if v % 7 == 0 {
smarm::yield_now();
}
}
});
let hb = spawn(move || {
for v in 0..N {
txb.send(v).unwrap();
if v % 5 == 0 {
smarm::yield_now();
}
}
});
// A closed arm stays permanently ready, so it must leave the arm
// set once observed (keeping it would starve the other arm at
// priority order) — select until the FIRST closure, then drain the
// survivor with plain recv.
let mut sum = 0i64;
let survivor: &smarm::Receiver<i64> = loop {
let i = select(&[&rxa, &rxb]);
let arm: &smarm::Receiver<i64> = if i == 0 { &rxa } else { &rxb };
match arm.try_recv() {
Ok(Some(v)) => sum += v,
Ok(None) => {} // defensive: ready arm already drained
Err(_) => break if i == 0 { &rxb } else { &rxa },
}
};
loop {
match survivor.recv() {
Ok(v) => sum += v,
Err(_) => break,
}
}
OUT.store(sum, Ordering::SeqCst);
ha.join().unwrap();
hb.join().unwrap();
});
assert_eq!(OUT.load(Ordering::SeqCst), 2 * (0..200i64).sum::<i64>());
}
#[test]
fn motivating_pattern_inbox_plus_monitor_down() {
// The gen_server/handle_info shape select unlocks: a server waiting on
// its request inbox AND a monitor Down channel in one park.
use smarm::monitor;
run(|| {
let (req_tx, req_rx) = channel::<i64>();
let worker = spawn(|| {
smarm::sleep(Duration::from_millis(10));
// worker exits -> Down fires
});
let m = monitor(worker.pid());
let _keep_inbox_open = req_tx;
let mut served = 0;
loop {
match select(&[&m.rx, &req_rx]) {
0 => {
let _down = m.rx.try_recv().unwrap().expect("Down message");
break;
}
_ => {
if let Ok(Some(_)) = req_rx.try_recv() {
served += 1;
}
}
}
}
assert_eq!(served, 0);
worker.join().unwrap();
});
}
// ---------------------------------------------------------------------------
// select_timeout
// ---------------------------------------------------------------------------
use smarm::select_timeout;
#[test]
fn select_timeout_returns_none_after_the_deadline() {
run(|| {
let (_keep_a, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
let t0 = Instant::now();
let r = select_timeout(&[&rxa, &rxb], Duration::from_millis(30));
assert_eq!(r, None);
assert!(t0.elapsed() >= Duration::from_millis(30));
});
}
#[test]
fn select_timeout_ready_arm_wins_without_arming_a_timer() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
txa.send(5).unwrap();
assert_eq!(select_timeout(&[&rxb, &rxa], Duration::from_millis(500)), Some(1));
assert_eq!(rxa.try_recv().unwrap(), Some(5));
});
}
#[test]
fn select_timeout_arm_beats_timer_and_stale_entry_stays_inert() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(5));
txa.send(1).unwrap();
});
let t0 = Instant::now();
let r = select_timeout(&[&rxa, &rxb], Duration::from_millis(200));
assert_eq!(r, Some(0));
assert!(t0.elapsed() < Duration::from_millis(200));
assert_eq!(rxa.try_recv().unwrap(), Some(1));
h.join().unwrap();
// The abandoned timer entry expires mid-sleep; a stale-epoch wake
// landing would return this one-shot park early.
let t1 = Instant::now();
smarm::sleep(Duration::from_millis(250));
assert!(
t1.elapsed() >= Duration::from_millis(250),
"one-shot park returned early: the stale select timer landed"
);
});
}
#[test]
fn select_timeout_timer_first_message_still_delivered_later() {
run(|| {
let (txa, rxa) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(60));
txa.send(9).unwrap();
});
assert_eq!(select_timeout(&[&rxa], Duration::from_millis(10)), None);
// The wait is over but the channel is intact: the late message
// arrives and a fresh select sees it.
assert_eq!(select(&[&rxa]), 0);
assert_eq!(rxa.try_recv().unwrap(), Some(9));
h.join().unwrap();
});
}
#[test]
fn select_timeout_zero_duration_polls() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::ZERO), None);
txa.send(3).unwrap();
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::ZERO), Some(0));
assert_eq!(rxa.try_recv().unwrap(), Some(3));
});
}
#[test]
fn select_timeout_closed_arm_is_ready_not_a_timeout() {
run(|| {
let (_keep_a, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
drop(txb);
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::from_millis(200)), Some(1));
assert!(rxb.try_recv().is_err());
});
}
+115
View File
@@ -0,0 +1,115 @@
//! Terminal wake: a scheduler thread blocked in the idle wait must be woken
//! when a sibling reaches the all-done verdict, or `rt.run` stalls in its
//! worker join.
//!
//! Mechanism (runtime.rs, `Pop::Idle`): an idle scheduler snapshots
//! `peek_deadline()` / `io_outstanding` and blocks in `poll_wake` (or
//! `thread::sleep`) on that snapshot. `enqueue` does not write the wake
//! pipe — only IO completions do — so the snapshot can go terminally stale:
//!
//! - An actor parked in `wait_readable` that is `request_stop`ped produces
//! NO completion (cancellation deregisters the waiter); a sibling that
//! blocked on `io_outstanding > 0` with no timers is in `poll(-1)` forever.
//! - An actor cancelled out of a long `sleep` leaves its timer entry
//! orphaned; a sibling that blocked on that deadline sleeps it out in full.
//!
//! In both cases the remaining work completes on the *other* scheduler
//! thread, which hits AllDone and returns — and nothing wakes the blocked
//! one. `Runtime::run` joins it: a permanent hang in the first case, a
//! full-deadline stall in the second.
//!
//! Both tests force the window deterministically: the root busy-spins
//! (creating no timer entries and occupying one scheduler thread) so the
//! other thread settles into the stale idle wait before the stop is issued.
use std::sync::mpsc;
use std::time::{Duration, Instant};
struct PipePair {
read: libc::c_int,
write: libc::c_int,
}
impl PipePair {
fn new() -> Self {
let mut fds: [libc::c_int; 2] = [0; 2];
let r = unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_CLOEXEC | libc::O_NONBLOCK) };
assert_eq!(r, 0, "pipe2 failed");
PipePair { read: fds[0], write: fds[1] }
}
}
impl Drop for PipePair {
fn drop(&mut self) {
unsafe {
libc::close(self.read);
libc::close(self.write);
}
}
}
/// Occupy the current scheduler thread without creating timer entries or
/// parking. A plain spin has no observation points, so the root stays
/// on-CPU and the sibling scheduler is left alone with the idle branch.
fn spin_for(d: Duration) {
let t0 = Instant::now();
while t0.elapsed() < d {
std::hint::spin_loop();
}
}
/// Run `body` under a 2-scheduler runtime on a watchdog thread; fail if
/// `Runtime::run` has not returned within `limit`.
fn run_with_watchdog(limit: Duration, body: impl FnOnce() + Send + 'static) {
let (done_tx, done_rx) = mpsc::channel::<()>();
std::thread::spawn(move || {
let rt = smarm::init(smarm::Config::exact(2));
rt.run(body);
let _ = done_tx.send(());
});
done_rx
.recv_timeout(limit)
.expect("Runtime::run did not return: idle scheduler thread was never woken at termination");
}
/// Permanent-hang variant: sibling blocked in `poll_wake(wake_fd, None)`
/// because `io_outstanding > 0` (one actor parked in `wait_readable`) and no
/// timers are pending. The waiter is then stop-cancelled — no IO completion
/// ever writes the wake pipe — and everything else finishes on the root's
/// thread. Without a terminal wake, `rt.run` never returns.
#[test]
fn run_returns_after_io_waiter_is_stop_cancelled() {
run_with_watchdog(Duration::from_secs(10), || {
let pipe = PipePair::new();
let rfd = pipe.read;
let h = smarm::spawn(move || {
// Never-readable fd (write end open, nothing written).
let _ = smarm::wait_readable(rfd);
});
// Let the waiter park and the sibling scheduler settle into the
// io_outstanding>0 / no-timers idle wait: poll(wake_fd, -1).
spin_for(Duration::from_millis(200));
smarm::request_stop(h.pid());
let _ = h.join();
drop(pipe);
});
}
/// Finite-stall variant: sibling blocked on an orphaned long timer
/// deadline. An actor cancelled out of `sleep(60s)` leaves its timer entry
/// behind (documented as harmless at AllDone — but a sibling that already
/// blocked on that deadline sleeps it out in full, stalling `rt.run` for
/// the better part of a minute).
#[test]
fn run_returns_after_long_sleeper_is_stop_cancelled() {
run_with_watchdog(Duration::from_secs(10), || {
let h = smarm::spawn(|| {
smarm::sleep(Duration::from_secs(60));
});
// Let the sleeper park and the sibling scheduler block on the 60s
// deadline: poll(wake_fd, ~60_000ms).
spin_for(Duration::from_millis(200));
smarm::request_stop(h.pid());
let _ = h.join();
});
}
+15 -15
View File
@@ -124,11 +124,11 @@ use smarm::pid::Pid;
use smarm::timer::{Reason, TimerTarget, Timers};
struct RecordingTarget {
calls: Mutex<Vec<(Pid, u64)>>,
calls: Mutex<Vec<(Pid, u32)>>,
}
impl TimerTarget for RecordingTarget {
fn on_timeout(&self, pid: Pid, seq: u64) {
self.calls.lock().unwrap().push((pid, seq));
fn on_timeout(&self, pid: Pid, epoch: u32) {
self.calls.lock().unwrap().push((pid, epoch));
}
}
@@ -137,9 +137,9 @@ fn timers_pop_due_returns_entries_in_deadline_order() {
let mut t = Timers::new();
let now = Instant::now();
// Insert out of order; pop_due should hand them back sorted by deadline.
t.insert_sleep(now + Duration::from_millis(30), Pid::new(0, 0));
t.insert_sleep(now + Duration::from_millis(10), Pid::new(1, 0));
t.insert_sleep(now + Duration::from_millis(20), Pid::new(2, 0));
t.insert_sleep(now + Duration::from_millis(30), Pid::new(0, 0), 1);
t.insert_sleep(now + Duration::from_millis(10), Pid::new(1, 0), 1);
t.insert_sleep(now + Duration::from_millis(20), Pid::new(2, 0), 1);
// Advance past all of them.
let due = t.pop_due(now + Duration::from_millis(50));
@@ -152,8 +152,8 @@ fn timers_pop_due_returns_entries_in_deadline_order() {
fn timers_only_pop_entries_whose_deadline_has_passed() {
let mut t = Timers::new();
let now = Instant::now();
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0));
t.insert_sleep(now + Duration::from_millis(100), Pid::new(1, 0));
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0), 1);
t.insert_sleep(now + Duration::from_millis(100), Pid::new(1, 0), 1);
let due = t.pop_due(now + Duration::from_millis(20));
assert_eq!(due.len(), 1);
@@ -169,11 +169,11 @@ fn timers_mix_sleep_and_wait_timeout_reasons() {
let target = Arc::new(RecordingTarget { calls: Mutex::new(Vec::new()) });
let now = Instant::now();
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0));
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0), 1);
t.insert(
now + Duration::from_millis(10),
Pid::new(1, 0),
Reason::WaitTimeout { target: target.clone(), wait_seq: 42 },
Reason::WaitTimeout { target: target.clone(), epoch: 42 },
);
let due = t.pop_due(now + Duration::from_millis(20));
@@ -181,11 +181,11 @@ fn timers_mix_sleep_and_wait_timeout_reasons() {
// Order: Sleep (5ms) first, WaitTimeout (10ms) second.
match &due[0].reason {
Reason::Sleep => {}
Reason::Sleep { .. } => {}
_ => panic!("first entry should be a Sleep"),
}
match &due[1].reason {
Reason::WaitTimeout { wait_seq, .. } => assert_eq!(*wait_seq, 42),
Reason::WaitTimeout { epoch, .. } => assert_eq!(*epoch, 42),
_ => panic!("second entry should be a WaitTimeout"),
}
}
@@ -197,9 +197,9 @@ fn same_deadline_entries_pop_in_insertion_order() {
let mut t = Timers::new();
let now = Instant::now();
let d = now + Duration::from_millis(10);
t.insert_sleep(d, Pid::new(0, 0));
t.insert_sleep(d, Pid::new(1, 0));
t.insert_sleep(d, Pid::new(2, 0));
t.insert_sleep(d, Pid::new(0, 0), 1);
t.insert_sleep(d, Pid::new(1, 0), 1);
t.insert_sleep(d, Pid::new(2, 0), 1);
let due = t.pop_due(now + Duration::from_millis(20));
let pids: Vec<u32> = due.iter().map(|e| e.pid.index()).collect();
+176
View File
@@ -0,0 +1,176 @@
//! RFC 005 wake-slot tests: correctness with the slot on, push-policy
//! discrimination (actor vs scheduler context, spawns), displacement, the
//! default-off contract, and per-run counter reset.
//!
//! The slot is same-thread plain ops behind the existing queue-op contract,
//! so there is nothing new to model-check (RFC 005 §Summary); these tests
//! pin the *policy* — who lands in the slot, who never does — which is
//! observable through the `slot_hits` / `slot_displacements` counters.
use smarm::channel::channel;
use smarm::runtime::{init, Config};
use smarm::spawn;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
/// Ping-pong over channels: the slot's home pattern. Returns total roundtrips.
fn ping_pong(pairs: usize, roundtrips: u64) -> impl FnOnce() + Send + 'static {
move || {
let handles: Vec<_> = (0..pairs)
.map(|_| {
spawn(move || {
let (tx_ab, rx_ab) = channel::<u64>();
let (tx_ba, rx_ba) = channel::<u64>();
let echo = spawn(move || {
for _ in 0..roundtrips {
let v = rx_ab.recv().expect("echo recv");
tx_ba.send(v + 1).expect("echo send");
}
});
for i in 0..roundtrips {
tx_ab.send(i).expect("ping send");
assert_eq!(rx_ba.recv().expect("ping recv"), i + 1);
}
let _ = echo.join();
})
})
.collect();
for h in handles {
h.join().expect("pair");
}
}
}
// ---------------------------------------------------------------------------
// Correctness: messaging workloads complete with the slot on, 1 and N threads
// ---------------------------------------------------------------------------
#[test]
fn ping_pong_correct_with_slot_on_single_thread() {
let rt = init(Config::exact(1).wake_slot(true));
rt.run(ping_pong(4, 200));
}
#[test]
fn ping_pong_correct_with_slot_on_multi_thread() {
let rt = init(Config::exact(4).wake_slot(true));
rt.run(ping_pong(8, 200));
}
// ---------------------------------------------------------------------------
// Push policy: actor-context wakes hit the slot; the default is off
// ---------------------------------------------------------------------------
#[test]
fn messaging_workload_exercises_the_slot() {
let rt = init(Config::exact(1).wake_slot(true));
rt.run(ping_pong(2, 100));
let stats = rt.stats();
assert!(
stats.slot_hits() > 0,
"send-wakes are actor-context unparks — the slot must see hits, got 0"
);
}
#[test]
fn slot_off_means_zero_slot_traffic() {
// Off explicitly…
let rt = init(Config::exact(1).wake_slot(false));
rt.run(ping_pong(2, 100));
assert_eq!(rt.stats().slot_hits(), 0);
assert_eq!(rt.stats().slot_displacements(), 0);
// …and off by default (RFC 005: default off until the shootout accepts).
let rt = init(Config::exact(1));
rt.run(ping_pong(2, 100));
assert_eq!(rt.stats().slot_hits(), 0, "wake_slot must default to OFF");
}
// ---------------------------------------------------------------------------
// Push policy: spawns and join-wakes (finalize = scheduler context) bypass
// ---------------------------------------------------------------------------
#[test]
fn spawn_join_workload_bypasses_the_slot() {
let rt = init(Config::exact(1).wake_slot(true));
rt.run(|| {
for _ in 0..20 {
let handles: Vec<_> = (0..50).map(|_| spawn(|| {})).collect();
for h in handles {
h.join().expect("trivial actor");
}
}
});
let stats = rt.stats();
assert_eq!(
stats.slot_hits(),
0,
"spawns go shared by policy and joiner wakes fire from finalize \
(scheduler context) — a pure spawn/join workload must never touch \
the slot"
);
assert_eq!(stats.slot_displacements(), 0);
}
// ---------------------------------------------------------------------------
// Displacement: newest wake takes the slot, occupant goes shared — and runs
// ---------------------------------------------------------------------------
#[test]
fn displaced_occupant_reaches_the_shared_queue_and_runs() {
// Single thread, strict FIFO (rq-mutex default): rx1 and rx2 park on
// their recvs before the sender runs; the sender's back-to-back sends
// then produce two actor-context wakes — the second displaces the first.
let rt = init(Config::exact(1).wake_slot(true));
let ran = Arc::new(AtomicU64::new(0));
let (r1, r2) = (ran.clone(), ran.clone());
rt.run(move || {
let (tx1, rx1) = channel::<u32>();
let (tx2, rx2) = channel::<u32>();
let h1 = spawn(move || {
assert_eq!(rx1.recv().expect("rx1"), 1);
r1.fetch_add(1, Ordering::Relaxed);
});
let h2 = spawn(move || {
assert_eq!(rx2.recv().expect("rx2"), 2);
r2.fetch_add(1, Ordering::Relaxed);
});
let sender = spawn(move || {
tx1.send(1).expect("send 1"); // rx1 → slot
tx2.send(2).expect("send 2"); // rx2 → slot, rx1 displaced → shared
});
sender.join().expect("sender");
h1.join().expect("h1");
h2.join().expect("h2");
});
assert_eq!(ran.load(Ordering::Relaxed), 2, "both receivers must run");
let stats = rt.stats();
assert!(
stats.slot_displacements() >= 1,
"back-to-back wakes of parked receivers must displace at least once"
);
assert!(stats.slot_hits() >= 1, "the displacing wake is slot-popped");
}
// ---------------------------------------------------------------------------
// Counters reset at the start of each run() on a reused Runtime
// ---------------------------------------------------------------------------
#[test]
fn slot_counters_reset_per_run() {
let rt = init(Config::exact(1).wake_slot(true));
rt.run(ping_pong(2, 100));
let first = rt.stats().slot_hits();
assert!(first > 0);
// A slot-bypassing run on the same handle must read 0, not `first`.
rt.run(|| {
let h = spawn(|| {});
h.join().expect("trivial");
});
assert_eq!(
rt.stats().slot_hits(),
0,
"counters are reset at run() start; the second run had no slot traffic"
);
}