Markk116/smarm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: Markk116/smarm:7a244d057c6f68379cee7528df82bb99df102026
Branches Tags
Markk116/smarm:master Markk116/smarm:arm-port
..
compare: Markk116/smarm:arm-port
Branches Tags
Markk116/smarm:master Markk116/smarm:arm-port
1 Commits
7a244d057c .. arm-port
Author SHA1 Message Date
smarm-dev 89fb13e29a feat(arch): port context switch + cycle counter to aarch64 
Extract the x86-64-specific context-switch shims, initial-stack layout,
and cycle counter out of context.rs into a target_arch-gated src/arch/
module, and add an aarch64 (AAPCS64) backend.

- src/arch/mod.rs: shared SCHEDULER_SP/ACTOR_SP thread-locals + the
  four-item backend contract (init_actor_stack, switch_to_{actor,
  scheduler}, read_cycle_counter); compile_error! on other arches.
- src/arch/x86_64.rs: existing implementation verbatim, plus the rdtsc
  cycle counter relocated here from preempt.rs.
- src/arch/aarch64.rs: x19-x30 stp/ldp context switch, lr-based return,
  CNTVCT_EL0 cycle counter (isb-serialised).
- context.rs: now a thin facade re-exporting crate::arch; public surface
  unchanged for scheduler/preempt/tests.
- preempt.rs: read_cycle_counter delegates to crate::arch; per-arch
  DEFAULT_TIMESLICE_CYCLES (TSC and CNTVCT have different units).
- tests/context.rs: cfg-gate the x86 callee-saved-register probe and add
  the aarch64 x23-x26 sibling.
- .cargo/config.toml: aarch64-unknown-linux-gnu cross linker.

No functional change on x86-64: arch/x86_64.rs is byte-for-byte the old
context.rs body, and tests/preempt.rs (incl. the XMM-not-saved guard from
14dc7a7) is untouched.
 2026-06-07 21:54:28 +00:00
42 changed files with 1857 additions and 6598 deletions
Expand all files Collapse all files
.cargo/config.toml
+2
View File
@@ -0,0 +1,2 @@
[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
.gitignore
-1
View File
@@ -1,4 +1,3 @@
target
Cargo.lock
smarm_trace.json
/bench_results/
Cargo.toml
-20
View File
@@ -4,24 +4,12 @@ version = "0.4.0"
edition = "2021"
rust-version = "1.95"
[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ["cfg(loom)"] }
[features]
default = ["rq-mutex"]
smarm-trace = []
# Run-queue selection: exactly one, compile-time (see src/run_queue.rs).
# Non-default variants need --no-default-features (features are additive).
rq-mutex = []
rq-mpmc = []
rq-striped = []
[dependencies]
libc = "0.2"
[target.'cfg(loom)'.dependencies]
loom = "0.7"
[dev-dependencies]
libc = "0.2"
tokio = { version = "1", features = ["rt", "rt-multi-thread", "macros", "sync", "time"] }
@@ -53,11 +41,3 @@ harness = false
[[bench]]
name = "tokio_favored"
harness = false
[[bench]]
name = "rq_micro"
harness = false
[[bench]]
name = "rq_runtime"
harness = false
README.md
+2 -3
View File
@@ -21,15 +21,14 @@ convenience wrapper around `runtime::init(Config::exact(1)).run(f)`.
| `pid` | `(index, generation)` PIDs; stale handles are detectable, not silent |
| `actor` | Trampoline + `catch_unwind` boundary at the actor entry point |
| `scheduler` | Run queue, slot table, spawn/join, parking, idle path |
| `channel` | Unbounded MPSC channel; `recv` parks the actor; `recv_timeout` bounds it; `select`/`select_timeout` park on many receivers at once (ready-index, priority order) |
| `channel` | Unbounded MPSC channel; `recv` parks the actor |
| `mutex` | `Mutex<T>` with mandatory timeout; FIFO waiters; parks the green thread |
| `timer` | Min-heap of `(deadline, reason)`; `Sleep` and `WaitTimeout` reasons |
| `io` | `block_on_io` for blocking work; `wait_readable`/`wait_writable` + `read`/`write` via epoll |
| `supervisor` | `Signal::Exit`/`Panic`/`Stopped` funnelled to a parent; `OneForOne`/`OneForAll`/`RestForOne` strategies + restart-intensity cap |
| `monitor` | `monitor(pid)` → `Monitor { id, target, rx }`; one-shot `Down` via `rx`; `demonitor(&m)` tears one registration down; unidirectional death notice |
| `link` | bidirectional `link`/`unlink`; abnormal death propagates (cooperative stop, or an `ExitSignal` message under `trap_exit`) |
| `gen_server` | `call`/`call_timeout` (sync request-reply) / `cast` (async) over one inbox; `handle_info` over static info arms + `handle_down` via `Watcher`-fed monitors, selected ahead of the inbox; `ServerRef`/`ServerBuilder` + `init`/`terminate` hooks; server-down via channel closure |
| `registry` | `register`/`whereis`/`name_of`: name ↔ pid bimap; lazy generation-checked cleanup |
| `gen_server` | `call` (sync request-reply) / `cast` (async) over one inbox; `ServerRef` + `init`/`terminate` hooks; server-down via channel closure |
## Quick taste
ROADMAP.md
-376
View File
@@ -1,376 +0,0 @@
# smarm — Roadmap
## v0.4 — Actor primitives ✅ DONE
### 1. Cooperative cancellation ✅ (`a8ddb4a`)
Stop flag on `Actor` behind `Arc<AtomicBool>`. Observation points:
`maybe_preempt`/`check!()` + the wakeup side of `park_current`/`yield_now`.
Sentinel = `StopSentinel` (zero-size), caught in trampoline → `Outcome::Stopped`.
`join()` on a stopped actor returns `Ok(())`.
Known gap (by design): a tight no-alloc loop without `check!()` cannot be
stopped, same as preemption.
### 2. one_for_all / rest_for_one + ordered shutdown ✅ (`351dc9c`)
`Strategy::{OneForOne,OneForAll,RestForOne}` via `.strategy()`. Triggering
child's `Restart` policy decides whether anything restarts; strategy decides
which siblings are cycled. Survivors stopped in reverse start order, awaited on
the existing `supervisor_channel` funnel, restarted in start order.
`Signal::Stopped(pid)` + `DownReason::Stopped` added (distinct from Exit).
Also fixed a latent bug (`e80334b`): orphaned timers were gating shutdown.
`live == 0` is now the shutdown condition; timer heap cleared on exit.
### 3. Links + trap_exit ✅ (`6581484`)
`Slot.links: Vec<Pid>` (bidirectional). On finalize: abnormal death →
`request_stop(peer)` unless peer traps, in which case deliver `ExitSignal`
message. Normal exit does not propagate. `trap_exit()` returns a dedicated
`Receiver<ExitSignal>` (distinct from monitor `Down` channel).
### 4. Selective receive ✅ (`03f3875`)
`Receiver::recv_match(pred)` + `try_recv_match`. Scans `VecDeque` front-to-back,
removes+returns first match, parks and re-scans on new arrivals. `pred` is
`Fn(&T) -> bool` (not `FnMut`; runs under the channel lock — keep it cheap).
### 5. Grab-bag ✅
- **demonitor** (`5181037`): `monitor()` → `Monitor { id, target, rx }`.
`demonitor(&m) -> Option<MonitorId>`. Dropped `Sender` released post-lock
(non-reentrant shared mutex discipline).
- **gen_server** (`55221e9`): `GenServer` trait, `call`/`cast` over one
`Envelope` inbox. `terminate` hook via drop guard. No `handle_info`, no call
timeout (deferred — needs cross-channel mailbox merge or per-recv deadline).
- **Named registry**: `register`/`whereis`/`send_by_name` — **not yet done**.
HashMap in `SharedState`. No dependencies; can land any time.
- **arm-port branch**: aarch64 context-switch backend extracted into
`src/arch/`. Untested on hardware. Merge only after on-device validation.
---
## v0.5 — Runtime decomposition & run-path scaling ✅ DONE
Goal: dismantle the single `Mutex<SharedState>` along the run path so the
runtime scales to dozens of cores, with crash isolation hardened so a torn
write or a stray cancellation can never poison shared runtime state.
Guiding rules established this cycle:
- **Lock count that matters is hot-path locks.** Cold lifecycle paths may keep a
small per-slot lock; the run path targets zero locks.
- **Lock ordering: io-before-shared, slot locks are leaves** (never hold two
slot locks at once).
- **No unwind inside a runtime critical section.** Stop sentinel only fires at
lock-free observation points.
- **Never hold a thread-local guard across a possible switch point.** An actor
can be preempted at any allocation and resume on a different OS thread.
`with_runtime`, `with_shared`, and every `RawMutex` guard disable preemption
for this reason.
### Phase 1 — State decomposition ✅ (`3c7e26b`)
`next_monitor_id` → `AtomicU64`. `timers` → own `Mutex<Timers>`. `io` → own
`Mutex<Option<IoThread>>`. `pending_closures` → folded into `Slot::pending_closure`.
`check_cancelled()` gated behind `PREEMPTION_ENABLED` (poison fix).
`SharedState` now holds only: `slots`, `free_list`, `run_queue`, `root_pid`.
### Phase 2 — Slot table split ✅ (`5e0c9d4`)
Fixed slab `Box<[Slot]>`. Generation packed into state word: one
`AtomicU64 = (gen << 32) | state` — gen check is atomic with every transition.
Per-slot CAS state machine `Vacant/Queued/Running/RunningNotified/Parked/Done`.
Per-slot `RawMutex` for cold collections. Free list → `RawMutex<Vec<u32>>`.
`live_actors` atomic; termination = io_out==0 && queue empty && live==0.
### Phase 3 — Pluggable run queue ✅ (`1b3b618`)
Compile-time selected via feature flags. Variants:
- `rq-mutex` — `Mutex<VecDeque>`, baseline.
- `rq-mpmc` — hand-rolled Vyukov bounded MPMC ring.
- `rq-striped` — M Vyukov rings, fetch-add ticket distribution.
All variants compile in every build; feature only picks the alias. Bounded rings
are sound via slab cap + at-most-once-enqueued invariant.
Queue ops require preemption disabled (a producer suspended mid-publish stalls
every consumer behind its cell — livelock).
### Phase 4 — Bench harness ✅ (`6d9f369`)
`benches/rq_micro.rs`: raw structure sweep. `benches/rq_runtime.rs`:
yield-storm, ping-pong-pairs, spawn-storm. `scripts/bench_rq.sh`: rebuilds per
feature, aggregates to `bench_results/summary.csv`.
### Phase 5 — Safety hardening & model checking ✅ (`039703d`)
State machine extracted to `src/slot_state.rs` for loom checking. Loom on the
rings via `src/sync_shim.rs`. `NoPreempt`/no-unwind/TLS-guard audit complete.
Invariant-assertion sweep: every `StateWord` transition self-asserts its
precondition.
---
## v0.6 — Actor ergonomics ✅ DONE
### 1. Channel mutex migration ✅ (`8ff6cf4`)
`channel::Inner<T>` migrated from `std::sync::Mutex` to `RawMutex`, closing
the cross-thread pthread-release hole (guard held with preemption enabled →
timeslice switch → unlock from a different OS thread, UB) and removing poison.
The strict leaf rule did not survive contact: finalize clones the
supervisor/trap senders and `monitor()` clones the Down sender, all under a
cold lock, and those senders *live in the slot* — the nesting is structural.
The check generalized to two classes: **Leaf → Channel**, at most one of
each, both violation directions debug-asserted at the acquisition site.
Channel critical sections call only the lock-free unpark protocol, so the
order is acyclic. `recv_match`'s user predicate still runs under the channel
lock — cheap/pure/channel-free, as documented.
### 2. Named registry ✅ (`2c7cf0b`)
`register` / `whereis` / `unregister` + the inverse `name_of(pid)`: a
**bimap** (two HashMaps in exact inverse, debug-asserted) under one Leaf
`RawMutex` in `RuntimeInner` — the planned "HashMap in SharedState" predated
the v0.5 decomposition. Erlang semantics: one name per pid, one pid per
name, NameTaken only against a *live* holder. Cleanup is lazy: liveness
rides the generation-checked slot word (pids are never reused), dead
bindings behave as absent and are pruned on contact — no `finalize_actor`
hook, no Slot field, no reset-in-three-places exposure.
`send_by_name` was dropped from scope deliberately: smarm has no per-pid
mailbox, so there is nothing generic to send *to*. The registry yields a
`Pid` for monitor/link/request_stop and for routing user-owned channels.
### 3. gen_server: call timeout ✅ (`134ff52`, `90b7040`)
Landed as two primitives, *without* the sketched monitor machinery:
- `Receiver::recv_timeout(Duration)` — the per-`recv` deadline, on the
existing `timer::Reason::WaitTimeout`/`TimerTarget` machinery exactly as
`Mutex::lock_timeout` uses it (per-wait seq; stale heap entries no-op on
seq mismatch; message-first race resolution; `Disconnected` distinct from
`Timeout`).
- `ServerRef::call_timeout` = send + `recv_timeout` on the reply channel.
Server death is already observable there (reply sender drops as the server
unwinds), so monitor + demonitor-on-timeout had nothing left to guarantee:
no registration exists, so nothing can leak. Timed-out requests are still
handled; the late reply's send fails harmlessly (Erlang semantics).
(`handle_info` still deferred — needs cross-channel mailbox merge; see Later.)
---
## v0.7 — select, on epoch-stamped consuming wakes ✅ DONE
Goal: cross-channel wait (`select`) without surrendering the precise-wake
invariant. Decision record: the alternative ("wakes are hints", loop-and-
recheck at every park site) was evaluated and rejected — it distributes the
soundness obligation across every present and future park site and turns
every stale wake into a wasted reschedule. Instead the slot protocol itself
gains *wait identity*: a park-epoch packed into the spare bits of the slot
word, so that a stale unpark dies at one failed CAS in the waker.
Protocol (the consuming-wake rules):
- Word becomes `(gen << 32) | (epoch << 8) | state` — 24-bit epoch in the
previously unused bits; every transition still one CAS, gen check still
atomic with it.
- `begin_wait` (actor-side, before registering with any waker) bumps the
epoch and returns it; registrations carry `(pid, epoch)`.
- Every successful wake transition **consumes** the epoch (bumps it):
`Parked(e) → Queued(e+1)`, `Running(e) → RunningNotified(e+1)`. A second
unpark stamped with the same epoch can never land — at most one wake per
wait, by construction. This is what makes select's *loser* arms harmless
without a cancellation pass: their stale registrations fire one futile CAS
on next send and self-clean.
- `unpark_at(pid, epoch)` = epoch-matched wake. Bare `unpark(pid)` remains as
the wildcard and is reserved for **terminal** wakes (`request_stop`), which
never return control to the code that parked.
- One-shot park sites (`Mutex::lock_timeout`, `sleep`, `block_on_io`,
`wait_fd`) stay exactly as written: precise wakes are preserved, so a wake
still *means* something.
### 1. Slot-word epoch + consuming unparks ✅ (`4913835`)
`slot_state.rs`: repack, `begin_wait`, epoch-matched/consuming `unpark`,
epoch-preserving claim/yield/park transitions. Loom: existing theorems
re-proved on the new word + a new stale-epoch theorem (an unpark stamped with
a consumed epoch can never enqueue *or* notify). Plumbing only — no caller
stamps yet; behaviour identical.
### 2. Stamp every registration-based wake; retire per-primitive wait seqs ✅ (`400854a`)
The epoch IS the wait identity, so `channel::Inner.{cur_wait,next_wait_seq}`
and `mutex`'s per-wait `seq` are subsumed: registrations become
`(pid, epoch)`, `TimerTarget::on_timeout(pid, epoch)`, wakers call
`unpark_at`. Sleep timers, io completions (`Blocking` + `FdReady`) and join
waiters stamped likewise. End-state invariant, auditable in one sentence:
**the only wildcard wake is `request_stop`.**
### 3. `select` — ready-index wait over multiple receivers ✅ (`00128f3`)
`select(&[&dyn Selectable]) -> usize`: register self in each arm's
`parked_receiver` (sequentially — the park protocol makes cross-arm atomicity
unnecessary), park once, wake, scan, return the first ready index; caller
`try_recv`s it. Sound as ready-index *because channels are single-receiver*:
nobody can steal between ready and recv. A closed arm counts as ready (its
`try_recv` reports the closure). Arms are scanned in order — priority order,
documented, like BEAM receive clauses; no fairness promise. No cancel pass
(see protocol above). Drop-guard deregistration on the unwind path only, to
keep tests' single-parked-receiver asserts honest.
### 4. `select_timeout` ✅ (`a0a93b6`)
A `WaitTimeout` timer stamped with the select's epoch, whose target is a
stateless `unpark_at`. The woken selector classifies the wake from state
alone: some arm ready → that arm; none ready → with precise wakes the timer
is the only remaining stamped waker → `None`. No flag, no shared waiter
struct, no registration to leak. The timer entry expires as a stale-epoch
no-op when an arm wins, per the no-cancellation convention in `timer.rs`.
### 5. Docs + integration proof ✅ (this commit)
README module table, gotchas rewrite (the "No `select`" bullet becomes the
consuming-wake invariant), and an integration test of the motivating
pattern: a server selecting over its inbox and a monitor `Down` channel
(`tests/select.rs::motivating_pattern_inbox_plus_monitor_down`).
Unblocked but deliberately out of scope: `gen_server::handle_info` (now an
API-design question, not a runtime question — see Later).
### Deviations from the plan
- **The no-park exit needed a protocol piece the plan missed.** A `select`
returning an arm ready at *registration* time leaves earlier arms holding
live-epoch registrations — a sender could still set RunningNotified and
fault the actor's next one-shot park. Landed as `scheduler::retire_wait`
(bump the epoch, eat a landed notification via the new
`StateWord::clear_notify`, re-observe the stop flag — in that order) plus
the loom theorem `retire_eats_late_arm_notification`. The parked path
needed nothing: the winning wake's consume already retires the epoch.
- **No Drop guard.** The planned unwind-path deregistration was superseded:
the single-receiver debug_asserts relax to "none, or my own pid", so a
leftover own-registration (consumed epoch — inert by theorem) is simply
overwritten by the receiver's next wait on that channel. Less machinery,
and the asserts still catch the real misuse (two actors on one channel).
- **A closed arm is ready *forever*** — callers must drop it from the arm
set once observed, or priority order starves every higher-indexed arm
(documented on `select`; found the hard way by a livelocking test).
---
## v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅ DONE
Goal: spend the `select` primitive on the server loop — out-of-band messages
(`handle_info`) and monitor `Down`s (`handle_down`) dispatched alongside
calls and casts — and close the v0.2 fd-hygiene hole in `io.rs`.
Design decisions (settled up front):
- **Info channels are static.** `type Info: Send + 'static` on the trait; a
`Vec<Receiver<G::Info>>` handed over at start. Users enum their own Info.
Dynamic info subscription is deliberately out: revisit against a consumer.
- **Down forwarding is dynamic, because monitors are.** You monitor a worker
you just spawned inside a handler, so down arms cannot ride the static
list. Mechanism: the loop owns a control arm (`Receiver<Monitor>`);
`init` grows a `&ServerCtx` parameter (breaking, no-op default body) whose
clonable `Watcher` the state stores and calls `watch(monitor)` from any
handler thereafter. Handler signatures otherwise untouched.
- **Priority order: downs → control → infos → inbox.** System messages
preempt; a hot inbox cannot starve a death notice. Within each class,
declaration order (select's documented priority semantics).
- **Closed arms drop silently** (the closed-arm-is-ready-forever gotcha).
Closed inbox still means graceful shutdown, unchanged.
### 1. handle_info — static info arms in the server loop ✅ (`e5d1b3b`)
`type Info` + `handle_info` (no-op default) on `GenServer`; `ServerBuilder`
(`Server::new(state).with_info(rx).under(sup).start()`) so start variants
don't multiply; `start`/`start_under` stay as wrappers. Loop becomes
`select` over `[infos…, inbox]`, dispatching by arm index.
### 2. handle_down — ServerCtx::watcher + control arm ✅ (`24b95c9`)
`handle_down` (no-op default), `ServerCtx`/`Watcher`, the control arm, and
the loop's live `Vec<Monitor>`. A `Down` retires its arm (monitors are
one-shot). Integration test: the motivating worker-pool pattern — a server
that spawns workers in a handler, watches them, and reacts to their deaths.
### 3. io fd hygiene — DEL on the unwind path ✅ (`f6969e5`)
A stopped actor unwinding out of `wait_fd`'s park leaks its `waiters` entry
and the kernel registration; the stale entry then fails every future
`wait_*` on that fd with `AlreadyExists` (the defensive bare-DEL in
`epoll_register` sits behind the `contains_key` check, so it never runs).
Fix where the invariant breaks: a drop guard in `wait_fd`, armed after a
successful register, disarmed on normal wake; on unwind it removes the
waiters entry iff it is still `(me, epoch)` and only then DELs the fd
(an entry consumed by a racing `FdReady` means the fd may already be
re-registered by someone else — leave it alone).
### Deviations from the plan
- **The plain-inbox fast path narrowed.** It now also requires the control
arm closed, so every server pays one select on its first iteration (the
ctx drops after `init`; a state that didn't clone the Watcher closes the
arm there and then). Servers that DO hold a Watcher select forever — the
price of dynamic watch.
- **The fd leak was worse than the v0.2 caveat admitted.** Not "at most one
stale wakeup": the stale `waiters` entry failed every future `wait_*` on
that fd with AlreadyExists, permanently (the defensive DEL sat behind the
`contains_key` check). The regression test exhibits it as a hang.
---
## Later
### Tunable scheduler idle policy (RFC 004)
Today idle schedulers fall back to `thread::sleep(100µs)`, putting ~100µs
worst-case latency on every cross-thread actor handoff. RFC 004 specifies a
bounded spinning-worker policy (Go/Tokio model): idle schedulers spin on a
cheap atomic for a configurable budget before parking, so a burst of work is
picked up in cache-coherency time rather than after a kernel reschedule.
Two new `Config` knobs: `spin_budget_cycles` (RDTSC cycles to spin; `0`
recovers today's behaviour) and `max_spinners` (cap on concurrent spinners;
default `N/2`). Single new atomic `n_spinning`. No change to the run-queue
data structure — lands on the current `Mutex<VecDeque>` and is independent of
any future queue topology work.
Full spec: `rfc_004-tunable-scheduler-idle-policy.md` (artefacts.kalsbeek.dev).
### Unbounded / configurable-bounded actor count
Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384).
Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling segment
sizes, append-only) once the cap is actually hit. Do not let it calcify.
### arm-port validation & merge
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
hardware. Build + run full test suite on an aarch64 device; check
`chained_spawn` / `yield_many` bench medians; merge and update README.
---
## Invariants & gotchas (respect these across all cycles)
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
`with_shared`. Never send on a channel while holding the shared lock. Pattern:
`mem::take` data under the lock, send after releasing. See `finalize_actor`.
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs →
unpark joiners → reclaim slot if `outstanding_handles==0`. Death notifications
always precede reclamation.
- **Slot lifecycle reset in THREE places:** `Slot::vacant()`, `reclaim_slot()`
(runtime.rs), slot-init block in `spawn_under` (scheduler.rs). Any new `Slot`
field must be reset in all three.
- **Pid = (index, generation).** Stale handles caught by generation mismatch in
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
- **The only wildcard wake is `request_stop`, and it is terminal.** Every
registration-based waker (channel sends, mutex grants, wait-timers, io
completions, joiner wakes, `select` arms) carries the wait's park-epoch
and wakes through `unpark_at`; every successful wake consumes the epoch.
Wakes are therefore *meaningful*: one-shot park sites interpret them
without loops, and `select` needs no cancellation pass. When adding a new
waker, decide which form it is — if its registration handle can outlive
the wait it was created for, it MUST be epoch-stamped; a wait that can
exit without parking MUST `retire_wait` first (see slot_state.rs).
- **`select` exists; a unified per-process mailbox still does not.** The
supervisor keeps its single `supervisor_channel` funnel; `recv_match`
stays per-channel. `select` composes channels at the wait, not into one
queue — gen_server's `handle_info`/`handle_down` (v0.8) are built on
exactly that composition, with documented arm priority (downs → control
→ infos → inbox) instead of mailbox FIFO. A hot higher-priority arm
starves lower ones by design; that's the contract.
- **Cooperative-only.** Preemption and cancellation both depend on the actor
reaching `check!()`/yield/alloc/blocking points.
- **Lock order is Leaf → Channel, one of each at most** (debug-asserted in
`raw_mutex.rs`). Leaf = cold locks / free list / stack pool / registry,
mutual leaves. A channel lock may be taken under a Leaf (finalize/monitor
clone senders living in slots); nothing may be locked under a channel lock.
- **Queue ops require preemption disabled.** A producer suspended mid-publish
stalls every consumer — livelock. `with_runtime`, `with_shared`, and
`RawMutex` guards all disable preemption for their span.
- **`run()` is single-thread** (`Config::exact(1)`); tests rely on deterministic
single-thread ordering. Multi-thread via `runtime::init(Config…)`.
benches/README.md
-89
View File
@@ -1,89 +0,0 @@
# Benches
Two families live here: **comparison benches** (smarm vs tokio, predating
v0.5) and the **run-queue shootout** (v0.5 phase 4). All are plain binaries
(`harness = false` in `Cargo.toml`), so `cargo bench` just builds in release
and runs `main()` — no criterion, no magic.
```
cargo bench --bench <name> # one bench
cargo bench # all of them (slow; rarely what you want)
```
## Catalog
| file | what it measures |
|---|---|
| `primes.rs` | Compute fan-out/fan-in: counts primes across W workers. Pure compute throughput + spawn/join/channel cost. |
| `multi_scheduler.rs` | The original cross-runtime matrix: smarm (1 thread / N threads) vs tokio (current_thread / multi_thread) on compute, ping-pong, and spawn throughput. |
| `general.rs` | Workloads where neither runtime has a structural edge. Large gaps here mean real per-task/per-yield overhead differences — watch these for regressions. |
| `smarm_favored.rs` | Workloads the stackful green-thread model is built for. Single-thread numbers isolate per-switch cost from contention. |
| `tokio_favored.rs` | Workloads tokio's model is built for. Expect to lose; the value is knowing *by how much* and catching the gap widening. |
| `rq_micro.rs` | Run-queue **structures** in isolation (no runtime, no actors): push/pop throughput sweeping thread count × producer:consumer ratio. Covers all three queue types in one binary — the types compile in every build; only the runtime's alias is feature-selected. |
| `rq_runtime.rs` | The **whole scheduler** with the compile-time-selected queue: yield-storm (pure queue churn), ping-pong-pairs (park/unpark latency), spawn-storm (slab + free list + queue churn), sweeping scheduler count. Comparing variants requires rebuilding per `rq-*` feature. |
## The run-queue shootout
One command; it rebuilds `rq_runtime` once per queue variant, runs `rq_micro`
once, and aggregates:
```
./scripts/bench_rq.sh
# on a big box:
SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh
```
Outputs land in `bench_results/` (gitignored): one full log per run, plus
`summary.csv` assembled from the machine-readable `RQCSV,...` lines every
config prints alongside the human table.
Manual single-variant runs need the feature dance (features are additive, so
the default `rq-mutex` must be switched off):
```
cargo bench --bench rq_runtime --no-default-features --features rq-striped
```
### Knobs (env vars, all optional)
| var | default | used by |
|---|---|---|
| `SMARM_BENCH_THREADS` | `"1 2 4"` | both — space-separated sweep |
| `SMARM_BENCH_RUNS` | `5` | both — repetitions; the **median** is reported |
| `SMARM_BENCH_ITEMS` | `200000` | `rq_micro` — items per measurement |
| `SMARM_BENCH_YIELD_ACTORS` / `_YIELDS` | `200` / `500` | `rq_runtime` yield-storm |
| `SMARM_BENCH_PAIRS` / `_ROUNDTRIPS` | `32` / `1000` | `rq_runtime` ping-pong |
| `SMARM_BENCH_SPAWNS` | `5000` | `rq_runtime` spawn-storm |
## Reading the numbers honestly
- **Core count is the experiment.** On a 1-core machine (CI, sandboxes) the
sweep only validates the harness and catches gross pathologies —
oversubscribed schedulers measure context-switch noise, not contention.
Variant decisions come from a many-core box.
- The striped queue *should lose* at low thread counts (ticket overhead with
no contention to amortize) — that's expected, not a bug.
- Medians over `SMARM_BENCH_RUNS` absorb scheduling noise but not thermal /
turbo drift; for publishable numbers, pin the CPU governor and run a warmup
pass first.
- `spawn-storm` batches joins (1024 at a time) to stay well under the slab
cap; if you raise `SMARM_BENCH_SPAWNS` massively, that batching is why it
still works.
## Adding a bench
1. `benches/<name>.rs` with a plain `main()`; print the house table (see any
existing bench) and, if it belongs to a sweep, a greppable CSV line with a
distinctive prefix (`RQCSV,` for the shootout family).
2. Register it in `Cargo.toml`:
```toml
[[bench]]
name = "<name>"
harness = false
```
3. Take parameters from `SMARM_BENCH_*` env vars with modest defaults — the
defaults must finish in seconds on one core, the env scales them up on
real hardware.
4. Report **medians**, and keep one measurement = one fresh runtime
(`init(Config::exact(t))` inside the measured closure constructor, the
`run()` inside the timed region) so runs don't contaminate each other.
benches/rq_micro.rs
-186
View File
@@ -1,186 +0,0 @@
//! Raw run-queue microbench (ROADMAP_v0.5 phase 4).
//!
//! Benches the three queue STRUCTURES directly — no runtime, no actors — to
//! isolate the data structure under contention. All three types compile in
//! every build, so this binary covers the whole matrix in one run; it does
//! NOT need the rq-* feature rebuild dance (that's `rq_runtime`).
//!
//! Sweeps thread count × producer:consumer ratio. Queues are sized to the
//! item count, so the occupancy contract holds trivially and producers never
//! block on capacity.
//!
//! Knobs (env):
//! SMARM_BENCH_THREADS space-separated sweep, default "1 2 4"
//! SMARM_BENCH_ITEMS items per measurement, default 200_000
//! SMARM_BENCH_RUNS repetitions per config (median reported), default 5
//!
//! Output: the house table, plus one machine-readable line per config:
//! RQCSV,micro,<structure>,<threads>,<p:c>,<items>,<median_us>,<items_per_s>
//!
//! NOTE: numbers from a 1-core sandbox only validate the harness; real
//! contention curves come from the many-core box (scripts/bench_rq.sh).
use smarm::pid::Pid;
use smarm::run_queue::{MpmcRing, MutexQueue, StripedRing};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Instant;
fn env_usize(key: &str, default: usize) -> usize {
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}
fn env_threads() -> Vec<usize> {
std::env::var("SMARM_BENCH_THREADS")
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
.unwrap_or_else(|_| vec![1, 2, 4])
}
/// Generic driver: `producers` threads push `items` total, `consumers`
/// threads pop until everything is accounted for. Returns elapsed µs.
fn drive<Q: Send + Sync + 'static>(
q: Arc<Q>,
push: fn(&Q, Pid),
pop: fn(&Q) -> Option<Pid>,
producers: usize,
consumers: usize,
items: usize,
) -> u128 {
let remaining = Arc::new(AtomicUsize::new(items));
let start = Instant::now();
let mut hs = Vec::new();
let per = items / producers;
for p in 0..producers {
let q = q.clone();
// Give the last producer the remainder.
let n = if p == producers - 1 { items - per * (producers - 1) } else { per };
hs.push(std::thread::spawn(move || {
let pid = Pid::new(p as u32, 0);
for _ in 0..n {
push(&q, pid);
}
}));
}
for _ in 0..consumers {
let q = q.clone();
let remaining = remaining.clone();
hs.push(std::thread::spawn(move || loop {
// Claim-then-pop so consumers exit promptly when the budget hits
// zero; the claim is backed out on a miss.
let r = remaining.load(Ordering::Relaxed);
if r == 0 {
return;
}
if pop(&q).is_some() {
remaining.fetch_sub(1, Ordering::Relaxed);
} else {
std::hint::spin_loop();
}
}));
}
for h in hs {
h.join().unwrap();
}
start.elapsed().as_micros()
}
/// Single-thread alternating push/pop (the T = 1 case).
fn drive_single<Q>(q: &Q, push: fn(&Q, Pid), pop: fn(&Q) -> Option<Pid>, items: usize) -> u128 {
let pid = Pid::new(0, 0);
let start = Instant::now();
for _ in 0..items {
push(q, pid);
assert!(pop(q).is_some());
}
start.elapsed().as_micros()
}
struct Case {
structure: &'static str,
threads: usize,
producers: usize,
consumers: usize,
}
fn ratios_for(threads: usize) -> Vec<(usize, usize)> {
if threads < 2 {
return vec![(1, 1)]; // label only; T=1 runs the alternating driver
}
let mut v = vec![(threads / 2, threads - threads / 2)]; // balanced
if threads >= 4 {
v.push((3 * threads / 4, threads - 3 * threads / 4)); // producer-heavy
v.push((threads / 4, threads - threads / 4)); // consumer-heavy
}
v
}
fn main() {
let threads_sweep = env_threads();
let items = env_usize("SMARM_BENCH_ITEMS", 200_000);
let runs = env_usize("SMARM_BENCH_RUNS", 5);
println!("\n{}", "=".repeat(86));
println!(" run-queue raw structures — items={items}, runs={runs} (median)");
println!("{}", "=".repeat(86));
println!(
"{:>10} | {:>7} | {:>7} | {:>10} | {:>14}",
"structure", "threads", "p:c", "median µs", "items/s"
);
println!("{}", "-".repeat(86));
let mut cases = Vec::new();
for &t in &threads_sweep {
for (p, c) in ratios_for(t) {
for s in ["mutex", "mpmc", "striped"] {
cases.push(Case { structure: s, threads: t, producers: p, consumers: c });
}
}
}
for case in cases {
let mut times: Vec<u128> = (0..runs)
.map(|_| {
// Fresh queue per run; capacity = items so pushes never stall.
match case.structure {
"mutex" => {
let q = Arc::new(MutexQueue::new(case.threads, items));
if case.threads < 2 {
drive_single(&*q, MutexQueue::push, MutexQueue::pop, items)
} else {
drive(q, MutexQueue::push, MutexQueue::pop, case.producers, case.consumers, items)
}
}
"mpmc" => {
let q = Arc::new(MpmcRing::with_capacity(items));
if case.threads < 2 {
drive_single(&*q, MpmcRing::push, MpmcRing::pop, items)
} else {
drive(q, MpmcRing::push, MpmcRing::pop, case.producers, case.consumers, items)
}
}
"striped" => {
let q = Arc::new(StripedRing::new(case.threads.max(1), items));
if case.threads < 2 {
drive_single(&*q, StripedRing::push, StripedRing::pop, items)
} else {
drive(q, StripedRing::push, StripedRing::pop, case.producers, case.consumers, items)
}
}
_ => unreachable!(),
}
})
.collect();
times.sort_unstable();
let median = times[times.len() / 2];
let per_s = (items as f64 / (median as f64 / 1e6)) as u64;
let ratio = format!("{}:{}", case.producers, case.consumers);
println!(
"{:>10} | {:>7} | {:>7} | {:>10} | {:>14}",
case.structure, case.threads, ratio, median, per_s
);
println!(
"RQCSV,micro,{},{},{},{},{},{}",
case.structure, case.threads, ratio, items, median, per_s
);
}
}
benches/rq_runtime.rs
-188
View File
@@ -1,188 +0,0 @@
//! Runtime-level run-queue benches (ROADMAP_v0.5 phase 4).
//!
//! These exercise the WHOLE scheduler with the compile-time-selected queue,
//! so comparing variants means rebuilding per rq-* feature — that's what
//! scripts/bench_rq.sh does. Workloads:
//!
//! yield-storm — N actors yield K times each. Pure queue churn:
//! every yield is a push + pop with nothing in between.
//! ping-pong-pairs — P channel pairs, M roundtrips each. Park/unpark
//! latency through the queue.
//! spawn-storm — S spawn+join of trivial actors. Slab + queue + free
//! list under churn.
//!
//! Knobs (env):
//! SMARM_BENCH_THREADS scheduler-count sweep, default "1 2 4"
//! SMARM_BENCH_RUNS repetitions per config (median), default 5
//! SMARM_BENCH_YIELD_ACTORS / _YIELDS default 200 / 500
//! SMARM_BENCH_PAIRS / _ROUNDTRIPS default 32 / 1000
//! SMARM_BENCH_SPAWNS default 5000
//!
//! Output: house table + one line per config:
//! RQCSV,runtime,<variant>,<bench>,<threads>,<work>,<median_us>,<ops_per_s>
//!
//! NOTE: a 1-core sandbox validates the harness, not the scaling story;
//! real curves come from the many-core box.
use smarm::runtime::{init, Config};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Instant;
fn variant() -> &'static str {
if cfg!(feature = "rq-mpmc") {
"rq-mpmc"
} else if cfg!(feature = "rq-striped") {
"rq-striped"
} else {
"rq-mutex"
}
}
fn env_usize(key: &str, default: usize) -> usize {
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}
fn env_threads() -> Vec<usize> {
std::env::var("SMARM_BENCH_THREADS")
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
.unwrap_or_else(|_| vec![1, 2, 4])
}
/// (total_ops, elapsed_µs) for one measured run.
fn yield_storm(threads: usize, actors: usize, yields: usize) -> (u64, u128) {
let rt = init(Config::exact(threads));
let start = Instant::now();
rt.run(move || {
let handles: Vec<_> = (0..actors)
.map(|_| {
smarm::spawn(move || {
for _ in 0..yields {
smarm::yield_now();
}
})
})
.collect();
for h in handles {
let _ = h.join();
}
});
((actors * yields) as u64, start.elapsed().as_micros())
}
fn ping_pong_pairs(threads: usize, pairs: usize, roundtrips: usize) -> (u64, u128) {
let rt = init(Config::exact(threads));
let total = Arc::new(AtomicU64::new(0));
let t2 = total.clone();
let start = Instant::now();
rt.run(move || {
let handles: Vec<_> = (0..pairs)
.map(|_| {
let total = t2.clone();
smarm::spawn(move || {
let (tx_ab, rx_ab) = smarm::channel::channel::<u64>();
let (tx_ba, rx_ba) = smarm::channel::channel::<u64>();
let n = roundtrips as u64;
let echo = smarm::spawn(move || {
for _ in 0..n {
let v = rx_ab.recv().expect("echo recv");
tx_ba.send(v + 1).expect("echo send");
}
});
for i in 0..n {
tx_ab.send(i).expect("ping send");
let v = rx_ba.recv().expect("ping recv");
assert_eq!(v, i + 1);
}
let _ = echo.join();
total.fetch_add(n, Ordering::Relaxed);
})
})
.collect();
for h in handles {
let _ = h.join();
}
});
(total.load(Ordering::Relaxed), start.elapsed().as_micros())
}
fn spawn_storm(threads: usize, spawns: usize) -> (u64, u128) {
let rt = init(Config::exact(threads));
let start = Instant::now();
rt.run(move || {
// Batches bound simultaneous liveness well below the slab cap.
const BATCH: usize = 1024;
let mut left = spawns;
while left > 0 {
let n = left.min(BATCH);
let handles: Vec<_> = (0..n).map(|_| smarm::spawn(|| {})).collect();
for h in handles {
let _ = h.join();
}
left -= n;
}
});
(spawns as u64, start.elapsed().as_micros())
}
fn main() {
let threads_sweep = env_threads();
let runs = env_usize("SMARM_BENCH_RUNS", 5);
let ya = env_usize("SMARM_BENCH_YIELD_ACTORS", 200);
let yy = env_usize("SMARM_BENCH_YIELDS", 500);
let pp = env_usize("SMARM_BENCH_PAIRS", 32);
let pr = env_usize("SMARM_BENCH_ROUNDTRIPS", 1000);
let ss = env_usize("SMARM_BENCH_SPAWNS", 5000);
println!("\n{}", "=".repeat(86));
println!(" runtime benches — variant={}, runs={runs} (median)", variant());
println!("{}", "=".repeat(86));
println!(
"{:>16} | {:>7} | {:>16} | {:>10} | {:>14}",
"bench", "threads", "work", "median µs", "ops/s"
);
println!("{}", "-".repeat(86));
type Bench = (&'static str, String, Box<dyn Fn(usize) -> (u64, u128)>);
let benches: Vec<Bench> = vec![
(
"yield-storm",
format!("{ya}x{yy}"),
Box::new(move |t| yield_storm(t, ya, yy)),
),
(
"ping-pong-pairs",
format!("{pp}x{pr}"),
Box::new(move |t| ping_pong_pairs(t, pp, pr)),
),
(
"spawn-storm",
format!("{ss}"),
Box::new(move |t| spawn_storm(t, ss)),
),
];
for (name, work, f) in &benches {
for &t in &threads_sweep {
let mut ops = 0u64;
let mut times: Vec<u128> = (0..runs)
.map(|_| {
let (o, us) = f(t);
ops = o;
us
})
.collect();
times.sort_unstable();
let median = times[times.len() / 2];
let per_s = (ops as f64 / (median as f64 / 1e6)) as u64;
println!(
"{:>16} | {:>7} | {:>16} | {:>10} | {:>14}",
name, t, work, median, per_s
);
println!(
"RQCSV,runtime,{},{},{},{},{},{}",
variant(), name, t, work, median, per_s
);
}
}
}
docs/smarm - Deep Dive.html
+439 -877
View File
File diff suppressed because it is too large Load Diff
scripts/bench_rq.sh
-31
View File
@@ -1,31 +0,0 @@
#!/usr/bin/env bash
# Run-queue shootout driver (ROADMAP_v0.5 phase 4).
#
# Rebuilds the runtime bench once per rq-* feature and runs the raw-structure
# microbench once (it covers all structures in a single binary). Results land
# in bench_results/ as full logs; the RQCSV lines are aggregated into
# bench_results/summary.csv for plotting.
#
# Tune the sweep for the box, e.g. on the 20-core machine:
# SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh
set -euo pipefail
cd "$(dirname "$0")/.."
OUT=bench_results
mkdir -p "$OUT"
: "${SMARM_BENCH_THREADS:=1 2 4}"
export SMARM_BENCH_THREADS
echo "== raw structures (one binary, all variants) =="
cargo bench --bench rq_micro 2>&1 | tee "$OUT/micro.txt"
for v in rq-mutex rq-mpmc rq-striped; do
echo "== runtime benches: $v =="
cargo bench --bench rq_runtime --no-default-features --features "$v" \
2>&1 | tee "$OUT/runtime-$v.txt"
done
echo "bench,kind,a,b,c,d,median_us,ops_per_s" > "$OUT/summary.csv"
grep -h '^RQCSV,' "$OUT"/*.txt | sed 's/^RQCSV,//' >> "$OUT/summary.csv"
echo
echo "Summary: $OUT/summary.csv ($(($(wc -l < "$OUT/summary.csv") - 1)) rows)"
src/actor.rs
+2 -2
View File
@@ -122,9 +122,9 @@ pub struct Actor {
/// The PID this actor was assigned at spawn time.
pub pid: Pid,
/// The stack the actor runs on. Dropped (munmap'd) when the actor dies.
/// (The saved stack pointer lives on the `Slot` as an atomic, not here:
/// it is hot scheduling state, read/written without the cold lock.)
pub stack: Stack,
/// The saved stack pointer. Updated on every yield.
pub sp: usize,
/// The PID of this actor's supervisor. Used to deliver `Signal` on death.
pub supervisor: Pid,
/// Cooperative-cancellation flag. `request_stop` sets it (and unparks a
src/arch/aarch64.rs
+165
View File
@@ -0,0 +1,165 @@
//! Cooperative context switching and cycle counter, aarch64 (AAPCS64).
//!
//! The aarch64 mirror of the x86-64 backend. The protocol is identical — save
//! callee-saved state, swap stack pointers via the shared `*_SP` thread-locals,
//! restore, return — but the mechanics differ from x86 in two ways worth
//! stating up front, because they drive the stack layout:
//!
//! 1. **Return is via the link register, not the stack.** x86 `ret` pops the
//! return address off the stack; aarch64 `ret` jumps to whatever is in
//! `x30` (lr). So the entry point is parked in the saved-`x30` slot, and
//! the shim restores `x30` and `ret`s to it. There is no "ret target word"
//! sitting on the stack the way there is on x86.
//!
//! 2. **sp must be 16-byte aligned at all times**, not just at call entry.
//! We save 12 registers (x19–x28, x29, x30) = 96 bytes, already a multiple
//! of 16, so the frame is aligned by construction.
//!
//! FP/SIMD registers (v8–v15, whose low 64 bits are callee-saved under AAPCS64)
//! are NOT saved, for the same reason the x86 backend skips XMM: every yield
//! goes through a Rust call boundary, so the compiler has already spilled any
//! live vector state. If we ever yield from a non-call-boundary, this breaks.
use super::{get_actor_sp, get_scheduler_sp, set_actor_sp, set_scheduler_sp};
// ---------------------------------------------------------------------------
// Initial stack layout
//
// Twelve 8-byte slots, all zero except the x30 slot which holds `entry`. The
// first `switch_to_actor` loads x19–x28/x29/x30 back and `ret`s to x30,
// landing at the top of `entry` with sp 16-aligned (the AAPCS64 requirement).
//
// Layout (high → low), relative to aligned_top = top & ~15:
//
// aligned_top - 8 : x30 = entry ← `ret` jumps here.
// aligned_top - 16 : x29 = 0 (fp)
// aligned_top - 24 : x28 = 0
// aligned_top - 32 : x27 = 0
// aligned_top - 40 : x26 = 0
// aligned_top - 48 : x25 = 0
// aligned_top - 56 : x24 = 0
// aligned_top - 64 : x23 = 0
// aligned_top - 72 : x22 = 0
// aligned_top - 80 : x21 = 0
// aligned_top - 88 : x20 = 0
// aligned_top - 96 : x19 = 0 ← initial sp (16-aligned)
//
// The restore order in the shim (ldp pairs, low→high) must match this exactly.
// ---------------------------------------------------------------------------
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
unsafe {
let aligned_top = top as usize & !15;
let mut sp = aligned_top;
sp -= 8; (sp as *mut usize).write(entry as usize); // x30 (lr) → entry
sp -= 8; (sp as *mut usize).write(0); // x29 (fp)
sp -= 8; (sp as *mut usize).write(0); // x28
sp -= 8; (sp as *mut usize).write(0); // x27
sp -= 8; (sp as *mut usize).write(0); // x26
sp -= 8; (sp as *mut usize).write(0); // x25
sp -= 8; (sp as *mut usize).write(0); // x24
sp -= 8; (sp as *mut usize).write(0); // x23
sp -= 8; (sp as *mut usize).write(0); // x22
sp -= 8; (sp as *mut usize).write(0); // x21
sp -= 8; (sp as *mut usize).write(0); // x20
sp -= 8; (sp as *mut usize).write(0); // x19 ← initial sp
sp
}
}
// ---------------------------------------------------------------------------
// Context switch shims
//
// Each shim:
// 1. Pushes x19–x28, x29, x30 as six 16-byte stp pairs (sp pre-decrement).
// 2. Moves sp into x0 and calls the Rust helper that stores it.
// 3. Calls the Rust helper that returns the *other* side's saved sp.
// 4. Moves that into sp.
// 5. Restores the twelve registers and rets (to the restored x30).
//
// The helper calls clobber x30 themselves, but that's fine: the outgoing x30
// was already saved to the stack in step 1, and step 5 restores the incoming
// side's x30 before `ret`. The push/pop register order is paired so that the
// `ldp` sequence reverses the `stp` sequence exactly.
// ---------------------------------------------------------------------------
#[unsafe(naked)]
unsafe extern "C" fn switch_to_actor_asm() {
core::arch::naked_asm!(
"stp x19, x20, [sp, #-96]!",
"stp x21, x22, [sp, #16]",
"stp x23, x24, [sp, #32]",
"stp x25, x26, [sp, #48]",
"stp x27, x28, [sp, #64]",
"stp x29, x30, [sp, #80]",
"mov x0, sp",
"bl {set_sched_sp}",
"bl {get_actor_sp}",
"mov sp, x0",
"ldp x21, x22, [sp, #16]",
"ldp x23, x24, [sp, #32]",
"ldp x25, x26, [sp, #48]",
"ldp x27, x28, [sp, #64]",
"ldp x29, x30, [sp, #80]",
"ldp x19, x20, [sp], #96",
"ret",
set_sched_sp = sym set_scheduler_sp,
get_actor_sp = sym get_actor_sp,
);
}
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
pub unsafe fn switch_to_actor() {
unsafe { switch_to_actor_asm() };
}
#[unsafe(naked)]
pub unsafe extern "C" fn switch_to_scheduler() {
core::arch::naked_asm!(
"stp x19, x20, [sp, #-96]!",
"stp x21, x22, [sp, #16]",
"stp x23, x24, [sp, #32]",
"stp x25, x26, [sp, #48]",
"stp x27, x28, [sp, #64]",
"stp x29, x30, [sp, #80]",
"mov x0, sp",
"bl {set_actor_sp}",
"bl {get_sched_sp}",
"mov sp, x0",
"ldp x21, x22, [sp, #16]",
"ldp x23, x24, [sp, #32]",
"ldp x25, x26, [sp, #48]",
"ldp x27, x28, [sp, #64]",
"ldp x29, x30, [sp, #80]",
"ldp x19, x20, [sp], #96",
"ret",
set_actor_sp = sym set_actor_sp,
get_sched_sp = sym get_scheduler_sp,
);
}
// ---------------------------------------------------------------------------
// Cycle counter
//
// CNTVCT_EL0 is the virtual count register, readable from EL0 on Linux. `isb`
// serialises so we don't sample the counter before prior instructions retire
// (the aarch64 analogue of x86's `lfence` before rdtsc).
//
// Unit: CNTVCT ticks, whose frequency is CNTFRQ_EL0 — typically 1–50 MHz,
// NOT the CPU clock. This is a different and much coarser unit than the x86
// TSC, which is why `TIMESLICE_CYCLES` must be tuned separately for aarch64.
// ---------------------------------------------------------------------------
#[inline(always)]
pub fn read_cycle_counter() -> u64 {
let cnt: u64;
unsafe {
core::arch::asm!(
"isb",
"mrs {cnt}, cntvct_el0",
cnt = out(reg) cnt,
options(nostack, nomem),
);
}
cnt
}
src/arch/mod.rs
+54
View File
@@ -0,0 +1,54 @@
//! Architecture abstraction layer.
//!
//! Everything that depends on the target ISA lives behind this module: the
//! cooperative context-switch shims, the initial-stack layout, and the
//! cycle counter used by the preemption timeslice. The rest of the runtime
//! talks only to the API re-exported here and never names a register.
//!
//! Each backend (`x86_64`, `aarch64`) exposes the same four items:
//!
//! - `init_actor_stack(top, entry) -> usize`
//! Build a fresh actor stack so the first `switch_to_actor` lands
//! inside `entry` with the ABI-correct stack alignment.
//! - `switch_to_actor()`
//! Resume the actor whose sp is in `ACTOR_SP`; returns when it yields.
//! - `switch_to_scheduler()`
//! Yield from the current actor back to its scheduler thread.
//! - `read_cycle_counter() -> u64`
//! Monotonic per-core cycle counter for the timeslice clock. The unit
//! is ISA-defined (x86 TSC ticks vs. aarch64 virtual-counter ticks),
//! so `TIMESLICE_CYCLES` must be tuned per-arch — see `preempt`.
//!
//! The `SCHEDULER_SP` / `ACTOR_SP` thread-locals are shared across backends
//! and live here, since the saved-stack-pointer protocol is identical
//! regardless of which registers the shim happens to push.
use std::cell::Cell;
thread_local! {
static SCHEDULER_SP: Cell<usize> = const { Cell::new(0) };
static ACTOR_SP: Cell<usize> = const { Cell::new(0) };
}
// Used by the naked shims (via `sym`) and by the scheduler/tests through the
// re-exports below. `pub` because the asm references them as symbols.
pub(crate) fn get_scheduler_sp() -> usize { SCHEDULER_SP.with(|c| c.get()) }
pub(crate) fn set_scheduler_sp(v: usize) { SCHEDULER_SP.with(|c| c.set(v)) }
pub fn get_actor_sp() -> usize { ACTOR_SP.with(|c| c.get()) }
pub fn set_actor_sp(v: usize) { ACTOR_SP.with(|c| c.set(v)) }
#[cfg(target_arch = "x86_64")]
mod x86_64;
#[cfg(target_arch = "x86_64")]
pub use x86_64::{init_actor_stack, read_cycle_counter, switch_to_actor, switch_to_scheduler};
#[cfg(target_arch = "aarch64")]
mod aarch64;
#[cfg(target_arch = "aarch64")]
pub use aarch64::{init_actor_stack, read_cycle_counter, switch_to_actor, switch_to_scheduler};
#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
compile_error!(
"smarm's context-switch and cycle-counter layer supports only x86_64 and \
aarch64. Add a backend in src/arch/ for this target."
);
src/arch/x86_64.rs
+118
View File
@@ -0,0 +1,118 @@
//! Cooperative context switching and cycle counter, x86-64 (SysV AMD64).
//!
//! Two naked-asm functions move execution between a scheduler thread and an
//! actor running on its own mmap'd stack. The compiler cannot do this; the
//! whole point of `#[unsafe(naked)]` is that we control every instruction.
//!
//! `init_actor_stack` builds the initial stack so that the first
//! `switch_to_actor` lands inside the entry function with `rsp % 16 == 8`
//! (the x86-64 ABI requirement at function entry).
use super::{get_actor_sp, get_scheduler_sp, set_actor_sp, set_scheduler_sp};
// ---------------------------------------------------------------------------
// Initial stack layout
//
// We start from aligned_top = top & ~15, then bias by 8 and push seven 8-byte
// slots (downward). The first `switch_to_actor` pops r15..rbx and `ret`s —
// landing in `entry` with rsp % 16 == 8 (the x86-64 ABI state at the point
// just after a `call`, which is what `entry` is compiled to expect).
//
// Layout (high → low), relative to aligned_top = top & ~15. Note the entry
// slot is at aligned_top - 16, NOT aligned_top - 8: the function does
// `(top & ~15) - 8` and *then* a `-= 8` before the first write, so the first
// stored word lands at aligned_top - 16. Verified by single-stepping the
// `ret` under llmdbg: entry sits at an address with %16 == 0, so the post-ret
// rsp is %16 == 8.
//
// aligned_top - 8 : (unused padding; keeps entry's slot %16 == 0)
// aligned_top - 16 : entry ptr ← `ret` target. Post-ret: rsp % 16 == 8.
// aligned_top - 24 : rbx = 0
// aligned_top - 32 : rbp = 0
// aligned_top - 40 : r12 = 0
// aligned_top - 48 : r13 = 0
// aligned_top - 56 : r14 = 0
// aligned_top - 64 : r15 = 0 ← initial rsp
// ---------------------------------------------------------------------------
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
unsafe {
let mut sp = (top as usize & !15) - 8;
sp -= 8; (sp as *mut usize).write(entry as usize); // ret target
sp -= 8; (sp as *mut usize).write(0); // rbx
sp -= 8; (sp as *mut usize).write(0); // rbp
sp -= 8; (sp as *mut usize).write(0); // r12
sp -= 8; (sp as *mut usize).write(0); // r13
sp -= 8; (sp as *mut usize).write(0); // r14
sp -= 8; (sp as *mut usize).write(0); // r15
sp
}
}
// ---------------------------------------------------------------------------
// Context switch shims
//
// Each shim:
// 1. Pushes the six callee-saved integer registers.
// 2. Snaps rsp into rdi and calls the Rust helper that stores it.
// 3. Calls the Rust helper that returns the *other* side's saved rsp.
// 4. Moves that into rsp.
// 5. Pops the six registers and rets.
//
// XMM registers are NOT saved here. We rely on every yield happening through
// a Rust call site, which means the compiler has spilled any live XMM state
// to the stack before we get here. (This is the same argument the compiler
// uses internally — callee-saved regs are what survive a `call`, and the
// SysV AMD64 ABI says XMM0–15 are all caller-saved.) If we ever yield from
// a place that isn't a Rust call boundary, this assumption breaks.
// ---------------------------------------------------------------------------
#[unsafe(naked)]
unsafe extern "C" fn switch_to_actor_asm() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_sched_sp}",
"call {get_actor_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_sched_sp = sym set_scheduler_sp,
get_actor_sp = sym get_actor_sp,
);
}
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
pub unsafe fn switch_to_actor() {
unsafe { switch_to_actor_asm() };
}
#[unsafe(naked)]
pub unsafe extern "C" fn switch_to_scheduler() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_actor_sp}",
"call {get_sched_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_actor_sp = sym set_actor_sp,
get_sched_sp = sym get_scheduler_sp,
);
}
// ---------------------------------------------------------------------------
// Cycle counter
//
// `lfence` serialises the instruction stream so we don't measure time before
// prior instructions retire. Unit: TSC ticks (≈ CPU base clock).
// ---------------------------------------------------------------------------
#[inline(always)]
pub fn read_cycle_counter() -> u64 {
unsafe {
core::arch::asm!("lfence", options(nostack, nomem, preserves_flags));
core::arch::x86_64::_rdtsc()
}
}
src/channel.rs
+22 -339
View File
@@ -1,23 +1,9 @@
//! Unbounded MPSC channels.
//!
//! Inner state is `Arc<RawMutex<Inner<T>>>` so channels can be sent across OS
//! Inner state is `Arc<Mutex<Inner<T>>>` so channels can be sent across OS
//! threads (required for the multi-scheduler runtime where a sender and
//! receiver may run on different scheduler threads simultaneously).
//!
//! ## Why `RawMutex` (Channel class), not `std::sync::Mutex`
//!
//! An actor holding a guard with preemption *enabled* can be timesliced
//! inside the critical section and resume on a different OS thread — the
//! pthread mutex would then be released from a thread that didn't lock it,
//! which is UB (Linux futexes happen to tolerate it, but it's not
//! guaranteed). `RawMutex` disables preemption for the guard's span and is
//! cross-thread-release sound by construction, closing the hole. It also
//! cannot poison. Channel locks form their own [`LockClass::Channel`]
//! (raw_mutex.rs): they may be taken under a cold (Leaf) lock — finalize and
//! `monitor()` clone senders that live in slots — but nothing may be locked
//! under them, which the debug build enforces. `recv_match` runs its user
//! predicate under this lock: keep it cheap, pure, and channel-free.
//!
//! Semantics:
//! - Senders are clonable; the last sender drop closes the channel.
//! - `Receiver::recv` on an empty open channel parks the receiver.
@@ -29,12 +15,11 @@
//! parked, the receiver is unparked.
use crate::pid::Pid;
use crate::raw_mutex::RawMutex;
use std::collections::VecDeque;
use std::sync::Arc;
use std::sync::{Arc, Mutex};
pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
let inner = Arc::new(RawMutex::new_channel(Inner {
let inner = Arc::new(Mutex::new(Inner {
queue: VecDeque::new(),
parked_receiver: None,
senders: 1,
@@ -45,24 +30,17 @@ pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
struct Inner<T> {
queue: VecDeque<T>,
/// The parked receiver's `(pid, park-epoch)`. The epoch is the slot
/// word's runtime-wide wait identity (see slot_state.rs): wakers call
/// `unpark_at(pid, epoch)`, so an entry left over from an already-woken
/// wait — a `select` loser arm, a satisfied `recv_timeout`'s timer — is
/// inert: the wake fails the word's epoch CAS and no-ops. This replaces
/// the old per-channel `cur_wait`/`next_wait_seq`/`timed_out` trio: wait
/// identity now exists exactly once, in the slot word.
parked_receiver: Option<(Pid, u32)>,
parked_receiver: Option<Pid>,
senders: usize,
receiver_alive: bool,
}
pub struct Sender<T> {
inner: Arc<RawMutex<Inner<T>>>,
inner: Arc<Mutex<Inner<T>>>,
}
pub struct Receiver<T> {
inner: Arc<RawMutex<Inner<T>>>,
inner: Arc<Mutex<Inner<T>>>,
}
#[derive(Debug, PartialEq, Eq)]
@@ -79,30 +57,9 @@ impl std::fmt::Display for RecvError {
impl std::error::Error for RecvError {}
/// Returned by [`Receiver::recv_timeout`].
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum RecvTimeoutError {
/// The deadline passed with no message available.
Timeout,
/// All senders dropped with no message available — the bounded analogue
/// of [`RecvError`].
Disconnected,
}
impl std::fmt::Display for RecvTimeoutError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
RecvTimeoutError::Timeout => write!(f, "recv timed out"),
RecvTimeoutError::Disconnected => write!(f, "channel closed"),
}
}
}
impl std::error::Error for RecvTimeoutError {}
impl<T> Clone for Sender<T> {
fn clone(&self) -> Self {
self.inner.lock().senders += 1;
self.inner.lock().unwrap().senders += 1;
Sender { inner: self.inner.clone() }
}
}
@@ -110,7 +67,7 @@ impl<T> Clone for Sender<T> {
impl<T> Drop for Sender<T> {
fn drop(&mut self) {
let unpark = {
let mut g = self.inner.lock();
let mut g = self.inner.lock().unwrap();
g.senders -= 1;
// Wake the parked receiver on the last sender drop regardless of
// whether the queue is empty. A plain `recv` only ever parks on an
@@ -124,31 +81,31 @@ impl<T> Drop for Sender<T> {
None
}
};
if let Some((pid, epoch)) = unpark {
crate::scheduler::unpark_at(pid, epoch);
if let Some(pid) = unpark {
crate::scheduler::unpark(pid);
}
}
}
impl<T> Drop for Receiver<T> {
fn drop(&mut self) {
self.inner.lock().receiver_alive = false;
self.inner.lock().unwrap().receiver_alive = false;
}
}
impl<T> Sender<T> {
pub fn send(&self, value: T) -> Result<(), SendError<T>> {
let unpark = {
let mut g = self.inner.lock();
let mut g = self.inner.lock().unwrap();
if !g.receiver_alive {
return Err(SendError(value));
}
g.queue.push_back(value);
g.parked_receiver.take()
};
if let Some((pid, epoch)) = unpark {
if let Some(pid) = unpark {
crate::te!(crate::trace::Event::Send { sender: crate::actor::current_pid().unwrap_or(crate::pid::Pid::new(u32::MAX, u32::MAX)), receiver: Some(pid) });
crate::scheduler::unpark_at(pid, epoch);
crate::scheduler::unpark(pid);
} else {
crate::te!(crate::trace::Event::Send { sender: crate::actor::current_pid().unwrap_or(crate::pid::Pid::new(u32::MAX, u32::MAX)), receiver: None });
}
@@ -160,7 +117,7 @@ impl<T> Receiver<T> {
pub fn recv(&self) -> Result<T, RecvError> {
loop {
{
let mut g = self.inner.lock();
let mut g = self.inner.lock().unwrap();
if let Some(v) = g.queue.pop_front() {
return Ok(v);
}
@@ -170,13 +127,10 @@ impl<T> Receiver<T> {
let me = crate::actor::current_pid()
.expect("recv() called outside an actor");
debug_assert!(
g.parked_receiver.is_none_or(|(p, _)| p == me),
g.parked_receiver.is_none(),
"channel has more than one receiver"
);
// begin_wait is lock-free — legal under the Channel lock;
// registering in the same critical section makes the epoch
// atomic with the senders' view of the registration.
g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
g.parked_receiver = Some(me);
crate::te!(crate::trace::Event::RecvPark(me));
}
// Release the lock before parking — the unparker will need it.
@@ -186,75 +140,6 @@ impl<T> Receiver<T> {
}
}
/// Bounded receive: like [`recv`](Self::recv), but gives up once
/// `timeout` has elapsed, returning [`RecvTimeoutError::Timeout`].
///
/// Built on the same timer machinery as `Mutex::lock_timeout`: the wait
/// registers a `WaitTimeout` entry stamped with the wait's park-epoch;
/// on expiry the channel (as the
/// [`TimerTarget`](crate::timer::TimerTarget)) checks whether *this*
/// wait is still parked and, only then, cancels it. A wake that races
/// the deadline resolves message-first: if a message is available when
/// the receiver runs, it is delivered even if the timer had already
/// fired. A satisfied or abandoned wait leaves its timer entry to expire
/// as a no-op (registration gone; epoch consumed), per the
/// no-cancellation convention in `timer.rs`.
///
/// The wake is classified from state alone — wakes are precise (the only
/// stamped wakers of this wait are a send, the last-sender drop, and the
/// timer; a stop wake unwinds out of `park_current` and never reaches
/// the classification), so: message queued → `Ok`; `senders == 0` →
/// `Disconnected`; neither → it was the timer → `Timeout`.
///
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-
/// due timer is drained, then reports `Timeout` unless a message was
/// already queued.
pub fn recv_timeout(&self, timeout: std::time::Duration) -> Result<T, RecvTimeoutError>
where
T: Send + 'static,
{
let me = crate::actor::current_pid()
.expect("recv_timeout() called outside an actor");
// Fast path + wait registration, one critical section.
let epoch;
{
let mut g = self.inner.lock();
if let Some(v) = g.queue.pop_front() {
return Ok(v);
}
if g.senders == 0 {
return Err(RecvTimeoutError::Disconnected);
}
debug_assert!(
g.parked_receiver.is_none_or(|(p, _)| p == me),
"channel has more than one receiver"
);
epoch = crate::scheduler::begin_wait();
g.parked_receiver = Some((me, epoch));
crate::te!(crate::trace::Event::RecvPark(me));
}
// Arm the timer after releasing the channel lock (insert takes the
// timers lock; never nest under a Channel lock). A send or even the
// timer itself may unpark us before we park — the RunningNotified
// protocol makes the park below return immediately in that case.
let deadline = crate::timer::deadline_from_now(timeout);
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = self.inner.clone();
crate::scheduler::insert_wait_timer(deadline, me, target, epoch);
crate::scheduler::park_current();
crate::te!(crate::trace::Event::RecvWake(crate::actor::current_pid().unwrap()));
let mut g = self.inner.lock();
if let Some(v) = g.queue.pop_front() {
return Ok(v);
}
if g.senders == 0 {
return Err(RecvTimeoutError::Disconnected);
}
Err(RecvTimeoutError::Timeout)
}
/// Selective receive: remove and return the first queued message for which
/// `pred` holds, leaving the rest in arrival order. If no queued message
/// matches, parks and re-scans on every send (a selective receiver may park
@@ -271,7 +156,7 @@ impl<T> Receiver<T> {
{
loop {
{
let mut g = self.inner.lock();
let mut g = self.inner.lock().unwrap();
if let Some(i) = g.queue.iter().position(|v| pred(v)) {
// position() found it, so remove() returns Some.
return Ok(g.queue.remove(i).unwrap());
@@ -283,10 +168,10 @@ impl<T> Receiver<T> {
let me = crate::actor::current_pid()
.expect("recv_match() called outside an actor");
debug_assert!(
g.parked_receiver.is_none_or(|(p, _)| p == me),
g.parked_receiver.is_none(),
"channel has more than one receiver"
);
g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
g.parked_receiver = Some(me);
crate::te!(crate::trace::Event::RecvPark(me));
}
// Release the lock before parking — the unparker will need it.
@@ -303,7 +188,7 @@ impl<T> Receiver<T> {
where
F: Fn(&T) -> bool,
{
let mut g = self.inner.lock();
let mut g = self.inner.lock().unwrap();
if let Some(i) = g.queue.iter().position(|v| pred(v)) {
return Ok(Some(g.queue.remove(i).unwrap()));
}
@@ -316,7 +201,7 @@ impl<T> Receiver<T> {
/// Non-blocking. `Ok(Some(v))` if a message was available, `Ok(None)` if
/// the channel is empty but open, `Err(RecvError)` if closed and drained.
pub fn try_recv(&self) -> Result<Option<T>, RecvError> {
let mut g = self.inner.lock();
let mut g = self.inner.lock().unwrap();
if let Some(v) = g.queue.pop_front() {
return Ok(Some(v));
}
@@ -326,205 +211,3 @@ impl<T> Receiver<T> {
Ok(None)
}
}
// ---------------------------------------------------------------------------
// TimerTarget — the expiry half of recv_timeout
// ---------------------------------------------------------------------------
impl<T: Send + 'static> crate::timer::TimerTarget for RawMutex<Inner<T>> {
fn on_timeout(&self, pid: Pid, epoch: u32) {
// Cancel the wait only if THIS wait (epoch match) is still
// registered. If a sender already took `parked_receiver`, the
// receiver is waking with a message — message wins, the timer
// no-ops. If a later wait by the same receiver is registered, the
// epoch mismatches — stale entry, no-op. (The unpark_at would fail
// its word CAS in either case anyway; checking under the lock keeps
// the registration bookkeeping exact.)
let unpark = {
let mut g = self.lock();
if g.parked_receiver == Some((pid, epoch)) {
g.parked_receiver = None;
true
} else {
false
}
};
// Unpark outside the channel lock — it may take the run-queue lock;
// legal under a Channel lock, but pointless to nest.
if unpark {
crate::scheduler::unpark_at(pid, epoch);
}
}
}
// ---------------------------------------------------------------------------
// select — ready-index wait over multiple receivers
// ---------------------------------------------------------------------------
mod sealed {
pub trait Sealed {}
}
impl<T> sealed::Sealed for Receiver<T> {}
/// An arm of a [`select`]. Implemented by [`Receiver`]; sealed, because the
/// registration contract below is part of the runtime's wake protocol.
///
/// Contract (all under the arm's own lock): `sel_register` checks-or-
/// registers atomically — if the arm is ready it does NOT register and
/// returns `false`; otherwise it publishes `(pid, epoch)` where its wakers
/// will find it. "Ready" means a receive would not park: a message is
/// queued, or the arm is closed.
pub trait Selectable: sealed::Sealed {
#[doc(hidden)]
fn sel_register(&self, pid: Pid, epoch: u32) -> bool;
#[doc(hidden)]
fn sel_ready(&self) -> bool;
}
impl<T> Selectable for Receiver<T> {
fn sel_register(&self, pid: Pid, epoch: u32) -> bool {
let mut g = self.inner.lock();
if !g.queue.is_empty() || g.senders == 0 {
return false;
}
debug_assert!(
g.parked_receiver.is_none_or(|(p, _)| p == pid),
"channel has more than one receiver"
);
g.parked_receiver = Some((pid, epoch));
true
}
fn sel_ready(&self) -> bool {
let g = self.inner.lock();
!g.queue.is_empty() || g.senders == 0
}
}
/// Park on every arm at once; return the index of the first ready one.
///
/// "Ready" means a receive on that arm would not park: a message is queued,
/// or the arm is **closed** (so the caller's `try_recv` observes the
/// disconnect — a dead arm is an event, not a hang). The caller consumes the
/// arm itself, typically via [`Receiver::try_recv`]; single-receiver
/// channels guarantee nothing can steal the message in between.
///
/// A closed arm stays ready *forever*: once its disconnect has been
/// observed, drop it from the arm set — under priority order it would
/// otherwise win every subsequent call and starve every higher-indexed arm.
///
/// Arms are scanned **in order**: index 0 is the highest priority, both on
/// the immediate-ready path and after a wake. This is a documented
/// guarantee (compose like BEAM receive clauses: put control channels
/// first), not an accident — and therefore there is NO fairness promise; a
/// saturated arm 0 starves arm 1 by design.
///
/// One actor may select on a channel and later `recv` on it (or select on
/// overlapping sets) freely. What stays illegal is what was always illegal:
/// two *different* actors receiving on one channel.
///
/// Built on the consuming-wake protocol (see slot_state.rs): all arms are
/// registered under one wait epoch; the winning wake consumes it, so losing
/// arms' registrations are inert and need no cancellation pass — they
/// self-clean at their wakers' failed CAS, or get overwritten by this
/// receiver's next wait on that channel.
///
/// Panics if `arms` is empty, or when called outside an actor.
pub fn select(arms: &[&dyn Selectable]) -> usize {
assert!(!arms.is_empty(), "select() on an empty arm list");
let me = crate::actor::current_pid().expect("select() called outside an actor");
loop {
let epoch = crate::scheduler::begin_wait();
if let Some(i) = register_arms(me, epoch, arms) {
return i;
}
crate::scheduler::park_current();
// Woken precisely: an arm's send (message) or last-sender drop
// (closure) consumed our epoch, and both leave their arm ready —
// return the first one, in priority order (which may be a
// different, higher-priority arm than the one that woke us; its
// message stays queued and re-reports ready on the next call).
for (i, arm) in arms.iter().enumerate() {
if arm.sel_ready() {
return i;
}
}
// Unreachable by protocol (a stop wake unwinds out of
// park_current). Defensive: re-open the wait and re-register —
// stale own-registrations are overwritten.
}
}
/// The registration pass shared by [`select`] and [`select_timeout`]:
/// check-or-register each arm, in priority order, each atomically under its
/// own lock. Cross-arm atomicity is unnecessary: an arm becoming ready
/// right after its registration wakes the caller through the protocol (the
/// prep-to-park window is closed by RunningNotified).
///
/// `Some(i)` = arm `i` was ready, the pass stopped, and the wait has been
/// RETIRED (no park may follow): earlier arms hold live-epoch
/// registrations, so the epoch is bumped, a landed notification eaten, and
/// a pending stop re-observed — without which a stale arm wake could fault
/// a later one-shot park. `None` = every arm registered; the caller parks.
fn register_arms(me: Pid, epoch: u32, arms: &[&dyn Selectable]) -> Option<usize> {
for (i, arm) in arms.iter().enumerate() {
if !arm.sel_register(me, epoch) {
crate::scheduler::retire_wait();
return Some(i);
}
}
None
}
/// The [`select_timeout`] timer target: stateless, because precise wakes
/// make classification a pure function of channel state. The entry is
/// stamped with the select's epoch; if an arm already won, this unpark dies
/// at the word's epoch CAS (the no-cancellation convention in `timer.rs`).
struct SelectTimeout;
impl crate::timer::TimerTarget for SelectTimeout {
fn on_timeout(&self, pid: Pid, epoch: u32) {
crate::scheduler::unpark_at(pid, epoch);
}
}
/// [`select`] with a deadline: returns `Some(index)` like `select`, or
/// `None` once `timeout` elapses with no arm ready.
///
/// All of `select`'s semantics carry over (priority order, closed arms
/// permanently ready, no fairness promise). The timeout is one more stamped
/// waker on the same wait epoch — nothing is registered in any arm for it,
/// so there is nothing to cancel or leak: an arm winning leaves the timer
/// entry to expire as a stale-epoch no-op; the timer winning leaves the
/// arms' registrations to self-clean exactly as a `select` loser's would.
///
/// The wake is classified from state alone (wakes are precise): some arm
/// ready → `Some` of the first, in priority order; none ready → the timer
/// was the only remaining stamped waker → `None`. A message that races the
/// deadline resolves message-first, as `recv_timeout` does.
///
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-due
/// timer is drained, then reports `None` unless an arm was already ready.
///
/// Panics if `arms` is empty, or when called outside an actor.
pub fn select_timeout(
arms: &[&dyn Selectable],
timeout: std::time::Duration,
) -> Option<usize> {
assert!(!arms.is_empty(), "select_timeout() on an empty arm list");
let me = crate::actor::current_pid().expect("select_timeout() called outside an actor");
let epoch = crate::scheduler::begin_wait();
if let Some(i) = register_arms(me, epoch, arms) {
return Some(i); // ready now: the timer was never armed
}
// Arm the timer after the registration pass, outside every Channel
// lock (insert takes the timers lock).
let deadline = crate::timer::deadline_from_now(timeout);
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = std::sync::Arc::new(SelectTimeout);
crate::scheduler::insert_wait_timer(deadline, me, target, epoch);
crate::scheduler::park_current();
// Woken precisely: an arm (ready below) or the timer (nothing ready).
arms.iter().position(|arm| arm.sel_ready())
}
src/context.rs
+11 -111
View File
@@ -1,114 +1,14 @@
//! Cooperative context switching, x86-64.
//! Cooperative context switching — public façade over the arch backend.
//!
//! Two naked-asm functions move execution between a scheduler thread and an
//! actor running on its own mmap'd stack. The compiler cannot do this; the
//! whole point of `#[unsafe(naked)]` is that we control every instruction.
//! The real implementation lives in `crate::arch`, selected per `target_arch`.
//! This module re-exports the stable surface (`init_actor_stack`, the two
//! `switch_to_*` shims, and the `*_actor_sp` accessors) so the scheduler,
//! `preempt`, and the `tests/context.rs` integration tests keep importing
//! `smarm::context::{...}` unchanged regardless of which ISA is built.
//!
//! `SCHEDULER_SP` and `ACTOR_SP` are thread-locals holding each side's saved
//! stack pointer. `init_actor_stack` builds the initial stack so that the
//! first `switch_to_actor` lands inside the entry function with `rsp % 16 == 8`
//! (the x86-64 ABI requirement at function entry).
//! See `crate::arch` for the contract every backend implements, and
//! `crate::arch::{x86_64, aarch64}` for the per-ISA register choreography.
use std::cell::Cell;
thread_local! {
static SCHEDULER_SP: Cell<usize> = const { Cell::new(0) };
static ACTOR_SP: Cell<usize> = const { Cell::new(0) };
}
fn get_scheduler_sp() -> usize { SCHEDULER_SP.with(|c| c.get()) }
fn set_scheduler_sp(v: usize) { SCHEDULER_SP.with(|c| c.set(v)) }
pub fn get_actor_sp() -> usize { ACTOR_SP.with(|c| c.get()) }
pub fn set_actor_sp(v: usize) { ACTOR_SP.with(|c| c.set(v)) }
// ---------------------------------------------------------------------------
// Initial stack layout
//
// We start from aligned_top = top & ~15, then bias by 8 and push seven 8-byte
// slots (downward). The first `switch_to_actor` pops r15..rbx and `ret`s —
// landing in `entry` with rsp % 16 == 8 (the x86-64 ABI state at the point
// just after a `call`, which is what `entry` is compiled to expect).
//
// Layout (high → low), relative to aligned_top = top & ~15. Note the entry
// slot is at aligned_top - 16, NOT aligned_top - 8: the function does
// `(top & ~15) - 8` and *then* a `-= 8` before the first write, so the first
// stored word lands at aligned_top - 16. Verified by single-stepping the
// `ret` under llmdbg: entry sits at an address with %16 == 0, so the post-ret
// rsp is %16 == 8.
//
// aligned_top - 8 : (unused padding; keeps entry's slot %16 == 0)
// aligned_top - 16 : entry ptr ← `ret` target. Post-ret: rsp % 16 == 8.
// aligned_top - 24 : rbx = 0
// aligned_top - 32 : rbp = 0
// aligned_top - 40 : r12 = 0
// aligned_top - 48 : r13 = 0
// aligned_top - 56 : r14 = 0
// aligned_top - 64 : r15 = 0 ← initial rsp
// ---------------------------------------------------------------------------
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
unsafe {
let mut sp = (top as usize & !15) - 8;
sp -= 8; (sp as *mut usize).write(entry as usize); // ret target
sp -= 8; (sp as *mut usize).write(0); // rbx
sp -= 8; (sp as *mut usize).write(0); // rbp
sp -= 8; (sp as *mut usize).write(0); // r12
sp -= 8; (sp as *mut usize).write(0); // r13
sp -= 8; (sp as *mut usize).write(0); // r14
sp -= 8; (sp as *mut usize).write(0); // r15
sp
}
}
// ---------------------------------------------------------------------------
// Context switch shims
//
// Each shim:
// 1. Pushes the six callee-saved integer registers.
// 2. Snaps rsp into rdi and calls the Rust helper that stores it.
// 3. Calls the Rust helper that returns the *other* side's saved rsp.
// 4. Moves that into rsp.
// 5. Pops the six registers and rets.
//
// XMM registers are NOT saved here. We rely on every yield happening through
// a Rust call site, which means the compiler has spilled any live XMM state
// to the stack before we get here. (This is the same argument the compiler
// uses internally — callee-saved regs are what survive a `call`, and the
// SysV AMD64 ABI says XMM0–15 are all caller-saved.) If we ever yield from
// a place that isn't a Rust call boundary, this assumption breaks.
// ---------------------------------------------------------------------------
#[unsafe(naked)]
unsafe extern "C" fn switch_to_actor_asm() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_sched_sp}",
"call {get_actor_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_sched_sp = sym set_scheduler_sp,
get_actor_sp = sym get_actor_sp,
);
}
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
pub unsafe fn switch_to_actor() {
unsafe { switch_to_actor_asm() };
}
#[unsafe(naked)]
pub unsafe extern "C" fn switch_to_scheduler() {
core::arch::naked_asm!(
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
"mov rdi, rsp",
"call {set_actor_sp}",
"call {get_sched_sp}",
"mov rsp, rax",
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
"ret",
set_actor_sp = sym set_actor_sp,
get_sched_sp = sym get_scheduler_sp,
);
}
pub use crate::arch::{
get_actor_sp, init_actor_stack, set_actor_sp, switch_to_actor, switch_to_scheduler,
};
src/gen_server.rs
+35 -258
View File
@@ -6,23 +6,14 @@
//! [`call`](ServerRef::call) (synchronous, returns a reply) or
//! [`cast`](ServerRef::cast) (fire-and-forget).
//!
//! ## One inbox, many arms
//! ## Why a single inbox
//!
//! Every call and cast travels the *same* inbox channel as an [`Envelope`]
//! (calls and casts are ordered relative to each other, like Erlang); the
//! server loop dispatches by variant. A `call` carries a freshly made
//! one-shot reply channel; the server sends the reply straight back down it.
//!
//! Out-of-band messages ride *separate* channels composed at the wait via
//! [`select`](crate::channel::select): info channels handed over at start
//! ([`ServerBuilder::with_info`]) are dispatched to
//! [`handle_info`](GenServer::handle_info). Arm priority is
//! **infos before inbox**, in declaration order — a hot inbox cannot starve
//! an out-of-band message; conversely a hot info channel CAN starve the
//! inbox, deliberately (system-message semantics). An info channel whose
//! senders are all gone is silently dropped from the arm set (per the
//! closed-arm-is-ready-forever rule on `select`); a closed *inbox* still
//! means graceful shutdown.
//! smarm has no `select` and no unified per-process mailbox (see the runtime
//! gotchas in `task.md`), so a server cannot wait on a "call channel" and a
//! "cast channel" simultaneously. Instead every message — call or cast —
//! travels the *same* inbox channel as an [`Envelope`], and the server loop
//! dispatches by variant. A `call` carries a freshly made one-shot reply
//! channel; the server sends the reply straight back down it.
//!
//! ## Server death
//!
@@ -45,12 +36,13 @@
//!
//! ## Not here (yet)
//!
//! No dynamic info subscription: the info-channel set is fixed at start.
//! Revisit against a real consumer. (Monitor `Down` forwarding is dynamic —
//! see `handle_down` — because monitors are inherently created at runtime.)
//! No `handle_info` and no call timeout. Both need machinery smarm currently
//! defers: a call timeout needs a per-`recv` deadline (cf. the deferred
//! `Signal::Timeout`), and `handle_info` needs the cross-channel mailbox merge
//! that selective receive deliberately left unmade. They are slated to land
//! together with timeouts.
use crate::channel::{channel, select, Receiver, Selectable, Sender};
use crate::monitor::{Down, Monitor};
use crate::channel::{channel, Receiver, Sender};
use crate::pid::Pid;
use crate::scheduler::{spawn, spawn_under};
@@ -66,17 +58,9 @@ pub trait GenServer: Send + 'static {
type Reply: Send + 'static;
/// Asynchronous request type (carried by [`ServerRef::cast`]).
type Cast: Send + 'static;
/// Out-of-band message type, delivered to [`handle_info`](Self::handle_info)
/// from the info channels registered at start
/// ([`ServerBuilder::with_info`]). Servers with several out-of-band
/// sources enum them up into one `Info`. Use `()` if unused.
type Info: Send + 'static;
/// Runs once inside the server actor before any message is handled. The
/// [`ServerCtx`] is the loop's one runtime hook: clone its [`Watcher`]
/// into the state here to be able to [`watch`](Watcher::watch) monitors
/// from any later handler.
fn init(&mut self, _ctx: &ServerCtx) {}
/// Runs once inside the server actor before any message is handled.
fn init(&mut self) {}
/// Handle a synchronous call and produce the reply sent back to the caller.
fn handle_call(&mut self, request: Self::Call) -> Self::Reply;
@@ -84,14 +68,6 @@ pub trait GenServer: Send + 'static {
/// Handle a fire-and-forget cast.
fn handle_cast(&mut self, request: Self::Cast);
/// Handle an out-of-band message from one of the info channels. Default:
/// drop it.
fn handle_info(&mut self, _info: Self::Info) {}
/// Handle a [`Down`] from a monitor handed to the loop via
/// [`Watcher::watch`]. Default: drop it.
fn handle_down(&mut self, _down: Down) {}
/// Runs as the server actor exits, on any exit path (see module docs).
fn terminate(&mut self) {}
}
@@ -124,19 +100,6 @@ pub enum CallError {
ServerDown,
}
/// Returned by [`ServerRef::call_timeout`].
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum CallTimeoutError {
/// The server was already gone, or died before replying.
ServerDown,
/// The deadline passed before a reply arrived. The request stays in the
/// server's inbox: it will still be *handled*, but the reply is discarded
/// (the abandoned reply channel's receiver is dropped, so the server's
/// reply send fails harmlessly). Erlang behaves the same way; design
/// idempotent calls accordingly.
Timeout,
}
/// Returned by [`ServerRef::cast`] when the server is no longer reachable.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum CastError {
@@ -161,33 +124,6 @@ impl<G: GenServer> ServerRef<G> {
reply_rx.recv().map_err(|_| CallError::ServerDown)
}
/// Bounded synchronous request-reply: like [`call`](Self::call), but
/// gives up after `timeout`, returning [`CallTimeoutError::Timeout`].
///
/// The roadmap sketched this as monitor + wait-reply-or-Down + demonitor;
/// that machinery is unnecessary here because server death is already
/// observable on the reply channel itself — the reply sender is dropped
/// as the server's stack unwinds, closing the channel and waking the
/// parked caller (see the module docs). So a bounded call is exactly
/// [`Receiver::recv_timeout`] on the reply channel: `Ok` on a reply,
/// `Disconnected` → [`CallTimeoutError::ServerDown`], `Timeout` →
/// [`CallTimeoutError::Timeout`]. Nothing is registered, so nothing can
/// leak on the timeout path by construction.
pub fn call_timeout(
&self,
request: G::Call,
timeout: std::time::Duration,
) -> Result<G::Reply, CallTimeoutError> {
let (reply_tx, reply_rx) = channel::<G::Reply>();
self.tx
.send(Envelope::Call(request, reply_tx))
.map_err(|_| CallTimeoutError::ServerDown)?;
reply_rx.recv_timeout(timeout).map_err(|e| match e {
crate::channel::RecvTimeoutError::Disconnected => CallTimeoutError::ServerDown,
crate::channel::RecvTimeoutError::Timeout => CallTimeoutError::Timeout,
})
}
/// Fire-and-forget request. Returns once the message is enqueued; does not
/// wait for the server to handle it. [`CastError::ServerDown`] if the inbox
/// is already closed.
@@ -198,109 +134,24 @@ impl<G: GenServer> ServerRef<G> {
}
}
/// The server loop's runtime hook, passed to [`GenServer::init`]. Currently
/// carries only the [`Watcher`]; opaque so fields can grow without breaking.
pub struct ServerCtx {
watcher: Watcher,
}
impl ServerCtx {
/// A clonable handle to the loop's monitor intake. Store it in the state
/// during `init` to watch monitors from later handlers.
pub fn watcher(&self) -> Watcher {
self.watcher.clone()
}
/// Shorthand for `ctx.watcher().watch(m)` when watching during `init`.
pub fn watch(&self, m: Monitor) {
self.watcher.watch(m)
}
}
/// Hands [`Monitor`]s to a server loop; their [`Down`]s are dispatched to
/// [`GenServer::handle_down`]. Down arms outrank every other arm (a death
/// notice cannot be starved), and a delivered `Down` retires its arm —
/// monitors are one-shot.
#[derive(Clone)]
pub struct Watcher {
tx: Sender<Monitor>,
}
impl Watcher {
/// Transfer `m` to the server loop. If the server is already gone the
/// monitor is silently dropped (its queued `Down`, if any, with it) —
/// there is no loop left to care.
pub fn watch(&self, m: Monitor) {
let _ = self.tx.send(m);
}
}
/// Configure-then-start construction for servers that need more than a bare
/// [`start`]: info channels, a supervisor. Consumed by [`start`](Self::start).
///
/// ```ignore
/// let server = ServerBuilder::new(state)
/// .with_info(events_rx)
/// .under(supervisor_pid)
/// .start();
/// ```
pub struct ServerBuilder<G: GenServer> {
state: G,
infos: Vec<Receiver<G::Info>>,
supervisor: Option<Pid>,
}
impl<G: GenServer> ServerBuilder<G> {
pub fn new(state: G) -> Self {
ServerBuilder { state, infos: Vec::new(), supervisor: None }
}
/// Add an out-of-band channel; messages arriving on it are dispatched to
/// [`GenServer::handle_info`]. Repeatable; arm priority is call order
/// (earlier = higher), and every info channel outranks the inbox.
pub fn with_info(mut self, rx: Receiver<G::Info>) -> Self {
self.infos.push(rx);
self
}
/// Spawn the server under an explicit supervisor pid (via [`spawn_under`])
/// so it slots into the supervision tree.
pub fn under(mut self, supervisor: Pid) -> Self {
self.supervisor = Some(supervisor);
self
}
/// Spawn the server actor and hand back its [`ServerRef`]. The server's
/// lifetime is governed by its refs, not by joining, so the backing join
/// handle is dropped.
pub fn start(self) -> ServerRef<G> {
let (tx, rx) = channel::<Envelope<G>>();
let ServerBuilder { state, infos, supervisor } = self;
let handle = match supervisor {
Some(sup) => spawn_under(sup, move || server_loop::<G>(rx, state, infos)),
None => spawn(move || server_loop::<G>(rx, state, infos)),
};
ServerRef { tx, pid: handle.pid() }
}
}
/// Spawn `state` as a server under the current actor (via [`spawn`]). Returns a
/// [`ServerRef`]. Shorthand for `ServerBuilder::new(state).start()`.
/// [`ServerRef`]; the server's lifetime is governed by its refs, not by
/// joining, so the backing join handle is dropped.
pub fn start<G: GenServer>(state: G) -> ServerRef<G> {
ServerBuilder::new(state).start()
let (tx, rx) = channel::<Envelope<G>>();
let handle = spawn(move || server_loop::<G>(rx, state));
ServerRef { tx, pid: handle.pid() }
}
/// Like [`start`], but spawns the server under an explicit supervisor pid (via
/// [`spawn_under`]) so it slots into the supervision tree.
pub fn start_under<G: GenServer>(supervisor: Pid, state: G) -> ServerRef<G> {
ServerBuilder::new(state).under(supervisor).start()
let (tx, rx) = channel::<Envelope<G>>();
let handle = spawn_under(supervisor, move || server_loop::<G>(rx, state));
ServerRef { tx, pid: handle.pid() }
}
fn server_loop<G: GenServer>(
rx: Receiver<Envelope<G>>,
state: G,
mut infos: Vec<Receiver<G::Info>>,
) {
fn server_loop<G: GenServer>(rx: Receiver<Envelope<G>>, state: G) {
// terminate() must run on every exit path (clean close, panic, stop), so it
// lives in this guard's Drop rather than after the loop.
struct Terminate<G: GenServer>(G);
@@ -310,96 +161,22 @@ fn server_loop<G: GenServer>(
}
}
fn dispatch<G: GenServer>(state: &mut G, env: Envelope<G>) {
match env {
Envelope::Call(request, reply_tx) => {
let reply = state.handle_call(request);
let mut guard = Terminate(state);
guard.0.init();
loop {
match rx.recv() {
Ok(Envelope::Call(request, reply_tx)) => {
let reply = guard.0.handle_call(request);
// The caller may have gone away (e.g. cancelled while parked);
// a failed reply send is not the server's problem.
let _ = reply_tx.send(reply);
}
Envelope::Cast(request) => state.handle_cast(request),
Ok(Envelope::Cast(request)) => guard.0.handle_cast(request),
// All ServerRefs dropped → inbox closed → graceful shutdown.
Err(_) => break,
}
}
let mut guard = Terminate(state);
// The control arm: Watchers feed Monitors to the loop through it. The
// ctx (and with it the loop's own sender) drops right after init — a
// state that didn't clone the Watcher closes the arm, the first select
// observes the closure, and the loop falls back to the plain-inbox park.
let (watch_tx, watch_rx) = channel::<Monitor>();
guard.0.init(&ServerCtx { watcher: Watcher { tx: watch_tx } });
let mut monitors: Vec<Monitor> = Vec::new();
let mut watch_open = true;
loop {
if monitors.is_empty() && !watch_open && infos.is_empty() {
// Nothing to select over: park on the inbox alone, exactly the
// pre-v0.8 loop.
match rx.recv() {
Ok(env) => dispatch(&mut guard.0, env),
// All ServerRefs dropped → inbox closed → graceful shutdown.
Err(_) => break,
}
} else {
// Arm priority: downs, then the control arm, then infos (each in
// declaration order), then the inbox. The arm slice is rebuilt
// per iteration because every set but the inbox shrinks or grows.
let nd = monitors.len();
let nw = watch_open as usize;
let i = {
let mut arms: Vec<&dyn Selectable> =
Vec::with_capacity(nd + nw + infos.len() + 1);
for m in &monitors {
arms.push(&m.rx);
}
if watch_open {
arms.push(&watch_rx);
}
for r in &infos {
arms.push(r);
}
arms.push(&rx);
select(&arms)
};
if i < nd {
// A Down retires its arm either way: delivered (one-shot) or
// closed without delivering (defensive; shouldn't happen).
let m = monitors.remove(i);
if let Ok(Some(down)) = m.rx.try_recv() {
guard.0.handle_down(down);
}
} else if i < nd + nw {
match watch_rx.try_recv() {
Ok(Some(m)) => monitors.push(m),
// Single-receiver: nothing can drain the arm between
// select's ready and our try_recv.
Ok(None) => debug_assert!(false, "ready control arm was empty"),
// Every Watcher gone: stop selecting on the arm.
Err(_) => watch_open = false,
}
} else if i < nd + nw + infos.len() {
let j = i - nd - nw;
match infos[j].try_recv() {
Ok(Some(info)) => guard.0.handle_info(info),
Ok(None) => debug_assert!(false, "ready info arm was empty"),
// Senders all gone: drop the arm, keep serving. `remove`
// (not swap_remove) — order is priority.
Err(_) => {
infos.remove(j);
}
}
} else {
match rx.try_recv() {
Ok(Some(env)) => dispatch(&mut guard.0, env),
Ok(None) => debug_assert!(false, "ready inbox was empty"),
Err(_) => break,
}
}
}
// Observation point so a server whose arms are never empty stays
// Observation point so a server whose inbox is never empty stays
// preemptible and cancellable.
crate::check!();
}
src/io.rs
+16 -22
View File
@@ -44,13 +44,11 @@
//!
//! Fd hygiene
//! ==========
//! An actor stopped while waiting on an fd unwinds out of `wait_fd`'s park;
//! a drop guard there (armed after a successful register, forgotten on a
//! normal wake) removes the `waiters` entry iff it is still that wait's
//! `(pid, epoch)` and only then `EPOLL_CTL_DEL`s the fd — an entry already
//! consumed by a racing `FdReady` means the fd may carry someone else's
//! fresh registration, which must be left alone. `epoll_register` keeps a
//! defensive bare DEL before ADD as belt-and-braces.
//! If an actor dies while waiting on an fd, the registration is leaked
//! (the fd stays in the epollfd, armed). EPOLLONESHOT bounds the damage:
//! at most one stale wakeup, after which the kernel disarms. The stale
//! wakeup hits a dead pid in `waiters` and is dropped. Acceptable for v0.2;
//! a future pass should DEL on actor death.
//!
//! Buffers used with `read`/`write` should be on fds opened with
//! `O_NONBLOCK`. If they aren't, the syscall may block the scheduler
@@ -86,9 +84,6 @@ use std::thread::JoinHandle as OsJoinHandle;
pub type IoResult = Result<Box<dyn Any + Send>, Box<dyn Any + Send>>;
struct Request {
/// The submitter's park-epoch — carried through to the `Blocking`
/// completion so the wake is epoch-matched.
epoch: u32,
pid: Pid,
/// The work to perform. Returns the wire-form result directly.
work: Box<dyn FnOnce() -> IoResult + Send>,
@@ -98,7 +93,7 @@ struct Request {
pub enum Completion {
/// A `block_on_io` closure has finished (Ok = return value, Err = panic
/// payload).
Blocking { pid: Pid, epoch: u32, result: IoResult },
Blocking { pid: Pid, result: IoResult },
/// An fd registered via `wait_readable`/`wait_writable` is ready. The
/// scheduler looks up the parked pid in `waiters`, unparks it, and
/// removes the entry. `pid` isn't in this variant because the epoll
@@ -136,7 +131,7 @@ pub struct IoThread {
/// One parked actor per registered fd. Populated by `wait_readable` /
/// `wait_writable` and drained by the scheduler when a `FdReady`
/// completion is processed.
pub waiters: HashMap<RawFd, (Pid, u32)>,
pub waiters: HashMap<RawFd, Pid>,
// ----- Threads -----
@@ -236,12 +231,12 @@ impl IoThread {
}
/// Hand a request to the pool. Increments `outstanding`.
pub fn submit(&mut self, pid: Pid, epoch: u32, work: Box<dyn FnOnce() -> IoResult + Send>) {
pub fn submit(&mut self, pid: Pid, work: Box<dyn FnOnce() -> IoResult + Send>) {
self.outstanding += 1;
// Send can only fail if the pool has hung up, which only happens
// on shutdown. submit during shutdown is a bug.
self.tx
.send(Request { pid, epoch, work })
.send(Request { pid, work })
.expect("io pool hung up unexpectedly");
}
@@ -270,7 +265,6 @@ impl IoThread {
&mut self,
fd: RawFd,
pid: Pid,
epoch: u32,
readable: bool,
writable: bool,
) -> io::Result<()> {
@@ -284,10 +278,10 @@ impl IoThread {
));
}
// Belt-and-braces: the unwind guard in `wait_fd` is responsible for
// cleaning up a stopped waiter's registration, but a bare DEL is
// harmless if the fd isn't registered (ENOENT) and removes any leak
// a path we haven't thought of might leave behind.
// Defensive cleanup: if a previous actor died while waiting on this
// fd, the kernel-side registration was leaked (we don't walk all
// waiters on actor death). A bare DEL is harmless if the fd isn't
// registered (ENOENT), and removes any leak.
unsafe {
libc::epoll_ctl(self.epollfd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut());
}
@@ -309,7 +303,7 @@ impl IoThread {
if r < 0 {
return Err(io::Error::last_os_error());
}
self.waiters.insert(fd, (pid, epoch));
self.waiters.insert(fd, pid);
Ok(())
}
@@ -375,7 +369,7 @@ fn pool_loop(
completions: Arc<Mutex<VecDeque<Completion>>>,
wake_write: RawFd,
) {
while let Ok(Request { pid, epoch, work }) = rx.recv() {
while let Ok(Request { pid, work }) = rx.recv() {
let result: IoResult = match panic::catch_unwind(panic::AssertUnwindSafe(work)) {
Ok(r) => r,
Err(payload) => Err(payload),
@@ -383,7 +377,7 @@ fn pool_loop(
completions
.lock()
.unwrap()
.push_back(Completion::Blocking { pid, epoch, result });
.push_back(Completion::Blocking { pid, result });
wake_scheduler(wake_write);
}
}
src/lib.rs
+3 -9
View File
@@ -11,6 +11,7 @@
//!
//! See `LOOM.md` for the design intent and the deferred-for-later list.
pub mod arch;
pub mod stack;
pub mod context;
pub mod preempt;
@@ -23,15 +24,9 @@ pub mod timer;
pub mod io;
pub mod mutex;
pub mod monitor;
pub mod registry;
pub mod link;
pub mod gen_server;
pub mod runtime;
pub(crate) mod raw_mutex;
pub(crate) mod slot_state;
pub(crate) mod sync_shim;
#[doc(hidden)] // pub only so benches/rq_micro.rs can drive the raw structures
pub mod run_queue;
pub mod trace;
// ---------------------------------------------------------------------------
@@ -45,13 +40,12 @@ static ALLOCATOR: preempt::PreemptingAllocator = preempt::PreemptingAllocator;
// Public API re-exports
// ---------------------------------------------------------------------------
pub use channel::{channel, select, select_timeout, Receiver, RecvError, RecvTimeoutError, Selectable, Sender};
pub use gen_server::{CallError, CallTimeoutError, CastError, GenServer, ServerBuilder, ServerCtx, ServerRef, Watcher};
pub use channel::{channel, Receiver, RecvError, Sender};
pub use gen_server::{CallError, CastError, GenServer, ServerRef};
pub use link::{link, trap_exit, unlink, ExitSignal};
pub use monitor::{demonitor, monitor, Down, DownReason, Monitor, MonitorId};
pub use mutex::{LockTimeout, Mutex, MutexGuard};
pub use pid::Pid;
pub use registry::{name_of, register, unregister, whereis, RegisterError};
pub use runtime::{init, Config, Runtime};
pub use scheduler::{
block_on_io, request_stop, run, self_pid, sleep, spawn, spawn_under, wait_readable,
src/link.rs
+44 -66
View File
@@ -49,9 +49,10 @@
//! [`Down`]: crate::monitor::Down
//! [`request_stop`]: crate::scheduler::request_stop
use crate::channel::{channel, Receiver};
use crate::channel::{channel, Receiver, Sender};
use crate::monitor::DownReason;
use crate::pid::Pid;
use crate::runtime::State;
use crate::scheduler::{request_stop, self_pid, with_runtime};
/// A linked peer's death, delivered to a trapping actor's inbox.
@@ -77,13 +78,11 @@ pub fn trap_exit() -> Receiver<ExitSignal> {
let (tx, rx) = channel::<ExitSignal>();
let me = self_pid();
with_runtime(|inner| {
if let Some(slot) = inner.slot_at(me) {
let mut cold = slot.cold.lock();
// Own slot: generation is necessarily current (we're running).
if let Some(actor) = cold.actor.as_mut() {
inner.with_shared(|s| {
if let Some(actor) = s.slot_mut(me).and_then(|slot| slot.actor.as_mut()) {
actor.trap = Some(tx);
}
}
})
});
rx
}
@@ -101,61 +100,45 @@ pub fn link(target: Pid) {
return;
}
// Cold locks are leaves: never hold two at once. The link is recorded one
// side at a time, TARGET FIRST — that ordering is what makes the race
// window sound:
//
// - Once `me` is in `target.links`, the target's death always reaches us
// (its finalize cascade walks that list). So after step 1 succeeds, the
// link semantics are already live.
// - If the target dies between step 1 and step 2, its cascade removes
// `target` from OUR links (a no-op, we haven't added it yet) and
// delivers the exit signal — correct, the link was established. Our
// subsequent step-2 insert leaves a stale `target` entry in `me.links`;
// stale entries are benign by construction (every cascade walk
// re-verifies the peer's word; `unlink` removes them like any other).
//
// The reverse order would be unsound: target dying in the window would
// walk its links WITHOUT us — a silently dead link that we believe is live.
let registered_on_target = with_runtime(|inner| match inner.slot_at(target) {
Some(slot) => {
let mut cold = slot.cold.lock();
if slot.is_live_for(target) && cold.actor.is_some() {
if !cold.links.contains(&me) {
cold.links.push(me);
// Under the lock: if the target is live, record the link both ways and
// return `None`. If it is gone, return the caller's trap sender (if any)
// so we can deliver the NoProc signal after releasing the lock.
let dead_action: Option<Option<Sender<ExitSignal>>> = with_runtime(|inner| {
inner.with_shared(|s| {
let target_live = matches!(
s.slot(target),
Some(slot) if slot.actor.is_some() && !matches!(slot.state, State::Done)
);
if target_live {
if let Some(slot) = s.slot_mut(me) {
if !slot.links.contains(&target) {
slot.links.push(target);
}
}
true
if let Some(slot) = s.slot_mut(target) {
if !slot.links.contains(&me) {
slot.links.push(me);
}
}
None
} else {
false
// Grab our own trap sender so the NoProc delivery (below)
// doesn't need a second lock acquisition.
Some(
s.slot(me)
.and_then(|slot| slot.actor.as_ref())
.and_then(|a| a.trap.clone()),
)
}
}
None => false,
});
if registered_on_target {
with_runtime(|inner| {
let slot = inner.slot_at(me).expect("link: own slot vanished");
let mut cold = slot.cold.lock();
if !cold.links.contains(&target) {
cold.links.push(target);
}
});
return;
}
// Target already gone: deliver NoProc to ourselves — as a message if
// trapping, otherwise as a cooperative stop.
let my_trap = with_runtime(|inner| {
inner.slot_at(me).and_then(|slot| {
let cold = slot.cold.lock();
cold.actor.as_ref().and_then(|a| a.trap.clone())
})
});
match my_trap {
Some(tx) => {
match dead_action {
None => {} // linked successfully
Some(Some(tx)) => {
let _ = tx.send(ExitSignal { from: target, reason: DownReason::NoProc });
}
None => request_stop(me),
Some(None) => request_stop(me),
}
}
@@ -169,18 +152,13 @@ pub fn unlink(target: Pid) {
return;
}
with_runtime(|inner| {
// One cold lock at a time (leaf rule). Order is immaterial here:
// a half-removed link is just a stale entry on one side, and stale
// entries are benign (re-verified on every cascade walk).
if let Some(slot) = inner.slot_at(me) {
let mut cold = slot.cold.lock();
cold.links.retain(|p| *p != target);
}
if let Some(slot) = inner.slot_at(target) {
let mut cold = slot.cold.lock();
if slot.generation() == target.generation() {
cold.links.retain(|p| *p != me);
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(me) {
slot.links.retain(|p| *p != target);
}
}
if let Some(slot) = s.slot_mut(target) {
slot.links.retain(|p| *p != me);
}
})
});
}
src/monitor.rs
+25 -28
View File
@@ -47,6 +47,7 @@
use crate::channel::{channel, Receiver, Sender};
use crate::pid::Pid;
use crate::runtime::State;
use crate::scheduler::with_runtime;
/// Why a monitored actor went down.
@@ -109,26 +110,24 @@ pub struct Monitor {
pub fn monitor(target: Pid) -> Monitor {
let (tx, rx) = channel::<Down>();
// Register under the target's cold lock. `tx.clone()` takes the channel's
// own lock — a Channel-class RawMutex, explicitly permitted *under* a Leaf
// (cold) lock by the lock order (see raw_mutex.rs). We must still not
// *send* under the lock, as `Sender::send` can unpark a parked receiver,
// and there's no reason to nest that.
// Register under the shared lock. We allocate the id and (if the target is
// live) clone the sender into its monitor list, keeping the original `tx`
// for the NoProc fallback. `tx.clone()` only touches the channel's own
// mutex, never the shared runtime mutex, so it is safe under the lock — but
// we must not *send* here, as `Sender::send` can call back in to unpark a
// parked receiver and the shared mutex is not reentrant.
let (id, registered) = with_runtime(|inner| {
let id = inner.alloc_monitor_id();
let registered = match inner.slot_at(target) {
Some(slot) => {
let mut cold = slot.cold.lock();
if slot.is_live_for(target) {
cold.monitors.push((id, tx.clone()));
inner.with_shared(|s| {
let id = s.alloc_monitor_id();
let registered = match s.slot_mut(target) {
Some(slot) if !matches!(slot.state, State::Done) => {
slot.monitors.push((id, tx.clone()));
true
} else {
false
}
}
None => false,
};
(id, registered)
_ => false,
};
(id, registered)
})
});
if !registered {
@@ -148,18 +147,16 @@ pub fn monitor(target: Pid) -> Monitor {
/// dropping the [`Monitor`] closes its receiver and the queued notice goes with
/// it — the analogue of Erlang's `demonitor(Ref, [flush])`.
pub fn demonitor(m: &Monitor) -> Option<MonitorId> {
// Remove the registration under the target's cold lock, but move the
// `Sender` *out* and let it drop only after the lock is released:
// dropping the last sender runs `Sender::drop`, which may unpark a parked
// receiver — legal under a cold lock, but pointless to nest.
// Remove the registration under the lock, but move the `Sender` *out* and
// let it drop only after the lock is released: dropping the last sender
// runs `Sender::drop`, which may unpark a parked receiver → `with_shared`,
// and the shared mutex is not reentrant.
let removed: Option<(MonitorId, Sender<Down>)> = with_runtime(|inner| {
let slot = inner.slot_at(m.target)?;
let mut cold = slot.cold.lock();
if slot.generation() != m.target.generation() {
return None; // slot reused; the Down already fired
}
let pos = cold.monitors.iter().position(|(mid, _)| *mid == m.id)?;
Some(cold.monitors.remove(pos))
inner.with_shared(|s| {
let slot = s.slot_mut(m.target)?;
let pos = slot.monitors.iter().position(|(mid, _)| *mid == m.id)?;
Some(slot.monitors.remove(pos))
})
});
// `removed`'s sender drops here, outside the lock.
removed.map(|(id, _sender)| id)
src/mutex.rs
+18 -24
View File
@@ -33,15 +33,13 @@ impl std::error::Error for LockTimeout {}
struct Wait {
pid: Pid,
/// The wait's park-epoch (slot-word wait identity, see slot_state.rs).
/// Grants and timeouts wake via `unpark_at(pid, epoch)`; a stale entry
/// can neither be granted by mistake nor wake the wrong wait.
epoch: u32,
seq: u64,
}
struct MutexState {
holder: Option<Pid>,
waiters: VecDeque<Wait>,
next_seq: u64,
default_timeout: Duration,
}
@@ -55,6 +53,7 @@ impl MutexCore {
state: StdMutex::new(MutexState {
holder: None,
waiters: VecDeque::new(),
next_seq: 0,
default_timeout,
}),
}
@@ -62,17 +61,17 @@ impl MutexCore {
}
impl TimerTarget for MutexCore {
fn on_timeout(&self, pid: Pid, epoch: u32) {
fn on_timeout(&self, pid: Pid, wait_seq: u64) {
let unpark = {
let mut st = self.state.lock().unwrap();
// Remove from waiters only if still there with matching epoch.
// Remove from waiters only if still there with matching seq.
// If the lock was already granted (holder == Some(pid)), the
// timer fired after the grant — treat as no-op; the actor
// will see `is_holder == true` and return Ok.
if st.holder == Some(pid) {
return;
}
let pos = st.waiters.iter().position(|w| w.pid == pid && w.epoch == epoch);
let pos = st.waiters.iter().position(|w| w.pid == pid && w.seq == wait_seq);
if pos.is_some() {
st.waiters.remove(pos.unwrap());
true
@@ -81,7 +80,7 @@ impl TimerTarget for MutexCore {
}
};
if unpark {
scheduler::unpark_at(pid, epoch);
scheduler::unpark(pid);
}
}
}
@@ -134,25 +133,20 @@ impl<T> Mutex<T> {
// Slow path: register as a waiter, set timeout, park.
let _np = scheduler::NoPreempt::enter();
let epoch = {
let seq = {
let mut st = self.core.state.lock().unwrap();
// begin_wait is lock-free — legal under the state lock; this
// makes the epoch atomic with the registration's visibility to
// grants and timeouts.
let epoch = scheduler::begin_wait();
st.waiters.push_back(Wait { pid: me, epoch });
epoch
let seq = st.next_seq;
st.next_seq = st.next_seq.wrapping_add(1);
st.waiters.push_back(Wait { pid: me, seq });
seq
};
let target: Arc<dyn TimerTarget> = self.core.clone();
let deadline = timer::deadline_from_now(timeout);
scheduler::insert_wait_timer(deadline, me, target, epoch);
scheduler::insert_wait_timer(deadline, me, target, seq);
scheduler::park_current();
// Resumed — precisely: only our grant or our timer can wake this
// wait (both epoch-stamped; a stop wake unwinds out of
// park_current). The one-shot interpretation below is therefore
// exhaustive. Are we the holder?
// Resumed. Are we the holder?
let is_holder = self.core.state.lock().unwrap().holder == Some(me);
if is_holder {
let value = self.value.lock().unwrap().take()
@@ -234,12 +228,12 @@ impl<T> Drop for MutexGuard<'_, T> {
let v = self.value.take().expect("MutexGuard: double drop");
*self.mutex.value.lock().unwrap() = Some(v);
let next = {
let next_pid = {
let mut st = self.mutex.core.state.lock().unwrap();
match st.waiters.pop_front() {
Some(w) => {
st.holder = Some(w.pid);
Some((w.pid, w.epoch))
Some(w.pid)
}
None => {
st.holder = None;
@@ -247,8 +241,8 @@ impl<T> Drop for MutexGuard<'_, T> {
}
}
};
if let Some((pid, epoch)) = next {
scheduler::unpark_at(pid, epoch);
if let Some(pid) = next_pid {
scheduler::unpark(pid);
}
}
}
src/preempt.rs
+18 -21
View File
@@ -30,7 +30,17 @@ use std::cell::Cell;
use std::sync::atomic::{AtomicBool, Ordering};
pub const DEFAULT_ALLOC_INTERVAL: u32 = 128;
// The timeslice is measured in `rdtsc()` ticks, and the tick *unit* differs by
// ISA (see `crate::arch::read_cycle_counter`). Both constants target ~100µs.
// x86-64 : TSC ≈ CPU base clock; 300_000 ticks ≈ 100µs on a 3 GHz core.
// aarch64: CNTVCT runs at CNTFRQ_EL0, commonly ~24 MHz; 2_400 ticks ≈ 100µs.
// A real implementation would read the frequency at startup and compute this;
// for now the constant is calibrated per-arch. Tune on-device if needed.
#[cfg(target_arch = "x86_64")]
pub const DEFAULT_TIMESLICE_CYCLES: u64 = 300_000; // ≈ 100µs on a 3 GHz CPU
#[cfg(target_arch = "aarch64")]
pub const DEFAULT_TIMESLICE_CYCLES: u64 = 2_400; // ≈ 100µs at a 24 MHz CNTVCT
thread_local! {
/// While `false`, the allocator hook is a no-op.
@@ -116,14 +126,12 @@ pub fn reset_timeslice() {
TIMESLICE_START.with(|c| c.set(rdtsc()));
}
/// Per-core cycle counter for the timeslice clock. Delegates to the active
/// arch backend (`rdtsc` on x86-64, `CNTVCT_EL0` on aarch64). The tick unit
/// is ISA-defined, so `DEFAULT_TIMESLICE_CYCLES` is calibrated per-arch below.
#[inline(always)]
pub fn rdtsc() -> u64 {
unsafe {
// SAFETY: x86-64 only. `lfence` serialises the instruction stream so
// we don't measure time before prior instructions retire.
core::arch::asm!("lfence", options(nostack, nomem, preserves_flags));
core::arch::x86_64::_rdtsc()
}
crate::arch::read_cycle_counter()
}
pub struct PreemptingAllocator;
@@ -171,22 +179,11 @@ pub fn maybe_preempt() {
let n = c.get();
if n == 0 {
c.set(CONFIGURED_ALLOC_INTERVAL.with(|i| i.get()));
// Cooperative cancellation shares the amortised cadence with the
// timeslice check. Observe a pending stop first: if we are being
// cancelled there is no point yielding, we unwind instead.
check_cancelled();
if PREEMPTION_ENABLED.with(|e| e.get()) {
// Cooperative cancellation shares the amortised cadence with
// the timeslice check, and shares its gate: while preemption
// is disabled (`NoPreempt`, `with_shared`, channel critical
// sections) the stop sentinel must NOT be raised, because an
// allocation-triggered unwind inside a region holding a
// `std::sync::Mutex` would poison it — one `request_stop` at
// the wrong moment would then cascade `lock().unwrap()`
// panics through every later user of that lock. Observation
// is merely deferred to the next enabled allocation or the
// wakeup side of the next park/yield, both of which are
// lock-free points by construction.
//
// Observe a pending stop first: if we are being cancelled
// there is no point yielding, we unwind instead.
check_cancelled();
let start = TIMESLICE_START.with(|s| s.get());
if rdtsc().saturating_sub(start) > CONFIGURED_TIMESLICE_CYCLES.with(|t| t.get()) {
// SAFETY: reachable only inside an actor (the scheduler
src/raw_mutex.rs
-360
View File
@@ -1,360 +0,0 @@
//! A minimal futex-based mutex that cannot poison.
//!
//! `std::sync::Mutex` poisons on unwind, turning one panic into a cascade of
//! `lock().unwrap()` panics in every later user. The runtime's internal
//! critical sections must never unwind anyway (the stop sentinel is gated
//! behind `PREEMPTION_ENABLED`, and the guard below disables preemption), so
//! poisoning buys nothing and costs a failure mode. This mutex has no poison
//! state by construction.
//!
//! Two further properties the runtime wants:
//!
//! - **The guard enters `NoPreempt`.** A timeslice switch while holding an OS
//! mutex would suspend the actor with the lock held, stalling every other
//! OS thread that touches it until the actor is resumed. Disabling
//! preemption for the (short) critical section keeps lock hold times
//! bounded. It also closes the unwind hole structurally: with
//! `PREEMPTION_ENABLED` false, `maybe_preempt` neither yields nor raises
//! the stop sentinel, so no allocation inside the critical section can
//! unwind it.
//! - **No std machinery.** One `AtomicU32` and two futex syscalls; friendlier
//! to an eventual embedded port than `std::sync::Mutex` (swap the futex for
//! a spin or WFE backend).
//!
//! Algorithm: the classic three-state futex mutex (Drepper, "Futexes Are
//! Tricky", mutex3). 0 = unlocked, 1 = locked, 2 = locked with (possible)
//! waiters. Uncontended lock/unlock is one CAS / one swap, no syscall.
//!
//! Lock-order position — two classes (see [`LockClass`]):
//!
//! - **Leaf**: slot cold locks, the free list, the stack pool, the name
//! registry. Mutual leaves — never hold two at once.
//! - **Channel**: a channel's internal lock. May be acquired *under* a Leaf
//! (finalize clones the supervisor/trap senders, and `monitor()` clones the
//! Down sender, all under a cold lock — structural, the sender lives in the
//! slot), but nothing may be acquired under a Channel lock: channel
//! critical sections call only the lock-free unpark protocol.
//!
//! So the total order is Leaf → Channel, one of each at most. Holding either
//! while pushing to the run queue is permitted (unpark from inside a cold or
//! channel section); the reverse — taking any `RawMutex` from inside a
//! run-queue op — cannot arise (queue ops call nothing).
use std::cell::UnsafeCell;
use std::ops::{Deref, DerefMut};
use std::sync::atomic::{AtomicU32, Ordering};
const UNLOCKED: u32 = 0;
const LOCKED: u32 = 1;
const CONTENDED: u32 = 2;
/// How many `pause` spins to burn before falling back to the futex. Critical
/// sections under this lock are tens of nanoseconds (push to a Vec, clone a
/// sender), so a short spin almost always avoids the syscall.
const SPIN_LIMIT: u32 = 64;
/// Which rung of the two-rung lock order a `RawMutex` occupies. Debug builds
/// enforce the order mechanically (see the module docs); release builds carry
/// no state and no checks.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum LockClass {
/// Runtime cold data: slot cold locks, free list, stack pool, registry.
/// Mutual leaves among themselves; a Channel lock may be taken under one.
Leaf,
/// A channel's internal lock. One at a time, nothing acquired under it;
/// may itself be acquired under a Leaf.
Channel,
}
// The ordering rules, mechanically enforced (debug builds): a deadlock from a
// violated order is a hang waiting for the right interleaving, so fail at the
// acquisition that violates it, not in the eventual hang.
#[cfg(debug_assertions)]
thread_local! {
static LEAVES_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
static CHANNELS_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
}
#[inline]
fn order_check_acquire(class: LockClass) {
#[cfg(debug_assertions)]
match class {
LockClass::Leaf => LEAVES_HELD.with(|l| {
debug_assert_eq!(
l.get(),
0,
"lock order violated: acquiring a Leaf RawMutex while already \
holding one (cold locks / free list / stack pool / registry \
are mutual leaves)"
);
CHANNELS_HELD.with(|c| {
debug_assert_eq!(
c.get(),
0,
"lock order violated: acquiring a Leaf RawMutex under a \
channel lock (order is Leaf -> Channel, never the reverse)"
);
});
l.set(l.get() + 1);
}),
LockClass::Channel => CHANNELS_HELD.with(|c| {
debug_assert_eq!(
c.get(),
0,
"lock order violated: acquiring a channel lock while already \
holding one (channel locks are mutual leaves)"
);
c.set(c.get() + 1);
}),
}
#[cfg(not(debug_assertions))]
let _ = class;
}
#[inline]
fn order_check_release(class: LockClass) {
#[cfg(debug_assertions)]
match class {
LockClass::Leaf => LEAVES_HELD.with(|c| c.set(c.get() - 1)),
LockClass::Channel => CHANNELS_HELD.with(|c| c.set(c.get() - 1)),
}
#[cfg(not(debug_assertions))]
let _ = class;
}
pub(crate) struct RawMutex<T> {
state: AtomicU32,
class: LockClass,
data: UnsafeCell<T>,
}
// SAFETY: standard mutex argument — exclusive access to `data` is mediated by
// `state`; `T: Send` suffices for both because `&RawMutex` only ever hands out
// access to one thread at a time.
unsafe impl<T: Send> Send for RawMutex<T> {}
unsafe impl<T: Send> Sync for RawMutex<T> {}
impl<T> RawMutex<T> {
/// A Leaf-class mutex — the default for runtime cold data.
pub(crate) const fn new(data: T) -> Self {
Self::with_class(data, LockClass::Leaf)
}
/// A Channel-class mutex — for channel internals only.
pub(crate) const fn new_channel(data: T) -> Self {
Self::with_class(data, LockClass::Channel)
}
pub(crate) const fn with_class(data: T, class: LockClass) -> Self {
Self {
state: AtomicU32::new(UNLOCKED),
class,
data: UnsafeCell::new(data),
}
}
#[inline]
pub(crate) fn lock(&self) -> RawMutexGuard<'_, T> {
// Enter NoPreempt *before* acquiring, so a preemption can't fire
// between acquisition and guard construction.
let prev_preempt = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
order_check_acquire(self.class);
if self
.state
.compare_exchange(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_err()
{
self.lock_slow();
}
RawMutexGuard { m: self, prev_preempt }
}
#[cold]
fn lock_slow(&self) {
// Bounded spin first: the expected hold time is far below the cost of
// a futex round trip.
let mut spins = 0;
loop {
let s = self.state.load(Ordering::Relaxed);
if s == UNLOCKED
&& self
.state
.compare_exchange_weak(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_ok()
{
return;
}
spins += 1;
if spins >= SPIN_LIMIT {
break;
}
std::hint::spin_loop();
}
// Futex path. Mark contended and sleep until woken; on wake, retake
// by swapping to CONTENDED (we cannot know whether other waiters
// remain, so we must conservatively keep the contended marker).
while self.state.swap(CONTENDED, Ordering::Acquire) != UNLOCKED {
futex_wait(&self.state, CONTENDED);
}
}
#[inline]
fn unlock(&self) {
if self.state.swap(UNLOCKED, Ordering::Release) == CONTENDED {
futex_wake(&self.state, 1);
}
}
}
pub(crate) struct RawMutexGuard<'a, T> {
m: &'a RawMutex<T>,
prev_preempt: bool,
}
impl<T> Deref for RawMutexGuard<'_, T> {
type Target = T;
#[inline]
fn deref(&self) -> &T {
// SAFETY: guard existence implies exclusive ownership of the lock.
unsafe { &*self.m.data.get() }
}
}
impl<T> DerefMut for RawMutexGuard<'_, T> {
#[inline]
fn deref_mut(&mut self) -> &mut T {
// SAFETY: as above, plus &mut self.
unsafe { &mut *self.m.data.get() }
}
}
impl<T> Drop for RawMutexGuard<'_, T> {
#[inline]
fn drop(&mut self) {
self.m.unlock();
order_check_release(self.m.class);
// Restore preemption only after the lock is released.
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(self.prev_preempt));
}
}
// ---------------------------------------------------------------------------
// futex (x86-64 Linux; master is x86-only, see arm-port branch)
// ---------------------------------------------------------------------------
fn futex_wait(state: &AtomicU32, expected: u32) {
// SAFETY: `state` is a valid, aligned u32 for the duration of the call.
// Spurious wakeups and EAGAIN (value already changed) are both handled by
// the caller's retry loop.
unsafe {
libc::syscall(
libc::SYS_futex,
state.as_ptr(),
libc::FUTEX_WAIT | libc::FUTEX_PRIVATE_FLAG,
expected,
std::ptr::null::<libc::timespec>(),
);
}
}
fn futex_wake(state: &AtomicU32, n: i32) {
// SAFETY: as above.
unsafe {
libc::syscall(
libc::SYS_futex,
state.as_ptr(),
libc::FUTEX_WAKE | libc::FUTEX_PRIVATE_FLAG,
n,
);
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[test]
fn uncontended_lock_unlock() {
let m = RawMutex::new(0u64);
for _ in 0..1000 {
*m.lock() += 1;
}
assert_eq!(*m.lock(), 1000);
}
#[test]
fn contended_counter_is_exact() {
const THREADS: usize = 8;
const PER: u64 = 50_000;
let m = Arc::new(RawMutex::new(0u64));
let hs: Vec<_> = (0..THREADS)
.map(|_| {
let m = m.clone();
std::thread::spawn(move || {
for _ in 0..PER {
*m.lock() += 1;
}
})
})
.collect();
for h in hs {
h.join().unwrap();
}
assert_eq!(*m.lock(), THREADS as u64 * PER);
}
#[test]
fn no_poison_on_unwind() {
let m = Arc::new(RawMutex::new(0u64));
let m2 = m.clone();
let _ = std::thread::spawn(move || {
let _g = m2.lock();
panic!("unwind while holding");
})
.join();
// A std Mutex would now be poisoned; this one just works.
*m.lock() += 1;
assert_eq!(*m.lock(), 1);
}
#[test]
fn channel_lock_nests_under_leaf() {
// The permitted ordering: Leaf -> Channel (finalize/monitor clone a
// sender under a cold lock). Must not trip the order check.
let leaf = RawMutex::new(0u64);
let chan = RawMutex::new_channel(0u64);
let _l = leaf.lock();
let _c = chan.lock();
}
#[cfg(debug_assertions)]
#[test]
#[should_panic(expected = "lock order violated")]
fn leaf_under_channel_is_rejected() {
let leaf = RawMutex::new(0u64);
let chan = RawMutex::new_channel(0u64);
let _c = chan.lock();
let _l = leaf.lock(); // Channel -> Leaf: forbidden
}
#[cfg(debug_assertions)]
#[test]
#[should_panic(expected = "lock order violated")]
fn two_leaves_are_rejected() {
let a = RawMutex::new(0u64);
let b = RawMutex::new(0u64);
let _ga = a.lock();
let _gb = b.lock(); // leaves are mutual: forbidden
}
#[cfg(debug_assertions)]
#[test]
#[should_panic(expected = "lock order violated")]
fn two_channel_locks_are_rejected() {
let a = RawMutex::new_channel(0u64);
let b = RawMutex::new_channel(0u64);
let _ga = a.lock();
let _gb = b.lock(); // channel locks are mutual leaves: forbidden
}
}
src/registry.rs
-169
View File
@@ -1,169 +0,0 @@
//! Named pid registry.
//!
//! `register(name, pid)` binds a name to a live actor; `whereis(name)` looks
//! it up; `name_of(pid)` is the inverse. Erlang semantics: at most one name
//! per pid, at most one pid per name, registering over a live binding is an
//! error. The registry is a *bimap* (two `HashMap`s kept in exact inverse
//! under one lock) so both lookup directions are O(1) — `name_of` exists so
//! supervisors, tracing, and diagnostics can label a pid without scanning.
//!
//! ## Cleanup is lazy
//!
//! There is no hook in `finalize_actor` and no name field in `Slot` (which
//! would buy into the reset-in-three-places slot invariant). Instead every
//! operation that touches a binding checks the bound pid's liveness via the
//! generation-checked slot word — a `Pid` is `(index, generation)` and a
//! generation is never reused, so a stale binding is *detectable*, never
//! misdirected. Bindings to dead actors behave as absent and are pruned on
//! contact: `whereis`/`name_of` return `None`, `register` treats the name as
//! free. The cost is that a dead binding lingers until something looks at it;
//! the payoff is zero coupling to the actor lifecycle.
//!
//! ## Locking
//!
//! One `RawMutex` (Leaf class) in `RuntimeInner`. Liveness checks under it
//! read only the atomic slot state word — no second lock is ever taken, so
//! the leaf rule holds trivially.
use crate::pid::Pid;
use crate::scheduler::with_runtime;
use std::collections::HashMap;
/// Why a [`register`] call was rejected.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RegisterError {
/// The name is bound to a different, still-live actor.
NameTaken { holder: Pid },
/// The pid already has a (live) registered name; one name per pid.
PidAlreadyRegistered { name: String },
/// The pid does not refer to a live actor.
NoProc,
}
impl std::fmt::Display for RegisterError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
RegisterError::NameTaken { holder } => {
write!(f, "name is already registered to live actor {holder}")
}
RegisterError::PidAlreadyRegistered { name } => {
write!(f, "pid is already registered as {name:?}")
}
RegisterError::NoProc => write!(f, "pid does not refer to a live actor"),
}
}
}
impl std::error::Error for RegisterError {}
/// The bimap. Invariant (held under the registry lock): `by_name` and
/// `by_pid` are exact inverses of each other at all times.
pub(crate) struct Registry {
by_name: HashMap<String, Pid>,
by_pid: HashMap<Pid, String>,
}
impl Registry {
pub(crate) fn new() -> Self {
Self { by_name: HashMap::new(), by_pid: HashMap::new() }
}
fn remove_binding(&mut self, name: &str, pid: Pid) {
self.by_name.remove(name);
self.by_pid.remove(&pid);
debug_assert_eq!(self.by_name.len(), self.by_pid.len(), "bimap out of sync");
}
fn insert_binding(&mut self, name: String, pid: Pid) {
self.by_pid.insert(pid, name.clone());
self.by_name.insert(name, pid);
debug_assert_eq!(self.by_name.len(), self.by_pid.len(), "bimap out of sync");
}
}
/// Is `pid` a live actor right now? Atomic slot-word read; no lock.
fn live(inner: &crate::runtime::RuntimeInner, pid: Pid) -> bool {
inner.slot_at(pid).is_some_and(|s| s.is_live_for(pid))
}
/// Bind `name` to the live actor `pid`.
///
/// Fails with [`RegisterError::NoProc`] if `pid` is not live,
/// [`RegisterError::NameTaken`] if the name is bound to a *live* actor
/// (a binding to a dead actor is evicted and the name treated as free), and
/// [`RegisterError::PidAlreadyRegistered`] if `pid` already has a name.
///
/// Panics if called outside `Runtime::run()`.
pub fn register(name: impl Into<String>, pid: Pid) -> Result<(), RegisterError> {
let name = name.into();
with_runtime(|inner| {
let mut reg = inner.registry.lock();
if !live(inner, pid) {
return Err(RegisterError::NoProc);
}
let prior = reg.by_name.get(&name).copied();
if let Some(holder) = prior {
if holder == pid {
return Ok(()); // already exactly this binding: idempotent
}
if live(inner, holder) {
return Err(RegisterError::NameTaken { holder });
}
// Stale binding: the holder died. Evict and treat the name as free.
reg.remove_binding(&name, holder);
}
if let Some(existing) = reg.by_pid.get(&pid) {
// `pid` is live (checked above) and pids are never reused, so this
// binding is necessarily current — one name per pid.
return Err(RegisterError::PidAlreadyRegistered { name: existing.clone() });
}
reg.insert_binding(name, pid);
Ok(())
})
}
/// Look up the pid bound to `name`. `None` if unbound, or if the bound actor
/// is no longer live (the stale binding is pruned on the way out).
pub fn whereis(name: &str) -> Option<Pid> {
with_runtime(|inner| {
let mut reg = inner.registry.lock();
let pid = *reg.by_name.get(name)?;
if live(inner, pid) {
Some(pid)
} else {
reg.remove_binding(name, pid);
None
}
})
}
/// The inverse lookup: the name `pid` is registered under, if any. `None` if
/// unregistered or no longer live (pruning the stale binding).
pub fn name_of(pid: Pid) -> Option<String> {
with_runtime(|inner| {
let mut reg = inner.registry.lock();
let name = reg.by_pid.get(&pid)?.clone();
if live(inner, pid) {
Some(name)
} else {
reg.remove_binding(&name, pid);
None
}
})
}
/// Remove the binding for `name`, returning the pid it was bound to if that
/// actor is still live. A binding to a dead actor is pruned and reported as
/// `None`, consistent with [`whereis`].
pub fn unregister(name: &str) -> Option<Pid> {
with_runtime(|inner| {
let mut reg = inner.registry.lock();
let pid = *reg.by_name.get(name)?;
reg.remove_binding(name, pid);
if live(inner, pid) {
Some(pid)
} else {
None
}
})
}
src/run_queue.rs
-558
View File
@@ -1,558 +0,0 @@
//! The run queue, selected at COMPILE TIME by mutually-exclusive cargo
//! features (no runtime dispatch — the scheduler's pop loop is the hottest
//! code in the runtime):
//!
//! - `rq-mutex` (default) — `Mutex<VecDeque>`. The control/baseline:
//! strictly FIFO, trivially correct, one global lock.
//! - `rq-mpmc` — a single hand-rolled Vyukov bounded MPMC ring (per-cell
//! sequence numbers). Strict FIFO, lock-free, one hot
//! enqueue/dequeue cache-line pair.
//! - `rq-striped` — M Vyukov rings with fetch-add ticket distribution.
//! *Relaxed* FIFO: ordering across stripes is bounded-skewed
//! (≈ one ring's worth of reordering per stripe), in exchange
//! for spreading the hot line M ways. Predicted winner at
//! high core counts; phase 4's shootout decides.
//!
//! Select non-default variants with `--no-default-features --features rq-…`
//! (cargo features are additive, so the default must be switched off).
//!
//! All variants are compiled unconditionally (so every build runs every
//! variant's unit tests); the feature only picks which one the runtime uses
//! via the [`RunQueue`] alias.
//!
//! # Contract (shared by all variants)
//!
//! - **Occupancy is bounded by `max_actors`.** A pid is in the queue at most
//! once (pushes pair 1:1 with transitions into `Queued`; only the
//! scheduler transitions `Queued → Running` — see the state-machine docs
//! in `runtime.rs`), and at most `max_actors` actors exist. The bounded
//! rings are sized ≥ `max_actors`, so **`push` is infallible**; a full
//! ring is an invariant violation and panics loudly rather than spinning.
//! - **Preemption must be disabled around every push/pop** (debug-asserted).
//! For the mutex variant this is the usual no-switch/no-unwind-under-lock
//! rule. For the rings it is *load-bearing in a sharper way*: a producer
//! suspended between claiming a cell and publishing its sequence number
//! stalls every consumer behind that cell — on a busy runtime that is a
//! livelock, since the suspended actor's own resume entry sits behind the
//! hole. Callers get this for free: every queue op happens inside
//! `with_runtime`/`try_with_runtime` (NoPreempt for their span since
//! phase 2) or on a scheduler thread between resumes (preemption off).
//! - **`pop() == None` is a snapshot, not a fence.** A push that is mid-
//! publish (or in a stripe the probe already passed) may be missed; the
//! caller's idle path sleeps ≤ 100µs and retries, so the cost is a bounded
//! latency blip, never a lost entry. Termination does not lean on this:
//! the all-clear is `live_actors == 0` (+ io quiescent), and `live == 0`
//! already implies the queue holds nothing actionable — see the argument
//! in `schedule_loop`.
//! - `len()` is approximate (stats only).
use crate::pid::Pid;
use crate::sync_shim::{AtomicUsize, Ordering, UnsafeCell};
use std::mem::MaybeUninit;
// ---------------------------------------------------------------------------
// Feature selection
// ---------------------------------------------------------------------------
#[cfg(not(any(feature = "rq-mutex", feature = "rq-mpmc", feature = "rq-striped")))]
compile_error!(
"smarm: no run queue selected. Enable exactly one of the features \
`rq-mutex` (default), `rq-mpmc`, `rq-striped`."
);
#[cfg(all(feature = "rq-mutex", feature = "rq-mpmc"))]
compile_error!(
"smarm: features `rq-mutex` and `rq-mpmc` are mutually exclusive \
(use --no-default-features to drop the default `rq-mutex`)."
);
#[cfg(all(feature = "rq-mutex", feature = "rq-striped"))]
compile_error!(
"smarm: features `rq-mutex` and `rq-striped` are mutually exclusive \
(use --no-default-features to drop the default `rq-mutex`)."
);
#[cfg(all(feature = "rq-mpmc", feature = "rq-striped"))]
compile_error!("smarm: features `rq-mpmc` and `rq-striped` are mutually exclusive.");
#[cfg(feature = "rq-mutex")]
pub(crate) type RunQueue = MutexQueue;
#[cfg(feature = "rq-mpmc")]
pub(crate) type RunQueue = MpmcRing;
#[cfg(feature = "rq-striped")]
pub(crate) type RunQueue = StripedRing;
#[inline]
fn assert_no_preempt() {
debug_assert!(
!crate::preempt::PREEMPTION_ENABLED.with(|c| c.get()),
"run-queue op with preemption enabled — a switch mid-op stalls or \
corrupts the queue; route through with_runtime or scheduler context"
);
}
// ---------------------------------------------------------------------------
// rq-mutex — the baseline
// ---------------------------------------------------------------------------
#[allow(dead_code)]
pub struct MutexQueue {
q: std::sync::Mutex<std::collections::VecDeque<Pid>>,
}
#[allow(dead_code)]
impl MutexQueue {
pub fn new(_threads: usize, max_actors: usize) -> Self {
Self {
// Pre-size: the queue can never outgrow the slab, and one
// allocation at init beats reallocating under the lock later.
q: std::sync::Mutex::new(std::collections::VecDeque::with_capacity(max_actors)),
}
}
pub fn push(&self, pid: Pid) {
assert_no_preempt();
self.q.lock().unwrap().push_back(pid);
}
pub fn pop(&self) -> Option<Pid> {
assert_no_preempt();
self.q.lock().unwrap().pop_front()
}
pub fn len(&self) -> u64 {
self.q.lock().unwrap().len() as u64
}
}
// ---------------------------------------------------------------------------
// rq-mpmc — Vyukov bounded MPMC ring
// ---------------------------------------------------------------------------
//
// Dmitry Vyukov's bounded MPMC queue: each cell carries a sequence number.
// A producer may write cell `i` when `seq == pos` (its turn); it publishes
// with `seq = pos + 1`. A consumer may read when `seq == pos + 1`; it
// releases the cell to the next lap with `seq = pos + capacity`. Producers
// and consumers each contend on one counter; cell handoff is a per-cell
// Acquire/Release pair, so unrelated push/pop pairs don't serialize.
/// Pad to a cache-line pair so the producer and consumer counters (and the
/// cells) don't false-share.
#[repr(align(128))]
struct CachePadded<T>(T);
struct Cell {
seq: AtomicUsize,
pid: UnsafeCell<MaybeUninit<Pid>>,
}
#[allow(dead_code)]
pub struct MpmcRing {
buf: Box<[Cell]>,
mask: usize,
enqueue_pos: CachePadded<AtomicUsize>,
dequeue_pos: CachePadded<AtomicUsize>,
}
// SAFETY: cells are handed off between threads via the per-cell seq
// (Release on publish, Acquire on claim); Pid is Copy + Send.
unsafe impl Send for MpmcRing {}
unsafe impl Sync for MpmcRing {}
#[allow(dead_code)]
impl MpmcRing {
pub fn new(_threads: usize, max_actors: usize) -> Self {
Self::with_capacity(max_actors)
}
/// Pub for the raw-structure microbench (sized to the op count so the
/// occupancy contract is trivially met there). Runtime code uses `new`.
pub fn with_capacity(min_cap: usize) -> Self {
// Occupancy ≤ max_actors (queue contract), so capacity = the next
// power of two ≥ max_actors can never overflow. (≥ 2 so mask works.)
let cap = min_cap.next_power_of_two().max(2);
let buf: Box<[Cell]> = (0..cap)
.map(|i| Cell {
seq: AtomicUsize::new(i),
pid: UnsafeCell::new(MaybeUninit::uninit()),
})
.collect();
Self {
buf,
mask: cap - 1,
enqueue_pos: CachePadded(AtomicUsize::new(0)),
dequeue_pos: CachePadded(AtomicUsize::new(0)),
}
}
pub fn push(&self, pid: Pid) {
assert_no_preempt();
assert!(
self.try_push(pid),
"smarm: run queue overflow — occupancy exceeded the slab bound, \
which the at-most-once-enqueued invariant forbids. This is a \
runtime bug (double enqueue), not a capacity tuning problem."
);
}
/// One full claim attempt; `false` only if the ring is full.
fn try_push(&self, pid: Pid) -> bool {
let mut pos = self.enqueue_pos.0.load(Ordering::Relaxed);
loop {
let cell = &self.buf[pos & self.mask];
let seq = cell.seq.load(Ordering::Acquire);
let diff = seq as isize - pos as isize;
if diff == 0 {
// Our turn: claim the position.
match self.enqueue_pos.0.compare_exchange_weak(
pos, pos + 1, Ordering::Relaxed, Ordering::Relaxed,
) {
Ok(_) => {
// SAFETY: the claim gives us exclusive write access
// to this cell until we publish below.
cell.pid.with_mut(|p| unsafe { (*p).write(pid) });
cell.seq.store(pos + 1, Ordering::Release);
return true;
}
Err(actual) => pos = actual,
}
} else if diff < 0 {
return false; // full — a whole lap behind
} else {
pos = self.enqueue_pos.0.load(Ordering::Relaxed);
}
}
}
pub fn pop(&self) -> Option<Pid> {
assert_no_preempt();
let mut pos = self.dequeue_pos.0.load(Ordering::Relaxed);
loop {
let cell = &self.buf[pos & self.mask];
let seq = cell.seq.load(Ordering::Acquire);
let diff = seq as isize - (pos + 1) as isize;
if diff == 0 {
match self.dequeue_pos.0.compare_exchange_weak(
pos, pos + 1, Ordering::Relaxed, Ordering::Relaxed,
) {
Ok(_) => {
// SAFETY: the claim gives us exclusive read access;
// the producer's Release publish made `pid` visible
// to our Acquire load of `seq`.
let pid = cell.pid.with(|p| unsafe { (*p).assume_init_read() });
// Release the cell for the next lap.
cell.seq.store(pos + self.mask + 1, Ordering::Release);
return Some(pid);
}
Err(actual) => pos = actual,
}
} else if diff < 0 {
// Empty (or the producer at this cell hasn't published yet —
// a snapshot miss the caller's idle-retry loop absorbs).
return None;
} else {
pos = self.dequeue_pos.0.load(Ordering::Relaxed);
}
}
}
pub fn len(&self) -> u64 {
let e = self.enqueue_pos.0.load(Ordering::Relaxed);
let d = self.dequeue_pos.0.load(Ordering::Relaxed);
e.saturating_sub(d) as u64
}
}
// ---------------------------------------------------------------------------
// rq-striped — M Vyukov rings, ticket-distributed
// ---------------------------------------------------------------------------
//
// Producers fetch-add a ticket and start probing at stripe `ticket % M`;
// consumers do the same with their own ticket. Under symmetric load the
// tickets spread producers and consumers uniformly, so each stripe sees
// ~1/M of the traffic and the single hot cache-line pair becomes M cooler
// ones. FIFO is relaxed: two pushes that land in different stripes can be
// popped in either order, with skew bounded by stripe occupancy imbalance.
//
// Push probes forward from its home stripe until a `try_push` succeeds.
// Σ stripe capacity ≥ 2 × max_actors while occupancy ≤ max_actors, so at
// every instant at least half the total capacity is free and the probe
// terminates (in practice on the first stripe).
#[allow(dead_code)]
pub struct StripedRing {
stripes: Box<[MpmcRing]>,
/// Stripe count minus one (count is a power of two).
stripe_mask: usize,
push_ticket: CachePadded<AtomicUsize>,
pop_ticket: CachePadded<AtomicUsize>,
}
#[allow(dead_code)]
impl StripedRing {
pub fn new(threads: usize, max_actors: usize) -> Self {
// One stripe per scheduler thread, rounded up to a power of two —
// more stripes than threads buys nothing (at most `threads` ops are
// in flight) and costs pop-probe latency when mostly empty.
let n = threads.max(1).next_power_of_two();
// Per-stripe capacity: 2 × max_actors / n in total, and never below
// a floor that keeps degenerate configs (tiny slab, many threads)
// trivially correct.
let per = ((2 * max_actors) / n).next_power_of_two().max(8);
let stripes: Box<[MpmcRing]> = (0..n).map(|_| MpmcRing::with_capacity(per)).collect();
Self {
stripes,
stripe_mask: n - 1,
push_ticket: CachePadded(AtomicUsize::new(0)),
pop_ticket: CachePadded(AtomicUsize::new(0)),
}
}
pub fn push(&self, pid: Pid) {
assert_no_preempt();
let home = self.push_ticket.0.fetch_add(1, Ordering::Relaxed);
// Probe from the home stripe; capacity headroom (Σ ≥ 2×occupancy)
// guarantees a free stripe exists, so the outer loop terminates.
// The retry-from-home lap handles the racy case where every stripe
// momentarily refused us.
loop {
for i in 0..=self.stripe_mask {
let s = &self.stripes[(home + i) & self.stripe_mask];
if s.try_push(pid) {
return;
}
}
std::hint::spin_loop();
}
}
pub fn pop(&self) -> Option<Pid> {
assert_no_preempt();
let home = self.pop_ticket.0.fetch_add(1, Ordering::Relaxed);
for i in 0..=self.stripe_mask {
if let Some(pid) = self.stripes[(home + i) & self.stripe_mask].pop() {
return Some(pid);
}
}
None // snapshot miss possible across stripes; idle-retry absorbs it
}
pub fn len(&self) -> u64 {
self.stripes.iter().map(|s| s.len()).sum()
}
}
// ---------------------------------------------------------------------------
// Tests — all variants, in every build (the feature only picks the alias)
// ---------------------------------------------------------------------------
#[cfg(all(test, not(loom)))]
mod tests {
use super::*;
use std::collections::HashSet;
use std::sync::Arc;
fn pid(i: u32) -> Pid {
Pid::new(i, 0)
}
fn fifo_smoke<Q>(q: &Q, push: impl Fn(&Q, Pid), pop: impl Fn(&Q) -> Option<Pid>) {
for i in 0..100 {
push(q, pid(i));
}
for i in 0..100 {
assert_eq!(pop(q), Some(pid(i)));
}
assert_eq!(pop(q), None);
}
#[test]
fn mutex_fifo() {
let q = MutexQueue::new(1, 1024);
fifo_smoke(&q, |q, p| q.push(p), |q| q.pop());
}
#[test]
fn mpmc_fifo_single_thread() {
let q = MpmcRing::new(1, 1024);
fifo_smoke(&q, |q, p| q.push(p), |q| q.pop());
}
#[test]
fn mpmc_wraps_many_laps() {
let q = MpmcRing::with_capacity(8);
for lap in 0..1000u32 {
for i in 0..8 {
q.push(pid(lap * 8 + i));
}
for i in 0..8 {
assert_eq!(q.pop(), Some(pid(lap * 8 + i)));
}
}
assert_eq!(q.pop(), None);
}
/// N producers, M consumers, every element exactly once. Run on plain OS
/// threads (PREEMPTION_ENABLED defaults false, satisfying the contract).
fn exactly_once<Q: Send + Sync + 'static>(
q: Q,
push: fn(&Q, Pid),
pop: fn(&Q) -> Option<Pid>,
producers: u32,
consumers: u32,
per_producer: u32,
) {
let q = Arc::new(q);
let total = (producers * per_producer) as usize;
let popped = Arc::new(std::sync::Mutex::new(Vec::with_capacity(total)));
let remaining = Arc::new(AtomicUsize::new(total));
let mut hs = Vec::new();
for p in 0..producers {
let q = q.clone();
hs.push(std::thread::spawn(move || {
for i in 0..per_producer {
push(&q, pid(p * per_producer + i));
}
}));
}
for _ in 0..consumers {
let q = q.clone();
let popped = popped.clone();
let remaining = remaining.clone();
hs.push(std::thread::spawn(move || {
let mut local = Vec::new();
while remaining.load(Ordering::Relaxed) > 0 {
if let Some(pid) = pop(&q) {
remaining.fetch_sub(1, Ordering::Relaxed);
local.push(pid);
} else {
std::hint::spin_loop();
}
}
popped.lock().unwrap().extend(local);
}));
}
for h in hs {
h.join().unwrap();
}
let popped = popped.lock().unwrap();
assert_eq!(popped.len(), total, "count mismatch");
let set: HashSet<u64> = popped.iter().map(|p| ((p.index() as u64) << 32) | p.generation() as u64).collect();
assert_eq!(set.len(), total, "duplicate or lost element");
assert_eq!(pop(&q), None);
}
#[test]
fn mpmc_exactly_once_contended() {
exactly_once(MpmcRing::new(8, 4096), |q, p| q.push(p), |q| q.pop(), 4, 4, 1000);
}
#[test]
fn striped_exactly_once_contended() {
exactly_once(StripedRing::new(8, 4096), |q, p| q.push(p), |q| q.pop(), 4, 4, 1000);
}
#[test]
fn striped_drains_after_skewed_load() {
// Hammer pushes from one thread (all tickets walk the stripes in
// order) and verify a single consumer sees every element.
let q = StripedRing::new(4, 64);
let mut seen = HashSet::new();
for i in 0..64 {
q.push(pid(i));
}
while let Some(p) = q.pop() {
assert!(seen.insert(p.index()));
}
assert_eq!(seen.len(), 64);
}
}
// ---------------------------------------------------------------------------
// loom model tests — RUSTFLAGS="--cfg loom" cargo test --lib --release
// ---------------------------------------------------------------------------
#[cfg(all(test, loom))]
mod loom_tests {
use super::*;
use loom::sync::Arc;
use loom::thread;
fn pid(i: u32) -> Pid {
Pid::new(i, 0)
}
/// Two producers, main-thread consumer: both elements arrive exactly
/// once, across every interleaving — including through a lap wraparound
/// (capacity 2 forces cell reuse).
#[test]
fn mpmc_two_producers_exactly_once() {
loom::model(|| {
let q = Arc::new(MpmcRing::with_capacity(2));
let mut hs = Vec::new();
for i in 0..2u32 {
let q = q.clone();
hs.push(thread::spawn(move || q.push(pid(i))));
}
let mut got = Vec::new();
while got.len() < 2 {
match q.pop() {
Some(p) => got.push(p.index()),
None => thread::yield_now(),
}
}
for h in hs {
h.join().unwrap();
}
got.sort_unstable();
assert_eq!(got, vec![0, 1]);
assert!(q.pop().is_none());
});
}
/// Producer races a consumer on a single element: the consumer either
/// gets it or sees a clean None — never a torn/duplicated element.
#[test]
fn mpmc_push_pop_race() {
loom::model(|| {
let q = Arc::new(MpmcRing::with_capacity(2));
let q2 = q.clone();
let prod = thread::spawn(move || q2.push(pid(7)));
let seen = q.pop();
prod.join().unwrap();
match seen {
Some(p) => {
assert_eq!(p.index(), 7);
assert!(q.pop().is_none());
}
None => assert_eq!(q.pop().map(|p| p.index()), Some(7)),
}
});
}
/// Striped: two producers landing in (potentially) different stripes,
/// main-thread consumer drains both exactly once.
#[test]
fn striped_two_producers_exactly_once() {
loom::model(|| {
let q = Arc::new(StripedRing::new(2, 4));
let mut hs = Vec::new();
for i in 0..2u32 {
let q = q.clone();
hs.push(thread::spawn(move || q.push(pid(i))));
}
let mut got = Vec::new();
while got.len() < 2 {
match q.pop() {
Some(p) => got.push(p.index()),
None => thread::yield_now(),
}
}
for h in hs {
h.join().unwrap();
}
got.sort_unstable();
assert_eq!(got, vec![0, 1]);
assert!(q.pop().is_none());
});
}
}
src/runtime.rs
+418 -663
View File
File diff suppressed because it is too large Load Diff
src/scheduler.rs
+134 -197
View File
@@ -21,39 +21,18 @@ use std::sync::Arc;
// ---------------------------------------------------------------------------
/// Borrow the current runtime. Panics if called outside `Runtime::run()`.
///
/// The whole span runs with preemption disabled. Two reasons, both load-
/// bearing:
///
/// - The `RUNTIME` thread-local borrow is live across `f`. A preemption-
/// driven context switch inside `f` can resume the actor on a DIFFERENT OS
/// thread; the borrow guard would then increment this thread's RefCell but
/// decrement the other's — a count underflow that leaves that thread's
/// RefCell permanently "mutably borrowed". Holding a thread-local guard
/// across a potential switch point is the one unforgivable sin of green
/// threads; disabling preemption makes "no switch inside `f`" structural
/// instead of an accident of which bodies happen to allocate.
/// - `f` is runtime bookkeeping. Suspending an actor halfway through it (or
/// unwinding via the stop sentinel, which shares the gate) is never wanted.
pub(crate) fn with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> R {
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| {
RUNTIME.with(|r| {
let b = r.borrow();
let inner = b.as_ref().expect("smarm: not inside Runtime::run()");
f(inner)
});
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
result
})
}
/// Borrow the runtime if present; returns `None` otherwise.
/// Used on cleanup paths (channel Drop during teardown).
/// Same preemption gate as [`with_runtime`]; same reasons.
pub(crate) fn try_with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> Option<R> {
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)));
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
result
RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)))
}
// ---------------------------------------------------------------------------
@@ -75,34 +54,22 @@ impl JoinHandle {
pub fn join(mut self) -> Result<(), JoinError> {
use crate::actor::Outcome;
use crate::runtime::State; // need State visibility
let me = current_pid().expect("join() called outside an actor");
loop {
// Check-Done-or-register-waiter is atomic under the target's cold
// lock; finalize publishes Done and takes the waiter list under
// the same lock, so we either see the outcome or are woken.
let outcome = with_runtime(|inner| {
let slot = inner.slot_at(self.pid)
.expect("join: pid index out of range");
let mut cold = slot.cold.lock();
match slot.status_for(self.pid) {
// Our outstanding handle pins the slot: it cannot be
// reclaimed (generation cannot change) while we hold it.
crate::slot_state::Status::Stale => {
panic!("join: target slot has been reused")
}
crate::slot_state::Status::Done => {
Some(cold.outcome.take().expect("Done slot must have outcome"))
}
crate::slot_state::Status::Live => {
// begin_wait is lock-free, legal under the cold lock;
// registering under it makes the epoch atomic with
// the check-Done-or-register linearization point.
cold.waiters.push((me, begin_wait()));
inner.with_shared(|s| {
let slot = s.slot_mut(self.pid)
.expect("join: target slot has been reused");
if matches!(slot.state, State::Done) {
Some(slot.outcome.take().expect("Done slot must have outcome"))
} else {
slot.waiters.push(me);
None
}
}
})
});
match outcome {
@@ -128,25 +95,20 @@ impl JoinHandle {
fn decrement_handle_count(&mut self) {
with_runtime(|inner| {
let should_reclaim = match inner.slot_at(self.pid) {
Some(slot) => {
let mut cold = slot.cold.lock();
match slot.status_for(self.pid) {
crate::slot_state::Status::Stale => false,
status => {
cold.outstanding_handles =
cold.outstanding_handles.saturating_sub(1);
cold.outstanding_handles == 0
&& status == crate::slot_state::Status::Done
}
inner.with_shared(|s| {
let should_reclaim = match s.slot_mut(self.pid) {
Some(slot) => {
slot.outstanding_handles =
slot.outstanding_handles.saturating_sub(1);
matches!(slot.state, crate::runtime::State::Done)
&& slot.outstanding_handles == 0
}
None => false,
};
if should_reclaim {
crate::runtime::reclaim_slot(s, self.pid);
}
None => false,
};
if should_reclaim {
// Re-verified inside; benign if finalize's reclaim won a race.
crate::runtime::reclaim_slot(inner, self.pid);
}
})
});
}
}
@@ -167,28 +129,56 @@ impl Drop for JoinHandle {
// ---------------------------------------------------------------------------
pub fn spawn(f: impl FnOnce() + Send + 'static) -> JoinHandle {
let parent = current_pid().unwrap_or_else(|| {
// Outside an actor but inside run(): the initial spawn. with_runtime
// panics with "not inside Runtime::run()" if there's no runtime at all.
with_runtime(|_| crate::runtime::ROOT_PID)
});
let parent = current_pid()
.or_else(|| with_runtime(|inner| inner.with_shared(|s| s.root_pid)))
.expect("spawn() before run()");
spawn_under(parent, f)
}
pub fn spawn_under(supervisor: Pid, f: impl FnOnce() + Send + 'static) -> JoinHandle {
// Stack + closure boxing happen before ANY runtime lock is taken: no
// syscall and no allocation ever stalls another scheduler thread.
let stack = with_runtime(|inner| inner.stack_pool.lock().pop())
// Try to reuse a stack from the pool; fall back to a fresh mmap if empty.
// Allocation happens before taking the shared lock so any syscall doesn't
// stall other scheduler threads.
let stack = with_runtime(|inner| inner.stack_pool.lock().unwrap().pop())
.unwrap_or_else(|| {
crate::stack::Stack::new(crate::runtime::ACTOR_STACK_SIZE)
.expect("stack allocation failed")
});
let sp = init_actor_stack(stack.top(), crate::actor::trampoline);
let closure: crate::runtime::Closure = Box::new(f);
let pid = with_runtime(|inner| {
let idx = inner.allocate_slot(); // panics loudly on slab exhaustion
crate::runtime::install_actor(inner, idx, sp, stack, supervisor, closure)
inner.with_shared(|s| {
let (idx, gen) = s.allocate_slot();
let pid = Pid::new(idx, gen);
let slot = &mut s.slots[idx as usize];
slot.actor = Some(crate::actor::Actor {
pid,
stack,
sp,
supervisor,
stop: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)),
trap: None,
});
slot.state = crate::runtime::State::Runnable;
slot.outstanding_handles = 1;
slot.outcome = None;
slot.waiters.clear();
slot.supervisor_channel = None;
slot.monitors.clear();
slot.links.clear();
slot.pending_unpark = false;
slot.pending_io_result = None;
s.run_queue.push_back(pid);
// Grow the closures vec to cover this slot index, then store.
let idx = idx as usize;
if s.pending_closures.len() <= idx {
s.pending_closures.resize_with(idx + 1, || None);
}
s.pending_closures[idx] = Some(Box::new(f) as crate::runtime::Closure);
crate::te!(crate::trace::Event::Spawn { parent: supervisor, child: pid });
crate::te!(crate::trace::Event::Enqueue(pid));
pid
})
});
JoinHandle { pid, consumed: false }
@@ -222,54 +212,39 @@ pub fn park_current() {
}
pub fn unpark(pid: Pid) {
// The whole protocol lives on the slot's packed word (gen + epoch +
// state in one CAS) — see slot_state.rs. No runtime lock unless we
// enqueue. WILDCARD form: reserved for terminal wakes (request_stop);
// every registration-based waker must use `unpark_at`.
let _ = try_with_runtime(|inner| inner.unpark(pid));
}
/// Epoch-matched unpark: wake `pid` only if its current wait is still the
/// one this waker registered for. The form every registration-based waker
/// (channel senders, mutex grants, wait-timers, io completions, joiner
/// wakes) must use — see slot_state.rs for the consuming-wake rules.
pub(crate) fn unpark_at(pid: Pid, epoch: u32) {
let _ = try_with_runtime(|inner| inner.unpark_at(pid, epoch));
}
/// Open a new wait for the CURRENT actor: bump its park-epoch and return
/// it. Call once per wait, before registering `(pid, epoch)` with any
/// waker. Lock-free (one CAS on the own slot word), so it is legal under
/// any lock, including a Channel-class lock.
pub(crate) fn begin_wait() -> u32 {
let me = current_pid().expect("begin_wait() called outside an actor");
with_runtime(|inner| inner.begin_wait(me))
}
/// Close the current actor's wait WITHOUT parking on it. For the no-park
/// exit of multi-registration waits (`select` finding an arm ready at
/// registration time): bumps the epoch so in-flight wakes die at their CAS,
/// eats a notification that already landed, then observes a pending stop —
/// in that order. After this, leftover registrations are stale-epoch and
/// self-clean at their wakers' failed CAS; nothing can fault the actor's
/// next one-shot park.
pub(crate) fn retire_wait() {
let me = current_pid().expect("retire_wait() called outside an actor");
with_runtime(|inner| inner.retire_wait(me));
// A request_stop that fired before the clear had its notification
// eaten, but it set the stop flag first — observe it here and unwind.
// One that fires after re-notifies a Running word as usual.
crate::preempt::check_cancelled();
let result = try_with_runtime(|inner| {
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
match slot.state {
crate::runtime::State::Parked => {
// Actor is suspended — safe to re-queue immediately.
slot.state = crate::runtime::State::Runnable;
s.run_queue.push_back(pid);
crate::te!(crate::trace::Event::UnparkDirect(pid));
crate::te!(crate::trace::Event::Enqueue(pid));
}
crate::runtime::State::Runnable => {
// Actor is still running (between registering its
// parked_receiver and calling park_current). Set the
// flag; the scheduler will re-queue after the Park
// yield instead of sleeping.
slot.pending_unpark = true;
crate::te!(crate::trace::Event::UnparkDeferred(pid));
}
crate::runtime::State::Done => {}
}
}
})
});
let _ = result;
}
/// Request cooperative cancellation of `pid`.
///
/// Sets the actor's stop flag and wakes it so it observes the flag promptly:
/// a parked target is re-queued; a running target is marked notified, so its
/// next park returns immediately to an observation point. The actor realises
/// the stop as a controlled unwind at its next observation point
/// (`check!()`/allocation, or the wakeup side of a blocking park),
/// terminating with `Outcome::Stopped`.
/// Sets the actor's stop flag and, if it is parked, wakes it so it observes
/// the flag promptly. The actor realises the stop as a controlled unwind at
/// its next observation point (`check!()`/allocation, or the wakeup side of a
/// blocking park), terminating with `Outcome::Stopped`.
///
/// This is best-effort and cooperative: an actor that never reaches an
/// observation point — a tight loop with no `check!()`, no allocation, and no
@@ -277,19 +252,21 @@ pub(crate) fn retire_wait() {
/// if `pid` is already gone.
pub fn request_stop(pid: Pid) {
let _ = try_with_runtime(|inner| {
if let Some(slot) = inner.slot_at(pid) {
{
let cold = slot.cold.lock();
// Verify under the cold lock: generation can't change while
// we hold it (reclaim takes the same lock).
if slot.generation() == pid.generation() {
if let Some(actor) = cold.actor.as_ref() {
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
}
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
if let Some(actor) = slot.actor.as_ref() {
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
}
// Wake a parked target so it reaches its post-park observation
// point now. A Runnable target will observe on its next yield;
// a Done target has nothing to stop.
if matches!(slot.state, crate::runtime::State::Parked) {
slot.state = crate::runtime::State::Runnable;
s.run_queue.push_back(pid);
crate::te!(crate::trace::Event::Enqueue(pid));
}
}
inner.unpark(pid);
}
})
});
}
@@ -319,9 +296,8 @@ impl Drop for NoPreempt {
pub fn sleep(duration: std::time::Duration) {
let me = current_pid().expect("sleep() called outside an actor");
let _np = NoPreempt::enter();
let epoch = begin_wait();
let deadline = crate::timer::deadline_from_now(duration);
with_runtime(|inner| inner.timers.lock().unwrap().insert_sleep(deadline, me, epoch));
with_runtime(|inner| inner.with_shared(|s| s.timers.insert_sleep(deadline, me)));
park_current();
}
@@ -329,14 +305,16 @@ pub fn insert_wait_timer(
deadline: std::time::Instant,
pid: Pid,
target: std::sync::Arc<dyn crate::timer::TimerTarget>,
epoch: u32,
wait_seq: u64,
) {
with_runtime(|inner| {
inner.timers.lock().unwrap().insert(
deadline,
pid,
crate::timer::Reason::WaitTimeout { target, epoch },
);
inner.with_shared(|s| {
s.timers.insert(
deadline,
pid,
crate::timer::Reason::WaitTimeout { target, wait_seq },
);
})
});
}
@@ -356,24 +334,19 @@ where
});
{
let _np = NoPreempt::enter();
let epoch = begin_wait();
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
io.as_mut().expect("io thread not started").submit(me, epoch, work);
});
with_runtime(|inner| inner.with_shared(|s| {
let io = s.io.as_mut().expect("io thread not started");
io.submit(me, work);
}));
park_current();
}
let result = with_runtime(|inner| {
let slot = inner.slot_at(me).expect("block_on_io: own slot vanished");
let mut cold = slot.cold.lock();
debug_assert_eq!(
slot.generation(), me.generation(),
"block_on_io: own slot reused mid-park"
);
cold.pending_io_result
let result = with_runtime(|inner| inner.with_shared(|s| {
s.slot_mut(me)
.expect("block_on_io: own slot vanished")
.pending_io_result
.take()
.expect("block_on_io: resumed without a result")
});
}));
match result {
Ok(any) => *any.downcast::<T>().expect("block_on_io: type mismatch"),
Err(payload) => std::panic::resume_unwind(payload),
@@ -391,44 +364,11 @@ pub fn wait_writable(fd: std::os::fd::RawFd) -> std::io::Result<()> {
fn wait_fd(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::Result<()> {
let me = current_pid().expect("wait_*() called outside an actor");
let _np = NoPreempt::enter();
let epoch = begin_wait();
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
io.as_mut().expect("io thread not started").epoll_register(fd, me, epoch, readable, writable)
})?;
// If a terminal stop unwinds us out of the park below, the registration
// must not outlive us: a stale `waiters` entry fails every future
// `wait_*` on this fd with AlreadyExists, and the kernel-side ADD leaks
// until the fd happens to be reused. Clean up iff the entry is still
// ours — a `FdReady` racing the stop may have consumed it already (it
// removes + DELs under the io lock), after which the fd may even carry
// ANOTHER actor's fresh registration; in that case touch nothing.
struct Dereg {
fd: std::os::fd::RawFd,
me: Pid,
epoch: u32,
}
impl Drop for Dereg {
fn drop(&mut self) {
with_runtime(|inner| {
let mut io = inner.io.lock().unwrap();
if let Some(io) = io.as_mut() {
if io.waiters.get(&self.fd) == Some(&(self.me, self.epoch)) {
io.waiters.remove(&self.fd);
io.epoll_deregister(self.fd);
}
}
});
}
}
let guard = Dereg { fd, me, epoch };
with_runtime(|inner| inner.with_shared(|s| {
let io = s.io.as_mut().expect("io thread not started");
io.epoll_register(fd, me, readable, writable)
}))?;
park_current();
// Normal wake: the FdReady path removed the entry and DEL'd the fd
// before the unpark, so the guard's check would be a guaranteed no-op —
// skip the io lock on the hot path. (Dereg owns nothing; forget leaks
// no resource.)
std::mem::forget(guard);
Ok(())
}
@@ -449,16 +389,13 @@ pub fn write(fd: std::os::fd::RawFd, buf: &[u8]) -> std::io::Result<usize> {
// ---------------------------------------------------------------------------
pub fn register_supervisor_channel(pid: Pid, sender: Sender<Signal>) {
with_runtime(|inner| {
let slot = inner.slot_at(pid)
.unwrap_or_else(|| panic!("register_supervisor_channel: pid {:?} not found", pid));
let mut cold = slot.cold.lock();
assert_eq!(
slot.generation(), pid.generation(),
"register_supervisor_channel: pid {:?} not found", pid
);
cold.supervisor_channel = Some(sender);
});
with_runtime(|inner| inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
slot.supervisor_channel = Some(sender);
} else {
panic!("register_supervisor_channel: pid {:?} not found", pid);
}
}));
}
// ---------------------------------------------------------------------------
src/slot_state.rs
-624
View File
@@ -1,624 +0,0 @@
//! The per-slot scheduling state machine, as a standalone unit.
//!
//! One atomic word packs `(generation << 32) | (epoch << 8) | state`; every
//! transition is a CAS on the packed word, so the generation check is atomic
//! with the transition — no ABA, no acting on a recycled slot. The diagram
//! and the full protocol rationale live in `runtime.rs`; this module is the
//! mechanism, factored out so that:
//!
//! - loom can model-check the production transitions directly (see the
//! `loom_tests` module; built with `RUSTFLAGS="--cfg loom"`), and
//! - every method asserts the precondition it relies on (`debug_assert!` —
//! these are hot paths), per the assert-the-invariants house rule.
//!
//! ## The park-epoch (wait identity)
//!
//! The middle 24 bits carry the slot's *park-epoch*: the identity of the
//! actor's current (or most recent) wait. The rules:
//!
//! - [`begin_wait`](StateWord::begin_wait) bumps the epoch and returns it;
//! the actor calls it once per wait, *before* registering itself with any
//! waker. Registrations carry `(pid, epoch)`.
//! - A wake may be **epoch-matched** (`unpark(gen, Some(epoch))`): it lands
//! only if the word still carries that epoch. Wakers whose registration
//! handle can outlive the wait it was created for (channel senders, mutex
//! grants, wait-timers) MUST use this form.
//! - Every successful wake **consumes** the epoch — `Parked(e) → Queued(e+1)`,
//! `Running(e) → RunningNotified(e+1)` — so at most one wake can ever land
//! per wait, by construction. A loser in a multi-waker race (e.g. the
//! non-winning arms of a `select`) fails the epoch check and no-ops; it can
//! neither steal a future wait's wake nor leave a pending notification that
//! would fault a later one-shot park (`Mutex::lock_timeout`, `sleep`,
//! `block_on_io`, `wait_fd` all rely on wakes being *meaningful*).
//! - The wildcard form (`unpark(gen, None)`) also consumes, and is reserved
//! for terminal wakes — `request_stop` — which never return control to the
//! code that parked.
//!
//! Epoch wrap (24 bits = 16.7M waits) is harmless: a collision would require
//! a taken registration to stay in flight across a full wrap of the *same
//! actor's* waits, and registrations are consumed at take-time under their
//! primitive's lock — the exposure is the taker's instruction window.
//!
//! Atomics come from `sync_shim` (std normally, `loom::sync` under
//! `cfg(loom)`).
use crate::sync_shim::{AtomicU64, Ordering};
pub(crate) const ST_VACANT: u64 = 0;
pub(crate) const ST_QUEUED: u64 = 1;
pub(crate) const ST_RUNNING: u64 = 2;
pub(crate) const ST_RUNNING_NOTIFIED: u64 = 3;
pub(crate) const ST_PARKED: u64 = 4;
pub(crate) const ST_DONE: u64 = 5;
/// Park-epoch width: 24 bits, packed at word bits 8..32.
pub(crate) const EPOCH_MASK: u32 = 0x00FF_FFFF;
#[inline]
pub(crate) const fn pack(gen: u32, epoch: u32, st: u64) -> u64 {
debug_assert!(epoch & !EPOCH_MASK == 0);
((gen as u64) << 32) | ((epoch as u64) << 8) | st
}
#[inline]
pub(crate) const fn word_gen(w: u64) -> u32 {
(w >> 32) as u32
}
#[inline]
pub(crate) const fn word_epoch(w: u64) -> u32 {
((w >> 8) as u32) & EPOCH_MASK
}
#[inline]
pub(crate) const fn word_state(w: u64) -> u64 {
w & 0xFF
}
/// What an unpark amounted to. The caller owns the side effects (enqueue,
/// trace events) — this module is pure state.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub(crate) enum Unpark {
/// Parked → Queued: the caller must enqueue the pid.
Enqueue,
/// Running → RunningNotified: the scheduler's park-return will re-queue.
Notified,
/// Stale generation, stale epoch, already queued/notified, done, or
/// vacant.
Noop,
}
/// A pid's-eye view of the slot, for cold paths that hold the slot lock.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub(crate) enum Status {
/// The generation no longer matches: the slot was reclaimed (and possibly
/// reused) — the pid is stale.
Stale,
/// The actor terminated; its outcome is (or was) in the slot.
Done,
/// Alive in some scheduling state (Queued / Running / Notified / Parked).
Live,
}
pub(crate) struct StateWord(AtomicU64);
impl StateWord {
pub(crate) fn new() -> Self {
Self(AtomicU64::new(pack(0, 0, ST_VACANT)))
}
#[inline]
pub(crate) fn load(&self) -> u64 {
self.0.load(Ordering::Acquire)
}
#[inline]
pub(crate) fn generation(&self) -> u32 {
word_gen(self.load())
}
#[inline]
pub(crate) fn status_for(&self, gen: u32) -> Status {
let w = self.load();
if word_gen(w) != gen {
return Status::Stale;
}
match word_state(w) {
ST_DONE => Status::Done,
// A matching generation on a Vacant slot is unreachable for any
// ISSUED pid — reclaim bumps the generation in the very store
// that vacates, and install publishes Queued before the pid
// escapes. But `Pid::new` is public, so a forged / never-issued
// pid (e.g. `Pid::new(5, 0)` against a fresh slab) can land
// here; for those, "no such actor" is the correct total answer.
ST_VACANT => Status::Stale,
_ => Status::Live,
}
}
/// Spawn-side publish: Vacant → Queued. The caller owns the vacant slot
/// exclusively (it popped the index from the free list), so this is a
/// plain Release store; it is the moment the actor becomes visible to
/// pops, unparks, and stops. The epoch starts at 0 for each occupancy
/// (`set_done` zeroes it; wait identity never crosses a lifetime).
pub(crate) fn publish_queued(&self, gen: u32) {
debug_assert_eq!(
self.load(),
pack(gen, 0, ST_VACANT),
"publish over a non-vacant slot"
);
self.0.store(pack(gen, 0, ST_QUEUED), Ordering::Release);
}
/// Scheduler pop-side claim: Queued → Running, epoch preserved. `false`
/// means the popped pid is stale — by the at-most-once-enqueued
/// invariant, a generation mismatch is the only possible failure
/// (asserted). Nothing can move a matching-gen word off Queued (wakes
/// no-op on Queued), so the CAS loop is single-shot in practice.
#[must_use]
pub(crate) fn try_claim(&self, gen: u32) -> bool {
loop {
let w = self.load();
if word_gen(w) != gen {
return false;
}
debug_assert_eq!(
word_state(w),
ST_QUEUED,
"queued pid found in unexpected state {} — double enqueue?",
word_state(w)
);
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), ST_RUNNING),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return true;
}
}
}
/// Yield return path: Running | RunningNotified → Queued, epoch
/// preserved. A notification that arrived mid-run coalesces into the
/// re-queue. Caller must enqueue. CAS loop because a notify can bump the
/// epoch between the read and the exchange.
pub(crate) fn yield_return(&self, gen: u32) {
loop {
let w = self.load();
debug_assert!(
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(w) == gen,
"yield return from invalid word {w:#x}"
);
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), ST_QUEUED),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return;
}
}
}
/// Park return path. `true` = actually parked. `false` = an unpark landed
/// in the prep-to-park window (RunningNotified); the word is already back
/// to Queued and the caller must enqueue — the lost-wakeup window,
/// closed. Epoch preserved on both paths (the notify already consumed
/// it).
#[must_use]
pub(crate) fn park_return(&self, gen: u32) -> bool {
loop {
let w = self.load();
debug_assert_eq!(word_gen(w), gen, "park return with stale gen");
let target = match word_state(w) {
ST_RUNNING => ST_PARKED,
ST_RUNNING_NOTIFIED => ST_QUEUED,
st => unreachable!("park return from invalid state {st}"),
};
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), target),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return target == ST_PARKED;
}
}
}
/// Open a new wait: bump the park-epoch and return it. Called by the
/// waiting actor itself (so the state is Running, or RunningNotified if
/// a terminal wake is already pending — the bump preserves the pending
/// notification), once per wait, BEFORE registering `(pid, epoch)` with
/// any waker.
#[must_use]
pub(crate) fn begin_wait(&self, gen: u32) -> u32 {
loop {
let w = self.load();
debug_assert!(
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(w) == gen,
"begin_wait from invalid word {w:#x}"
);
let next = word_epoch(w).wrapping_add(1) & EPOCH_MASK;
if self
.0
.compare_exchange(
w,
pack(gen, next, word_state(w)),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return next;
}
}
}
/// The unpark protocol — the one way anything outside the scheduler makes
/// an actor runnable. See [`Unpark`] for the caller's obligations.
///
/// `want = Some(epoch)` is the epoch-matched form: lands only if the word
/// still carries that epoch (i.e. the wait it was registered for is still
/// the current, un-woken wait). `want = None` is the wildcard, reserved
/// for terminal wakes. Both forms CONSUME the epoch on success.
#[must_use]
pub(crate) fn unpark(&self, gen: u32, want: Option<u32>) -> Unpark {
loop {
let w = self.load();
if word_gen(w) != gen {
return Unpark::Noop;
}
if let Some(e) = want {
if word_epoch(w) != e {
return Unpark::Noop;
}
}
let bumped = word_epoch(w).wrapping_add(1) & EPOCH_MASK;
match word_state(w) {
ST_PARKED => {
if self
.0
.compare_exchange(
w,
pack(gen, bumped, ST_QUEUED),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return Unpark::Enqueue;
}
}
ST_RUNNING => {
if self
.0
.compare_exchange(
w,
pack(gen, bumped, ST_RUNNING_NOTIFIED),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return Unpark::Notified;
}
}
_ => return Unpark::Noop, // Queued | Notified | Done | Vacant
}
}
}
/// Eat a pending notification: RunningNotified → Running, epoch
/// preserved; no-op on Running. Called by the RUNNING actor itself, on
/// the no-park exit of a wait it registered for but never parked on
/// (`select` returning a ready arm at registration time), AFTER bumping
/// the epoch and BEFORE re-checking its stop flag:
///
/// - post-bump, the only wakers that can have set RunningNotified are
/// ones stamped with the just-retired epoch (a select arm) or a
/// terminal wildcard (`request_stop`);
/// - the caller's stop-flag check AFTER the clear catches the terminal
/// case (the flag is set before the wake fires), so eating its
/// notification loses nothing — and a stop arriving later re-notifies
/// a Running word as usual;
/// - what remains eaten is exactly the stale arm wake that would
/// otherwise fault the actor's next one-shot park.
///
/// Returns whether a notification was eaten.
pub(crate) fn clear_notify(&self, gen: u32) -> bool {
loop {
let w = self.load();
debug_assert!(
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(w) == gen,
"clear_notify from invalid word {w:#x}"
);
if word_state(w) != ST_RUNNING_NOTIFIED {
return false;
}
if self
.0
.compare_exchange(
w,
pack(gen, word_epoch(w), ST_RUNNING),
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
return true;
}
}
}
/// Finalize: Running | RunningNotified → Done, epoch zeroed (wait
/// identity never crosses an occupancy). Called by the scheduler that
/// just ran the actor to completion (so those are the only legal prior
/// states), under the slot's cold lock so join's check-or-register is
/// linearized against it.
pub(crate) fn set_done(&self, gen: u32) {
let prev = self.0.swap(pack(gen, 0, ST_DONE), Ordering::AcqRel);
debug_assert!(
matches!(word_state(prev), ST_RUNNING | ST_RUNNING_NOTIFIED)
&& word_gen(prev) == gen,
"finalize from invalid word {prev:#x}"
);
}
/// Reclaim: Done → Vacant(gen + 1). The generation bump IS the reclaim:
/// every stale pid is dead from this store onwards. Caller holds the cold
/// lock and has verified eligibility (asserted).
pub(crate) fn reclaim(&self, gen: u32) {
debug_assert_eq!(
self.load(),
pack(gen, 0, ST_DONE),
"reclaim of a non-Done slot"
);
self.0
.store(pack(gen.wrapping_add(1), 0, ST_VACANT), Ordering::Release);
}
}
// ---------------------------------------------------------------------------
// loom model tests — RUSTFLAGS="--cfg loom" cargo test --lib --release
// ---------------------------------------------------------------------------
#[cfg(all(test, loom))]
mod loom_tests {
use super::*;
use loom::sync::atomic::{AtomicBool, AtomicUsize};
use loom::sync::Arc;
use loom::thread;
use std::sync::atomic::Ordering as O;
/// THE lost-wakeup theorem. A waiter registers a condition check then
/// parks (as every parking site does); a waker sets the condition then
/// unparks. In every interleaving the waiter must end up runnable —
/// parked-forever-with-condition-set must be unreachable.
#[test]
fn no_lost_wakeup_park_vs_unpark() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0)); // scheduler claimed: actor Running
let epoch = word.begin_wait(0); // actor opens the wait
let ready = Arc::new(AtomicBool::new(false));
let enqueues = Arc::new(AtomicUsize::new(0));
// Waker: make the condition true, then wake the registered wait.
let w = word.clone();
let r = ready.clone();
let e = enqueues.clone();
let waker = thread::spawn(move || {
r.store(true, O::SeqCst);
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
e.fetch_add(1, O::SeqCst);
}
});
// Waiter (as the scheduler executes it): re-check the condition,
// park only if still false; a Notified park-return re-queues.
let parked = if ready.load(O::SeqCst) {
false // condition already visible: doesn't park at all
} else if word.park_return(0) {
true
} else {
enqueues.fetch_add(1, O::SeqCst); // notified → re-queued
false
};
waker.join().unwrap();
let w = word.load();
if parked {
// Parked is only a FINAL state if the waker's unpark moved it
// back to Queued (+ one enqueue). Parked-and-stays-parked
// would be the lost wakeup.
assert_eq!(word_state(w), ST_QUEUED, "lost wakeup: parked forever");
assert_eq!(enqueues.load(O::SeqCst), 1);
} else {
// Never more than one enqueue (at-most-once-enqueued).
assert!(enqueues.load(O::SeqCst) <= 1);
}
});
}
/// Two concurrent unparkers, one parked actor: exactly one wins the
/// enqueue (at-most-once), regardless of interleaving. Both stamped with
/// the live epoch — the consuming bump is what serializes them.
#[test]
fn two_unparkers_one_enqueue() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0));
let epoch = word.begin_wait(0);
assert!(word.park_return(0)); // actor parked
let enqueues = Arc::new(AtomicUsize::new(0));
let mut hs = Vec::new();
for _ in 0..2 {
let w = word.clone();
let e = enqueues.clone();
hs.push(thread::spawn(move || {
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
e.fetch_add(1, O::SeqCst);
}
}));
}
for h in hs {
h.join().unwrap();
}
assert_eq!(enqueues.load(O::SeqCst), 1);
assert_eq!(word_state(word.load()), ST_QUEUED);
});
}
/// The stale-epoch theorem — what `select`'s loser arms lean on. An
/// actor opens a wait, two registered wakers race it (against the park
/// itself, covering the prep-to-park window); afterwards the actor is
/// runnable exactly once, and a LATE waker still stamped with the
/// consumed epoch can neither enqueue nor notify — in every
/// interleaving. (Under wildcard semantics the late waker would corrupt
/// the actor's NEXT one-shot park; this is the theorem that buys
/// `Mutex::lock_timeout`/`sleep`/`block_on_io` their unchanged code.)
#[test]
fn consumed_epoch_unpark_never_lands() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0));
let epoch = word.begin_wait(0);
// Two arms race the wake, concurrent with the park itself.
let enqueues = Arc::new(AtomicUsize::new(0));
let mut hs = Vec::new();
for _ in 0..2 {
let w = word.clone();
let e = enqueues.clone();
hs.push(thread::spawn(move || {
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
e.fetch_add(1, O::SeqCst);
}
}));
}
let mut runnable_via_notify = false;
if !word.park_return(0) {
runnable_via_notify = true; // notified in prep-to-park
}
for h in hs {
h.join().unwrap();
}
// Exactly one path made the actor runnable.
let direct = enqueues.load(O::SeqCst);
if runnable_via_notify {
assert_eq!(direct, 0, "woken twice: notify AND enqueue");
} else {
assert_eq!(direct, 1, "parked forever, or woken twice");
}
assert_eq!(word_state(word.load()), ST_QUEUED);
// The actor runs again. A waker still holding the OLD epoch —
// a select loser arm firing later — must be a strict no-op,
// not a pending notification.
assert!(word.try_claim(0));
assert_eq!(word.unpark(0, Some(epoch)), Unpark::Noop);
assert_eq!(word_state(word.load()), ST_RUNNING, "stale epoch notified a live run");
});
}
/// The retire theorem — `select`'s no-park exit. An actor opens a wait
/// and registers, then finds an arm ready and returns WITHOUT parking;
/// a loser arm's waker fires concurrently, stamped with the live epoch.
/// The exit retires the wait (bump, then eat): in every interleaving
/// the run ends on a clean Running word — no pending notification
/// survives to fault the actor's next one-shot park — and the waker
/// never enqueues.
#[test]
fn retire_eats_late_arm_notification() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
assert!(word.try_claim(0));
let epoch = word.begin_wait(0); // select opens + registers
let w = word.clone();
let waker = thread::spawn(move || w.unpark(0, Some(epoch)));
// No-park exit: bump (invalidates in-flight wakes), then eat
// (consumes one that already landed).
let _ = word.begin_wait(0);
word.clear_notify(0);
assert_ne!(waker.join().unwrap(), Unpark::Enqueue);
assert_eq!(
word_state(word.load()),
ST_RUNNING,
"stale arm wake survived the retire"
);
});
}
/// The ABA theorem: a stale-generation unpark racing reclaim + reuse can
/// never touch the slot's new occupant.
#[test]
fn stale_unpark_never_hits_reused_slot() {
loom::model(|| {
let word = Arc::new(StateWord::new());
// Gen-0 actor runs to completion.
word.publish_queued(0);
assert!(word.try_claim(0));
let w = word.clone();
let stale = thread::spawn(move || w.unpark(0, None));
// Scheduler: finalize, reclaim, and a new spawn reuses the slot.
word.set_done(0);
word.reclaim(0);
word.publish_queued(1);
// The stale unpark may have squeezed in only while gen 0 was
// still Running (→ Notified) — in which case set_done's swap
// absorbed it — or it observed Done/Vacant/gen-1 and no-op'd.
// Either way it must never claim an enqueue.
assert_ne!(stale.join().unwrap(), Unpark::Enqueue);
// And the new occupant is exactly where its spawn put it.
assert_eq!(word.load(), pack(1, 0, ST_QUEUED));
});
}
/// Unpark racing the claim itself: whatever the interleaving, the actor
/// is Running or RunningNotified afterwards and nobody enqueued (it was
/// never parked).
#[test]
fn unpark_vs_claim_coalesces() {
loom::model(|| {
let word = Arc::new(StateWord::new());
word.publish_queued(0);
let w = word.clone();
let unparker = thread::spawn(move || w.unpark(0, None));
assert!(word.try_claim(0)); // the entry is ours; claim must win
let r = unparker.join().unwrap();
assert_ne!(r, Unpark::Enqueue);
let st = word_state(word.load());
assert!(matches!(st, ST_RUNNING | ST_RUNNING_NOTIFIED));
});
}
}
src/sync_shim.rs
-37
View File
@@ -1,37 +0,0 @@
//! std vs loom indirection for the modules that loom model-checks
//! (`slot_state`, `run_queue`). Everything else uses std paths directly —
//! the full runtime (context switches, futexes, real TLS) is not loom-able
//! and is never executed under `cfg(loom)`.
//!
//! Build the loom models with: `RUSTFLAGS="--cfg loom" cargo test --lib --release`
#[cfg(loom)]
pub(crate) use loom::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
#[cfg(not(loom))]
pub(crate) use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
/// `UnsafeCell` with loom's `with`/`with_mut` access API; pass-through cost
/// is zero in normal builds (`#[inline]`, newtype over std's cell).
#[cfg(loom)]
pub(crate) use loom::cell::UnsafeCell;
#[cfg(not(loom))]
pub(crate) struct UnsafeCell<T>(std::cell::UnsafeCell<T>);
#[cfg(not(loom))]
impl<T> UnsafeCell<T> {
pub(crate) fn new(v: T) -> Self {
Self(std::cell::UnsafeCell::new(v))
}
#[inline]
pub(crate) fn with<R>(&self, f: impl FnOnce(*const T) -> R) -> R {
f(self.0.get())
}
#[inline]
pub(crate) fn with_mut<R>(&self, f: impl FnOnce(*mut T) -> R) -> R {
f(self.0.get())
}
}
src/timer.rs
+16 -19
View File
@@ -18,9 +18,9 @@
//! No cancellation. When a non-timer wakeup happens (e.g. lock granted
//! before timeout), the timer entry is left in the heap. It will be popped
//! eventually and the dispatch will observe "actor is no longer parked /
//! the wait's epoch was consumed" and no-op. Cost is ~32 bytes per stale
//! entry plus a few cycles on pop; acceptable given the upper bound is "one
//! entry per parked actor".
//! wait_seq is stale" and no-op. Cost is ~32 bytes per stale entry plus a
//! few cycles on pop; acceptable given the upper bound is "one entry per
//! parked actor".
//!
//! Stale pids (slot reused since the timer was inserted) are filtered on
//! pop by the scheduler — same convention as the run queue.
@@ -35,21 +35,18 @@ use std::time::{Duration, Instant};
///
/// Held inside `Entry`, dispatched by the scheduler in `pop_due`.
pub enum Reason {
/// `sleep(d)`. Wake `pid` via the epoch-matched unpark: if anything
/// else (necessarily a terminal wake) already consumed the wait, the
/// entry is stale and no-ops at the CAS.
Sleep { epoch: u32 },
/// A bounded wait (`Mutex::lock_timeout`, `Receiver::recv_timeout`,
/// `select_timeout`). On expiry the scheduler calls
/// `target.on_timeout(pid, epoch)`. The target then decides whether
/// `pid` was actually still waiting (registration still present under
/// its lock), and if so takes the registration and unparks via
/// `unpark_at`. The epoch is the slot-word park-epoch — the runtime-wide
/// wait identity — so a stale entry is doubly inert: the registration
/// check misses, and even a racing unpark fails the word's epoch CAS.
/// `loom::sleep(d)`. Unpark `pid` unconditionally (modulo the usual
/// "still parked?" check the scheduler applies).
Sleep,
/// A bounded wait — currently only `Mutex::lock_timeout`. On expiry the
/// scheduler calls `target.on_timeout(pid, wait_seq)`. The target then
/// decides whether `pid` was actually still waiting, and if so unparks
/// it with whatever error the wait was bounded for. `wait_seq` lets the
/// target tell apart "this wait" from "a later wait by the same actor
/// on the same target".
WaitTimeout {
target: Arc<dyn TimerTarget>,
epoch: u32,
wait_seq: u64,
},
}
@@ -58,7 +55,7 @@ pub enum Reason {
/// Implementors: do not touch `SchedulerState` other than via the public
/// `unpark` / channel APIs. The scheduler is mid-iteration when this fires.
pub trait TimerTarget: Send + Sync {
fn on_timeout(&self, pid: Pid, epoch: u32);
fn on_timeout(&self, pid: Pid, wait_seq: u64);
}
pub struct Entry {
@@ -107,8 +104,8 @@ impl Timers {
}
/// Insert a `Sleep` timer. Convenience for the common case.
pub fn insert_sleep(&mut self, deadline: Instant, pid: Pid, epoch: u32) {
self.insert(deadline, pid, Reason::Sleep { epoch });
pub fn insert_sleep(&mut self, deadline: Instant, pid: Pid) {
self.insert(deadline, pid, Reason::Sleep);
}
/// Insert an arbitrary timer entry.
task.md
+263
View File
@@ -0,0 +1,263 @@
# smarm — task.md (next steps)
Handoff for a future session reusing this sandbox. Read top to bottom once
before starting; the gotchas section is hard-won and will save you a faceplant.
## Resume the environment
- Repo: `smarm`. Two branches (the old single `arm-port` stack was split):
- `master` — the mainline, and HEAD. Carries roadmap #1–#5: cooperative
cancellation, supervisor strategies (one_for_one/all, rest_for_one) + the
orphaned-timer shutdown fix, links/trap_exit, selective receive, gen_server,
and demonitor/`MonitorId`. Tagged `v0.4.0`. x86-64 Linux only.
- `arm-port` — `master` plus a single commit: `feat(arch): aarch64 context
switch + cycle counter`. Extracts the x86-64 context-switch / stack-init /
cycle-counter out of `context.rs` into a `target_arch`-gated `src/arch/`
(x86_64 + aarch64 backends) and adds an AAPCS64 backend. ⚠️ UNTESTED: never
built or run on real ARM hardware. The x86-64 path is unchanged
(`arch/x86_64.rs` is the old `context.rs` body verbatim), so the x86 suite
passing says nothing about the aarch64 backend. Build + test on-device
before trusting it.
- Toolchain is installed but NOT on PATH in a fresh shell. First line of every
session: `. "$HOME/.cargo/env"` (rustc/cargo 1.96).
- Build `cargo build` · all tests `cargo test` · one suite `cargo test --test monitor`.
- Bench probe `cargo bench --bench general` (custom print-only harness; compiles
tokio in release the first time — slow — but `target/` persists across git
checkouts so it's paid once).
- Perf regression check: `git checkout <pre-change-sha>` →
`cargo bench --bench general | tee before.txt` → `git checkout arm-port` →
run again → diff the `smarm 1-thread` medians for `chained_spawn` and
`yield_many` (those exercise spawn/finalize/scheduler). Numbers are noisy on
this shared CPU; treat as "regression beyond noise?" not a precise delta.
## Roadmap (dependency order)
### 1. Cooperative cancellation — the keystone ✅ DONE (`a8ddb4a`)
Everything below (one_for_all/rest_for_one, links) needs a *safe* way to stop a
running peer. Forcible teardown of another green thread's stack is unsound here
(shared heap + Drop). So: cooperative stop the actor observes and unwinds itself.
Shipped as designed (sentinel unwind, not Result-threading). Notes for what
came next / future readers:
- Stop flag lives on `Actor` behind `Arc<AtomicBool>` (fresh per spawn), NOT a
`Slot` field — sidesteps the three-place reset, at the cost of one small
alloc per spawn. The scheduler hands the resume path a raw `*const AtomicBool`
(no per-resume refcount traffic); `yield_many` bench stayed at baseline,
`chained_spawn` ~+6% from that alloc (left as-is; move to a `Slot` field if it
ever matters).
- Observation points: amortised `maybe_preempt`/`check!()` path + the wakeup
side of `park_current`/`yield_now`. Sentinel = `StopSentinel` (zero-size),
recognised in the trampoline → `Outcome::Stopped`. `join()` on a stopped actor
returns `Ok(())` (no payload to propagate; reason is on the monitor channel).
- Documented gaps confirmed by tests: no-observation-point loop can't be stopped
(same as preemption); a user `catch_unwind` can swallow the sentinel but the
flag stays set so the next yield re-raises.
Original plan, for reference:
- Add a per-actor stop flag (Slot field + atomic, or check via shared state).
- `request_stop(pid)`: set the flag, unpark if parked.
- Realize the stop as a **controlled unwind**: when the scheduler resumes a
stop-requested actor, inject a sentinel panic (dedicated payload type) so the
existing `trampoline` `catch_unwind` tears the stack down and runs Drop. The
trampoline recognizes the sentinel and reports a new `Outcome::Stopped`
(distinct from a user `Panic`). This avoids changing every blocking-op
signature.
- Alternative considered: thread `Result<_, Cancelled>` through recv/sleep/
lock/io. Rejected — large API churn. Go with the sentinel unwind.
- Caveat to document: user code with its own `catch_unwind` can swallow the
sentinel (cf. Erlang `catch`); re-check the flag at the next yield and/or
re-raise. And a tight no-alloc loop without `check!()` can't be stopped —
same inherent limitation as preemption.
- Observation points: `maybe_preempt()`/`check!()` (cheap flag check) and the
blocking parks (recv/sleep/mutex/io) on the stop-driven unpark.
- Tests: looping actor on `check!()` gets stopped → `Outcome::Stopped`, Drop
guards ran; parked-on-recv actor gets stopped; no-check loop documents the gap.
### 2. one_for_all / rest_for_one + ordered shutdown ✅ DONE (`351dc9c`)
Shipped. What landed vs the plan:
- `Strategy::{OneForOne,OneForAll,RestForOne}` selected via `.strategy()`,
default `OneForOne`. The struct keeps the `OneForOne` name (compat; existing
tests untouched) — a rename to `Supervisor` is a deferred refactor.
- The *triggering* child's `Restart` policy decides whether anything restarts;
the strategy decides which live siblings are cycled (all / index-> after the
failed one). Survivors are `request_stop`'d in reverse start order, awaited on
the existing `supervisor_channel` funnel (no new channel, no `select`),
restarted in start order. One failure = one intensity tick regardless of group
size. Out-of-band signals during an await are stashed and replayed.
- `Signal::Stopped(pid)` + `DownReason::Stopped` added (kept distinct from Exit,
as planned). A `Stopped` signal counts as abnormal for the restart decision.
- Ordered shutdown: on cap-trip / mailbox-close-with-survivors, stop remaining
children in reverse start order and await them (no-op on the normal exit).
- ⚠️ Surfaced + fixed a latent keystone bug (`e80334b`): a cancelled
sleeping/timeout actor orphans its timer entry, and the scheduler's shutdown
check counted pending timers → `run()` hung until the dead actor's deadline
fired (a `sleep(30s)` sleeper hung shutdown 30s). Fix: timers no longer gate
shutdown (`live == 0` already implies nothing a timer could wake); heap is
cleared on exit. Independent of the supervisor work.
- Tests: all-restart (sibling cycled despite clean exit), suffix-restart
(prefix child left alone), reverse-order teardown.
Original plan, for reference:
- one_for_all: on any child failure, `request_stop` all siblings, await their
termination signals, restart all per spec.
- rest_for_one: stop+restart the failed child and those started after it.
- Supervisor shutdown: stop children in reverse start order.
- Decide signal surface: add `Signal::Stopped(pid)` + `DownReason::Stopped`
rather than folding into Exit (clearer for the supervisor's await logic).
- Tests: all-restart, suffix-restart, reverse-order shutdown.
### 3. Links + trap_exit ✅ DONE (`6581484`)
- `Slot.links: Vec<Pid>` (bidirectional); `link`/`unlink`; `trap_exit()` flag
lives on `Actor` (fresh per spawn → a restarted child starts un-trapped, and
no fourth slot-reset site).
- On finalize, reverse links are cleared under the lock (always — keeps the
cascade acyclic), then for each linked peer: abnormal death (`Panic`/
`Stopped`) → `request_stop(peer)` unless peer traps, in which case deliver an
`ExitSignal` *message* instead. Normal exit does NOT propagate. Linking an
already-dead pid delivers an immediate `NoProc` signal (message if trapping,
else `request_stop(self)` — not a silent no-op).
- Resolved: the trap inbox is a **dedicated** channel (`trap_exit() ->
Receiver<ExitSignal>`), distinct from the monitor `Down` channel; `ExitSignal`
reuses `DownReason` and carries no panic payload (joiner-only, as with
monitors). `spawn_link` deferred to #5.
- Tests (`tests/link.rs`): linked pair one panics → other stopped (+Drop ran);
trap_exit → other gets a message and survives; normal exit doesn't propagate;
dead-pid link stops a non-trapper / messages a trapper; `unlink` prevents
propagation.
### 4. Selective receive (independent track) ✅ DONE (`03f3875`)
Shipped. What landed vs the plan:
- `Receiver::recv_match(pred) -> Result<T, RecvError>` scans the queued
`VecDeque` front-to-back, removes+returns the first match, leaves the rest in
arrival order; parks and re-scans when nothing matches. `try_recv_match` (the
non-blocking variant, mirroring `try_recv`) rolled in same commit.
- Wakeup turned out cheaper than feared: `Sender::send` *already* took
`parked_receiver` on every push, so "wake on ANY send" needed no send-side
change. The one real edit was relaxing `Sender::drop` to unpark the parked
receiver on the last-sender drop regardless of queue emptiness — a selective
receiver can park on a non-empty no-match queue and must wake to observe
closure. No-op for plain `recv` (only ever parks on an empty queue); stress
suite stays green.
- `pred` is `Fn(&T) -> bool` (not `FnMut`) on purpose: it's re-run from scratch
on every scan, so a stateful predicate would re-count surprisingly. It runs
under the channel lock — keep it cheap/pure, don't re-enter the channel.
- Close semantics: `recv_match` returns `Err(RecvError)` only when closed AND no
queued message matches; a match is still returned on a closed channel.
Non-matches are left for a later `recv`.
- Tests (`tests/selective_recv.rs`): out-of-order match pulled first; non-matches
remain in order; park-on-non-empty then wake on a match; closed-with-only-
non-matches → Err; closed-but-match-present → match; `try_recv_match` states.
Original plan, for reference:
- Add `Receiver::recv_match(pred) -> T`: scan the queued `VecDeque`, remove+return
first match, leave the rest in order; park and re-scan on new arrivals.
- This changes channel wakeup: a parked selective receiver must wake on ANY send
(not just empty→nonempty) and re-scan. Touches `channel.rs` carefully — the
stress tests guard lost-wakeup invariants; keep them green.
- Tests: messages arrive out of interest-order; match pulled first; non-matches
remain for a later `recv`.
### 5. Grab-bag (each its own small commit)
- `spawn_link`: spawn-and-link atomically (deferred from #3); thin wrapper over
`spawn_under` + `link`, but do it under one lock so there's no window where
the child dies before the link is recorded.
- `demonitor`: needs a per-monitor id to remove a specific sender. Decide the
monitor API NOW before more code depends on it — likely return a
`Monitor { id, rx }` instead of a bare `Receiver<Down>`.
✅ DONE (this commit). What landed vs the plan:
- `monitor()` now returns `Monitor { id, target, rx }` (added `target` over
the sketched `{id, rx}` so `demonitor` jumps straight to the slot instead of
scanning every slot for the id). `MonitorId(u64)` is opaque, from a
monotonic `next_monitor_id` counter on `SharedState`, bumped under the
shared lock in `monitor()` — no atomics, deterministic, never reused.
- `Slot.monitors: Vec<(MonitorId, Sender<Down>)>`. The three slot-reset sites
were untouched — they `.clear()`/`Vec::new()`, which is element-type-
agnostic, so no new reset obligation. `finalize_actor` just destructures
`(_, m)` and sends as before.
- `demonitor(&Monitor) -> Option<MonitorId>`: `Some(id)` when a live
registration was found+removed, `None` when it had already fired (drained on
finalize), was `NoProc`, or the slot was reclaimed. Chose `Option<MonitorId>`
over a bare bool — names which registration went. Generation half of the pid
makes a stale demonitor a clean no-op: a recycled slot index fails
`slot_mut`'s generation check, so it can never strip a different actor's
monitor.
- ⚠️ Reentrancy: the removed `Sender` is `remove`d out of the Vec under the
lock but **dropped after the lock is released** — `Sender::drop` can unpark a
parked receiver → `with_shared`, and the shared mutex is non-reentrant. Same
discipline as `finalize_actor`.
- "Flush" (discard a `Down` the target already queued) falls out of dropping
the `Monitor`: `demonitor(&m); drop(m)`. That's the cleanup the still-to-come
gen_server **call timeout** wants — monitor the server, wait reply-or-Down-
or-deadline, then demonitor+drop so a timed-out call leaks no registration
and no stale `Down`.
- Perf: touches `Slot` + `finalize_actor`, but `chained_spawn`/`yield_many`
register no monitors, so the Vec stays empty (take-empty is identical cost,
finalize loop runs zero times). before/after `general` probe medians within
noise. Tests (`tests/monitor.rs`): demonitor-stops-delivery, one-of-many
(siblings untouched), after-fire-is-None.
- Named registry: `register(name,pid)`/`whereis`/`send_by_name`; a
`HashMap<String,Pid>` in `SharedState`.
- gen_server-style call/cast: request-reply correlation as a thin layer over
channels (`call` sends `{req, reply_tx}`, awaits `reply_rx`); no runtime change.
✅ DONE (`a4fcf6c`). What landed vs the plan:
- `GenServer` trait on the state value: assoc `Call`/`Reply`/`Cast` types,
required `handle_call`/`handle_cast`, optional `init`/`terminate` hooks.
`ServerRef<G>` is a clonable inbox sender + `pid()`; `start` / `start_under`.
- One inbox, not two: a single `Envelope { Call(req, reply_tx) | Cast }`
channel, dispatched by variant. Forced by no-`select`/no-unified-mailbox —
a server can't wait on a call channel and a cast channel at once.
- Server-down falls out of channel closure (no monitor needed): `send` fails
if the inbox is gone; the reply sender drops on the server's unwind so a
parked caller wakes to `Err`. Both → `Call/CastError::ServerDown`.
- `terminate` runs via a drop guard → fires on *every* exit path (clean inbox
close, handler panic, `request_stop`), not just the clean one. Caveat: it
may run mid-unwind, so keep it non-blocking (a panic inside it during an
unwind double-panics → abort).
- No `handle_info`, no call timeout — both deferred to land with timeouts
(`handle_info` needs the still-unmade cross-channel mailbox merge; a call
timeout needs a per-`recv` deadline / `Signal::Timeout`).
- Pure additive layer (no Slot/scheduler/spawn/finalize change) → no perf
check run. Tests (`tests/gen_server.rs`): cast→call roundtrip, init/terminate
ordering, both server-down paths.
- Docs: README now points at the experimental, untested aarch64 port on the
`arm-port` branch. The module table still calls `context` x86-64-only — true
for `master`, since the `src/arch/` split rides on `arm-port`. Fold the arch/
split and ARM64-supported wording into the README module table + build section
once `arm-port` is validated on hardware and merged.
## Gotchas / invariants (respect these)
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
`with_shared`. NEVER send on a channel while holding the shared lock. Pattern:
`mem::take` the senders/data under the lock, send after releasing. See
`finalize_actor` (supervisor signal + monitor Downs both sent post-lock).
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
Done/outcome → recycle stack (post-lock) → deliver supervisor Signal + monitor
Downs (post-lock) → unpark joiners → reclaim slot iff `outstanding_handles==0`.
Death notifications always precede reclamation, so a pid carried in a
Signal/Down is still matchable even as its slot is about to be reused.
- **Slot lifecycle is reset in THREE places** — `Slot::vacant()`,
`reclaim_slot()` (runtime.rs), and the slot-init block in `spawn_under`
(scheduler.rs). Any new Slot field must be reset in all three (monitors was).
- **Pid = (index, generation);** stale handles caught by generation mismatch in
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
- **No `select`, no unified per-process mailbox.** Why the supervisor uses the
single `supervisor_channel` funnel rather than N monitor channels. trap_exit
resolved this by giving each trapping actor a dedicated `Receiver<ExitSignal>`
inbox (see #3); selective receive (#4) stayed *per-channel* (`recv_match`
scans one channel's queue) rather than introducing a cross-channel mailbox —
if selective receive ever needs to span the monitor/trap inboxes too, that
cross-channel merge is the still-unmade decision.
- **Cooperative-only**: preemption and (future) cancellation both depend on the
actor reaching `check!()`/yield/alloc/blocking points.
- `run()` is single-thread (`Config::exact(1)`); tests rely on deterministic
single-thread ordering (parent runs until it parks). Multi-thread via
`runtime::init(Config…)`.
## Workflow expectations (from the human)
- TDD: write the failing test first, then implement.
- Commit incrementally with conventional-commit messages; keep each commit a
reviewable unit (they diff in their IDE and are the filter to the codebase).
- Run the full suite before each commit; check perf when a change touches
Slot/scheduler/spawn/finalize hot paths.
tests/README.md
-120
View File
@@ -1,120 +0,0 @@
# Tests
Integration tests for the runtime. Each file owns one feature area or one
class of bug. Everything here runs under plain `cargo test`; the loom model
tests are the exception — they live **in the library** (`src/slot_state.rs`,
`src/run_queue.rs`), not in this directory, because loom must compile the
production code with shimmed atomics (see "Loom" below).
## Running
```
cargo test # debug build — RUN THIS ONE: all invariant asserts live
cargo test --release # what users actually execute (LTO, no debug_asserts)
```
Debug builds are not just "slower tests": the runtime self-checks its
invariants only there — every `StateWord` transition asserts its
precondition, `enqueue` asserts the exact `(gen, Queued)` word, `RawMutex`
enforces the never-two-cold-locks leaf rule with a per-thread held-count,
`live_actors` checks for double-finalize underflow. A green release run with
a red debug run means an invariant broke without (yet) corrupting behavior —
treat it as a real failure.
### The queue-variant matrix
The run queue is compile-time selected; the suite must pass under all three
(features are additive, so drop the default first):
```
cargo test # rq-mutex (default)
cargo test --no-default-features --features rq-mpmc
cargo test --no-default-features --features rq-striped
```
### Loom (model checking)
```
RUSTFLAGS="--cfg loom" cargo test --lib --release
```
Exhaustively explores interleavings of the slot state machine
(`src/slot_state.rs`: lost-wakeup, at-most-once-enqueue, the stale-pid ABA
theorem, unpark-vs-claim) and the ring queues (`src/run_queue.rs`:
exactly-once through lap wraparound, push/pop races). Models run the
production transitions through `src/sync_shim.rs` — std atomics normally,
`loom::sync` under `--cfg loom`. `RawMutex` is deliberately not modeled:
futexes can't be, and it's the textbook Drepper mutex3 with stress and
unwind-safety tests of its own.
### Trace feature
`cargo test --features smarm-trace` exists mainly to catch bit-rot in the
`te!()` call sites; run it after touching scheduler paths.
### Before a runtime-core PR
The full matrix, in rough order of bug-finding power per minute:
1. `cargo test` (debug, default variant)
2. debug under `rq-mpmc` and `rq-striped`
3. `cargo test --release`
4. loom
5. `cargo build --features smarm-trace`
## Catalog
**Low-level units (no scheduler)**
| file | covers |
|---|---|
| `context.rs` | `init_actor_stack` + the naked-asm context-switch shims, poked directly |
| `stack.rs` | the mmap'd stack allocator |
| `pid.rs` | pid packing/equality |
**Feature areas (run under a real runtime)**
| file | covers |
|---|---|
| `runtime.rs` | `Config`, `Runtime::run`, re-running a runtime, correctness under genuine parallelism |
| `scheduler.rs` | spawn / join / panic delivery / `yield_now` / `self_pid` |
| `channel.rs` | send/recv (recv parks, so these need the runtime) |
| `selective_recv.rs` | `recv_match` / `try_recv_match` |
| `mutex.rs` | the actor-blocking `Mutex<T>` (lock parks) |
| `timer.rs` | `sleep` ordering — time-sensitive, generous tolerances by design |
| `io.rs` | `block_on_io`: blocking closures on the pool while the actor parks |
| `io_epoll.rs` | `wait_readable` / `wait_writable` + the `read`/`write` sugar |
| `preempt.rs` | explicit preemption via `smarm::check!()` |
| `cancel.rs` | cooperative cancellation (`request_stop`) — the keystone semantics |
| `monitor.rs` | `monitor` delivers exactly one `Down`; `demonitor` |
| `link.rs` | bidirectional links + `trap_exit` |
| `supervisor.rs` | one-for-one supervision |
| `gen_server.rs` | call/cast round-trips, lifecycle callbacks, server-down detection |
**Regression & stress**
| file | covers |
|---|---|
| `stress.rs` | lost wakeups, pid-table pressure, thundering herds, panic isolation under concurrency. Where the phase-2 RefCell-migration bug was caught. |
| `poison_stop.rs` | `request_stop` racing an alloc-under-lock must not poison/abort. See its header for the full story. |
| `many_timers_multi_thread.rs` | multi-thread sleep-timer lost-wakeup regression |
## Conventions
- **Each test owns its runtime.** `init(Config::exact(N))` + `rt.run(...)`;
never share a `Runtime` between tests. Oversubscription (`exact(4)` on one
core) is deliberate — forced interleaving at yield points is how
single-core CI finds races at all.
- **Regression tests must be validated against the bug.** A regression test
that passes with the bug reintroduced is documentation, not a test.
Reintroduce the fix's inverse locally and watch it fail before trusting it
(`poison_stop.rs` went through exactly this: its first version never fired
the sentinel under a lock, and was rewritten until it SIGABRT'd pre-fix).
- **Stochastic tests get the odds stacked.** Use `Config::alloc_interval(1)`
to make every allocation an observation point, many actors, and both
phases of any every-other-allocation cadence (see
`poison_stop::self_stop_during_spawn...`).
- **Time-based assertions use ordering, not durations.** Assert
"didn't return instantly" / "A woke before B", with generous tolerances;
CI machines are slow and noisy.
- New invariants added to the runtime should come with the assert at the
point of reliance (debug_assert on hot paths) *and*, where the invariant is
a protocol, a loom model in the owning module — that combination is what
made phases 2–5 land without a single post-merge race so far.
tests/channel.rs
-186
View File
@@ -108,189 +108,3 @@ fn recv_returns_err_when_all_senders_dropped() {
assert!(saw_err.load(std::sync::atomic::Ordering::SeqCst));
}
#[test]
fn channel_ops_interleaved_with_monitor_churn_multi_thread() {
// Regression for the RawMutex migration: monitor registration clones the
// Down sender under the target's cold (Leaf) lock, which now nests a
// Channel-class lock under it. Debug builds enforce the Leaf -> Channel
// ordering on every acquisition, so driving channels, monitors, and actor
// death concurrently across schedulers makes any ordering regression
// panic here rather than deadlock in the field.
use std::sync::atomic::{AtomicI64, Ordering};
use std::sync::Arc;
let total = Arc::new(AtomicI64::new(0));
let total2 = total.clone();
smarm::init(smarm::Config::exact(4)).run(move || {
let (tx, rx) = channel::<i64>();
let consumer = spawn(move || {
let mut sum = 0;
while let Ok(v) = rx.recv() {
sum += v;
}
OUT.with(|c| c.set(sum)); // not asserted cross-thread; see total
total2.fetch_add(sum, Ordering::Relaxed);
});
let mut handles = Vec::new();
for i in 0..32i64 {
let tx = tx.clone();
handles.push(spawn(move || {
// Short-lived target whose death fires the monitor below.
let t = spawn(move || {
tx.send(i).unwrap();
});
let m = smarm::monitor(t.pid());
t.join().unwrap();
// Down delivery exercises send-from-finalize.
let d = m.rx.recv().unwrap();
assert_eq!(d.reason, smarm::DownReason::Exit);
}));
}
drop(tx);
for h in handles {
h.join().unwrap();
}
consumer.join().unwrap();
});
assert_eq!(total.load(std::sync::atomic::Ordering::Relaxed), (0..32).sum::<i64>());
}
// ---------------------------------------------------------------------------
// recv_timeout
// ---------------------------------------------------------------------------
use smarm::RecvTimeoutError;
use std::time::{Duration, Instant};
#[test]
fn recv_timeout_returns_queued_message_immediately() {
run(|| {
let (tx, rx) = channel::<i64>();
tx.send(5).unwrap();
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(5));
});
}
#[test]
fn recv_timeout_times_out_on_silent_channel() {
run(|| {
let (_tx, rx) = channel::<i64>();
let start = Instant::now();
let r = rx.recv_timeout(Duration::from_millis(50));
assert_eq!(r, Err(RecvTimeoutError::Timeout));
assert!(start.elapsed() >= Duration::from_millis(50));
});
}
#[test]
fn recv_timeout_wakes_promptly_on_send() {
run(|| {
let (tx, rx) = channel::<i64>();
let h = spawn(move || {
let start = Instant::now();
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(9));
// Far below the timeout: the send woke us, not the deadline.
assert!(start.elapsed() < Duration::from_secs(1));
});
smarm::yield_now();
tx.send(9).unwrap();
h.join().unwrap();
});
}
#[test]
fn recv_timeout_reports_disconnected_on_close() {
run(|| {
let (tx, rx) = channel::<i64>();
let h = spawn(move || {
assert_eq!(
rx.recv_timeout(Duration::from_secs(10)),
Err(RecvTimeoutError::Disconnected)
);
});
smarm::yield_now();
drop(tx);
h.join().unwrap();
});
}
#[test]
fn recv_timeout_zero_duration_is_a_bounded_poll() {
run(|| {
let (_tx, rx) = channel::<i64>();
assert_eq!(rx.recv_timeout(Duration::ZERO), Err(RecvTimeoutError::Timeout));
});
}
#[test]
fn channel_remains_usable_after_a_timeout() {
// The stale timer entry from the first (timed-out) wait must not cancel
// or corrupt later waits — seq isolation.
run(|| {
let (tx, rx) = channel::<i64>();
assert_eq!(
rx.recv_timeout(Duration::from_millis(10)),
Err(RecvTimeoutError::Timeout)
);
// Plain recv still works...
tx.send(1).unwrap();
assert_eq!(rx.recv(), Ok(1));
// ...and so does a second bounded wait, woken by a send.
let h = spawn(move || {
tx.send(2).unwrap();
});
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(2));
h.join().unwrap();
});
}
#[test]
fn recv_timeout_many_waiters_multi_thread() {
// Mixed outcomes under real parallelism: half the channels get fed,
// half time out; every actor must resolve correctly.
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;
let got = Arc::new(AtomicU32::new(0));
let timed_out = Arc::new(AtomicU32::new(0));
let (got2, timed_out2) = (got.clone(), timed_out.clone());
smarm::init(smarm::Config::exact(4)).run(move || {
let mut handles = Vec::new();
for i in 0..24i64 {
let (tx, rx) = channel::<i64>();
let got = got2.clone();
let timed_out = timed_out2.clone();
handles.push(spawn(move || match rx.recv_timeout(Duration::from_millis(100)) {
Ok(v) => {
assert_eq!(v, i);
got.fetch_add(1, Ordering::Relaxed);
}
Err(RecvTimeoutError::Timeout) => {
timed_out.fetch_add(1, Ordering::Relaxed);
}
Err(e) => panic!("unexpected: {e}"),
}));
if i % 2 == 0 {
handles.push(spawn(move || {
tx.send(i).unwrap();
}));
}
// odd i: tx drops here -> Disconnected, not Timeout! Keep it alive
// instead by leaking the sender into a holder actor that outlives
// the deadline.
else {
handles.push(spawn(move || {
smarm::sleep(Duration::from_millis(200));
drop(tx);
}));
}
}
for h in handles {
h.join().unwrap();
}
});
assert_eq!(got.load(std::sync::atomic::Ordering::Relaxed), 12);
assert_eq!(timed_out.load(std::sync::atomic::Ordering::Relaxed), 12);
}
tests/context.rs
+34
View File
@@ -58,6 +58,7 @@ use std::sync::OnceLock;
static REG_BEFORE: OnceLock<[u64; 4]> = OnceLock::new();
static REG_AFTER: OnceLock<[u64; 4]> = OnceLock::new();
#[cfg(target_arch = "x86_64")]
extern "C-unwind" fn actor_reg_check() {
unsafe {
let s0: u64 = 0xAAAA_BBBB_0000_0001;
@@ -83,6 +84,39 @@ extern "C-unwind" fn actor_reg_check() {
}
}
// aarch64 sibling: same intent, AAPCS64 callee-saved integer registers.
// LLVM reserves x19 (and uses x29/x30 as fp/lr), so those can't be named as
// asm operands. We use x23–x26 instead — callee-saved and bindable — moving
// known values into them, yielding, then reading them back, exactly mirroring
// the x86 test's move-into-r12..r15 pattern. The shim saves x23–x26 in its
// `stp x23,x24` / `stp x25,x26` pairs, so a clean round-trip proves they
// survive the switch.
#[cfg(target_arch = "aarch64")]
extern "C-unwind" fn actor_reg_check() {
unsafe {
let s0: u64 = 0xAAAA_BBBB_0000_0001;
let s1: u64 = 0xCCCC_DDDD_0000_0002;
let s2: u64 = 0xEEEE_FFFF_0000_0003;
let s3: u64 = 0x1111_2222_0000_0004;
core::arch::asm!(
"mov x23, {s0}", "mov x24, {s1}", "mov x25, {s2}", "mov x26, {s3}",
s0 = in(reg) s0, s1 = in(reg) s1, s2 = in(reg) s2, s3 = in(reg) s3,
out("x23") _, out("x24") _, out("x25") _, out("x26") _,
);
REG_BEFORE.set([s0, s1, s2, s3]).ok();
switch_to_scheduler();
let a0: u64; let a1: u64; let a2: u64; let a3: u64;
core::arch::asm!(
"mov {a0}, x23", "mov {a1}, x24", "mov {a2}, x25", "mov {a3}, x26",
a0 = out(reg) a0, a1 = out(reg) a1, a2 = out(reg) a2, a3 = out(reg) a3,
);
REG_AFTER.set([a0, a1, a2, a3]).ok();
switch_to_scheduler();
}
}
#[test]
fn callee_saved_registers_survive_yield() {
let stack = Stack::new(64 * 1024).unwrap();
tests/gen_server.rs
+2 -304
View File
@@ -1,7 +1,7 @@
//! gen_server tests: call round-trip, cast, lifecycle callbacks, and the two
//! server-down detection paths (reply-channel close vs. inbox-send failure).
use smarm::gen_server::{start, CallError, GenServer, ServerBuilder};
use smarm::gen_server::{start, CallError, GenServer};
use smarm::run;
use std::sync::{Arc, Mutex};
@@ -26,7 +26,6 @@ impl GenServer for Counter {
type Call = Req;
type Reply = i64;
type Cast = Op;
type Info = ();
fn handle_call(&mut self, req: Req) -> i64 {
match req {
@@ -69,9 +68,8 @@ impl GenServer for Lifecycle {
type Call = ();
type Reply = ();
type Cast = ();
type Info = ();
fn init(&mut self, _ctx: &smarm::gen_server::ServerCtx) {
fn init(&mut self) {
self.log.lock().unwrap().push("init");
}
@@ -137,303 +135,3 @@ fn call_after_server_gone_is_server_down() {
});
assert_eq!(*got.lock().unwrap(), Some(Err(CallError::ServerDown)));
}
// ---------------------------------------------------------------------------
// call_timeout
// ---------------------------------------------------------------------------
use smarm::gen_server::CallTimeoutError;
use std::time::{Duration, Instant};
/// Replies after sleeping `delay_ms` (parking the server actor, not the OS
/// thread), so callers can race a deadline against the reply.
struct Slow;
impl GenServer for Slow {
type Call = u64; // delay in ms
type Reply = u64;
type Cast = ();
type Info = ();
fn handle_call(&mut self, delay_ms: u64) -> u64 {
if delay_ms > 0 {
smarm::sleep(Duration::from_millis(delay_ms));
}
delay_ms
}
fn handle_cast(&mut self, _: ()) {}
}
#[test]
fn call_timeout_returns_reply_within_deadline() {
run(|| {
let srv = start(Slow);
assert_eq!(srv.call_timeout(0, Duration::from_secs(10)), Ok(0));
});
}
#[test]
fn call_timeout_times_out_on_slow_handler() {
run(|| {
let srv = start(Slow);
let start_t = Instant::now();
let r = srv.call_timeout(500, Duration::from_millis(50));
assert_eq!(r, Err(CallTimeoutError::Timeout));
let elapsed = start_t.elapsed();
// Gave up at the deadline, not at the reply.
assert!(elapsed >= Duration::from_millis(50));
assert!(elapsed < Duration::from_millis(500));
});
}
#[test]
fn server_survives_an_abandoned_call_and_late_reply_is_discarded() {
run(|| {
let srv = start(Slow);
assert_eq!(
srv.call_timeout(100, Duration::from_millis(20)),
Err(CallTimeoutError::Timeout)
);
// The timed-out request is still handled; its reply send fails
// harmlessly (receiver dropped). The server must keep serving, and
// the late reply must not leak into THIS call's reply channel.
assert_eq!(srv.call_timeout(0, Duration::from_secs(10)), Ok(0));
// Plain unbounded call still fine too.
assert_eq!(srv.call(0), Ok(0));
});
}
#[test]
fn call_timeout_to_dead_server_is_server_down_not_timeout() {
struct Bomb;
impl GenServer for Bomb {
type Call = ();
type Info = ();
type Reply = ();
type Cast = ();
fn handle_call(&mut self, _: ()) {
panic!("kaboom");
}
fn handle_cast(&mut self, _: ()) {}
}
run(|| {
let srv = start(Bomb);
// Dies mid-call: reply channel closes -> ServerDown (even though the
// generous deadline never fires).
assert_eq!(
srv.call_timeout((), Duration::from_secs(10)),
Err(CallTimeoutError::ServerDown)
);
// Already gone: inbox send fails -> ServerDown.
assert_eq!(
srv.call_timeout((), Duration::from_secs(10)),
Err(CallTimeoutError::ServerDown)
);
});
}
// ---------------------------------------------------------------------------
// handle_info: out-of-band channels selected alongside the inbox (v0.8)
// ---------------------------------------------------------------------------
/// Logs every message it handles, in order; a call reads the log back.
struct Logger {
log: Vec<&'static str>,
}
impl GenServer for Logger {
type Call = ();
type Reply = Vec<&'static str>;
type Cast = ();
type Info = &'static str;
fn handle_call(&mut self, _: ()) -> Vec<&'static str> {
self.log.clone()
}
fn handle_cast(&mut self, _: ()) {
self.log.push("cast");
}
fn handle_info(&mut self, info: &'static str) {
self.log.push(info);
}
}
// An info message is dispatched to handle_info, interleaved with normal
// service.
#[test]
fn info_is_dispatched() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (info_tx, info_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(info_rx)
.start();
info_tx.send("info").unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["info"]);
}
// Arm priority: with a cast AND an info both queued before the server first
// runs, the info is handled first — info arms outrank the inbox. Relies on
// run()'s deterministic single-thread ordering.
#[test]
fn info_outranks_inbox() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (info_tx, info_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(info_rx)
.start();
// The server actor hasn't run yet: both messages are queued before
// its first select. Inbox first in *send* order, info first in *arm*
// order — arm order must win.
server.cast(()).unwrap();
info_tx.send("info").unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["info", "cast"]);
}
// Two info channels: declaration order is priority order.
#[test]
fn info_arms_keep_declaration_priority() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (hi_tx, hi_rx) = smarm::channel::<&'static str>();
let (lo_tx, lo_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(hi_rx)
.with_info(lo_rx)
.start();
// Sent low-priority first; handled high-priority first.
lo_tx.send("lo").unwrap();
hi_tx.send("hi").unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["hi", "lo"]);
}
// A closed info arm is silently dropped and the server keeps serving; the
// closure does NOT reach handle_info and does NOT starve the inbox (the
// closed-arm-is-ready-forever gotcha).
#[test]
fn closed_info_arm_is_dropped_silently() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let (info_tx, info_rx) = smarm::channel::<&'static str>();
let server = ServerBuilder::new(Logger { log: Vec::new() })
.with_info(info_rx)
.start();
drop(info_tx); // closed before the server's first select
server.cast(()).unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec!["cast"]);
}
// ---------------------------------------------------------------------------
// handle_down: monitors handed to the loop via Watcher (v0.8)
// ---------------------------------------------------------------------------
use smarm::gen_server::Watcher;
use smarm::{monitor, spawn, DownReason, Pid};
/// The motivating pattern: a server that spawns workers from a handler,
/// watches them, and logs their deaths.
struct Pool {
watcher: Option<Watcher>,
log: Vec<DownReason>,
}
enum PoolCast {
SpawnDoomedWorker,
Watch(Pid),
}
impl GenServer for Pool {
type Call = ();
type Reply = Vec<DownReason>;
type Cast = PoolCast;
type Info = ();
fn init(&mut self, ctx: &smarm::gen_server::ServerCtx) {
self.watcher = Some(ctx.watcher());
}
fn handle_call(&mut self, _: ()) -> Vec<DownReason> {
self.log.clone()
}
fn handle_cast(&mut self, cast: PoolCast) {
let watcher = self.watcher.as_ref().expect("init ran first");
match cast {
PoolCast::SpawnDoomedWorker => {
let h = spawn(|| panic!("worker died"));
watcher.watch(monitor(h.pid()));
}
PoolCast::Watch(pid) => watcher.watch(monitor(pid)),
}
}
fn handle_down(&mut self, down: smarm::Down) {
self.log.push(down.reason);
}
}
// A worker spawned and watched from inside a handler delivers its Down to
// handle_down. Down arms outrank the inbox, so the death is in the log by
// the time the follow-up call is answered.
#[test]
fn worker_pool_down_reaches_handle_down() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let server = start(Pool { watcher: None, log: Vec::new() });
server.cast(PoolCast::SpawnDoomedWorker).unwrap();
let _ = server.call(()).unwrap(); // sync point: cast handled, worker live
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec![DownReason::Panic]);
}
// Watching an already-dead pid yields an immediate NoProc Down, and the down
// arm outranks the inbox: the call cast *after* the watch still observes it.
#[test]
fn watch_dead_pid_is_noproc_down() {
let got = Arc::new(Mutex::new(Vec::new()));
let got2 = got.clone();
run(move || {
let h = spawn(|| {});
let dead = h.pid();
h.join().unwrap();
let server = start(Pool { watcher: None, log: Vec::new() });
server.cast(PoolCast::Watch(dead)).unwrap();
*got2.lock().unwrap() = server.call(()).unwrap();
});
assert_eq!(*got.lock().unwrap(), vec![DownReason::NoProc]);
}
// A state that never clones the Watcher closes the control arm; the loop
// falls back to the plain-inbox park and keeps serving. (Every pre-v0.8 test
// in this file also exercises this path.)
#[test]
fn unused_ctx_closes_control_arm_silently() {
let got = Arc::new(Mutex::new(0i64));
let got2 = got.clone();
run(move || {
let server = start(Counter { n: 0 });
server.cast(Op::Add(2)).unwrap();
server.cast(Op::Add(40)).unwrap();
*got2.lock().unwrap() = server.call(Req::Get).unwrap();
});
assert_eq!(*got.lock().unwrap(), 42);
}
tests/io_epoll.rs
-43
View File
@@ -322,46 +322,3 @@ fn wait_writable_on_empty_pipe_returns_quickly() {
elapsed
);
}
// ---------------------------------------------------------------------------
// Fd hygiene on actor death (v0.8)
// ---------------------------------------------------------------------------
// An actor stopped while parked on an fd must not leak its `waiters` entry:
// before the unwind-path guard in wait_fd, the stale entry made every
// future wait_*() on that fd fail with AlreadyExists, forever.
#[test]
fn stopped_waiter_does_not_poison_the_fd() {
let outcome = Arc::new(StdMutex::new(None::<u8>));
let outcome2 = outcome.clone();
run(move || {
let p = Pipe::new();
let (rfd, wfd) = (p.read, p.write);
// First waiter parks on the (empty) pipe and is stopped in place.
let h = smarm::spawn(move || {
wait_readable(rfd).unwrap();
unreachable!("the pipe is never written while this actor lives");
});
yield_now(); // let it reach the park
smarm::request_stop(h.pid());
h.join().unwrap(); // Ok(()): stopped, not panicked
// Second waiter on the SAME fd must be able to register...
let seen = Arc::new(AtomicU32::new(0));
let seen2 = seen.clone();
let h2 = smarm::spawn(move || {
wait_readable(rfd).unwrap();
let mut buf = [0u8; 1];
assert_eq!(raw_read(rfd, &mut buf), 1);
seen2.store(buf[0] as u32, Ordering::SeqCst);
});
yield_now(); // ...and park (a failed register would panic the unwrap)
// ...and actually be woken by readiness.
assert_eq!(raw_write(wfd, b"x"), 1);
h2.join().unwrap();
*outcome2.lock().unwrap() = Some(seen.load(Ordering::SeqCst) as u8);
});
assert_eq!(*outcome.lock().unwrap(), Some(b'x'));
}
tests/mutex.rs
+1 -2
View File
@@ -1,5 +1,4 @@
//! `smarm::Mutex<T>` (the actor-blocking mutex) tests. All run under the
//! scheduler because `lock()`
//! `loom::Mutex<T>` tests. All run under the scheduler because `lock()`
//! needs to be able to park.
use smarm::{run, spawn, yield_now, LockTimeout, Mutex};
tests/poison_stop.rs
-89
View File
@@ -1,89 +0,0 @@
//! Regression: request_stop racing an alloc-under-lock must not poison any
//! runtime mutex. Before the fix, check_cancelled() fired regardless of
//! PREEMPTION_ENABLED, so a sentinel unwind could trigger inside with_shared
//! (which allocates: run_queue.push_back, waiters.push, etc), poisoning the
//! shared mutex and cascading lock().unwrap() panics.
use smarm::runtime::{init, Config};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
#[test]
fn stop_storm_does_not_poison_runtime() {
// alloc_interval(1): every allocation is an observation point, so pre-fix
// the sentinel fires inside with_shared's alloc sites (run_queue.push_back,
// waiters.push, ...) with near-certainty rather than 1-in-128.
let rt = init(Config::exact(4).alloc_interval(1));
let completed = Arc::new(AtomicUsize::new(0));
let c = completed.clone();
rt.run(move || {
// Spawn many short actors that allocate + yield heavily (hammering
// with_shared), and request_stop each one mid-flight from siblings.
let mut handles = Vec::new();
for _ in 0..200 {
let h = smarm::spawn(|| {
for _ in 0..50 {
let _v: Vec<u8> = Vec::with_capacity(64); // alloc → maybe_preempt
smarm::yield_now();
}
});
handles.push(h);
}
// Cancel half of them; the others run to completion. If any lock got
// poisoned the runtime would panic on a subsequent lock().unwrap().
for (i, h) in handles.iter().enumerate() {
if i % 2 == 0 {
smarm::request_stop(h.pid());
}
}
for h in handles {
let _ = h.join();
}
c.fetch_add(1, Ordering::SeqCst);
});
assert_eq!(completed.load(Ordering::SeqCst), 1, "root completed cleanly");
}
/// The sharper repro: a stop-flagged actor whose *next allocation* is the
/// `Box::new(closure)` inside `spawn`'s `with_shared` critical section.
/// Pre-fix, the ungated `check_cancelled` in `maybe_preempt` raises the
/// sentinel right there, unwinding while the global shared mutex is held →
/// poisoned → every later `with_shared` panics on every scheduler thread and
/// the whole runtime collapses. Post-fix, observation is deferred to the next
/// lock-free point and each actor dies a clean `Outcome::Stopped`.
///
/// `request_stop(self_pid())` sets the flag without parking (we're running),
/// and the capturing closure makes `Box::new(f)` a real allocation. With
/// `alloc_interval(1)` the check fires every other allocation, so each actor
/// has ~50% odds of the check landing on the under-lock alloc; 64 actors make
/// a pre-fix escape astronomically unlikely.
#[test]
fn self_stop_during_spawn_does_not_poison_shared_mutex() {
let rt = init(Config::exact(4).alloc_interval(1));
let completed = Arc::new(AtomicUsize::new(0));
let c = completed.clone();
rt.run(move || {
let mut handles = Vec::new();
for i in 0..64usize {
let h = smarm::spawn(move || {
// Vary pre-stop allocation count to cover both phases of the
// every-other-allocation check cadence.
for _ in 0..(i % 2) {
let _phase: Vec<u8> = Vec::with_capacity(8);
}
let payload = vec![0u8; 64]; // captured → Box::new(f) allocates
smarm::request_stop(smarm::self_pid());
let grandchild = smarm::spawn(move || drop(payload));
let _ = grandchild.join();
});
handles.push(h);
}
for h in handles {
// Stopped reports Ok from join; the point is that join itself
// (with_shared) doesn't panic on a poisoned mutex.
let _ = h.join();
}
c.fetch_add(1, Ordering::SeqCst);
});
assert_eq!(completed.load(Ordering::SeqCst), 1, "root completed cleanly");
}
tests/registry.rs
-221
View File
@@ -1,221 +0,0 @@
//! Named registry tests. Run under the scheduler: registration requires a
//! live runtime and live actors.
use smarm::{channel, name_of, register, run, spawn, unregister, whereis, RegisterError};
#[test]
fn register_whereis_roundtrip() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("worker", h.pid()).unwrap();
assert_eq!(whereis("worker"), Some(h.pid()));
assert_eq!(name_of(h.pid()).as_deref(), Some("worker"));
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn register_is_idempotent_for_same_binding() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("svc", h.pid()).unwrap();
// Same name, same pid: a no-op Ok, not NameTaken.
register("svc", h.pid()).unwrap();
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn duplicate_name_on_live_holder_is_rejected() {
run(|| {
let (tx, rx) = channel::<()>();
let (tx2, rx2) = channel::<()>();
let a = spawn(move || {
rx.recv().unwrap();
});
let b = spawn(move || {
rx2.recv().unwrap();
});
register("svc", a.pid()).unwrap();
assert_eq!(
register("svc", b.pid()),
Err(RegisterError::NameTaken { holder: a.pid() })
);
tx.send(()).unwrap();
tx2.send(()).unwrap();
a.join().unwrap();
b.join().unwrap();
});
}
#[test]
fn one_name_per_pid() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("first", h.pid()).unwrap();
assert_eq!(
register("second", h.pid()),
Err(RegisterError::PidAlreadyRegistered { name: "first".into() })
);
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn registering_a_dead_pid_is_noproc() {
run(|| {
let h = spawn(|| {});
let pid = h.pid();
h.join().unwrap();
assert_eq!(register("ghost", pid), Err(RegisterError::NoProc));
});
}
#[test]
fn binding_evaporates_on_death_and_name_is_reusable() {
run(|| {
let a = spawn(|| {});
let pid_a = a.pid();
register("svc", pid_a).unwrap();
a.join().unwrap();
// Dead holder: both lookup directions report unbound.
assert_eq!(whereis("svc"), None);
assert_eq!(name_of(pid_a), None);
// And the name is free for a successor.
let (tx, rx) = channel::<()>();
let b = spawn(move || {
rx.recv().unwrap();
});
register("svc", b.pid()).unwrap();
assert_eq!(whereis("svc"), Some(b.pid()));
tx.send(()).unwrap();
b.join().unwrap();
});
}
#[test]
fn register_over_a_dead_holder_succeeds_without_lookup_in_between() {
// The eviction path inside register() itself (not via whereis pruning).
run(|| {
let a = spawn(|| {});
let pid_a = a.pid();
register("svc", pid_a).unwrap();
a.join().unwrap();
let (tx, rx) = channel::<()>();
let b = spawn(move || {
rx.recv().unwrap();
});
register("svc", b.pid()).unwrap();
assert_eq!(whereis("svc"), Some(b.pid()));
assert_eq!(name_of(pid_a), None);
tx.send(()).unwrap();
b.join().unwrap();
});
}
#[test]
fn unregister_frees_both_directions() {
run(|| {
let (tx, rx) = channel::<()>();
let h = spawn(move || {
rx.recv().unwrap();
});
register("svc", h.pid()).unwrap();
assert_eq!(unregister("svc"), Some(h.pid()));
assert_eq!(whereis("svc"), None);
assert_eq!(name_of(h.pid()), None);
assert_eq!(unregister("svc"), None);
// The pid may take a new name afterwards.
register("svc2", h.pid()).unwrap();
assert_eq!(name_of(h.pid()).as_deref(), Some("svc2"));
tx.send(()).unwrap();
h.join().unwrap();
});
}
#[test]
fn whereis_from_another_actor_and_usable_with_runtime_apis() {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
let stopped = Arc::new(AtomicBool::new(false));
let stopped2 = stopped.clone();
run(move || {
let (tx, rx) = channel::<()>();
let svc = spawn(move || {
// Parks forever; only a request_stop ends it.
let _ = rx.recv();
});
register("stoppable", svc.pid()).unwrap();
let stopped3 = stopped2.clone();
let client = spawn(move || {
let pid = whereis("stoppable").expect("name must resolve cross-actor");
// The registry's pids plug into the rest of the runtime API.
smarm::request_stop(pid);
stopped3.store(true, Ordering::Relaxed);
});
client.join().unwrap();
svc.join().unwrap(); // a stopped actor joins Ok
drop(tx);
});
assert!(stopped.load(Ordering::Relaxed));
}
#[test]
fn registry_under_concurrent_churn_multi_thread() {
// Many actors racing to claim the same names while holders die: the
// bimap invariant (debug-asserted internally) and Erlang semantics must
// hold under real parallelism.
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;
let wins = Arc::new(AtomicU32::new(0));
let wins2 = wins.clone();
smarm::init(smarm::Config::exact(4)).run(move || {
let mut handles = Vec::new();
for round in 0..8 {
let name = format!("contested-{}", round % 2);
for _ in 0..8 {
let name = name.clone();
let wins = wins2.clone();
handles.push(spawn(move || {
let me = smarm::self_pid();
match register(&name, me) {
Ok(()) => {
wins.fetch_add(1, Ordering::Relaxed);
smarm::yield_now();
// May have been pruned-by-contact never; we are
// alive, so our binding must still resolve to us.
assert_eq!(whereis(&name), Some(me));
assert_eq!(unregister(&name), Some(me));
}
Err(RegisterError::NameTaken { .. }) => {}
Err(e) => panic!("unexpected register error: {e}"),
}
}));
}
}
for h in handles {
h.join().unwrap();
}
});
// At least one registration per name must have succeeded.
assert!(wins.load(Ordering::Relaxed) >= 2);
}
tests/select.rs
-340
View File
@@ -1,340 +0,0 @@
//! select tests. Beyond the functional cases, the *_stays_precise tests are
//! soundness probes for the consuming-wake protocol: they fire stale loser-
//! arm wakes at an actor and then run a one-shot park (`sleep`, whose early
//! return would be observable as a short elapsed time) to prove the stale
//! wake died at its epoch CAS instead of corrupting the next wait.
use smarm::{channel, run, select, spawn};
use std::sync::atomic::{AtomicI64, Ordering};
use std::time::{Duration, Instant};
static OUT: AtomicI64 = AtomicI64::new(0);
#[test]
fn ready_arm_returns_immediately_without_parking() {
OUT.store(0, Ordering::SeqCst);
run(|| {
let (txa, rxa) = channel::<i64>();
let (_txb, rxb) = channel::<i64>();
txa.send(42).unwrap();
let i = select(&[&rxb, &rxa]);
assert_eq!(i, 1);
OUT.store(rxa.try_recv().unwrap().expect("ready arm must hold a message"), Ordering::SeqCst);
});
assert_eq!(OUT.load(Ordering::SeqCst), 42);
}
#[test]
fn lower_index_wins_when_several_arms_are_ready() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
txa.send(1).unwrap();
txb.send(2).unwrap();
// Documented priority order: index 0 first, regardless of send order.
assert_eq!(select(&[&rxa, &rxb]), 0);
assert_eq!(select(&[&rxb, &rxa]), 0);
});
}
#[test]
fn parks_until_any_arm_fires() {
OUT.store(0, Ordering::SeqCst);
run(|| {
let (txa, _rxa_keepalive) = (channel::<i64>().0, ());
let _hold = txa; // arm a: sender alive, never sends
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let _keep_a_open = txa;
let h = spawn(move || {
smarm::sleep(Duration::from_millis(10));
txb.send(7).unwrap();
});
let i = select(&[&rxa, &rxb]);
assert_eq!(i, 1);
OUT.store(rxb.try_recv().unwrap().unwrap(), Ordering::SeqCst);
h.join().unwrap();
});
assert_eq!(OUT.load(Ordering::SeqCst), 7);
}
#[test]
fn closed_arm_counts_as_ready() {
run(|| {
let (_keep, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
drop(txb);
let i = select(&[&rxa, &rxb]);
assert_eq!(i, 1);
// The caller observes the closure on the arm itself.
assert!(rxb.try_recv().is_err());
});
}
#[test]
fn closure_while_parked_wakes_the_select() {
run(|| {
let (_keep, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(10));
drop(txb);
});
assert_eq!(select(&[&rxa, &rxb]), 1);
assert!(rxb.try_recv().is_err());
h.join().unwrap();
});
}
#[test]
fn loser_arm_wake_after_parked_select_stays_precise() {
// Both arms registered; arm 0 wins the wake; arm 1's sender then fires
// a wake stamped with the consumed epoch. The subsequent sleep is a
// one-shot park: an early return would mean the stale wake landed.
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(10));
txa.send(1).unwrap(); // wins
txb.send(2).unwrap(); // loser arm: stale-epoch wake
});
assert_eq!(select(&[&rxa, &rxb]), 0);
assert_eq!(rxa.try_recv().unwrap(), Some(1));
let t0 = Instant::now();
smarm::sleep(Duration::from_millis(40));
assert!(
t0.elapsed() >= Duration::from_millis(40),
"one-shot park returned early: a stale loser-arm wake landed"
);
// The loser's message was never lost.
assert_eq!(select(&[&rxa, &rxb]), 1);
assert_eq!(rxb.try_recv().unwrap(), Some(2));
h.join().unwrap();
});
}
#[test]
fn no_park_exit_retires_the_wait() {
// Arm 1 is ready at registration time, so select returns WITHOUT
// parking while arm 0 holds a live-epoch registration. Arm 0's sender
// then fires. retire_wait must have invalidated/eaten that wake;
// the sleep proves it.
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
txb.send(2).unwrap();
assert_eq!(select(&[&rxa, &rxb]), 1);
assert_eq!(rxb.try_recv().unwrap(), Some(2));
let h = spawn(move || {
txa.send(1).unwrap(); // fires at the (retired) select epoch
});
h.join().unwrap();
let t0 = Instant::now();
smarm::sleep(Duration::from_millis(40));
assert!(
t0.elapsed() >= Duration::from_millis(40),
"one-shot park returned early: the no-park exit leaked a wake"
);
assert_eq!(rxa.try_recv().unwrap(), Some(1));
});
}
#[test]
fn select_then_plain_recv_on_a_loser_arm() {
// A leftover own-registration from a select must not trip the next
// direct wait on that channel (it is overwritten, not asserted away).
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
txa.send(1).unwrap();
assert_eq!(select(&[&rxa, &rxb]), 0); // rxb may keep a registration
assert_eq!(rxa.try_recv().unwrap(), Some(1));
let h = spawn(move || {
smarm::sleep(Duration::from_millis(5));
txb.send(9).unwrap();
});
assert_eq!(rxb.recv().unwrap(), 9);
h.join().unwrap();
});
}
#[test]
fn select_loop_drains_two_producers_completely() {
const N: i64 = 200;
OUT.store(0, Ordering::SeqCst);
run(|| {
let (txa, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
let ha = spawn(move || {
for v in 0..N {
txa.send(v).unwrap();
if v % 7 == 0 {
smarm::yield_now();
}
}
});
let hb = spawn(move || {
for v in 0..N {
txb.send(v).unwrap();
if v % 5 == 0 {
smarm::yield_now();
}
}
});
// A closed arm stays permanently ready, so it must leave the arm
// set once observed (keeping it would starve the other arm at
// priority order) — select until the FIRST closure, then drain the
// survivor with plain recv.
let mut sum = 0i64;
let survivor: &smarm::Receiver<i64> = loop {
let i = select(&[&rxa, &rxb]);
let arm: &smarm::Receiver<i64> = if i == 0 { &rxa } else { &rxb };
match arm.try_recv() {
Ok(Some(v)) => sum += v,
Ok(None) => {} // defensive: ready arm already drained
Err(_) => break if i == 0 { &rxb } else { &rxa },
}
};
loop {
match survivor.recv() {
Ok(v) => sum += v,
Err(_) => break,
}
}
OUT.store(sum, Ordering::SeqCst);
ha.join().unwrap();
hb.join().unwrap();
});
assert_eq!(OUT.load(Ordering::SeqCst), 2 * (0..200i64).sum::<i64>());
}
#[test]
fn motivating_pattern_inbox_plus_monitor_down() {
// The gen_server/handle_info shape select unlocks: a server waiting on
// its request inbox AND a monitor Down channel in one park.
use smarm::monitor;
run(|| {
let (req_tx, req_rx) = channel::<i64>();
let worker = spawn(|| {
smarm::sleep(Duration::from_millis(10));
// worker exits -> Down fires
});
let m = monitor(worker.pid());
let _keep_inbox_open = req_tx;
let mut served = 0;
loop {
match select(&[&m.rx, &req_rx]) {
0 => {
let _down = m.rx.try_recv().unwrap().expect("Down message");
break;
}
_ => {
if let Ok(Some(_)) = req_rx.try_recv() {
served += 1;
}
}
}
}
assert_eq!(served, 0);
worker.join().unwrap();
});
}
// ---------------------------------------------------------------------------
// select_timeout
// ---------------------------------------------------------------------------
use smarm::select_timeout;
#[test]
fn select_timeout_returns_none_after_the_deadline() {
run(|| {
let (_keep_a, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
let t0 = Instant::now();
let r = select_timeout(&[&rxa, &rxb], Duration::from_millis(30));
assert_eq!(r, None);
assert!(t0.elapsed() >= Duration::from_millis(30));
});
}
#[test]
fn select_timeout_ready_arm_wins_without_arming_a_timer() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
txa.send(5).unwrap();
assert_eq!(select_timeout(&[&rxb, &rxa], Duration::from_millis(500)), Some(1));
assert_eq!(rxa.try_recv().unwrap(), Some(5));
});
}
#[test]
fn select_timeout_arm_beats_timer_and_stale_entry_stays_inert() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(5));
txa.send(1).unwrap();
});
let t0 = Instant::now();
let r = select_timeout(&[&rxa, &rxb], Duration::from_millis(200));
assert_eq!(r, Some(0));
assert!(t0.elapsed() < Duration::from_millis(200));
assert_eq!(rxa.try_recv().unwrap(), Some(1));
h.join().unwrap();
// The abandoned timer entry expires mid-sleep; a stale-epoch wake
// landing would return this one-shot park early.
let t1 = Instant::now();
smarm::sleep(Duration::from_millis(250));
assert!(
t1.elapsed() >= Duration::from_millis(250),
"one-shot park returned early: the stale select timer landed"
);
});
}
#[test]
fn select_timeout_timer_first_message_still_delivered_later() {
run(|| {
let (txa, rxa) = channel::<i64>();
let h = spawn(move || {
smarm::sleep(Duration::from_millis(60));
txa.send(9).unwrap();
});
assert_eq!(select_timeout(&[&rxa], Duration::from_millis(10)), None);
// The wait is over but the channel is intact: the late message
// arrives and a fresh select sees it.
assert_eq!(select(&[&rxa]), 0);
assert_eq!(rxa.try_recv().unwrap(), Some(9));
h.join().unwrap();
});
}
#[test]
fn select_timeout_zero_duration_polls() {
run(|| {
let (txa, rxa) = channel::<i64>();
let (_keep_b, rxb) = channel::<i64>();
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::ZERO), None);
txa.send(3).unwrap();
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::ZERO), Some(0));
assert_eq!(rxa.try_recv().unwrap(), Some(3));
});
}
#[test]
fn select_timeout_closed_arm_is_ready_not_a_timeout() {
run(|| {
let (_keep_a, rxa) = channel::<i64>();
let (txb, rxb) = channel::<i64>();
drop(txb);
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::from_millis(200)), Some(1));
assert!(rxb.try_recv().is_err());
});
}
tests/timer.rs
+15 -15
View File
@@ -124,11 +124,11 @@ use smarm::pid::Pid;
use smarm::timer::{Reason, TimerTarget, Timers};
struct RecordingTarget {
calls: Mutex<Vec<(Pid, u32)>>,
calls: Mutex<Vec<(Pid, u64)>>,
}
impl TimerTarget for RecordingTarget {
fn on_timeout(&self, pid: Pid, epoch: u32) {
self.calls.lock().unwrap().push((pid, epoch));
fn on_timeout(&self, pid: Pid, seq: u64) {
self.calls.lock().unwrap().push((pid, seq));
}
}
@@ -137,9 +137,9 @@ fn timers_pop_due_returns_entries_in_deadline_order() {
let mut t = Timers::new();
let now = Instant::now();
// Insert out of order; pop_due should hand them back sorted by deadline.
t.insert_sleep(now + Duration::from_millis(30), Pid::new(0, 0), 1);
t.insert_sleep(now + Duration::from_millis(10), Pid::new(1, 0), 1);
t.insert_sleep(now + Duration::from_millis(20), Pid::new(2, 0), 1);
t.insert_sleep(now + Duration::from_millis(30), Pid::new(0, 0));
t.insert_sleep(now + Duration::from_millis(10), Pid::new(1, 0));
t.insert_sleep(now + Duration::from_millis(20), Pid::new(2, 0));
// Advance past all of them.
let due = t.pop_due(now + Duration::from_millis(50));
@@ -152,8 +152,8 @@ fn timers_pop_due_returns_entries_in_deadline_order() {
fn timers_only_pop_entries_whose_deadline_has_passed() {
let mut t = Timers::new();
let now = Instant::now();
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0), 1);
t.insert_sleep(now + Duration::from_millis(100), Pid::new(1, 0), 1);
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0));
t.insert_sleep(now + Duration::from_millis(100), Pid::new(1, 0));
let due = t.pop_due(now + Duration::from_millis(20));
assert_eq!(due.len(), 1);
@@ -169,11 +169,11 @@ fn timers_mix_sleep_and_wait_timeout_reasons() {
let target = Arc::new(RecordingTarget { calls: Mutex::new(Vec::new()) });
let now = Instant::now();
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0), 1);
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0));
t.insert(
now + Duration::from_millis(10),
Pid::new(1, 0),
Reason::WaitTimeout { target: target.clone(), epoch: 42 },
Reason::WaitTimeout { target: target.clone(), wait_seq: 42 },
);
let due = t.pop_due(now + Duration::from_millis(20));
@@ -181,11 +181,11 @@ fn timers_mix_sleep_and_wait_timeout_reasons() {
// Order: Sleep (5ms) first, WaitTimeout (10ms) second.
match &due[0].reason {
Reason::Sleep { .. } => {}
Reason::Sleep => {}
_ => panic!("first entry should be a Sleep"),
}
match &due[1].reason {
Reason::WaitTimeout { epoch, .. } => assert_eq!(*epoch, 42),
Reason::WaitTimeout { wait_seq, .. } => assert_eq!(*wait_seq, 42),
_ => panic!("second entry should be a WaitTimeout"),
}
}
@@ -197,9 +197,9 @@ fn same_deadline_entries_pop_in_insertion_order() {
let mut t = Timers::new();
let now = Instant::now();
let d = now + Duration::from_millis(10);
t.insert_sleep(d, Pid::new(0, 0), 1);
t.insert_sleep(d, Pid::new(1, 0), 1);
t.insert_sleep(d, Pid::new(2, 0), 1);
t.insert_sleep(d, Pid::new(0, 0));
t.insert_sleep(d, Pid::new(1, 0));
t.insert_sleep(d, Pid::new(2, 0));
let due = t.pop_due(now + Duration::from_millis(20));
let pids: Vec<u32> = due.iter().map(|e| e.pid.index()).collect();