Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 smarm-dev
|
89fb13e29a |
@@ -0,0 +1,2 @@
|
||||
[target.aarch64-unknown-linux-gnu]
|
||||
linker = "aarch64-linux-gnu-gcc"
|
||||
@@ -1,6 +1,3 @@
|
||||
target
|
||||
Cargo.lock
|
||||
smarm_trace.json
|
||||
/bench_results/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
-24
@@ -4,24 +4,12 @@ version = "0.4.0"
|
||||
edition = "2021"
|
||||
rust-version = "1.95"
|
||||
|
||||
[lints.rust]
|
||||
unexpected_cfgs = { level = "warn", check-cfg = ["cfg(loom)"] }
|
||||
|
||||
[features]
|
||||
default = ["rq-mutex"]
|
||||
smarm-trace = []
|
||||
# Run-queue selection: exactly one, compile-time (see src/run_queue.rs).
|
||||
# Non-default variants need --no-default-features (features are additive).
|
||||
rq-mutex = []
|
||||
rq-mpmc = []
|
||||
rq-striped = []
|
||||
|
||||
[dependencies]
|
||||
libc = "0.2"
|
||||
|
||||
[target.'cfg(loom)'.dependencies]
|
||||
loom = "0.7"
|
||||
|
||||
[dev-dependencies]
|
||||
libc = "0.2"
|
||||
tokio = { version = "1", features = ["rt", "rt-multi-thread", "macros", "sync", "time"] }
|
||||
@@ -53,15 +41,3 @@ harness = false
|
||||
[[bench]]
|
||||
name = "tokio_favored"
|
||||
harness = false
|
||||
|
||||
[[bench]]
|
||||
name = "rq_micro"
|
||||
harness = false
|
||||
|
||||
[[bench]]
|
||||
name = "rq_runtime"
|
||||
harness = false
|
||||
|
||||
[[bench]]
|
||||
name = "spin_sweep"
|
||||
harness = false
|
||||
|
||||
@@ -21,15 +21,14 @@ convenience wrapper around `runtime::init(Config::exact(1)).run(f)`.
|
||||
| `pid` | `(index, generation)` PIDs; stale handles are detectable, not silent |
|
||||
| `actor` | Trampoline + `catch_unwind` boundary at the actor entry point |
|
||||
| `scheduler` | Run queue, slot table, spawn/join, parking, idle path |
|
||||
| `channel` | Unbounded MPSC channel; `recv` parks the actor; `recv_timeout` bounds it; `select`/`select_timeout` park on many receivers at once (ready-index, priority order) |
|
||||
| `channel` | Unbounded MPSC channel; `recv` parks the actor |
|
||||
| `mutex` | `Mutex<T>` with mandatory timeout; FIFO waiters; parks the green thread |
|
||||
| `timer` | Min-heap of `(deadline, reason)`; `Sleep` and `WaitTimeout` reasons |
|
||||
| `io` | `block_on_io` for blocking work; `wait_readable`/`wait_writable` + `read`/`write` via epoll |
|
||||
| `supervisor` | `Signal::Exit`/`Panic`/`Stopped` funnelled to a parent; `OneForOne`/`OneForAll`/`RestForOne` strategies + restart-intensity cap |
|
||||
| `monitor` | `monitor(pid)` → `Monitor { id, target, rx }`; one-shot `Down` via `rx`; `demonitor(&m)` tears one registration down; unidirectional death notice |
|
||||
| `link` | bidirectional `link`/`unlink`; abnormal death propagates (cooperative stop, or an `ExitSignal` message under `trap_exit`) |
|
||||
| `gen_server` | `call`/`call_timeout` (sync request-reply) / `cast` (async) over one inbox; `handle_info` over static info arms + `handle_down` via `Watcher`-fed monitors, selected ahead of the inbox; `ServerRef`/`ServerBuilder` + `init`/`terminate` hooks; server-down via channel closure |
|
||||
| `registry` | `register`/`whereis`/`name_of`: name ↔ pid bimap; lazy generation-checked cleanup |
|
||||
| `gen_server` | `call` (sync request-reply) / `cast` (async) over one inbox; `ServerRef` + `init`/`terminate` hooks; server-down via channel closure |
|
||||
|
||||
## Quick taste
|
||||
|
||||
|
||||
-259
@@ -1,259 +0,0 @@
|
||||
# smarm — Roadmap
|
||||
|
||||
## Shipped (compacted — full cycle plans and deviation records live in git history)
|
||||
|
||||
Cycles before v0.7 (v0.4 actor primitives, v0.5 runtime decomposition &
|
||||
pluggable run queue, v0.6 actor ergonomics): see `git log ROADMAP.md`.
|
||||
|
||||
### v0.7 — select, on epoch-stamped consuming wakes ✅
|
||||
A 24-bit park-epoch packed into the slot word gives every wait an identity:
|
||||
registrations carry `(pid, epoch)`, every successful wake **consumes** the
|
||||
epoch, stale wakes die at one failed CAS. Subsumed the per-primitive wait
|
||||
seqs (channel, mutex, timer); the only wildcard wake left is `request_stop`
|
||||
(terminal). On top: `select`/`select_timeout` — ready-index wait over many
|
||||
receivers, priority order, no cancellation pass. Loom theorems re-proved on
|
||||
the new word. Notable deviations: `retire_wait` for the no-park exit path
|
||||
(plan missed it); the Drop-guard deregistration was superseded by relaxed
|
||||
single-receiver asserts; a closed arm is ready *forever* (documented gotcha).
|
||||
Commits `4913835`…`00128f3`, `a0a93b6`.
|
||||
|
||||
### v0.8 — gen_server: handle_info / handle_down + io fd hygiene ✅
|
||||
Spent `select` on the server loop: static info arms (`type Info`,
|
||||
`ServerBuilder::with_info`) and dynamic monitor forwarding
|
||||
(`ServerCtx`/`Watcher` + a control arm), priority downs → control → infos →
|
||||
inbox. Closed the v0.2 fd hole: a drop guard in `wait_fd` DELs the kernel
|
||||
registration on unwind (the leak was worse than documented — a stale waiters
|
||||
entry permanently poisoned the fd). Deviation: the plain-inbox fast path
|
||||
narrowed; servers holding a `Watcher` select forever.
|
||||
Commits `e5d1b3b`, `24b95c9`, `f6969e5`.
|
||||
|
||||
---
|
||||
|
||||
## Decision record — queue topology 🔒 CLOSED (2026-06-10)
|
||||
|
||||
The run-queue shootout (harness `6d9f369`, 24-core sweep 1–24 schedulers,
|
||||
report `bench_report_rq_shootout.html`) landed in **RFC 005's World 3**: the
|
||||
three queue variants are within 10–15% of each other in `rq_runtime` at every
|
||||
scheduler count ≥ 4, on all three workloads.
|
||||
Consequences:
|
||||
- **`rq-mutex` stays the default** — simplest correct, no capacity
|
||||
constraints, locking model already integrated.
|
||||
- **Feature plumbing stays as is.** All three variants keep compiling in
|
||||
every build; `rq-mpmc`/`rq-striped` remain selectable for benching.
|
||||
- **Reopening is benchmark-driven only.** The report documents the
|
||||
conditional upgrade paths if a future workload qualifies: mpmc for
|
||||
message-passing-dominant loads at N ≤ 8; striped for high-contention balanced push/pop at N ≥ 16. Neither is a scheduler workload as measured.
|
||||
- **Effort redirects to the wake path**: RFC 005 (billed as a latency patch,
|
||||
per its own World 3 framing), RFC 004, and eventually per-switch cost.
|
||||
|
||||
---
|
||||
|
||||
## v0.9 — Wake-path latency
|
||||
|
||||
Goal: attack per-wake latency — the measured ceiling — with the two specified
|
||||
mechanisms, each benched against a clean baseline before the next lands.
|
||||
Order: RFC 005 first (the slot is measured against the just-benched idle
|
||||
policy), then RFC 004 (measured against the slot-accepted baseline), then an
|
||||
interaction pass.
|
||||
|
||||
Full specs: `rfc_005-wake-slot.md`, `rfc_004-tunable-scheduler-idle-policy.md`
|
||||
(artefacts.kalsbeek.dev).
|
||||
|
||||
### 1. Wake slot (RFC 005)
|
||||
Per-scheduler, thread-local, capacity-one wake cache, checked before the
|
||||
shared queue. Runtime-selected via `Config { wake_slot: bool }`, default off
|
||||
until accepted (one binary benches both arms).
|
||||
- **Push policy:** slot-eligible iff the wake originates from actor context
|
||||
(`current_pid().is_some()`). Scheduler-context wakes (timer/IO drain) and
|
||||
spawns always go shared.
|
||||
- **Displacement:** newest wake takes the slot, occupant pushed shared (Go
|
||||
semantics); the branch swap is the fallback if benches look pathological.
|
||||
- **Pop order:** slot, then shared. Slot-popped actors inherit the waker's
|
||||
remaining timeslice — a handoff chain is bounded by one slice, so the
|
||||
shared queue is consulted at least once per slice per scheduler (the
|
||||
starvation bound, zero new counters).
|
||||
- **Invariant care:** the slot push replaces `run_queue.push` at the tail of
|
||||
the `Parked → Queued` CAS; at-most-once-enqueued holds verbatim (pid in
|
||||
slot ⊕ shared queue). Stall blast radius grows by exactly one actor.
|
||||
- Counters: `slot_hits`, `slot_displacements`.
|
||||
|
||||
### 2. Slot shootout
|
||||
Extend `rq_runtime`/`bench_rq.sh` with the slot on/off dimension (cheap —
|
||||
it's a Config knob, not a feature rebuild). Same sweep as the rq shootout.
|
||||
- **ping-pong-pairs** — target metric; expect the win, single- and
|
||||
multi-scheduler.
|
||||
- **yield-storm** — regression guard; yields never touch the slot, any delta
|
||||
is pop-path overhead.
|
||||
- **spawn-storm** — neutrality check; spawns bypass the slot by policy.
|
||||
Acceptance flips the default on and re-baselines for RFC 004.
|
||||
This is done, see results annotated in RFC005.
|
||||
|
||||
### 3. Spinning workers (RFC 004)
|
||||
Bounded spin-before-park for idle schedulers, killing the ~100 µs
|
||||
`thread::sleep` worst case on cross-thread handoff. Two Config knobs:
|
||||
`spin_budget_cycles` (0 recovers today's behaviour) and `max_spinners`
|
||||
(default N/2); one new atomic `n_spinning`. No run-queue changes — lands on
|
||||
the frozen `rq-mutex` substrate, independent of the slot mechanically.
|
||||
Credit for the idea: Dennis Gustafsson - [Parallelizing the physics solver BSC 2025](https://www.youtube.com/watch?v=Kvsvd67XUKw).
|
||||
|
||||
### 4. RFC 004 bench + interaction pass
|
||||
Re-run the sweep with spinning enabled, slot on and off. The known
|
||||
interaction: once idle pickup latency ≪ slice, the slot's latency trade
|
||||
(occupant waits out the waker's slice while other schedulers idle) may stop
|
||||
paying — RFC 005's documented escape hatch is Go's "spinners exist → push
|
||||
shared instead" check. In scope only if the data demands it.
|
||||
|
||||
---
|
||||
|
||||
## Later
|
||||
### Highest priority
|
||||
#### Process groups: the primitive pubsub & channels should have sat on
|
||||
Context: urus is a webserver written on top of smarm to provide a testing target.
|
||||
A named pid→multiset map with monitor-backed removal: `registry.rs` generalised
|
||||
from name↔pid *bimap* to name→*multiset*, the death hook reused verbatim. Local
|
||||
first. The point is that urus pubsub collapses into a pg consumer (`subscribe` =
|
||||
join, `broadcast` = send-to-members) instead of being a bespoke mechanism, and the
|
||||
same group set reads two ways — fan-out (all members) vs discovery/pool (one
|
||||
member), with different netsplit consequences. Urus shipped pubsub/channels predate this
|
||||
and want reframing on top of it. Foundational, so early in the post-v0.9 stack.
|
||||
Needs an RFC.
|
||||
|
||||
#### Per-switch cost (context shims, epoch protocol)
|
||||
The shootout's residual: per-wake latency is 0.16–0.18 µs at N=1 and
|
||||
0.8–1.2 µs at N=8+, dominated by the context-switch shims and the epoch
|
||||
protocol, not the queue. On current evidence this is the larger constant —
|
||||
"the whole game" alongside the v0.9 work — but there is no spec yet. Needs a
|
||||
profiling spike (where do the cycles actually go per park/unpark round-trip)
|
||||
and then an RFC before it can be scheduled.
|
||||
|
||||
#### send_after / cancel_timer
|
||||
Message-delivery timer on the existing min-heap (`timer.rs`): deliver a value to a
|
||||
channel at a deadline, cancellable. Unlocks the gen_server idioms with no clean
|
||||
expression today — heartbeat, debounce, retry backoff, session expiry. Small;
|
||||
|
||||
#### Introspection — process_info / get_state / tree dump
|
||||
`trace.rs` is the seed. What an actor is parked on, queue depth, stack size; a
|
||||
gen_server state snapshot; a supervision-tree walk. The native edge over tokio —
|
||||
actors already carry pid, name, parent where tokio tasks are anonymous — so it
|
||||
costs little and differentiates a lot. Needs an RFC.
|
||||
|
||||
#### Worker pool behaviour
|
||||
Supervised, interchangeable workers with restart semantics over a shared inbox
|
||||
(poolboy / NimblePool shape) — distinct from connection pools (bb8/deadpool), which
|
||||
pool *resources*, not *supervised processes*. Sits on `supervisor.rs` +
|
||||
`gen_server.rs`. Needs an RFC.
|
||||
|
||||
### Medium Priority
|
||||
|
||||
#### Demand-driven pipelines — GenStage / Broadway shape
|
||||
Supervised producer/consumer stages where consumers signal demand upstream, with
|
||||
batching, ack, partitioning. The clearest thing hex has and crates.io lacks (stream
|
||||
combinators and bounded channels are not a supervised demand-contract stage graph),
|
||||
and the natural fit for ingestion-shaped workloads. Builds on channels + gen_server
|
||||
+ supervisor. Needs an RFC.
|
||||
#### Unwakeable idle sleep when io is absent (terminal-wake residual)
|
||||
The `(Some(deadline), None)` idle branch — timers pending, io subsystem never
|
||||
initialized — blocks in `thread::sleep` with no wake mechanism at all. The
|
||||
terminal wake (writes the wake pipe at AllDone) cannot reach it: no io, no
|
||||
pipe. Same stall as the fixed bug, in any no-io runtime: a sibling that
|
||||
blocked on an orphaned deadline sleeps it out in full after everything else
|
||||
finished. Candidates, mutually exclusive: (a) clamp the sleep (cheap, but
|
||||
turns idle into periodic wakeups), or (b) park the branch on a condvar/futex
|
||||
the AllDone path signals — and at that point consider making the condvar the
|
||||
idle primitive for the no-io runtime generally (a cross-thread unpark could
|
||||
signal it too, see below). Decide before any no-io deployment.
|
||||
#### Cross-thread unpark
|
||||
`RuntimeInner::enqueue` does not wake idle sibling schedulers — only io
|
||||
completions write the wake pipe. Mid-flight this is masked (the enqueuing
|
||||
thread is awake and eats the work itself), but it costs parallelism: work
|
||||
enqueued by a busy thread waits until the sibling's idle poll times out. Needs bench evidence (does the shared-queue handoff latency actually show up?) before a mechanism is picked.
|
||||
#### Unbounded / configurable-bounded actor count
|
||||
Fixed slab with a loud assert (`Config::max_actors(n)`, default 16 384).
|
||||
Revisit with a segmented slab (array of `AtomicPtr<Segment>`, doubling segment
|
||||
sizes, append-only) once the cap is actually hit. Do not let it calcify.
|
||||
#### arm-port validation & merge
|
||||
`arm-port` branch carries an AAPCS64 context-switch backend, never run on
|
||||
hardware. Build + run full test suite on an aarch64 device; check
|
||||
`chained_spawn` / `yield_many` bench medians; merge and update README.
|
||||
|
||||
### Low priority
|
||||
#### gen_statem — postponement + state timeouts only
|
||||
A thin layer over gen_server, not a new behaviour. The (state, event) dispatch
|
||||
matrix is free from the type system and not worth porting. The two mechanisms that
|
||||
are: event **postponement** (defer events in the wrong state, replay on transition
|
||||
— selective receive, codified) and **state timeouts** (auto-cancel on state
|
||||
change). Device-connection FSMs are the canonical use. Wants send_after underneath.
|
||||
Needs an RFC.
|
||||
|
||||
#### Clustering — distribution epic
|
||||
Sequenced deliberately after v0.9 and the per-switch-cost spike. A fat stack of
|
||||
RFCs, not one. Spine settled in discussion; decisions still open:
|
||||
- **Explicit remote boundary, never transparency.** Serialization colours *edges*
|
||||
(channel types), not functions — local edges stay zero-copy `Send`, only remote
|
||||
edges take a `RemoteRef<T: Serialize + DeserializeOwned>`. No hidden latency when
|
||||
a peer migrates; the refactor is visible by construction.
|
||||
- **One binary, role as runtime config** (`ROLE=… REGION=… SEEDS=…`); a build-hash
|
||||
handshake enforces same-binary type identity and sidesteps cross-version type
|
||||
agreement. Roles select which supervision subtree mounts.
|
||||
- **Distributed pg falls out of local pg + a membership/gossip layer**, and
|
||||
distributed pubsub falls out of that for free; per-member metadata (region, load)
|
||||
enables fly-style nearest-member routing.
|
||||
- **Migratable gen_servers** as a sub-layer: only behaviours migrate (a raw actor's
|
||||
stack is opaque; a gen_server *between callbacks* is just its `State`), gated by
|
||||
`Serialize` bounds + an `on_arrive` reacquire hook, addressed by name not pid. The
|
||||
BEAM can't do this — leaning on the behaviour layer is what buys it. Requires `State` to be serializable, so should probaby spec a `trait MigratableGenServer: GenServer where Self::State: Migratable` , or something to that extent, so we can lean on the type system to make sure we don't accidentally make state that cannot be serialised.
|
||||
- **CRDT presence** is the high-value, genuinely-hard layer above distributed pg,
|
||||
kept *out* of the pg primitive (the pg2 strong-consistency lesson). Furthest out. Can maybe defer to rust ecosystem
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Look into
|
||||
|
||||
### app actors block AllDone; no external stop path
|
||||
Agent working on urus (see same git server as smarm) reported a lazily spawned actor never returning, blocking program shutdown. Maybe we should do something about it. Agent worked around it by giving the actor an atomic bool to spin on. See urus example crud for exact impl.
|
||||
|
||||
---
|
||||
|
||||
## Invariants & gotchas (respect these across all cycles)
|
||||
|
||||
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
|
||||
`with_shared`. Never send on a channel while holding the shared lock. Pattern:
|
||||
`mem::take` data under the lock, send after releasing. See `finalize_actor`.
|
||||
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
|
||||
Done/outcome → recycle stack → deliver supervisor Signal + monitor Downs →
|
||||
unpark joiners → reclaim slot if `outstanding_handles==0`. Death notifications
|
||||
always precede reclamation.
|
||||
- **Slot lifecycle reset in THREE places:** `Slot::vacant()`, `reclaim_slot()`
|
||||
(runtime.rs), slot-init block in `spawn_under` (scheduler.rs). Any new `Slot`
|
||||
field must be reset in all three.
|
||||
- **Pid = (index, generation).** Stale handles caught by generation mismatch in
|
||||
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
|
||||
- **The only wildcard wake is `request_stop`, and it is terminal.** Every
|
||||
registration-based waker (channel sends, mutex grants, wait-timers, io
|
||||
completions, joiner wakes, `select` arms) carries the wait's park-epoch
|
||||
and wakes through `unpark_at`; every successful wake consumes the epoch.
|
||||
Wakes are therefore *meaningful*: one-shot park sites interpret them
|
||||
without loops, and `select` needs no cancellation pass. When adding a new
|
||||
waker, decide which form it is — if its registration handle can outlive
|
||||
the wait it was created for, it MUST be epoch-stamped; a wait that can
|
||||
exit without parking MUST `retire_wait` first (see slot_state.rs).
|
||||
- **`select` exists; a unified per-process mailbox still does not.** The
|
||||
supervisor keeps its single `supervisor_channel` funnel; `recv_match`
|
||||
stays per-channel. `select` composes channels at the wait, not into one
|
||||
queue — gen_server's `handle_info`/`handle_down` (v0.8) are built on
|
||||
exactly that composition, with documented arm priority (downs → control
|
||||
→ infos → inbox) instead of mailbox FIFO. A hot higher-priority arm
|
||||
starves lower ones by design; that's the contract.
|
||||
- **Cooperative-only.** Preemption and cancellation both depend on the actor
|
||||
reaching `check!()`/yield/alloc/blocking points.
|
||||
- **Lock order is Leaf → Channel, one of each at most** (debug-asserted in
|
||||
`raw_mutex.rs`). Leaf = cold locks / free list / stack pool / registry,
|
||||
mutual leaves. A channel lock may be taken under a Leaf (finalize/monitor
|
||||
clone senders living in slots); nothing may be locked under a channel lock.
|
||||
- **Queue ops require preemption disabled.** A producer suspended mid-publish
|
||||
stalls every consumer — livelock. `with_runtime`, `with_shared`, and
|
||||
`RawMutex` guards all disable preemption for their span.
|
||||
- **`run()` is single-thread** (`Config::exact(1)`); tests rely on deterministic
|
||||
single-thread ordering. Multi-thread via `runtime::init(Config…)`.
|
||||
|
||||
@@ -1,154 +0,0 @@
|
||||
# Benches
|
||||
|
||||
Two families live here: **comparison benches** (smarm vs tokio, predating
|
||||
v0.5) and the **run-queue shootout** (v0.5 phase 4). All are plain binaries
|
||||
(`harness = false` in `Cargo.toml`), so `cargo bench` just builds in release
|
||||
and runs `main()` — no criterion, no magic.
|
||||
|
||||
```
|
||||
cargo bench --bench <name> # one bench
|
||||
cargo bench # all of them (slow; rarely what you want)
|
||||
```
|
||||
|
||||
## Catalog
|
||||
|
||||
| file | what it measures |
|
||||
|---|---|
|
||||
| `primes.rs` | Compute fan-out/fan-in: counts primes across W workers. Pure compute throughput + spawn/join/channel cost. |
|
||||
| `multi_scheduler.rs` | The original cross-runtime matrix: smarm (1 thread / N threads) vs tokio (current_thread / multi_thread) on compute, ping-pong, and spawn throughput. |
|
||||
| `general.rs` | Workloads where neither runtime has a structural edge. Large gaps here mean real per-task/per-yield overhead differences — watch these for regressions. |
|
||||
| `smarm_favored.rs` | Workloads the stackful green-thread model is built for. Single-thread numbers isolate per-switch cost from contention. |
|
||||
| `tokio_favored.rs` | Workloads tokio's model is built for. Expect to lose; the value is knowing *by how much* and catching the gap widening. |
|
||||
| `rq_micro.rs` | Run-queue **structures** in isolation (no runtime, no actors): push/pop throughput sweeping thread count × producer:consumer ratio. Covers all three queue types in one binary — the types compile in every build; only the runtime's alias is feature-selected. |
|
||||
| `rq_runtime.rs` | The **whole scheduler** with the compile-time-selected queue: yield-storm (pure queue churn), ping-pong-pairs (park/unpark latency), spawn-storm (slab + free list + queue churn), sweeping scheduler count. Comparing variants requires rebuilding per `rq-*` feature. |
|
||||
| `spin_sweep.rs` | RFC 004 spinning workers: wake **latency vs idle-CPU** over `spin_budget_cycles`. Four workloads (remote-wake, fork-join, half-load, pure-idle) × budget × `max_spinners` × scheduler count; latency percentiles plus cores-busy from `getrusage`. The data that picks the budget default and on/off posture. |
|
||||
|
||||
## The run-queue shootout
|
||||
|
||||
One command; it rebuilds `rq_runtime` once per queue variant, runs `rq_micro`
|
||||
once, and aggregates:
|
||||
|
||||
```
|
||||
./scripts/bench_rq.sh
|
||||
# on a big box:
|
||||
SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh
|
||||
```
|
||||
|
||||
Outputs land in `bench_results/` (gitignored): one full log per run, plus
|
||||
`summary.csv` assembled from the machine-readable `RQCSV,...` lines every
|
||||
config prints alongside the human table.
|
||||
|
||||
Manual single-variant runs need the feature dance (features are additive, so
|
||||
the default `rq-mutex` must be switched off):
|
||||
|
||||
```
|
||||
cargo bench --bench rq_runtime --no-default-features --features rq-striped
|
||||
```
|
||||
|
||||
### Knobs (env vars, all optional)
|
||||
|
||||
| var | default | used by |
|
||||
|---|---|---|
|
||||
| `SMARM_BENCH_THREADS` | `"1 2 4"` | both — space-separated sweep |
|
||||
| `SMARM_BENCH_RUNS` | `5` | both — repetitions; the **median** is reported |
|
||||
| `SMARM_BENCH_ITEMS` | `200000` | `rq_micro` — items per measurement |
|
||||
| `SMARM_BENCH_YIELD_ACTORS` / `_YIELDS` | `200` / `500` | `rq_runtime` yield-storm |
|
||||
| `SMARM_BENCH_PAIRS` / `_ROUNDTRIPS` | `32` / `1000` | `rq_runtime` ping-pong |
|
||||
| `SMARM_BENCH_SPAWNS` | `5000` | `rq_runtime` spawn-storm |
|
||||
|
||||
## The spin sweep (RFC 004)
|
||||
|
||||
Spinning workers trade idle CPU for wake latency: an idle worker polls
|
||||
`queue_len` for up to `spin_budget_cycles` (at most `max_spinners` at once)
|
||||
before falling back to a futex park, so work that lands during the spin is
|
||||
claimed with no syscall. `spin_budget_cycles = 0` is the feature fully off —
|
||||
the historical `thread::sleep` park, exactly preserved. `spin_sweep` measures
|
||||
both sides of that trade across the budget so the curve can pick the default.
|
||||
|
||||
```
|
||||
cargo bench --bench spin_sweep
|
||||
# on a big box (scale the work up from the sub-second sandbox defaults):
|
||||
SMARM_BENCH_THREADS="1 2 4 8 16" SMARM_BENCH_RUNS=9 \
|
||||
SMARM_SPIN_BUDGET="0 1000 5000 20000 100000 500000 2000000" \
|
||||
SMARM_SPIN_ROUNDS=2000 SMARM_SPIN_BURSTS=500 SMARM_SPIN_ITEMS=5000 \
|
||||
cargo bench --bench spin_sweep | tee bench_results/spin.txt
|
||||
grep '^SPINCSV' bench_results/spin.txt > bench_results/spin.csv
|
||||
```
|
||||
|
||||
Two axes per config: **latency** (p50/p90/p99/min/max, pooled across runs) and
|
||||
**cost** (cores-busy = process CPU-seconds via `getrusage(RUSAGE_SELF)` over
|
||||
wall-seconds; `1.0` = one core pegged, up to `max_spinners` when every spinner
|
||||
is hot). The four workloads:
|
||||
|
||||
- **remote-wake** — the latency target. A consumer parks on `recv`; a producer
|
||||
on another scheduler timestamps just before `send`. Latency is send→receipt.
|
||||
- **fork-join** — fan-out wake latency: root forks `FANOUT` actors, joins them,
|
||||
gap, repeat.
|
||||
- **half-load** — the cost target. Work dispatched every `PERIOD_US`, so workers
|
||||
idle a fraction of each period; cores-busy climbs with budget as spin fills
|
||||
the gaps. Spin cost scales with **park frequency** × budget, so this (not
|
||||
pure-idle) is where the cost curve lives.
|
||||
- **pure-idle** — a floor check: one actor sleeps, nothing else runs. Confirms
|
||||
budget 0 stays parked and a large budget doesn't leak *continuous* CPU (a
|
||||
spinner spends its budget once per park, so a truly idle window barely moves).
|
||||
|
||||
The crossover the sweep reveals: spinning only wins when work arrives while a
|
||||
worker is still inside its budget. With a fixed inter-arrival gap, budgets whose
|
||||
spin-time (`budget_cycles / TSC_freq`) exceeds the gap start catching work
|
||||
before the park — so `SMARM_SPIN_GAP_US` sets the knee; tune it against your
|
||||
box's TSC frequency.
|
||||
|
||||
### Knobs (env vars, all optional)
|
||||
|
||||
| var | default | used by |
|
||||
|---|---|---|
|
||||
| `SMARM_BENCH_THREADS` | `"1 2"` | scheduler-count sweep |
|
||||
| `SMARM_BENCH_RUNS` | `3` | runs per config (latency pooled, cores median) |
|
||||
| `SMARM_SPIN_BUDGET` | `"0 1000 10000 100000 1000000"` | `spin_budget_cycles` sweep |
|
||||
| `SMARM_MAX_SPINNERS` | `"-"` | `max_spinners` sweep; `-` = runtime default (N/2) |
|
||||
| `SMARM_SPIN_GAP_US` | `150` | idle gap between rounds/bursts (sets the latency knee) |
|
||||
| `SMARM_SPIN_ROUNDS` | `100` | remote-wake rounds |
|
||||
| `SMARM_SPIN_FANOUT` / `_BURSTS` | `64` / `30` | fork-join |
|
||||
| `SMARM_SPIN_ITEMS` / `_PERIOD_US` | `150` / `300` | half-load |
|
||||
| `SMARM_SPIN_WINDOW_MS` | `25` | pure-idle window |
|
||||
|
||||
Output is the house table plus a greppable `SPINCSV,...` line per config.
|
||||
|
||||
**On 1 core spinning is pathological** — a hot spinner starves the very thread
|
||||
that would enqueue work, so the optimal budget there is 0 and you only ever see
|
||||
the cost, never the benefit (the sweep will show remote-wake latency getting
|
||||
*worse* with budget on one core). One core validates the harness; the real curve
|
||||
comes from the many-core box.
|
||||
|
||||
## Reading the numbers honestly
|
||||
|
||||
- **Core count is the experiment.** On a 1-core machine (CI, sandboxes) the
|
||||
sweep only validates the harness and catches gross pathologies —
|
||||
oversubscribed schedulers measure context-switch noise, not contention.
|
||||
Variant decisions come from a many-core box.
|
||||
- The striped queue *should lose* at low thread counts (ticket overhead with
|
||||
no contention to amortize) — that's expected, not a bug.
|
||||
- Medians over `SMARM_BENCH_RUNS` absorb scheduling noise but not thermal /
|
||||
turbo drift; for publishable numbers, pin the CPU governor and run a warmup
|
||||
pass first.
|
||||
- `spawn-storm` batches joins (1024 at a time) to stay well under the slab
|
||||
cap; if you raise `SMARM_BENCH_SPAWNS` massively, that batching is why it
|
||||
still works.
|
||||
|
||||
## Adding a bench
|
||||
|
||||
1. `benches/<name>.rs` with a plain `main()`; print the house table (see any
|
||||
existing bench) and, if it belongs to a sweep, a greppable CSV line with a
|
||||
distinctive prefix (`RQCSV,` for the shootout family).
|
||||
2. Register it in `Cargo.toml`:
|
||||
```toml
|
||||
[[bench]]
|
||||
name = "<name>"
|
||||
harness = false
|
||||
```
|
||||
3. Take parameters from `SMARM_BENCH_*` env vars with modest defaults — the
|
||||
defaults must finish in seconds on one core, the env scales them up on
|
||||
real hardware.
|
||||
4. Report **medians**, and keep one measurement = one fresh runtime
|
||||
(`init(Config::exact(t))` inside the measured closure constructor, the
|
||||
`run()` inside the timed region) so runs don't contaminate each other.
|
||||
+142
-142
@@ -2,313 +2,313 @@
|
||||
"chained_spawn": {
|
||||
"smarm 1-thread": {
|
||||
"result": 1000,
|
||||
"median": 413,
|
||||
"min": 410,
|
||||
"max": 439
|
||||
"median": 266,
|
||||
"min": 242,
|
||||
"max": 351
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 1000,
|
||||
"median": 909,
|
||||
"min": 888,
|
||||
"max": 951
|
||||
"median": 742,
|
||||
"min": 696,
|
||||
"max": 860
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 1000,
|
||||
"median": 62,
|
||||
"min": 61,
|
||||
"max": 62
|
||||
"max": 68
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 1000,
|
||||
"median": 197,
|
||||
"min": 194,
|
||||
"max": 210
|
||||
"median": 190,
|
||||
"min": 169,
|
||||
"max": 207
|
||||
}
|
||||
},
|
||||
"yield_many": {
|
||||
"smarm 1-thread": {
|
||||
"result": 200000,
|
||||
"median": 16475,
|
||||
"min": 16393,
|
||||
"max": 16732
|
||||
"median": 19071,
|
||||
"min": 18776,
|
||||
"max": 19396
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 200000,
|
||||
"median": 148708,
|
||||
"min": 111213,
|
||||
"max": 156462
|
||||
"median": 172454,
|
||||
"min": 166246,
|
||||
"max": 174230
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 200000,
|
||||
"median": 4751,
|
||||
"min": 4740,
|
||||
"max": 5259
|
||||
"median": 4737,
|
||||
"min": 4644,
|
||||
"max": 5065
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 200000,
|
||||
"median": 8320,
|
||||
"min": 7862,
|
||||
"max": 8882
|
||||
"median": 8738,
|
||||
"min": 7852,
|
||||
"max": 9770
|
||||
}
|
||||
},
|
||||
"fan_out_compute": {
|
||||
"smarm 1-thread": {
|
||||
"result": 33860,
|
||||
"median": 13453,
|
||||
"min": 13305,
|
||||
"max": 15077
|
||||
"median": 13234,
|
||||
"min": 13196,
|
||||
"max": 13390
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 33860,
|
||||
"median": 2451,
|
||||
"min": 2330,
|
||||
"max": 2520
|
||||
"median": 2244,
|
||||
"min": 2162,
|
||||
"max": 2380
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 33860,
|
||||
"median": 14019,
|
||||
"min": 12339,
|
||||
"max": 14045
|
||||
"median": 14049,
|
||||
"min": 14035,
|
||||
"max": 14300
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 33860,
|
||||
"median": 1500,
|
||||
"min": 1426,
|
||||
"max": 1600
|
||||
"median": 1474,
|
||||
"min": 1285,
|
||||
"max": 1823
|
||||
}
|
||||
},
|
||||
"ping_pong_oneshot": {
|
||||
"smarm 1-thread": {
|
||||
"result": 1000,
|
||||
"median": 898,
|
||||
"min": 782,
|
||||
"max": 920
|
||||
"median": 751,
|
||||
"min": 727,
|
||||
"max": 913
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 1000,
|
||||
"median": 1494,
|
||||
"min": 1489,
|
||||
"max": 1546
|
||||
"median": 1308,
|
||||
"min": 1227,
|
||||
"max": 1396
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 1000,
|
||||
"median": 396,
|
||||
"min": 389,
|
||||
"max": 409
|
||||
"median": 407,
|
||||
"min": 400,
|
||||
"max": 444
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 1000,
|
||||
"median": 10382,
|
||||
"min": 9559,
|
||||
"max": 10924
|
||||
"median": 10869,
|
||||
"min": 8683,
|
||||
"max": 11688
|
||||
}
|
||||
},
|
||||
"spawn_storm_busy": {
|
||||
"smarm 1-thread": {
|
||||
"result": 10000,
|
||||
"median": 106232,
|
||||
"min": 105627,
|
||||
"max": 107693
|
||||
"median": 112045,
|
||||
"min": 99936,
|
||||
"max": 117329
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 10000,
|
||||
"median": 49198,
|
||||
"min": 48894,
|
||||
"max": 52606
|
||||
"median": 137105,
|
||||
"min": 130852,
|
||||
"max": 147707
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 10000,
|
||||
"median": 1140,
|
||||
"min": 1018,
|
||||
"max": 1169
|
||||
"median": 1128,
|
||||
"min": 1123,
|
||||
"max": 1435
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 10000,
|
||||
"median": 18650,
|
||||
"min": 16486,
|
||||
"max": 19396
|
||||
"median": 19674,
|
||||
"min": 16013,
|
||||
"max": 27234
|
||||
}
|
||||
},
|
||||
"mpsc_contention": {
|
||||
"smarm 1-thread": {
|
||||
"result": 320000,
|
||||
"median": 6173,
|
||||
"min": 5446,
|
||||
"max": 6226
|
||||
"median": 3667,
|
||||
"min": 3608,
|
||||
"max": 4126
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 320000,
|
||||
"median": 36329,
|
||||
"min": 35121,
|
||||
"max": 37590
|
||||
"median": 45681,
|
||||
"min": 31908,
|
||||
"max": 51287
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 320000,
|
||||
"median": 5563,
|
||||
"min": 5527,
|
||||
"max": 6239
|
||||
"median": 6228,
|
||||
"min": 6210,
|
||||
"max": 6514
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 320000,
|
||||
"median": 63966,
|
||||
"min": 59972,
|
||||
"max": 67534
|
||||
"median": 66173,
|
||||
"min": 42208,
|
||||
"max": 83255
|
||||
}
|
||||
},
|
||||
"many_timers": {
|
||||
"smarm 1-thread": {
|
||||
"result": 10000,
|
||||
"median": 107826,
|
||||
"min": 107093,
|
||||
"max": 119034
|
||||
"median": 119988,
|
||||
"min": 107308,
|
||||
"max": 123557
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 10000,
|
||||
"median": 96539,
|
||||
"min": 95413,
|
||||
"max": 97804
|
||||
"median": 218842,
|
||||
"min": 182009,
|
||||
"max": 256988
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 10000,
|
||||
"median": 12584,
|
||||
"min": 12539,
|
||||
"max": 12627
|
||||
"median": 12432,
|
||||
"min": 12308,
|
||||
"max": 13468
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 10000,
|
||||
"median": 16183,
|
||||
"min": 16024,
|
||||
"max": 16541
|
||||
"median": 16311,
|
||||
"min": 15026,
|
||||
"max": 16897
|
||||
}
|
||||
},
|
||||
"multi_thread_scaling": {
|
||||
"smarm 1-thread": {
|
||||
"result": 33860,
|
||||
"median": 15083,
|
||||
"min": 15071,
|
||||
"max": 15283
|
||||
"median": 14908,
|
||||
"min": 14857,
|
||||
"max": 15218
|
||||
},
|
||||
"smarm 2-thread": {
|
||||
"result": 33860,
|
||||
"median": 8070,
|
||||
"min": 8003,
|
||||
"max": 8096
|
||||
"median": 7834,
|
||||
"min": 7717,
|
||||
"max": 8033
|
||||
},
|
||||
"smarm 4-thread": {
|
||||
"result": 33860,
|
||||
"median": 4460,
|
||||
"min": 4454,
|
||||
"max": 4516
|
||||
"median": 4393,
|
||||
"min": 4326,
|
||||
"max": 4435
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 33860,
|
||||
"median": 2333,
|
||||
"min": 2294,
|
||||
"max": 2348
|
||||
"median": 2173,
|
||||
"min": 2068,
|
||||
"max": 2405
|
||||
},
|
||||
"tokio multi 1-thread": {
|
||||
"result": 33860,
|
||||
"median": 14504,
|
||||
"min": 14193,
|
||||
"max": 14562
|
||||
"median": 14432,
|
||||
"min": 14219,
|
||||
"max": 14763
|
||||
},
|
||||
"tokio multi 2-thread": {
|
||||
"result": 33860,
|
||||
"median": 7297,
|
||||
"min": 7289,
|
||||
"max": 7398
|
||||
"median": 7333,
|
||||
"min": 7222,
|
||||
"max": 7477
|
||||
},
|
||||
"tokio multi 4-thread": {
|
||||
"result": 33860,
|
||||
"median": 3795,
|
||||
"min": 3756,
|
||||
"max": 3799
|
||||
"median": 3741,
|
||||
"min": 3681,
|
||||
"max": 3876
|
||||
},
|
||||
"tokio multi 24-thread": {
|
||||
"result": 33860,
|
||||
"median": 1544,
|
||||
"min": 1520,
|
||||
"max": 1610
|
||||
"median": 1513,
|
||||
"min": 1375,
|
||||
"max": 1979
|
||||
}
|
||||
},
|
||||
"deep_recursion": {
|
||||
"smarm 1-thread": {
|
||||
"result": 1,
|
||||
"median": 226,
|
||||
"min": 222,
|
||||
"max": 243
|
||||
"median": 102,
|
||||
"min": 96,
|
||||
"max": 123
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 1,
|
||||
"median": 745,
|
||||
"min": 744,
|
||||
"max": 776
|
||||
"median": 597,
|
||||
"min": 576,
|
||||
"max": 682
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 1,
|
||||
"median": 11,
|
||||
"min": 10,
|
||||
"max": 13
|
||||
"median": 13,
|
||||
"min": 11,
|
||||
"max": 35
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 1,
|
||||
"median": 53,
|
||||
"min": 53,
|
||||
"max": 57
|
||||
"median": 56,
|
||||
"min": 46,
|
||||
"max": 65
|
||||
}
|
||||
},
|
||||
"yield_in_hot_loop": {
|
||||
"smarm 1-thread": {
|
||||
"result": 1000000,
|
||||
"median": 64849,
|
||||
"min": 64396,
|
||||
"max": 65283
|
||||
"median": 80680,
|
||||
"min": 80308,
|
||||
"max": 81845
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 1000000,
|
||||
"median": 68507,
|
||||
"min": 62018,
|
||||
"max": 72341
|
||||
"median": 72606,
|
||||
"min": 72154,
|
||||
"max": 77206
|
||||
}
|
||||
},
|
||||
"uncontended_channel": {
|
||||
"smarm 1-thread": {
|
||||
"result": 1000000,
|
||||
"median": 11949,
|
||||
"min": 11928,
|
||||
"max": 13596
|
||||
"median": 9257,
|
||||
"min": 9223,
|
||||
"max": 12049
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 1000000,
|
||||
"median": 15083,
|
||||
"min": 15038,
|
||||
"max": 16994
|
||||
"median": 16925,
|
||||
"min": 16848,
|
||||
"max": 17019
|
||||
}
|
||||
},
|
||||
"catch_unwind_panics": {
|
||||
"smarm 1-thread": {
|
||||
"result": 10000,
|
||||
"median": 110932,
|
||||
"min": 110182,
|
||||
"max": 124147
|
||||
"median": 116821,
|
||||
"min": 111345,
|
||||
"max": 128261
|
||||
},
|
||||
"smarm 24-thread": {
|
||||
"result": 10000,
|
||||
"median": 13665,
|
||||
"min": 13172,
|
||||
"max": 13784
|
||||
"median": 117487,
|
||||
"min": 107011,
|
||||
"max": 129307
|
||||
},
|
||||
"tokio current_thread": {
|
||||
"result": 10000,
|
||||
"median": 10431,
|
||||
"min": 9346,
|
||||
"max": 10915
|
||||
"median": 10425,
|
||||
"min": 10141,
|
||||
"max": 10604
|
||||
},
|
||||
"tokio multi-thread": {
|
||||
"result": 10000,
|
||||
"median": 6171,
|
||||
"min": 5626,
|
||||
"max": 6555
|
||||
"median": 6418,
|
||||
"min": 3715,
|
||||
"max": 7144
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run-queue shootout driver (ROADMAP_v0.5 phase 4; RFC 005 slot dimension
|
||||
# added for the v0.9 slot shootout).
|
||||
#
|
||||
# Rebuilds the runtime bench once per rq-* feature and runs the raw-structure
|
||||
# microbench once (it covers all structures in a single binary). The RFC 005
|
||||
# wake slot is a runtime Config knob, NOT a feature — each rq_runtime binary
|
||||
# sweeps slot off/on internally (SMARM_BENCH_SLOT, default "0 1"). Results
|
||||
# land in bench_results/ as full logs; the RQCSV lines are aggregated into
|
||||
# bench_results/summary.csv and the RQSLOT counter lines (slot hits /
|
||||
# displacements, slot-on configs only) into bench_results/slot_counters.csv.
|
||||
#
|
||||
# Tune the sweep for the box, e.g. on the 20-core machine:
|
||||
# SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh
|
||||
# Slot-only re-run against the frozen rq-mutex substrate:
|
||||
# SMARM_BENCH_SLOT="0 1" cargo bench --bench rq_runtime
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
OUT=bench_results
|
||||
mkdir -p "$OUT"
|
||||
: "${SMARM_BENCH_THREADS:=1 2 4}"
|
||||
: "${SMARM_BENCH_SLOT:=0 1}"
|
||||
export SMARM_BENCH_THREADS SMARM_BENCH_SLOT
|
||||
|
||||
echo "== raw structures (one binary, all variants) =="
|
||||
cargo bench --bench rq_micro 2>&1 | tee "$OUT/micro.txt"
|
||||
|
||||
for v in rq-mutex rq-mpmc rq-striped; do
|
||||
echo "== runtime benches: $v (slot sweep: $SMARM_BENCH_SLOT) =="
|
||||
cargo bench --bench rq_runtime --no-default-features --features "$v" \
|
||||
2>&1 | tee "$OUT/runtime-$v.txt"
|
||||
done
|
||||
|
||||
# runtime rows: kind,variant,slot,bench,threads,work,median_us,ops_per_s
|
||||
# micro rows: kind,structure,threads,p:c,items,median_us,items_per_s (one
|
||||
# column narrower, as before — split on kind when plotting)
|
||||
echo "kind,a,b,c,d,e,median_us,ops_per_s" > "$OUT/summary.csv"
|
||||
grep -h '^RQCSV,' "$OUT"/*.txt | sed 's/^RQCSV,//' >> "$OUT/summary.csv"
|
||||
|
||||
echo "variant,bench,threads,slot_hits,slot_displacements" > "$OUT/slot_counters.csv"
|
||||
grep -h '^RQSLOT,' "$OUT"/*.txt | sed 's/^RQSLOT,//' >> "$OUT/slot_counters.csv" || true
|
||||
|
||||
echo
|
||||
echo "Summary: $OUT/summary.csv ($(($(wc -l < "$OUT/summary.csv") - 1)) rows)"
|
||||
echo "Slot counters: $OUT/slot_counters.csv ($(($(wc -l < "$OUT/slot_counters.csv") - 1)) rows)"
|
||||
+7
-18
@@ -29,13 +29,6 @@ fn available_threads() -> usize {
|
||||
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
|
||||
}
|
||||
|
||||
fn env_sets() -> u32 {
|
||||
std::env::var("SMARM_BENCH_SETS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(5)
|
||||
}
|
||||
|
||||
fn print_header(title: &str) {
|
||||
println!("\n{}", "=".repeat(80));
|
||||
println!(" {title}");
|
||||
@@ -48,17 +41,14 @@ fn print_header(title: &str) {
|
||||
}
|
||||
|
||||
fn run_n<F: FnMut() -> (u64, u128)>(name: &str, n: u32, mut f: F) {
|
||||
let sets = env_sets();
|
||||
let mut times = Vec::with_capacity((n * sets) as usize);
|
||||
let mut times = Vec::new();
|
||||
let mut last = 0u64;
|
||||
// One warmup before all sets, discarded.
|
||||
// One warmup iteration, discarded.
|
||||
let _ = f();
|
||||
for _ in 0..sets {
|
||||
for _ in 0..n {
|
||||
let (v, t) = f();
|
||||
times.push(t);
|
||||
last = v;
|
||||
}
|
||||
for _ in 0..n {
|
||||
let (v, t) = f();
|
||||
times.push(t);
|
||||
last = v;
|
||||
}
|
||||
times.sort_unstable();
|
||||
let median = times[times.len() / 2];
|
||||
@@ -416,8 +406,7 @@ fn main() {
|
||||
let n = available_threads();
|
||||
println!("smarm general benchmarks");
|
||||
println!("available parallelism: {n} threads");
|
||||
let sets = env_sets();
|
||||
println!("ITERS={ITERS}×{sets} sets = {} samples (+1 warmup, discarded)", ITERS * sets);
|
||||
println!("ITERS={ITERS} (+1 warmup, discarded)");
|
||||
println!(
|
||||
"CHAIN_DEPTH={CHAIN_DEPTH}, YIELD_TASKS={YIELD_TASKS}×{YIELD_ROUNDS}, \
|
||||
PRIME_N={PRIME_N}/{PRIME_WORKERS} workers, PP_ROUNDS={PP_ROUNDS}"
|
||||
|
||||
@@ -1,186 +0,0 @@
|
||||
//! Raw run-queue microbench (ROADMAP_v0.5 phase 4).
|
||||
//!
|
||||
//! Benches the three queue STRUCTURES directly — no runtime, no actors — to
|
||||
//! isolate the data structure under contention. All three types compile in
|
||||
//! every build, so this binary covers the whole matrix in one run; it does
|
||||
//! NOT need the rq-* feature rebuild dance (that's `rq_runtime`).
|
||||
//!
|
||||
//! Sweeps thread count × producer:consumer ratio. Queues are sized to the
|
||||
//! item count, so the occupancy contract holds trivially and producers never
|
||||
//! block on capacity.
|
||||
//!
|
||||
//! Knobs (env):
|
||||
//! SMARM_BENCH_THREADS space-separated sweep, default "1 2 4"
|
||||
//! SMARM_BENCH_ITEMS items per measurement, default 200_000
|
||||
//! SMARM_BENCH_RUNS repetitions per config (median reported), default 5
|
||||
//!
|
||||
//! Output: the house table, plus one machine-readable line per config:
|
||||
//! RQCSV,micro,<structure>,<threads>,<p:c>,<items>,<median_us>,<items_per_s>
|
||||
//!
|
||||
//! NOTE: numbers from a 1-core sandbox only validate the harness; real
|
||||
//! contention curves come from the many-core box (scripts/bench_rq.sh).
|
||||
|
||||
use smarm::pid::Pid;
|
||||
use smarm::run_queue::{MpmcRing, MutexQueue, StripedRing};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
fn env_usize(key: &str, default: usize) -> usize {
|
||||
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
|
||||
}
|
||||
|
||||
fn env_threads() -> Vec<usize> {
|
||||
std::env::var("SMARM_BENCH_THREADS")
|
||||
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
|
||||
.unwrap_or_else(|_| vec![1, 2, 4])
|
||||
}
|
||||
|
||||
/// Generic driver: `producers` threads push `items` total, `consumers`
|
||||
/// threads pop until everything is accounted for. Returns elapsed µs.
|
||||
fn drive<Q: Send + Sync + 'static>(
|
||||
q: Arc<Q>,
|
||||
push: fn(&Q, Pid),
|
||||
pop: fn(&Q) -> Option<Pid>,
|
||||
producers: usize,
|
||||
consumers: usize,
|
||||
items: usize,
|
||||
) -> u128 {
|
||||
let remaining = Arc::new(AtomicUsize::new(items));
|
||||
let start = Instant::now();
|
||||
let mut hs = Vec::new();
|
||||
let per = items / producers;
|
||||
for p in 0..producers {
|
||||
let q = q.clone();
|
||||
// Give the last producer the remainder.
|
||||
let n = if p == producers - 1 { items - per * (producers - 1) } else { per };
|
||||
hs.push(std::thread::spawn(move || {
|
||||
let pid = Pid::new(p as u32, 0);
|
||||
for _ in 0..n {
|
||||
push(&q, pid);
|
||||
}
|
||||
}));
|
||||
}
|
||||
for _ in 0..consumers {
|
||||
let q = q.clone();
|
||||
let remaining = remaining.clone();
|
||||
hs.push(std::thread::spawn(move || loop {
|
||||
// Claim-then-pop so consumers exit promptly when the budget hits
|
||||
// zero; the claim is backed out on a miss.
|
||||
let r = remaining.load(Ordering::Relaxed);
|
||||
if r == 0 {
|
||||
return;
|
||||
}
|
||||
if pop(&q).is_some() {
|
||||
remaining.fetch_sub(1, Ordering::Relaxed);
|
||||
} else {
|
||||
std::hint::spin_loop();
|
||||
}
|
||||
}));
|
||||
}
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
start.elapsed().as_micros()
|
||||
}
|
||||
|
||||
/// Single-thread alternating push/pop (the T = 1 case).
|
||||
fn drive_single<Q>(q: &Q, push: fn(&Q, Pid), pop: fn(&Q) -> Option<Pid>, items: usize) -> u128 {
|
||||
let pid = Pid::new(0, 0);
|
||||
let start = Instant::now();
|
||||
for _ in 0..items {
|
||||
push(q, pid);
|
||||
assert!(pop(q).is_some());
|
||||
}
|
||||
start.elapsed().as_micros()
|
||||
}
|
||||
|
||||
struct Case {
|
||||
structure: &'static str,
|
||||
threads: usize,
|
||||
producers: usize,
|
||||
consumers: usize,
|
||||
}
|
||||
|
||||
fn ratios_for(threads: usize) -> Vec<(usize, usize)> {
|
||||
if threads < 2 {
|
||||
return vec![(1, 1)]; // label only; T=1 runs the alternating driver
|
||||
}
|
||||
let mut v = vec![(threads / 2, threads - threads / 2)]; // balanced
|
||||
if threads >= 4 {
|
||||
v.push((3 * threads / 4, threads - 3 * threads / 4)); // producer-heavy
|
||||
v.push((threads / 4, threads - threads / 4)); // consumer-heavy
|
||||
}
|
||||
v
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let threads_sweep = env_threads();
|
||||
let items = env_usize("SMARM_BENCH_ITEMS", 200_000);
|
||||
let runs = env_usize("SMARM_BENCH_RUNS", 5);
|
||||
|
||||
println!("\n{}", "=".repeat(86));
|
||||
println!(" run-queue raw structures — items={items}, runs={runs} (median)");
|
||||
println!("{}", "=".repeat(86));
|
||||
println!(
|
||||
"{:>10} | {:>7} | {:>7} | {:>10} | {:>14}",
|
||||
"structure", "threads", "p:c", "median µs", "items/s"
|
||||
);
|
||||
println!("{}", "-".repeat(86));
|
||||
|
||||
let mut cases = Vec::new();
|
||||
for &t in &threads_sweep {
|
||||
for (p, c) in ratios_for(t) {
|
||||
for s in ["mutex", "mpmc", "striped"] {
|
||||
cases.push(Case { structure: s, threads: t, producers: p, consumers: c });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for case in cases {
|
||||
let mut times: Vec<u128> = (0..runs)
|
||||
.map(|_| {
|
||||
// Fresh queue per run; capacity = items so pushes never stall.
|
||||
match case.structure {
|
||||
"mutex" => {
|
||||
let q = Arc::new(MutexQueue::new(case.threads, items));
|
||||
if case.threads < 2 {
|
||||
drive_single(&*q, MutexQueue::push, MutexQueue::pop, items)
|
||||
} else {
|
||||
drive(q, MutexQueue::push, MutexQueue::pop, case.producers, case.consumers, items)
|
||||
}
|
||||
}
|
||||
"mpmc" => {
|
||||
let q = Arc::new(MpmcRing::with_capacity(items));
|
||||
if case.threads < 2 {
|
||||
drive_single(&*q, MpmcRing::push, MpmcRing::pop, items)
|
||||
} else {
|
||||
drive(q, MpmcRing::push, MpmcRing::pop, case.producers, case.consumers, items)
|
||||
}
|
||||
}
|
||||
"striped" => {
|
||||
let q = Arc::new(StripedRing::new(case.threads.max(1), items));
|
||||
if case.threads < 2 {
|
||||
drive_single(&*q, StripedRing::push, StripedRing::pop, items)
|
||||
} else {
|
||||
drive(q, StripedRing::push, StripedRing::pop, case.producers, case.consumers, items)
|
||||
}
|
||||
}
|
||||
_ => unreachable!(),
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
times.sort_unstable();
|
||||
let median = times[times.len() / 2];
|
||||
let per_s = (items as f64 / (median as f64 / 1e6)) as u64;
|
||||
let ratio = format!("{}:{}", case.producers, case.consumers);
|
||||
println!(
|
||||
"{:>10} | {:>7} | {:>7} | {:>10} | {:>14}",
|
||||
case.structure, case.threads, ratio, median, per_s
|
||||
);
|
||||
println!(
|
||||
"RQCSV,micro,{},{},{},{},{},{}",
|
||||
case.structure, case.threads, ratio, items, median, per_s
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,252 +0,0 @@
|
||||
//! Runtime-level run-queue benches (ROADMAP_v0.5 phase 4; slot dimension
|
||||
//! added for the v0.9 slot shootout, RFC 005).
|
||||
//!
|
||||
//! These exercise the WHOLE scheduler with the compile-time-selected queue,
|
||||
//! so comparing variants means rebuilding per rq-* feature — that's what
|
||||
//! scripts/bench_rq.sh does. The RFC 005 wake slot is a *runtime* Config
|
||||
//! knob, so one binary benches both arms; the slot on/off sweep happens
|
||||
//! inside this binary. Workloads:
|
||||
//!
|
||||
//! yield-storm — N actors yield K times each. Pure queue churn:
|
||||
//! every yield is a push + pop with nothing in between.
|
||||
//! Slot role: REGRESSION GUARD — yields never touch the
|
||||
//! slot, any slot-on delta is pop-path overhead.
|
||||
//! ping-pong-pairs — P channel pairs, M roundtrips each. Park/unpark
|
||||
//! latency through the queue.
|
||||
//! Slot role: TARGET METRIC — every send-wake is an
|
||||
//! actor-context unpark, the slot's home pattern.
|
||||
//! spawn-storm — S spawn+join of trivial actors. Slab + queue + free
|
||||
//! list under churn.
|
||||
//! Slot role: NEUTRALITY CHECK — spawns bypass the slot
|
||||
//! by policy; join wakes fire from finalize (scheduler
|
||||
//! context), also shared.
|
||||
//!
|
||||
//! Knobs (env):
|
||||
//! SMARM_BENCH_THREADS scheduler-count sweep, default "1 2 4"
|
||||
//! SMARM_BENCH_SLOT wake-slot sweep, default "0 1" (off then on)
|
||||
//! SMARM_BENCH_RUNS repetitions per config (median), default 5
|
||||
//! SMARM_BENCH_YIELD_ACTORS / _YIELDS default 200 / 500
|
||||
//! SMARM_BENCH_PAIRS / _ROUNDTRIPS default 32 / 1000
|
||||
//! SMARM_BENCH_SPAWNS default 5000
|
||||
//!
|
||||
//! Output: house table + one line per config:
|
||||
//! RQCSV,runtime,<variant>,<slot>,<bench>,<threads>,<work>,<median_us>,<ops_per_s>
|
||||
//! plus, for slot-on configs, the RFC 005 observability counters:
|
||||
//! RQSLOT,<variant>,<bench>,<threads>,<slot_hits>,<slot_displacements>
|
||||
//! (hits/displacements are taken from the same run as the median time).
|
||||
//!
|
||||
//! NOTE: a 1-core sandbox validates the harness, not the scaling story;
|
||||
//! real curves come from the many-core box.
|
||||
|
||||
use smarm::runtime::{init, Config};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
fn variant() -> &'static str {
|
||||
if cfg!(feature = "rq-mpmc") {
|
||||
"rq-mpmc"
|
||||
} else if cfg!(feature = "rq-striped") {
|
||||
"rq-striped"
|
||||
} else {
|
||||
"rq-mutex"
|
||||
}
|
||||
}
|
||||
|
||||
fn env_usize(key: &str, default: usize) -> usize {
|
||||
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
|
||||
}
|
||||
|
||||
fn env_threads() -> Vec<usize> {
|
||||
std::env::var("SMARM_BENCH_THREADS")
|
||||
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
|
||||
.unwrap_or_else(|_| vec![1, 2, 4])
|
||||
}
|
||||
|
||||
fn env_slots() -> Vec<bool> {
|
||||
std::env::var("SMARM_BENCH_SLOT")
|
||||
.map(|v| {
|
||||
v.split_whitespace()
|
||||
.filter_map(|t| match t {
|
||||
"0" | "off" | "false" => Some(false),
|
||||
"1" | "on" | "true" => Some(true),
|
||||
_ => None,
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_else(|_| vec![false, true])
|
||||
}
|
||||
|
||||
/// One measured run: (total_ops, elapsed_µs, slot_hits, slot_displacements).
|
||||
struct Sample {
|
||||
ops: u64,
|
||||
us: u128,
|
||||
hits: u64,
|
||||
displacements: u64,
|
||||
}
|
||||
|
||||
fn yield_storm(threads: usize, slot: bool, actors: usize, yields: usize) -> Sample {
|
||||
let rt = init(Config::exact(threads).wake_slot(slot));
|
||||
let start = Instant::now();
|
||||
rt.run(move || {
|
||||
let handles: Vec<_> = (0..actors)
|
||||
.map(|_| {
|
||||
smarm::spawn(move || {
|
||||
for _ in 0..yields {
|
||||
smarm::yield_now();
|
||||
}
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
for h in handles {
|
||||
let _ = h.join();
|
||||
}
|
||||
});
|
||||
let us = start.elapsed().as_micros();
|
||||
let stats = rt.stats();
|
||||
Sample {
|
||||
ops: (actors * yields) as u64,
|
||||
us,
|
||||
hits: stats.slot_hits(),
|
||||
displacements: stats.slot_displacements(),
|
||||
}
|
||||
}
|
||||
|
||||
fn ping_pong_pairs(threads: usize, slot: bool, pairs: usize, roundtrips: usize) -> Sample {
|
||||
let rt = init(Config::exact(threads).wake_slot(slot));
|
||||
let total = Arc::new(AtomicU64::new(0));
|
||||
let t2 = total.clone();
|
||||
let start = Instant::now();
|
||||
rt.run(move || {
|
||||
let handles: Vec<_> = (0..pairs)
|
||||
.map(|_| {
|
||||
let total = t2.clone();
|
||||
smarm::spawn(move || {
|
||||
let (tx_ab, rx_ab) = smarm::channel::channel::<u64>();
|
||||
let (tx_ba, rx_ba) = smarm::channel::channel::<u64>();
|
||||
let n = roundtrips as u64;
|
||||
let echo = smarm::spawn(move || {
|
||||
for _ in 0..n {
|
||||
let v = rx_ab.recv().expect("echo recv");
|
||||
tx_ba.send(v + 1).expect("echo send");
|
||||
}
|
||||
});
|
||||
for i in 0..n {
|
||||
tx_ab.send(i).expect("ping send");
|
||||
let v = rx_ba.recv().expect("ping recv");
|
||||
assert_eq!(v, i + 1);
|
||||
}
|
||||
let _ = echo.join();
|
||||
total.fetch_add(n, Ordering::Relaxed);
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
for h in handles {
|
||||
let _ = h.join();
|
||||
}
|
||||
});
|
||||
let us = start.elapsed().as_micros();
|
||||
let stats = rt.stats();
|
||||
Sample {
|
||||
ops: total.load(Ordering::Relaxed),
|
||||
us,
|
||||
hits: stats.slot_hits(),
|
||||
displacements: stats.slot_displacements(),
|
||||
}
|
||||
}
|
||||
|
||||
fn spawn_storm(threads: usize, slot: bool, spawns: usize) -> Sample {
|
||||
let rt = init(Config::exact(threads).wake_slot(slot));
|
||||
let start = Instant::now();
|
||||
rt.run(move || {
|
||||
// Batches bound simultaneous liveness well below the slab cap.
|
||||
const BATCH: usize = 1024;
|
||||
let mut left = spawns;
|
||||
while left > 0 {
|
||||
let n = left.min(BATCH);
|
||||
let handles: Vec<_> = (0..n).map(|_| smarm::spawn(|| {})).collect();
|
||||
for h in handles {
|
||||
let _ = h.join();
|
||||
}
|
||||
left -= n;
|
||||
}
|
||||
});
|
||||
let us = start.elapsed().as_micros();
|
||||
let stats = rt.stats();
|
||||
Sample {
|
||||
ops: spawns as u64,
|
||||
us,
|
||||
hits: stats.slot_hits(),
|
||||
displacements: stats.slot_displacements(),
|
||||
}
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let threads_sweep = env_threads();
|
||||
let slot_sweep = env_slots();
|
||||
let runs = env_usize("SMARM_BENCH_RUNS", 5);
|
||||
let ya = env_usize("SMARM_BENCH_YIELD_ACTORS", 200);
|
||||
let yy = env_usize("SMARM_BENCH_YIELDS", 500);
|
||||
let pp = env_usize("SMARM_BENCH_PAIRS", 32);
|
||||
let pr = env_usize("SMARM_BENCH_ROUNDTRIPS", 1000);
|
||||
let ss = env_usize("SMARM_BENCH_SPAWNS", 5000);
|
||||
|
||||
println!("\n{}", "=".repeat(106));
|
||||
println!(
|
||||
" runtime benches — variant={}, runs={runs} (median)",
|
||||
variant()
|
||||
);
|
||||
println!("{}", "=".repeat(106));
|
||||
println!(
|
||||
"{:>16} | {:>7} | {:>4} | {:>16} | {:>10} | {:>14} | {:>10} | {:>9}",
|
||||
"bench", "threads", "slot", "work", "median µs", "ops/s", "slot hits", "displaced"
|
||||
);
|
||||
println!("{}", "-".repeat(106));
|
||||
|
||||
type Bench = (&'static str, String, Box<dyn Fn(usize, bool) -> Sample>);
|
||||
let benches: Vec<Bench> = vec![
|
||||
(
|
||||
"yield-storm",
|
||||
format!("{ya}x{yy}"),
|
||||
Box::new(move |t, s| yield_storm(t, s, ya, yy)),
|
||||
),
|
||||
(
|
||||
"ping-pong-pairs",
|
||||
format!("{pp}x{pr}"),
|
||||
Box::new(move |t, s| ping_pong_pairs(t, s, pp, pr)),
|
||||
),
|
||||
(
|
||||
"spawn-storm",
|
||||
format!("{ss}"),
|
||||
Box::new(move |t, s| spawn_storm(t, s, ss)),
|
||||
),
|
||||
];
|
||||
|
||||
for (name, work, f) in &benches {
|
||||
for &t in &threads_sweep {
|
||||
for &slot in &slot_sweep {
|
||||
let mut samples: Vec<Sample> = (0..runs).map(|_| f(t, slot)).collect();
|
||||
// Median by elapsed time; report the counters from that
|
||||
// same run so hits/time stay paired.
|
||||
samples.sort_unstable_by_key(|s| s.us);
|
||||
let mid = &samples[samples.len() / 2];
|
||||
let per_s = (mid.ops as f64 / (mid.us as f64 / 1e6)) as u64;
|
||||
let slot_str = if slot { "on" } else { "off" };
|
||||
println!(
|
||||
"{:>16} | {:>7} | {:>4} | {:>16} | {:>10} | {:>14} | {:>10} | {:>9}",
|
||||
name, t, slot_str, work, mid.us, per_s, mid.hits, mid.displacements
|
||||
);
|
||||
println!(
|
||||
"RQCSV,runtime,{},{},{},{},{},{},{}",
|
||||
variant(), slot_str, name, t, work, mid.us, per_s
|
||||
);
|
||||
if slot {
|
||||
println!(
|
||||
"RQSLOT,{},{},{},{},{}",
|
||||
variant(), name, t, mid.hits, mid.displacements
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -40,13 +40,6 @@ fn available_threads() -> usize {
|
||||
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
|
||||
}
|
||||
|
||||
fn env_sets() -> u32 {
|
||||
std::env::var("SMARM_BENCH_SETS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(5)
|
||||
}
|
||||
|
||||
fn print_header(title: &str) {
|
||||
println!("\n{}", "=".repeat(80));
|
||||
println!(" {title}");
|
||||
@@ -59,17 +52,13 @@ fn print_header(title: &str) {
|
||||
}
|
||||
|
||||
fn run_n<F: FnMut() -> (u64, u128)>(name: &str, n: u32, mut f: F) {
|
||||
let sets = env_sets();
|
||||
let mut times = Vec::with_capacity((n * sets) as usize);
|
||||
let mut times = Vec::new();
|
||||
let mut last = 0u64;
|
||||
// One warmup before all sets, discarded.
|
||||
let _ = f();
|
||||
for _ in 0..sets {
|
||||
for _ in 0..n {
|
||||
let (v, t) = f();
|
||||
times.push(t);
|
||||
last = v;
|
||||
}
|
||||
let _ = f(); // warmup
|
||||
for _ in 0..n {
|
||||
let (v, t) = f();
|
||||
times.push(t);
|
||||
last = v;
|
||||
}
|
||||
times.sort_unstable();
|
||||
let median = times[times.len() / 2];
|
||||
@@ -396,8 +385,7 @@ fn main() {
|
||||
let n = available_threads();
|
||||
println!("smarm smarm-favored benchmarks");
|
||||
println!("available parallelism: {n} threads");
|
||||
let sets = env_sets();
|
||||
println!("ITERS={ITERS}×{sets} sets = {} samples (+1 warmup, discarded)", ITERS * sets);
|
||||
println!("ITERS={ITERS} (+1 warmup, discarded)");
|
||||
println!(
|
||||
"RECURSE_DEPTH={RECURSE_DEPTH}, HOT_YIELDS={HOT_YIELDS}×2, \
|
||||
UNCONT_MSGS={UNCONT_MSGS}, PANIC_TASKS={PANIC_TASKS}"
|
||||
|
||||
@@ -1,435 +0,0 @@
|
||||
//! RFC 004 spinning-worker measurement tool — the latency-vs-idle-CPU curve
|
||||
//! over `spin_budget_cycles`.
|
||||
//!
|
||||
//! Spinning workers trade idle CPU for wake latency: an idle worker burns
|
||||
//! cycles polling `queue_len` (up to `spin_budget_cycles`, capped at
|
||||
//! `max_spinners` concurrent spinners) so that incoming work is claimed with
|
||||
//! no futex syscall. `spin_budget_cycles = 0` is the feature fully off — the
|
||||
//! historical `thread::sleep` park, exactly preserved. This binary measures
|
||||
//! both sides of that trade so the sweep can pick the budget default (RFC 004
|
||||
//! Q3) and the on/off posture (Q4).
|
||||
//!
|
||||
//! Workloads — each one is a fresh runtime per sample (no cross-contamination):
|
||||
//!
|
||||
//! remote-wake — a consumer blocks on recv (its worker idles); a producer
|
||||
//! on another scheduler timestamps just before send. Latency
|
||||
//! = send -> receipt, the quantity spinning is meant to cut.
|
||||
//! A fixed gap (SMARM_SPIN_GAP_US) between rounds lets the
|
||||
//! consumer re-idle each round. THE LATENCY TARGET.
|
||||
//! fork-join — root forks FANOUT trivial actors and joins them (one
|
||||
//! burst), gap, repeat. Latency = fork -> all-joined; stresses
|
||||
//! the fan-out wake. Secondary latency target.
|
||||
//! half-load — work dispatched every PERIOD_US for ITEMS items, so
|
||||
//! workers idle a fraction of each period. THE COST TARGET:
|
||||
//! cores-busy climbs with budget as spin fills the idle gaps.
|
||||
//! pure-idle — one actor sleeps WINDOW_MS, nothing else runs. The cleanest
|
||||
//! single picture of spin burn: cores-busy is ~0 at budget 0
|
||||
//! and climbs toward max_spinners as the budget grows.
|
||||
//!
|
||||
//! Two axes, reported together per config:
|
||||
//! latency — p50 / p90 / p99 / min / max over the POOLED rounds of every run
|
||||
//! (pooling gives a stabler tail than a median-of-percentiles).
|
||||
//! cost — cores-busy = process CPU-seconds (getrusage RUSAGE_SELF, all
|
||||
//! runtime threads) / wall-seconds over the measured window. 1.0 =
|
||||
//! one core pegged; approaches max_spinners when every spinner is
|
||||
//! hot. Reported as the median over runs.
|
||||
//!
|
||||
//! The crossover the sweep reveals: spinning only wins when work arrives while
|
||||
//! a worker is still inside its budget. With a fixed inter-arrival gap, that
|
||||
//! means budgets whose spin-time (budget_cycles / TSC_freq) exceeds the gap
|
||||
//! start catching work before the park; smaller budgets park and pay the full
|
||||
//! wake. So GAP_US sets the knee — tune it against your box's TSC frequency.
|
||||
//!
|
||||
//! Knobs (env):
|
||||
//! SMARM_BENCH_THREADS scheduler-count sweep default "1 2"
|
||||
//! SMARM_BENCH_RUNS runs per config (pooled/median) default 3
|
||||
//! SMARM_SPIN_BUDGET spin_budget_cycles sweep default "0 1000 10000 100000 1000000"
|
||||
//! SMARM_MAX_SPINNERS max_spinners sweep; "-" = runtime default (N/2) default "-"
|
||||
//! SMARM_SPIN_GAP_US idle gap between rounds/bursts default 150
|
||||
//! SMARM_SPIN_ROUNDS remote-wake rounds default 100
|
||||
//! SMARM_SPIN_FANOUT fork-join actors per burst default 64
|
||||
//! SMARM_SPIN_BURSTS fork-join bursts default 30
|
||||
//! SMARM_SPIN_ITEMS half-load dispatch count default 150
|
||||
//! SMARM_SPIN_PERIOD_US half-load inter-dispatch period default 300
|
||||
//! SMARM_SPIN_WINDOW_MS pure-idle window default 25
|
||||
//!
|
||||
//! Output: house table + one greppable line per config:
|
||||
//! SPINCSV,<variant>,<workload>,<threads>,<budget>,<spinners>,<gap_us>,
|
||||
//! <n>,<p50_us>,<p90_us>,<p99_us>,<min_us>,<max_us>,<cores_busy>,<ops_s>
|
||||
//! ("spinners" is the literal "default" when SMARM_MAX_SPINNERS is "-".)
|
||||
//!
|
||||
//! NOTE: on a 1-core box (CI, sandboxes) spinning is *pathological* — a hot
|
||||
//! spinner starves the very thread that would enqueue work, so the optimal
|
||||
//! budget there is 0 and you only ever see the cost, never the benefit. This
|
||||
//! binary on one core validates the harness and catches gross pathologies; the
|
||||
//! real latency-vs-idle-CPU curve comes from the many-core box.
|
||||
|
||||
use smarm::runtime::{init, Config};
|
||||
use smarm::{channel, sleep, spawn};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
// --------------------------------------------------------------------------
|
||||
// env helpers (house style, matching rq_runtime.rs)
|
||||
// --------------------------------------------------------------------------
|
||||
|
||||
fn variant() -> &'static str {
|
||||
if cfg!(feature = "rq-mpmc") {
|
||||
"rq-mpmc"
|
||||
} else if cfg!(feature = "rq-striped") {
|
||||
"rq-striped"
|
||||
} else {
|
||||
"rq-mutex"
|
||||
}
|
||||
}
|
||||
|
||||
fn env_usize(key: &str, default: usize) -> usize {
|
||||
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
|
||||
}
|
||||
|
||||
fn env_u64(key: &str, default: u64) -> u64 {
|
||||
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
|
||||
}
|
||||
|
||||
fn env_threads() -> Vec<usize> {
|
||||
std::env::var("SMARM_BENCH_THREADS")
|
||||
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
|
||||
.unwrap_or_else(|_| vec![1, 2])
|
||||
}
|
||||
|
||||
fn env_budgets() -> Vec<u64> {
|
||||
std::env::var("SMARM_SPIN_BUDGET")
|
||||
.map(|v| v.split_whitespace().filter_map(|t| t.parse().ok()).collect())
|
||||
.unwrap_or_else(|_| vec![0, 1_000, 10_000, 100_000, 1_000_000])
|
||||
}
|
||||
|
||||
/// `None` means "leave the runtime default (N/2)"; encoded in env as "-".
|
||||
fn env_spinners() -> Vec<Option<u32>> {
|
||||
std::env::var("SMARM_MAX_SPINNERS")
|
||||
.map(|v| {
|
||||
v.split_whitespace()
|
||||
.map(|t| if t == "-" { None } else { t.parse().ok() })
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_else(|_| vec![None])
|
||||
}
|
||||
|
||||
fn make_config(threads: usize, budget: u64, spinners: Option<u32>) -> Config {
|
||||
let cfg = Config::exact(threads).spin_budget_cycles(budget);
|
||||
match spinners {
|
||||
Some(n) => cfg.max_spinners(n),
|
||||
None => cfg,
|
||||
}
|
||||
}
|
||||
|
||||
// --------------------------------------------------------------------------
|
||||
// CPU accounting — process-wide (all runtime threads) via getrusage
|
||||
// --------------------------------------------------------------------------
|
||||
|
||||
/// Total CPU seconds (user + system) consumed by the whole process so far.
|
||||
/// RUSAGE_SELF aggregates every thread, which is exactly what "how much did
|
||||
/// the runtime burn" needs — the spinners are separate threads from the one
|
||||
/// reading this.
|
||||
fn cpu_seconds() -> f64 {
|
||||
// SAFETY: getrusage with RUSAGE_SELF and a valid out-pointer; we check the
|
||||
// return code and only read the (now-initialised) timeval fields.
|
||||
let mut ru: libc::rusage = unsafe { std::mem::zeroed() };
|
||||
let rc = unsafe { libc::getrusage(libc::RUSAGE_SELF, &mut ru) };
|
||||
assert_eq!(rc, 0, "getrusage(RUSAGE_SELF) failed");
|
||||
let tv = |t: libc::timeval| t.tv_sec as f64 + t.tv_usec as f64 * 1e-6;
|
||||
tv(ru.ru_utime) + tv(ru.ru_stime)
|
||||
}
|
||||
|
||||
// --------------------------------------------------------------------------
|
||||
// One sample = one fresh runtime run
|
||||
// --------------------------------------------------------------------------
|
||||
|
||||
struct Sample {
|
||||
/// Per-event latencies in nanoseconds (empty for pure-idle).
|
||||
lat_ns: Vec<u64>,
|
||||
/// CPU-seconds / wall-seconds over the measured window.
|
||||
cores_busy: f64,
|
||||
/// Completed work units (for ops/s); 0 where not meaningful.
|
||||
ops: u64,
|
||||
/// Wall-seconds of the measured window (for ops/s).
|
||||
wall_s: f64,
|
||||
}
|
||||
|
||||
/// Shared scratch the root closure writes and the caller reads after `run()`
|
||||
/// returns. (`run` takes a `FnOnce` and yields no value, so we hand results
|
||||
/// out through an Arc rather than a return.)
|
||||
#[derive(Default)]
|
||||
struct Scratch {
|
||||
lat_ns: Vec<u64>,
|
||||
cpu_delta: f64,
|
||||
wall_s: f64,
|
||||
ops: u64,
|
||||
}
|
||||
|
||||
fn finish(scratch: Arc<Mutex<Scratch>>) -> Sample {
|
||||
let mut s = scratch.lock().unwrap();
|
||||
let cores_busy = if s.wall_s > 0.0 { s.cpu_delta / s.wall_s } else { 0.0 };
|
||||
Sample {
|
||||
lat_ns: std::mem::take(&mut s.lat_ns),
|
||||
cores_busy,
|
||||
ops: s.ops,
|
||||
wall_s: s.wall_s,
|
||||
}
|
||||
}
|
||||
|
||||
fn remote_wake(threads: usize, budget: u64, spinners: Option<u32>, rounds: usize, gap: Duration) -> Sample {
|
||||
let rt = init(make_config(threads, budget, spinners));
|
||||
let scratch = Arc::new(Mutex::new(Scratch::default()));
|
||||
let out = scratch.clone();
|
||||
rt.run(move || {
|
||||
let (tx, rx) = channel::channel::<Instant>();
|
||||
let lat_sink = out.clone();
|
||||
// Consumer parks on recv; its worker idles (spins or parks per budget)
|
||||
// until each send wakes it. It records the wake latency itself.
|
||||
let consumer = spawn(move || {
|
||||
let mut lats = Vec::with_capacity(rounds);
|
||||
for _ in 0..rounds {
|
||||
let sent = rx.recv().expect("remote-wake recv");
|
||||
lats.push(sent.elapsed().as_nanos() as u64);
|
||||
}
|
||||
lat_sink.lock().unwrap().lat_ns = lats;
|
||||
});
|
||||
// Producer: sleep the gap (so the consumer's worker re-idles), then
|
||||
// timestamp and send. The send is the wake event under measurement.
|
||||
let cpu0 = cpu_seconds();
|
||||
let w0 = Instant::now();
|
||||
for _ in 0..rounds {
|
||||
sleep(gap);
|
||||
tx.send(Instant::now()).expect("remote-wake send");
|
||||
}
|
||||
let wall = w0.elapsed().as_secs_f64();
|
||||
let cpu = cpu_seconds() - cpu0;
|
||||
let _ = consumer.join();
|
||||
let mut s = out.lock().unwrap();
|
||||
s.cpu_delta = cpu;
|
||||
s.wall_s = wall;
|
||||
s.ops = rounds as u64;
|
||||
});
|
||||
finish(scratch)
|
||||
}
|
||||
|
||||
fn fork_join(threads: usize, budget: u64, spinners: Option<u32>, fanout: usize, bursts: usize, gap: Duration) -> Sample {
|
||||
let rt = init(make_config(threads, budget, spinners));
|
||||
let scratch = Arc::new(Mutex::new(Scratch::default()));
|
||||
let out = scratch.clone();
|
||||
rt.run(move || {
|
||||
let mut lats = Vec::with_capacity(bursts);
|
||||
let cpu0 = cpu_seconds();
|
||||
let w0 = Instant::now();
|
||||
for _ in 0..bursts {
|
||||
// Gap lets the workers park between bursts, so the fan-out has real
|
||||
// wakes to do rather than landing on already-hot workers.
|
||||
sleep(gap);
|
||||
let t = Instant::now();
|
||||
let handles: Vec<_> = (0..fanout).map(|_| spawn(|| {})).collect();
|
||||
for h in handles {
|
||||
let _ = h.join();
|
||||
}
|
||||
lats.push(t.elapsed().as_nanos() as u64);
|
||||
}
|
||||
let wall = w0.elapsed().as_secs_f64();
|
||||
let cpu = cpu_seconds() - cpu0;
|
||||
let mut s = out.lock().unwrap();
|
||||
s.lat_ns = lats;
|
||||
s.cpu_delta = cpu;
|
||||
s.wall_s = wall;
|
||||
s.ops = (bursts * fanout) as u64;
|
||||
});
|
||||
finish(scratch)
|
||||
}
|
||||
|
||||
fn half_load(threads: usize, budget: u64, spinners: Option<u32>, items: usize, period: Duration) -> Sample {
|
||||
let rt = init(make_config(threads, budget, spinners));
|
||||
let scratch = Arc::new(Mutex::new(Scratch::default()));
|
||||
let out = scratch.clone();
|
||||
rt.run(move || {
|
||||
let cpu0 = cpu_seconds();
|
||||
let w0 = Instant::now();
|
||||
// Dispatch one trivial actor every `period`. Between dispatches the
|
||||
// workers have nothing to do, so they idle for a fraction of each
|
||||
// period — that idle fraction is what spinning fills (or doesn't).
|
||||
let mut handles = Vec::with_capacity(items);
|
||||
for i in 0..items {
|
||||
let target = period * i as u32;
|
||||
let elapsed = w0.elapsed();
|
||||
if target > elapsed {
|
||||
sleep(target - elapsed);
|
||||
}
|
||||
handles.push(spawn(|| {}));
|
||||
}
|
||||
for h in handles {
|
||||
let _ = h.join();
|
||||
}
|
||||
let wall = w0.elapsed().as_secs_f64();
|
||||
let cpu = cpu_seconds() - cpu0;
|
||||
let mut s = out.lock().unwrap();
|
||||
s.cpu_delta = cpu;
|
||||
s.wall_s = wall;
|
||||
s.ops = items as u64;
|
||||
});
|
||||
finish(scratch)
|
||||
}
|
||||
|
||||
fn pure_idle(threads: usize, budget: u64, spinners: Option<u32>, window: Duration) -> Sample {
|
||||
let rt = init(make_config(threads, budget, spinners));
|
||||
let scratch = Arc::new(Mutex::new(Scratch::default()));
|
||||
let out = scratch.clone();
|
||||
rt.run(move || {
|
||||
// One actor sleeps; every other worker thread has nothing to do and
|
||||
// spins-or-parks per budget for the whole window.
|
||||
let cpu0 = cpu_seconds();
|
||||
let w0 = Instant::now();
|
||||
sleep(window);
|
||||
let wall = w0.elapsed().as_secs_f64();
|
||||
let cpu = cpu_seconds() - cpu0;
|
||||
let mut s = out.lock().unwrap();
|
||||
s.cpu_delta = cpu;
|
||||
s.wall_s = wall;
|
||||
s.ops = 0;
|
||||
});
|
||||
finish(scratch)
|
||||
}
|
||||
|
||||
// --------------------------------------------------------------------------
|
||||
// stats
|
||||
// --------------------------------------------------------------------------
|
||||
|
||||
/// Nearest-rank percentile over an already-sorted slice. `p` in [0, 100].
|
||||
fn pct(sorted: &[u64], p: f64) -> u64 {
|
||||
if sorted.is_empty() {
|
||||
return 0;
|
||||
}
|
||||
let idx = ((p / 100.0) * (sorted.len() - 1) as f64).round() as usize;
|
||||
sorted[idx.min(sorted.len() - 1)]
|
||||
}
|
||||
|
||||
fn median_f64(xs: &mut [f64]) -> f64 {
|
||||
if xs.is_empty() {
|
||||
return 0.0;
|
||||
}
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
struct Aggregate {
|
||||
n: usize,
|
||||
p50_us: f64,
|
||||
p90_us: f64,
|
||||
p99_us: f64,
|
||||
min_us: f64,
|
||||
max_us: f64,
|
||||
cores_busy: f64,
|
||||
ops_s: f64,
|
||||
}
|
||||
|
||||
/// Pool latencies across all runs (stabler tail); take the median cores-busy
|
||||
/// and median ops/s across runs.
|
||||
fn aggregate(samples: Vec<Sample>) -> Aggregate {
|
||||
let mut pooled: Vec<u64> = Vec::new();
|
||||
let mut cores: Vec<f64> = Vec::new();
|
||||
let mut ops_s: Vec<f64> = Vec::new();
|
||||
for s in &samples {
|
||||
pooled.extend_from_slice(&s.lat_ns);
|
||||
cores.push(s.cores_busy);
|
||||
if s.wall_s > 0.0 && s.ops > 0 {
|
||||
ops_s.push(s.ops as f64 / s.wall_s);
|
||||
}
|
||||
}
|
||||
pooled.sort_unstable();
|
||||
let ns_to_us = |ns: u64| ns as f64 / 1000.0;
|
||||
Aggregate {
|
||||
n: pooled.len(),
|
||||
p50_us: ns_to_us(pct(&pooled, 50.0)),
|
||||
p90_us: ns_to_us(pct(&pooled, 90.0)),
|
||||
p99_us: ns_to_us(pct(&pooled, 99.0)),
|
||||
min_us: pooled.first().map(|&v| ns_to_us(v)).unwrap_or(0.0),
|
||||
max_us: pooled.last().map(|&v| ns_to_us(v)).unwrap_or(0.0),
|
||||
cores_busy: median_f64(&mut cores),
|
||||
ops_s: median_f64(&mut ops_s),
|
||||
}
|
||||
}
|
||||
|
||||
// --------------------------------------------------------------------------
|
||||
// driver
|
||||
// --------------------------------------------------------------------------
|
||||
|
||||
fn main() {
|
||||
let threads_sweep = env_threads();
|
||||
let budgets = env_budgets();
|
||||
let spinners_sweep = env_spinners();
|
||||
let runs = env_usize("SMARM_BENCH_RUNS", 3);
|
||||
let gap = Duration::from_micros(env_u64("SMARM_SPIN_GAP_US", 150));
|
||||
let rounds = env_usize("SMARM_SPIN_ROUNDS", 100);
|
||||
let fanout = env_usize("SMARM_SPIN_FANOUT", 64);
|
||||
let bursts = env_usize("SMARM_SPIN_BURSTS", 30);
|
||||
let items = env_usize("SMARM_SPIN_ITEMS", 150);
|
||||
let period = Duration::from_micros(env_u64("SMARM_SPIN_PERIOD_US", 300));
|
||||
let window = Duration::from_millis(env_u64("SMARM_SPIN_WINDOW_MS", 25));
|
||||
|
||||
// (name, closure producing one Sample for a (threads,budget,spinners) point)
|
||||
type Workload = (&'static str, Box<dyn Fn(usize, u64, Option<u32>) -> Sample>);
|
||||
let workloads: Vec<Workload> = vec![
|
||||
("remote-wake", Box::new(move |t, b, s| remote_wake(t, b, s, rounds, gap))),
|
||||
("fork-join", Box::new(move |t, b, s| fork_join(t, b, s, fanout, bursts, gap))),
|
||||
("half-load", Box::new(move |t, b, s| half_load(t, b, s, items, period))),
|
||||
("pure-idle", Box::new(move |t, b, s| pure_idle(t, b, s, window))),
|
||||
];
|
||||
|
||||
let width = 122;
|
||||
println!("\n{}", "=".repeat(width));
|
||||
println!(
|
||||
" spin sweep — variant={}, runs={runs} (latency pooled, cores median), gap={}µs",
|
||||
variant(),
|
||||
gap.as_micros()
|
||||
);
|
||||
println!("{}", "=".repeat(width));
|
||||
println!(
|
||||
"{:>11} | {:>3} | {:>9} | {:>8} | {:>6} | {:>9} | {:>9} | {:>9} | {:>9} | {:>9} | {:>7} | {:>10}",
|
||||
"workload", "thr", "budget", "spinners", "n", "p50 µs", "p90 µs", "p99 µs", "min µs", "max µs", "cores", "ops/s"
|
||||
);
|
||||
println!("{}", "-".repeat(width));
|
||||
|
||||
for (name, run_one) in &workloads {
|
||||
for &threads in &threads_sweep {
|
||||
for &budget in &budgets {
|
||||
for &spinners in &spinners_sweep {
|
||||
let samples: Vec<Sample> =
|
||||
(0..runs).map(|_| run_one(threads, budget, spinners)).collect();
|
||||
let a = aggregate(samples);
|
||||
let sp_str = match spinners {
|
||||
Some(n) => n.to_string(),
|
||||
None => "default".to_string(),
|
||||
};
|
||||
// Latency columns are blank for the no-latency probe.
|
||||
let has_lat = a.n > 0;
|
||||
let lat = |v: f64| if has_lat { format!("{v:.2}") } else { "-".to_string() };
|
||||
println!(
|
||||
"{:>11} | {:>3} | {:>9} | {:>8} | {:>6} | {:>9} | {:>9} | {:>9} | {:>9} | {:>9} | {:>7.3} | {:>10}",
|
||||
name,
|
||||
threads,
|
||||
budget,
|
||||
sp_str,
|
||||
a.n,
|
||||
lat(a.p50_us),
|
||||
lat(a.p90_us),
|
||||
lat(a.p99_us),
|
||||
lat(a.min_us),
|
||||
lat(a.max_us),
|
||||
a.cores_busy,
|
||||
if a.ops_s > 0.0 { format!("{:.0}", a.ops_s) } else { "-".to_string() },
|
||||
);
|
||||
println!(
|
||||
"SPINCSV,{},{},{},{},{},{},{},{:.3},{:.3},{:.3},{:.3},{:.3},{:.4},{:.0}",
|
||||
variant(), name, threads, budget, sp_str, gap.as_micros(),
|
||||
a.n, a.p50_us, a.p90_us, a.p99_us, a.min_us, a.max_us, a.cores_busy, a.ops_s
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+5
-137
@@ -50,11 +50,6 @@ SWEEP_GRID = [
|
||||
(128, 1_200_000),
|
||||
]
|
||||
|
||||
# Number of independent cargo bench processes per measurement point.
|
||||
# Each process is a fully isolated run (fresh warmup, cold caches, new PID),
|
||||
# so the final median is a median of independent samples — robust to OS noise.
|
||||
BENCH_SETS = 5
|
||||
|
||||
# Regression threshold: warn if median is more than this % worse than baseline.
|
||||
REGRESSION_THRESHOLD_PCT = 10
|
||||
|
||||
@@ -108,12 +103,9 @@ def parse_output(text: str) -> dict[str, dict[str, dict]]:
|
||||
# Running
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def run_benches_once(env_extra: dict[str, str] | None = None) -> dict[str, dict[str, dict]]:
|
||||
"""Run all BENCHES once and return merged parsed results."""
|
||||
def run_benches(env_extra: dict[str, str] | None = None) -> dict[str, dict[str, dict]]:
|
||||
"""Run all BENCHES and return merged parsed results."""
|
||||
env = os.environ.copy()
|
||||
# Each process does exactly one set of ITERS samples — no within-process
|
||||
# accumulation; the caller handles multi-set aggregation.
|
||||
env["SMARM_BENCH_SETS"] = "1"
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
|
||||
@@ -137,45 +129,6 @@ def run_benches_once(env_extra: dict[str, str] | None = None) -> dict[str, dict[
|
||||
return all_results
|
||||
|
||||
|
||||
def run_benches(env_extra: dict[str, str] | None = None, sets: int = BENCH_SETS) -> dict[str, dict[str, dict]]:
|
||||
"""Run BENCH_SETS independent processes and return median-of-medians per label.
|
||||
|
||||
Each set is a separate cargo bench invocation with its own warmup and OS
|
||||
context, so samples are statistically independent. The final median and
|
||||
min/max are computed over the per-set medians.
|
||||
"""
|
||||
# Accumulate per-set medians: {bench: {label: [median_set1, median_set2, ...]}}
|
||||
accumulated: dict[str, dict[str, list[int]]] = {}
|
||||
last_result: dict[str, dict[str, int]] = {}
|
||||
|
||||
for i in range(sets):
|
||||
print(f" set {i + 1}/{sets}…", flush=True)
|
||||
set_results = run_benches_once(env_extra)
|
||||
for bench, labels in set_results.items():
|
||||
accumulated.setdefault(bench, {})
|
||||
last_result.setdefault(bench, {})
|
||||
for label, data in labels.items():
|
||||
accumulated[bench].setdefault(label, [])
|
||||
accumulated[bench][label].append(data["median"])
|
||||
last_result[bench][label] = data["result"]
|
||||
|
||||
# Collapse to final stats.
|
||||
final: dict[str, dict[str, dict]] = {}
|
||||
for bench, labels in accumulated.items():
|
||||
final[bench] = {}
|
||||
for label, medians in labels.items():
|
||||
medians.sort()
|
||||
mid = medians[len(medians) // 2]
|
||||
final[bench][label] = {
|
||||
"result": last_result[bench][label],
|
||||
"median": mid,
|
||||
"min": medians[0],
|
||||
"max": medians[-1],
|
||||
}
|
||||
|
||||
return final
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Baseline JSON
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -243,90 +196,6 @@ def check_regressions(current: dict, baseline: dict) -> bool:
|
||||
# Pretty print
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _threads(label: str) -> int | None:
|
||||
"""Worker-thread count implied by a runtime label.
|
||||
|
||||
tokio's `current_thread` is a single-threaded executor (1); an explicit
|
||||
`multi N-thread` is N; a bare `multi-thread` has no count (None) — tokio's
|
||||
default work-stealing pool, paired against smarm's widest config.
|
||||
"""
|
||||
if "current_thread" in label:
|
||||
return 1
|
||||
m = re.search(r"(\d+)-thread", label)
|
||||
return int(m.group(1)) if m else None
|
||||
|
||||
|
||||
def vs_tokio(results: dict) -> list[tuple]:
|
||||
"""Per bench, like-for-like smarm-vs-tokio rows matched by thread count.
|
||||
|
||||
Exact thread-count matches are paired directly (smarm 1-thread vs tokio
|
||||
current_thread, smarm 4-thread vs tokio multi 4-thread, …). tokio's bare
|
||||
`multi-thread` (no explicit count) is paired against the widest unmatched
|
||||
smarm multi-thread config. Lower median µs = faster; ratio = tokio_med /
|
||||
smarm_med, so ratio > 1 means smarm is that many times faster.
|
||||
|
||||
Returns rows of (bench, smarm_label, smarm_med, tokio_label, tokio_med,
|
||||
ratio, winner). Benches without a comparable pair are skipped.
|
||||
"""
|
||||
rows: list[tuple] = []
|
||||
for bench, runtimes in sorted(results.items()):
|
||||
smarm: dict[int, tuple[str, int]] = {}
|
||||
tokio: dict[int, tuple[str, int]] = {}
|
||||
tokio_default: tuple[str, int] | None = None # bare 'multi-thread'
|
||||
for label, data in runtimes.items():
|
||||
n = _threads(label)
|
||||
if label.startswith("smarm"):
|
||||
if n is not None:
|
||||
smarm[n] = (label, data["median"])
|
||||
elif label.startswith("tokio"):
|
||||
if n is None:
|
||||
tokio_default = (label, data["median"])
|
||||
else:
|
||||
tokio[n] = (label, data["median"])
|
||||
|
||||
def row(s: tuple[str, int], t: tuple[str, int]):
|
||||
s_label, s_med = s
|
||||
t_label, t_med = t
|
||||
if s_med == 0:
|
||||
return None
|
||||
ratio = t_med / s_med
|
||||
return (bench, s_label, s_med, t_label, t_med, ratio,
|
||||
"smarm" if ratio >= 1.0 else "tokio")
|
||||
|
||||
matched: set[int] = set()
|
||||
for n in sorted(set(smarm) & set(tokio)):
|
||||
r = row(smarm[n], tokio[n])
|
||||
if r:
|
||||
rows.append(r)
|
||||
matched.add(n)
|
||||
# tokio's default multi pool vs the widest smarm config not already paired.
|
||||
if tokio_default is not None:
|
||||
rem = [n for n in smarm if n not in matched and n > 1]
|
||||
if rem:
|
||||
r = row(smarm[max(rem)], tokio_default)
|
||||
if r:
|
||||
rows.append(r)
|
||||
return rows
|
||||
|
||||
|
||||
def print_vs_tokio(results: dict) -> None:
|
||||
"""Human summary + greppable VSTOKIO lines (best smarm vs best tokio)."""
|
||||
rows = vs_tokio(results)
|
||||
if not rows:
|
||||
return
|
||||
print("\n vs tokio (like-for-like by thread count; ratio>1 = smarm faster, lower µs better)")
|
||||
print(f" {'-'*78}")
|
||||
for bench, s_label, s_med, t_label, t_med, ratio, winner in rows:
|
||||
print(
|
||||
f" {bench:<22} {s_label} {s_med}µs vs {t_label} {t_med}µs"
|
||||
f" → {ratio:.2f}x ({winner})"
|
||||
)
|
||||
# Machine-readable, one line per bench:
|
||||
# VSTOKIO,<bench>,<smarm_label>,<smarm_us>,<tokio_label>,<tokio_us>,<ratio>,<winner>
|
||||
for bench, s_label, s_med, t_label, t_med, ratio, winner in rows:
|
||||
print(f"VSTOKIO,{bench},{s_label},{s_med},{t_label},{t_med},{ratio:.3f},{winner}")
|
||||
|
||||
|
||||
def print_results(results: dict, label: str = "") -> None:
|
||||
if label:
|
||||
print(f"\n{'='*70}")
|
||||
@@ -341,7 +210,6 @@ def print_results(results: dict, label: str = "") -> None:
|
||||
f" {rt_label:>28} | {data['result']:>10} | "
|
||||
f"{data['median']:>10} | {data['min']:>8} | {data['max']:>8}"
|
||||
)
|
||||
print_vs_tokio(results)
|
||||
|
||||
|
||||
def print_sweep_table(sweep_results: list[tuple[int, int, dict]]) -> None:
|
||||
@@ -384,7 +252,7 @@ def cmd_run(args) -> None:
|
||||
["cargo", "build", "--release", "--benches"],
|
||||
cwd=REPO, check=True, capture_output=True,
|
||||
)
|
||||
print(f"Running benches ({BENCH_SETS} independent sets)…")
|
||||
print("Running benches…")
|
||||
results = run_benches()
|
||||
print_results(results, "Results (default knobs)")
|
||||
if args.save_baseline:
|
||||
@@ -398,7 +266,7 @@ def cmd_regress(args) -> None:
|
||||
["cargo", "build", "--release", "--benches"],
|
||||
cwd=REPO, check=True, capture_output=True,
|
||||
)
|
||||
print(f"Running benches ({BENCH_SETS} independent sets)…")
|
||||
print("Running benches…")
|
||||
current = run_benches()
|
||||
print_results(current, "Current results")
|
||||
print(f"\nRegression check (threshold: >{REGRESSION_THRESHOLD_PCT}% slower than baseline)")
|
||||
@@ -420,7 +288,7 @@ def cmd_sweep(args) -> None:
|
||||
|
||||
for interval, cycles in SWEEP_GRID:
|
||||
tag = f"alloc_interval={interval}, timeslice_cycles={cycles}"
|
||||
print(f" Running: {tag} ({BENCH_SETS} sets)…", flush=True)
|
||||
print(f" Running: {tag} …", flush=True)
|
||||
env_extra = {
|
||||
"SMARM_ALLOC_INTERVAL": str(interval),
|
||||
"SMARM_TIMESLICE_CYCLES": str(cycles),
|
||||
|
||||
@@ -39,13 +39,6 @@ fn available_threads() -> usize {
|
||||
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
|
||||
}
|
||||
|
||||
fn env_sets() -> u32 {
|
||||
std::env::var("SMARM_BENCH_SETS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(5)
|
||||
}
|
||||
|
||||
fn print_header(title: &str) {
|
||||
println!("\n{}", "=".repeat(80));
|
||||
println!(" {title}");
|
||||
@@ -58,17 +51,13 @@ fn print_header(title: &str) {
|
||||
}
|
||||
|
||||
fn run_n<F: FnMut() -> (u64, u128)>(name: &str, n: u32, mut f: F) {
|
||||
let sets = env_sets();
|
||||
let mut times = Vec::with_capacity((n * sets) as usize);
|
||||
let mut times = Vec::new();
|
||||
let mut last = 0u64;
|
||||
// One warmup before all sets, discarded.
|
||||
let _ = f();
|
||||
for _ in 0..sets {
|
||||
for _ in 0..n {
|
||||
let (v, t) = f();
|
||||
times.push(t);
|
||||
last = v;
|
||||
}
|
||||
let _ = f(); // warmup
|
||||
for _ in 0..n {
|
||||
let (v, t) = f();
|
||||
times.push(t);
|
||||
last = v;
|
||||
}
|
||||
times.sort_unstable();
|
||||
let median = times[times.len() / 2];
|
||||
@@ -445,8 +434,7 @@ fn main() {
|
||||
let n = available_threads();
|
||||
println!("smarm tokio-favored benchmarks");
|
||||
println!("available parallelism: {n} threads");
|
||||
let sets = env_sets();
|
||||
println!("ITERS={ITERS}×{sets} sets = {} samples (+1 warmup, discarded)", ITERS * sets);
|
||||
println!("ITERS={ITERS} (+1 warmup, discarded)");
|
||||
println!(
|
||||
"STORM_BACKGROUND={STORM_BACKGROUND}, STORM_SPAWN={STORM_SPAWN}, \
|
||||
MPSC={MPSC_PRODUCERS}×{MPSC_PER_PRODUCER}, \
|
||||
|
||||
+439
-877
File diff suppressed because it is too large
Load Diff
+2
-2
@@ -122,9 +122,9 @@ pub struct Actor {
|
||||
/// The PID this actor was assigned at spawn time.
|
||||
pub pid: Pid,
|
||||
/// The stack the actor runs on. Dropped (munmap'd) when the actor dies.
|
||||
/// (The saved stack pointer lives on the `Slot` as an atomic, not here:
|
||||
/// it is hot scheduling state, read/written without the cold lock.)
|
||||
pub stack: Stack,
|
||||
/// The saved stack pointer. Updated on every yield.
|
||||
pub sp: usize,
|
||||
/// The PID of this actor's supervisor. Used to deliver `Signal` on death.
|
||||
pub supervisor: Pid,
|
||||
/// Cooperative-cancellation flag. `request_stop` sets it (and unparks a
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
//! Cooperative context switching and cycle counter, aarch64 (AAPCS64).
|
||||
//!
|
||||
//! The aarch64 mirror of the x86-64 backend. The protocol is identical — save
|
||||
//! callee-saved state, swap stack pointers via the shared `*_SP` thread-locals,
|
||||
//! restore, return — but the mechanics differ from x86 in two ways worth
|
||||
//! stating up front, because they drive the stack layout:
|
||||
//!
|
||||
//! 1. **Return is via the link register, not the stack.** x86 `ret` pops the
|
||||
//! return address off the stack; aarch64 `ret` jumps to whatever is in
|
||||
//! `x30` (lr). So the entry point is parked in the saved-`x30` slot, and
|
||||
//! the shim restores `x30` and `ret`s to it. There is no "ret target word"
|
||||
//! sitting on the stack the way there is on x86.
|
||||
//!
|
||||
//! 2. **sp must be 16-byte aligned at all times**, not just at call entry.
|
||||
//! We save 12 registers (x19–x28, x29, x30) = 96 bytes, already a multiple
|
||||
//! of 16, so the frame is aligned by construction.
|
||||
//!
|
||||
//! FP/SIMD registers (v8–v15, whose low 64 bits are callee-saved under AAPCS64)
|
||||
//! are NOT saved, for the same reason the x86 backend skips XMM: every yield
|
||||
//! goes through a Rust call boundary, so the compiler has already spilled any
|
||||
//! live vector state. If we ever yield from a non-call-boundary, this breaks.
|
||||
|
||||
use super::{get_actor_sp, get_scheduler_sp, set_actor_sp, set_scheduler_sp};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Initial stack layout
|
||||
//
|
||||
// Twelve 8-byte slots, all zero except the x30 slot which holds `entry`. The
|
||||
// first `switch_to_actor` loads x19–x28/x29/x30 back and `ret`s to x30,
|
||||
// landing at the top of `entry` with sp 16-aligned (the AAPCS64 requirement).
|
||||
//
|
||||
// Layout (high → low), relative to aligned_top = top & ~15:
|
||||
//
|
||||
// aligned_top - 8 : x30 = entry ← `ret` jumps here.
|
||||
// aligned_top - 16 : x29 = 0 (fp)
|
||||
// aligned_top - 24 : x28 = 0
|
||||
// aligned_top - 32 : x27 = 0
|
||||
// aligned_top - 40 : x26 = 0
|
||||
// aligned_top - 48 : x25 = 0
|
||||
// aligned_top - 56 : x24 = 0
|
||||
// aligned_top - 64 : x23 = 0
|
||||
// aligned_top - 72 : x22 = 0
|
||||
// aligned_top - 80 : x21 = 0
|
||||
// aligned_top - 88 : x20 = 0
|
||||
// aligned_top - 96 : x19 = 0 ← initial sp (16-aligned)
|
||||
//
|
||||
// The restore order in the shim (ldp pairs, low→high) must match this exactly.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
|
||||
unsafe {
|
||||
let aligned_top = top as usize & !15;
|
||||
let mut sp = aligned_top;
|
||||
sp -= 8; (sp as *mut usize).write(entry as usize); // x30 (lr) → entry
|
||||
sp -= 8; (sp as *mut usize).write(0); // x29 (fp)
|
||||
sp -= 8; (sp as *mut usize).write(0); // x28
|
||||
sp -= 8; (sp as *mut usize).write(0); // x27
|
||||
sp -= 8; (sp as *mut usize).write(0); // x26
|
||||
sp -= 8; (sp as *mut usize).write(0); // x25
|
||||
sp -= 8; (sp as *mut usize).write(0); // x24
|
||||
sp -= 8; (sp as *mut usize).write(0); // x23
|
||||
sp -= 8; (sp as *mut usize).write(0); // x22
|
||||
sp -= 8; (sp as *mut usize).write(0); // x21
|
||||
sp -= 8; (sp as *mut usize).write(0); // x20
|
||||
sp -= 8; (sp as *mut usize).write(0); // x19 ← initial sp
|
||||
sp
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Context switch shims
|
||||
//
|
||||
// Each shim:
|
||||
// 1. Pushes x19–x28, x29, x30 as six 16-byte stp pairs (sp pre-decrement).
|
||||
// 2. Moves sp into x0 and calls the Rust helper that stores it.
|
||||
// 3. Calls the Rust helper that returns the *other* side's saved sp.
|
||||
// 4. Moves that into sp.
|
||||
// 5. Restores the twelve registers and rets (to the restored x30).
|
||||
//
|
||||
// The helper calls clobber x30 themselves, but that's fine: the outgoing x30
|
||||
// was already saved to the stack in step 1, and step 5 restores the incoming
|
||||
// side's x30 before `ret`. The push/pop register order is paired so that the
|
||||
// `ldp` sequence reverses the `stp` sequence exactly.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[unsafe(naked)]
|
||||
unsafe extern "C" fn switch_to_actor_asm() {
|
||||
core::arch::naked_asm!(
|
||||
"stp x19, x20, [sp, #-96]!",
|
||||
"stp x21, x22, [sp, #16]",
|
||||
"stp x23, x24, [sp, #32]",
|
||||
"stp x25, x26, [sp, #48]",
|
||||
"stp x27, x28, [sp, #64]",
|
||||
"stp x29, x30, [sp, #80]",
|
||||
"mov x0, sp",
|
||||
"bl {set_sched_sp}",
|
||||
"bl {get_actor_sp}",
|
||||
"mov sp, x0",
|
||||
"ldp x21, x22, [sp, #16]",
|
||||
"ldp x23, x24, [sp, #32]",
|
||||
"ldp x25, x26, [sp, #48]",
|
||||
"ldp x27, x28, [sp, #64]",
|
||||
"ldp x29, x30, [sp, #80]",
|
||||
"ldp x19, x20, [sp], #96",
|
||||
"ret",
|
||||
set_sched_sp = sym set_scheduler_sp,
|
||||
get_actor_sp = sym get_actor_sp,
|
||||
);
|
||||
}
|
||||
|
||||
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
|
||||
pub unsafe fn switch_to_actor() {
|
||||
unsafe { switch_to_actor_asm() };
|
||||
}
|
||||
|
||||
#[unsafe(naked)]
|
||||
pub unsafe extern "C" fn switch_to_scheduler() {
|
||||
core::arch::naked_asm!(
|
||||
"stp x19, x20, [sp, #-96]!",
|
||||
"stp x21, x22, [sp, #16]",
|
||||
"stp x23, x24, [sp, #32]",
|
||||
"stp x25, x26, [sp, #48]",
|
||||
"stp x27, x28, [sp, #64]",
|
||||
"stp x29, x30, [sp, #80]",
|
||||
"mov x0, sp",
|
||||
"bl {set_actor_sp}",
|
||||
"bl {get_sched_sp}",
|
||||
"mov sp, x0",
|
||||
"ldp x21, x22, [sp, #16]",
|
||||
"ldp x23, x24, [sp, #32]",
|
||||
"ldp x25, x26, [sp, #48]",
|
||||
"ldp x27, x28, [sp, #64]",
|
||||
"ldp x29, x30, [sp, #80]",
|
||||
"ldp x19, x20, [sp], #96",
|
||||
"ret",
|
||||
set_actor_sp = sym set_actor_sp,
|
||||
get_sched_sp = sym get_scheduler_sp,
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cycle counter
|
||||
//
|
||||
// CNTVCT_EL0 is the virtual count register, readable from EL0 on Linux. `isb`
|
||||
// serialises so we don't sample the counter before prior instructions retire
|
||||
// (the aarch64 analogue of x86's `lfence` before rdtsc).
|
||||
//
|
||||
// Unit: CNTVCT ticks, whose frequency is CNTFRQ_EL0 — typically 1–50 MHz,
|
||||
// NOT the CPU clock. This is a different and much coarser unit than the x86
|
||||
// TSC, which is why `TIMESLICE_CYCLES` must be tuned separately for aarch64.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[inline(always)]
|
||||
pub fn read_cycle_counter() -> u64 {
|
||||
let cnt: u64;
|
||||
unsafe {
|
||||
core::arch::asm!(
|
||||
"isb",
|
||||
"mrs {cnt}, cntvct_el0",
|
||||
cnt = out(reg) cnt,
|
||||
options(nostack, nomem),
|
||||
);
|
||||
}
|
||||
cnt
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
//! Architecture abstraction layer.
|
||||
//!
|
||||
//! Everything that depends on the target ISA lives behind this module: the
|
||||
//! cooperative context-switch shims, the initial-stack layout, and the
|
||||
//! cycle counter used by the preemption timeslice. The rest of the runtime
|
||||
//! talks only to the API re-exported here and never names a register.
|
||||
//!
|
||||
//! Each backend (`x86_64`, `aarch64`) exposes the same four items:
|
||||
//!
|
||||
//! - `init_actor_stack(top, entry) -> usize`
|
||||
//! Build a fresh actor stack so the first `switch_to_actor` lands
|
||||
//! inside `entry` with the ABI-correct stack alignment.
|
||||
//! - `switch_to_actor()`
|
||||
//! Resume the actor whose sp is in `ACTOR_SP`; returns when it yields.
|
||||
//! - `switch_to_scheduler()`
|
||||
//! Yield from the current actor back to its scheduler thread.
|
||||
//! - `read_cycle_counter() -> u64`
|
||||
//! Monotonic per-core cycle counter for the timeslice clock. The unit
|
||||
//! is ISA-defined (x86 TSC ticks vs. aarch64 virtual-counter ticks),
|
||||
//! so `TIMESLICE_CYCLES` must be tuned per-arch — see `preempt`.
|
||||
//!
|
||||
//! The `SCHEDULER_SP` / `ACTOR_SP` thread-locals are shared across backends
|
||||
//! and live here, since the saved-stack-pointer protocol is identical
|
||||
//! regardless of which registers the shim happens to push.
|
||||
|
||||
use std::cell::Cell;
|
||||
|
||||
thread_local! {
|
||||
static SCHEDULER_SP: Cell<usize> = const { Cell::new(0) };
|
||||
static ACTOR_SP: Cell<usize> = const { Cell::new(0) };
|
||||
}
|
||||
|
||||
// Used by the naked shims (via `sym`) and by the scheduler/tests through the
|
||||
// re-exports below. `pub` because the asm references them as symbols.
|
||||
pub(crate) fn get_scheduler_sp() -> usize { SCHEDULER_SP.with(|c| c.get()) }
|
||||
pub(crate) fn set_scheduler_sp(v: usize) { SCHEDULER_SP.with(|c| c.set(v)) }
|
||||
pub fn get_actor_sp() -> usize { ACTOR_SP.with(|c| c.get()) }
|
||||
pub fn set_actor_sp(v: usize) { ACTOR_SP.with(|c| c.set(v)) }
|
||||
|
||||
#[cfg(target_arch = "x86_64")]
|
||||
mod x86_64;
|
||||
#[cfg(target_arch = "x86_64")]
|
||||
pub use x86_64::{init_actor_stack, read_cycle_counter, switch_to_actor, switch_to_scheduler};
|
||||
|
||||
#[cfg(target_arch = "aarch64")]
|
||||
mod aarch64;
|
||||
#[cfg(target_arch = "aarch64")]
|
||||
pub use aarch64::{init_actor_stack, read_cycle_counter, switch_to_actor, switch_to_scheduler};
|
||||
|
||||
#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
|
||||
compile_error!(
|
||||
"smarm's context-switch and cycle-counter layer supports only x86_64 and \
|
||||
aarch64. Add a backend in src/arch/ for this target."
|
||||
);
|
||||
@@ -0,0 +1,118 @@
|
||||
//! Cooperative context switching and cycle counter, x86-64 (SysV AMD64).
|
||||
//!
|
||||
//! Two naked-asm functions move execution between a scheduler thread and an
|
||||
//! actor running on its own mmap'd stack. The compiler cannot do this; the
|
||||
//! whole point of `#[unsafe(naked)]` is that we control every instruction.
|
||||
//!
|
||||
//! `init_actor_stack` builds the initial stack so that the first
|
||||
//! `switch_to_actor` lands inside the entry function with `rsp % 16 == 8`
|
||||
//! (the x86-64 ABI requirement at function entry).
|
||||
|
||||
use super::{get_actor_sp, get_scheduler_sp, set_actor_sp, set_scheduler_sp};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Initial stack layout
|
||||
//
|
||||
// We start from aligned_top = top & ~15, then bias by 8 and push seven 8-byte
|
||||
// slots (downward). The first `switch_to_actor` pops r15..rbx and `ret`s —
|
||||
// landing in `entry` with rsp % 16 == 8 (the x86-64 ABI state at the point
|
||||
// just after a `call`, which is what `entry` is compiled to expect).
|
||||
//
|
||||
// Layout (high → low), relative to aligned_top = top & ~15. Note the entry
|
||||
// slot is at aligned_top - 16, NOT aligned_top - 8: the function does
|
||||
// `(top & ~15) - 8` and *then* a `-= 8` before the first write, so the first
|
||||
// stored word lands at aligned_top - 16. Verified by single-stepping the
|
||||
// `ret` under llmdbg: entry sits at an address with %16 == 0, so the post-ret
|
||||
// rsp is %16 == 8.
|
||||
//
|
||||
// aligned_top - 8 : (unused padding; keeps entry's slot %16 == 0)
|
||||
// aligned_top - 16 : entry ptr ← `ret` target. Post-ret: rsp % 16 == 8.
|
||||
// aligned_top - 24 : rbx = 0
|
||||
// aligned_top - 32 : rbp = 0
|
||||
// aligned_top - 40 : r12 = 0
|
||||
// aligned_top - 48 : r13 = 0
|
||||
// aligned_top - 56 : r14 = 0
|
||||
// aligned_top - 64 : r15 = 0 ← initial rsp
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
|
||||
unsafe {
|
||||
let mut sp = (top as usize & !15) - 8;
|
||||
sp -= 8; (sp as *mut usize).write(entry as usize); // ret target
|
||||
sp -= 8; (sp as *mut usize).write(0); // rbx
|
||||
sp -= 8; (sp as *mut usize).write(0); // rbp
|
||||
sp -= 8; (sp as *mut usize).write(0); // r12
|
||||
sp -= 8; (sp as *mut usize).write(0); // r13
|
||||
sp -= 8; (sp as *mut usize).write(0); // r14
|
||||
sp -= 8; (sp as *mut usize).write(0); // r15
|
||||
sp
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Context switch shims
|
||||
//
|
||||
// Each shim:
|
||||
// 1. Pushes the six callee-saved integer registers.
|
||||
// 2. Snaps rsp into rdi and calls the Rust helper that stores it.
|
||||
// 3. Calls the Rust helper that returns the *other* side's saved rsp.
|
||||
// 4. Moves that into rsp.
|
||||
// 5. Pops the six registers and rets.
|
||||
//
|
||||
// XMM registers are NOT saved here. We rely on every yield happening through
|
||||
// a Rust call site, which means the compiler has spilled any live XMM state
|
||||
// to the stack before we get here. (This is the same argument the compiler
|
||||
// uses internally — callee-saved regs are what survive a `call`, and the
|
||||
// SysV AMD64 ABI says XMM0–15 are all caller-saved.) If we ever yield from
|
||||
// a place that isn't a Rust call boundary, this assumption breaks.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[unsafe(naked)]
|
||||
unsafe extern "C" fn switch_to_actor_asm() {
|
||||
core::arch::naked_asm!(
|
||||
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
|
||||
"mov rdi, rsp",
|
||||
"call {set_sched_sp}",
|
||||
"call {get_actor_sp}",
|
||||
"mov rsp, rax",
|
||||
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
|
||||
"ret",
|
||||
set_sched_sp = sym set_scheduler_sp,
|
||||
get_actor_sp = sym get_actor_sp,
|
||||
);
|
||||
}
|
||||
|
||||
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
|
||||
pub unsafe fn switch_to_actor() {
|
||||
unsafe { switch_to_actor_asm() };
|
||||
}
|
||||
|
||||
#[unsafe(naked)]
|
||||
pub unsafe extern "C" fn switch_to_scheduler() {
|
||||
core::arch::naked_asm!(
|
||||
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
|
||||
"mov rdi, rsp",
|
||||
"call {set_actor_sp}",
|
||||
"call {get_sched_sp}",
|
||||
"mov rsp, rax",
|
||||
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
|
||||
"ret",
|
||||
set_actor_sp = sym set_actor_sp,
|
||||
get_sched_sp = sym get_scheduler_sp,
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cycle counter
|
||||
//
|
||||
// `lfence` serialises the instruction stream so we don't measure time before
|
||||
// prior instructions retire. Unit: TSC ticks (≈ CPU base clock).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[inline(always)]
|
||||
pub fn read_cycle_counter() -> u64 {
|
||||
unsafe {
|
||||
core::arch::asm!("lfence", options(nostack, nomem, preserves_flags));
|
||||
core::arch::x86_64::_rdtsc()
|
||||
}
|
||||
}
|
||||
+22
-466
@@ -1,23 +1,9 @@
|
||||
//! Unbounded MPSC channels.
|
||||
//!
|
||||
//! Inner state is `Arc<RawMutex<Inner<T>>>` so channels can be sent across OS
|
||||
//! Inner state is `Arc<Mutex<Inner<T>>>` so channels can be sent across OS
|
||||
//! threads (required for the multi-scheduler runtime where a sender and
|
||||
//! receiver may run on different scheduler threads simultaneously).
|
||||
//!
|
||||
//! ## Why `RawMutex` (Channel class), not `std::sync::Mutex`
|
||||
//!
|
||||
//! An actor holding a guard with preemption *enabled* can be timesliced
|
||||
//! inside the critical section and resume on a different OS thread — the
|
||||
//! pthread mutex would then be released from a thread that didn't lock it,
|
||||
//! which is UB (Linux futexes happen to tolerate it, but it's not
|
||||
//! guaranteed). `RawMutex` disables preemption for the guard's span and is
|
||||
//! cross-thread-release sound by construction, closing the hole. It also
|
||||
//! cannot poison. Channel locks form their own [`LockClass::Channel`]
|
||||
//! (raw_mutex.rs): they may be taken under a cold (Leaf) lock — finalize and
|
||||
//! `monitor()` clone senders that live in slots — but nothing may be locked
|
||||
//! under them, which the debug build enforces. `recv_match` runs its user
|
||||
//! predicate under this lock: keep it cheap, pure, and channel-free.
|
||||
//!
|
||||
//! Semantics:
|
||||
//! - Senders are clonable; the last sender drop closes the channel.
|
||||
//! - `Receiver::recv` on an empty open channel parks the receiver.
|
||||
@@ -29,12 +15,11 @@
|
||||
//! parked, the receiver is unparked.
|
||||
|
||||
use crate::pid::Pid;
|
||||
use crate::raw_mutex::RawMutex;
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::Arc;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
|
||||
let inner = Arc::new(RawMutex::new_channel(Inner {
|
||||
let inner = Arc::new(Mutex::new(Inner {
|
||||
queue: VecDeque::new(),
|
||||
parked_receiver: None,
|
||||
senders: 1,
|
||||
@@ -45,24 +30,17 @@ pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
|
||||
|
||||
struct Inner<T> {
|
||||
queue: VecDeque<T>,
|
||||
/// The parked receiver's `(pid, park-epoch)`. The epoch is the slot
|
||||
/// word's runtime-wide wait identity (see slot_state.rs): wakers call
|
||||
/// `unpark_at(pid, epoch)`, so an entry left over from an already-woken
|
||||
/// wait — a `select` loser arm, a satisfied `recv_timeout`'s timer — is
|
||||
/// inert: the wake fails the word's epoch CAS and no-ops. This replaces
|
||||
/// the old per-channel `cur_wait`/`next_wait_seq`/`timed_out` trio: wait
|
||||
/// identity now exists exactly once, in the slot word.
|
||||
parked_receiver: Option<(Pid, u32)>,
|
||||
parked_receiver: Option<Pid>,
|
||||
senders: usize,
|
||||
receiver_alive: bool,
|
||||
}
|
||||
|
||||
pub struct Sender<T> {
|
||||
inner: Arc<RawMutex<Inner<T>>>,
|
||||
inner: Arc<Mutex<Inner<T>>>,
|
||||
}
|
||||
|
||||
pub struct Receiver<T> {
|
||||
inner: Arc<RawMutex<Inner<T>>>,
|
||||
inner: Arc<Mutex<Inner<T>>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
@@ -79,30 +57,9 @@ impl std::fmt::Display for RecvError {
|
||||
|
||||
impl std::error::Error for RecvError {}
|
||||
|
||||
/// Returned by [`Receiver::recv_timeout`].
|
||||
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
|
||||
pub enum RecvTimeoutError {
|
||||
/// The deadline passed with no message available.
|
||||
Timeout,
|
||||
/// All senders dropped with no message available — the bounded analogue
|
||||
/// of [`RecvError`].
|
||||
Disconnected,
|
||||
}
|
||||
|
||||
impl std::fmt::Display for RecvTimeoutError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
RecvTimeoutError::Timeout => write!(f, "recv timed out"),
|
||||
RecvTimeoutError::Disconnected => write!(f, "channel closed"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for RecvTimeoutError {}
|
||||
|
||||
impl<T> Clone for Sender<T> {
|
||||
fn clone(&self) -> Self {
|
||||
self.inner.lock().senders += 1;
|
||||
self.inner.lock().unwrap().senders += 1;
|
||||
Sender { inner: self.inner.clone() }
|
||||
}
|
||||
}
|
||||
@@ -110,7 +67,7 @@ impl<T> Clone for Sender<T> {
|
||||
impl<T> Drop for Sender<T> {
|
||||
fn drop(&mut self) {
|
||||
let unpark = {
|
||||
let mut g = self.inner.lock();
|
||||
let mut g = self.inner.lock().unwrap();
|
||||
g.senders -= 1;
|
||||
// Wake the parked receiver on the last sender drop regardless of
|
||||
// whether the queue is empty. A plain `recv` only ever parks on an
|
||||
@@ -124,31 +81,31 @@ impl<T> Drop for Sender<T> {
|
||||
None
|
||||
}
|
||||
};
|
||||
if let Some((pid, epoch)) = unpark {
|
||||
crate::scheduler::unpark_at(pid, epoch);
|
||||
if let Some(pid) = unpark {
|
||||
crate::scheduler::unpark(pid);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> Drop for Receiver<T> {
|
||||
fn drop(&mut self) {
|
||||
self.inner.lock().receiver_alive = false;
|
||||
self.inner.lock().unwrap().receiver_alive = false;
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> Sender<T> {
|
||||
pub fn send(&self, value: T) -> Result<(), SendError<T>> {
|
||||
let unpark = {
|
||||
let mut g = self.inner.lock();
|
||||
let mut g = self.inner.lock().unwrap();
|
||||
if !g.receiver_alive {
|
||||
return Err(SendError(value));
|
||||
}
|
||||
g.queue.push_back(value);
|
||||
g.parked_receiver.take()
|
||||
};
|
||||
if let Some((pid, epoch)) = unpark {
|
||||
if let Some(pid) = unpark {
|
||||
crate::te!(crate::trace::Event::Send { sender: crate::actor::current_pid().unwrap_or(crate::pid::Pid::new(u32::MAX, u32::MAX)), receiver: Some(pid) });
|
||||
crate::scheduler::unpark_at(pid, epoch);
|
||||
crate::scheduler::unpark(pid);
|
||||
} else {
|
||||
crate::te!(crate::trace::Event::Send { sender: crate::actor::current_pid().unwrap_or(crate::pid::Pid::new(u32::MAX, u32::MAX)), receiver: None });
|
||||
}
|
||||
@@ -160,7 +117,7 @@ impl<T> Receiver<T> {
|
||||
pub fn recv(&self) -> Result<T, RecvError> {
|
||||
loop {
|
||||
{
|
||||
let mut g = self.inner.lock();
|
||||
let mut g = self.inner.lock().unwrap();
|
||||
if let Some(v) = g.queue.pop_front() {
|
||||
return Ok(v);
|
||||
}
|
||||
@@ -170,13 +127,10 @@ impl<T> Receiver<T> {
|
||||
let me = crate::actor::current_pid()
|
||||
.expect("recv() called outside an actor");
|
||||
debug_assert!(
|
||||
g.parked_receiver.is_none_or(|(p, _)| p == me),
|
||||
g.parked_receiver.is_none(),
|
||||
"channel has more than one receiver"
|
||||
);
|
||||
// begin_wait is lock-free — legal under the Channel lock;
|
||||
// registering in the same critical section makes the epoch
|
||||
// atomic with the senders' view of the registration.
|
||||
g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
|
||||
g.parked_receiver = Some(me);
|
||||
crate::te!(crate::trace::Event::RecvPark(me));
|
||||
}
|
||||
// Release the lock before parking — the unparker will need it.
|
||||
@@ -186,75 +140,6 @@ impl<T> Receiver<T> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Bounded receive: like [`recv`](Self::recv), but gives up once
|
||||
/// `timeout` has elapsed, returning [`RecvTimeoutError::Timeout`].
|
||||
///
|
||||
/// Built on the same timer machinery as `Mutex::lock_timeout`: the wait
|
||||
/// registers a `WaitTimeout` entry stamped with the wait's park-epoch;
|
||||
/// on expiry the channel (as the
|
||||
/// [`TimerTarget`](crate::timer::TimerTarget)) checks whether *this*
|
||||
/// wait is still parked and, only then, cancels it. A wake that races
|
||||
/// the deadline resolves message-first: if a message is available when
|
||||
/// the receiver runs, it is delivered even if the timer had already
|
||||
/// fired. A satisfied or abandoned wait leaves its timer entry to expire
|
||||
/// as a no-op (registration gone; epoch consumed), per the
|
||||
/// no-cancellation convention in `timer.rs`.
|
||||
///
|
||||
/// The wake is classified from state alone — wakes are precise (the only
|
||||
/// stamped wakers of this wait are a send, the last-sender drop, and the
|
||||
/// timer; a stop wake unwinds out of `park_current` and never reaches
|
||||
/// the classification), so: message queued → `Ok`; `senders == 0` →
|
||||
/// `Disconnected`; neither → it was the timer → `Timeout`.
|
||||
///
|
||||
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-
|
||||
/// due timer is drained, then reports `Timeout` unless a message was
|
||||
/// already queued.
|
||||
pub fn recv_timeout(&self, timeout: std::time::Duration) -> Result<T, RecvTimeoutError>
|
||||
where
|
||||
T: Send + 'static,
|
||||
{
|
||||
let me = crate::actor::current_pid()
|
||||
.expect("recv_timeout() called outside an actor");
|
||||
|
||||
// Fast path + wait registration, one critical section.
|
||||
let epoch;
|
||||
{
|
||||
let mut g = self.inner.lock();
|
||||
if let Some(v) = g.queue.pop_front() {
|
||||
return Ok(v);
|
||||
}
|
||||
if g.senders == 0 {
|
||||
return Err(RecvTimeoutError::Disconnected);
|
||||
}
|
||||
debug_assert!(
|
||||
g.parked_receiver.is_none_or(|(p, _)| p == me),
|
||||
"channel has more than one receiver"
|
||||
);
|
||||
epoch = crate::scheduler::begin_wait();
|
||||
g.parked_receiver = Some((me, epoch));
|
||||
crate::te!(crate::trace::Event::RecvPark(me));
|
||||
}
|
||||
|
||||
// Arm the timer after releasing the channel lock (insert takes the
|
||||
// timers lock; never nest under a Channel lock). A send or even the
|
||||
// timer itself may unpark us before we park — the RunningNotified
|
||||
// protocol makes the park below return immediately in that case.
|
||||
let deadline = crate::timer::deadline_from_now(timeout);
|
||||
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = self.inner.clone();
|
||||
crate::scheduler::insert_wait_timer(deadline, me, target, epoch);
|
||||
|
||||
crate::scheduler::park_current();
|
||||
crate::te!(crate::trace::Event::RecvWake(crate::actor::current_pid().unwrap()));
|
||||
let mut g = self.inner.lock();
|
||||
if let Some(v) = g.queue.pop_front() {
|
||||
return Ok(v);
|
||||
}
|
||||
if g.senders == 0 {
|
||||
return Err(RecvTimeoutError::Disconnected);
|
||||
}
|
||||
Err(RecvTimeoutError::Timeout)
|
||||
}
|
||||
|
||||
/// Selective receive: remove and return the first queued message for which
|
||||
/// `pred` holds, leaving the rest in arrival order. If no queued message
|
||||
/// matches, parks and re-scans on every send (a selective receiver may park
|
||||
@@ -271,7 +156,7 @@ impl<T> Receiver<T> {
|
||||
{
|
||||
loop {
|
||||
{
|
||||
let mut g = self.inner.lock();
|
||||
let mut g = self.inner.lock().unwrap();
|
||||
if let Some(i) = g.queue.iter().position(|v| pred(v)) {
|
||||
// position() found it, so remove() returns Some.
|
||||
return Ok(g.queue.remove(i).unwrap());
|
||||
@@ -283,10 +168,10 @@ impl<T> Receiver<T> {
|
||||
let me = crate::actor::current_pid()
|
||||
.expect("recv_match() called outside an actor");
|
||||
debug_assert!(
|
||||
g.parked_receiver.is_none_or(|(p, _)| p == me),
|
||||
g.parked_receiver.is_none(),
|
||||
"channel has more than one receiver"
|
||||
);
|
||||
g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
|
||||
g.parked_receiver = Some(me);
|
||||
crate::te!(crate::trace::Event::RecvPark(me));
|
||||
}
|
||||
// Release the lock before parking — the unparker will need it.
|
||||
@@ -303,7 +188,7 @@ impl<T> Receiver<T> {
|
||||
where
|
||||
F: Fn(&T) -> bool,
|
||||
{
|
||||
let mut g = self.inner.lock();
|
||||
let mut g = self.inner.lock().unwrap();
|
||||
if let Some(i) = g.queue.iter().position(|v| pred(v)) {
|
||||
return Ok(Some(g.queue.remove(i).unwrap()));
|
||||
}
|
||||
@@ -316,7 +201,7 @@ impl<T> Receiver<T> {
|
||||
/// Non-blocking. `Ok(Some(v))` if a message was available, `Ok(None)` if
|
||||
/// the channel is empty but open, `Err(RecvError)` if closed and drained.
|
||||
pub fn try_recv(&self) -> Result<Option<T>, RecvError> {
|
||||
let mut g = self.inner.lock();
|
||||
let mut g = self.inner.lock().unwrap();
|
||||
if let Some(v) = g.queue.pop_front() {
|
||||
return Ok(Some(v));
|
||||
}
|
||||
@@ -326,332 +211,3 @@ impl<T> Receiver<T> {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// TimerTarget — the expiry half of recv_timeout
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
impl<T: Send + 'static> crate::timer::TimerTarget for RawMutex<Inner<T>> {
|
||||
fn on_timeout(&self, pid: Pid, epoch: u32) {
|
||||
// Cancel the wait only if THIS wait (epoch match) is still
|
||||
// registered. If a sender already took `parked_receiver`, the
|
||||
// receiver is waking with a message — message wins, the timer
|
||||
// no-ops. If a later wait by the same receiver is registered, the
|
||||
// epoch mismatches — stale entry, no-op. (The unpark_at would fail
|
||||
// its word CAS in either case anyway; checking under the lock keeps
|
||||
// the registration bookkeeping exact.)
|
||||
let unpark = {
|
||||
let mut g = self.lock();
|
||||
if g.parked_receiver == Some((pid, epoch)) {
|
||||
g.parked_receiver = None;
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
};
|
||||
// Unpark outside the channel lock — it may take the run-queue lock;
|
||||
// legal under a Channel lock, but pointless to nest.
|
||||
if unpark {
|
||||
crate::scheduler::unpark_at(pid, epoch);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// select — ready-index wait over multiple receivers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub(crate) mod sealed {
|
||||
pub trait Sealed {}
|
||||
}
|
||||
impl<T> sealed::Sealed for Receiver<T> {}
|
||||
|
||||
/// An arm of a [`select`]. Implemented by [`Receiver`]; sealed, because the
|
||||
/// registration contract below is part of the runtime's wake protocol.
|
||||
///
|
||||
/// Contract (all under the arm's own lock): `sel_register` checks-or-
|
||||
/// registers atomically — if the arm is ready it does NOT register and
|
||||
/// returns `Ok(false)`; otherwise it publishes `(pid, epoch)` where its
|
||||
/// wakers will find it and returns `Ok(true)`. "Ready" means a receive
|
||||
/// would not park: a message is queued, or the arm is closed. `Err` means
|
||||
/// the arm could not register at all (only fd arms can fail; channel
|
||||
/// registration is infallible) — the wait must be retired and earlier
|
||||
/// eager-cleanup arms unregistered.
|
||||
pub trait Selectable: sealed::Sealed {
|
||||
#[doc(hidden)]
|
||||
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool>;
|
||||
#[doc(hidden)]
|
||||
fn sel_ready(&self) -> bool;
|
||||
/// Remove this arm's `(pid, epoch)` registration if — and only if — it
|
||||
/// is still in place. Default no-op: a losing channel arm's stale
|
||||
/// registration is inert (its wakers die at the epoch CAS; the next
|
||||
/// wait overwrites the slot). Fd arms override this: their staleness
|
||||
/// poisons the fd (waiters entry + kernel-side ONESHOT registration)
|
||||
/// and needs an eager cleanup pass.
|
||||
#[doc(hidden)]
|
||||
fn sel_unregister(&self, _pid: Pid, _epoch: u32) {}
|
||||
/// Whether this arm requires the eager cleanup pass at all. Gates the
|
||||
/// post-wake `sel_unregister` sweep so channel-only selects keep
|
||||
/// today's zero-cancellation hot path.
|
||||
#[doc(hidden)]
|
||||
fn sel_eager_cleanup(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> Selectable for Receiver<T> {
|
||||
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool> {
|
||||
let mut g = self.inner.lock();
|
||||
if !g.queue.is_empty() || g.senders == 0 {
|
||||
return Ok(false);
|
||||
}
|
||||
debug_assert!(
|
||||
g.parked_receiver.is_none_or(|(p, _)| p == pid),
|
||||
"channel has more than one receiver"
|
||||
);
|
||||
g.parked_receiver = Some((pid, epoch));
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
fn sel_ready(&self) -> bool {
|
||||
let g = self.inner.lock();
|
||||
!g.queue.is_empty() || g.senders == 0
|
||||
}
|
||||
}
|
||||
|
||||
/// Park on every arm at once; return the index of the first ready one.
|
||||
///
|
||||
/// "Ready" means a receive on that arm would not park: a message is queued,
|
||||
/// or the arm is **closed** (so the caller's `try_recv` observes the
|
||||
/// disconnect — a dead arm is an event, not a hang). The caller consumes the
|
||||
/// arm itself, typically via [`Receiver::try_recv`]; single-receiver
|
||||
/// channels guarantee nothing can steal the message in between.
|
||||
///
|
||||
/// A closed arm stays ready *forever*: once its disconnect has been
|
||||
/// observed, drop it from the arm set — under priority order it would
|
||||
/// otherwise win every subsequent call and starve every higher-indexed arm.
|
||||
///
|
||||
/// Arms are scanned **in order**: index 0 is the highest priority, both on
|
||||
/// the immediate-ready path and after a wake. This is a documented
|
||||
/// guarantee (compose like BEAM receive clauses: put control channels
|
||||
/// first), not an accident — and therefore there is NO fairness promise; a
|
||||
/// saturated arm 0 starves arm 1 by design.
|
||||
///
|
||||
/// One actor may select on a channel and later `recv` on it (or select on
|
||||
/// overlapping sets) freely. What stays illegal is what was always illegal:
|
||||
/// two *different* actors receiving on one channel.
|
||||
///
|
||||
/// Built on the consuming-wake protocol (see slot_state.rs): all arms are
|
||||
/// registered under one wait epoch; the winning wake consumes it, so losing
|
||||
/// arms' registrations are inert and need no cancellation pass — they
|
||||
/// self-clean at their wakers' failed CAS, or get overwritten by this
|
||||
/// receiver's next wait on that channel.
|
||||
///
|
||||
/// Panics if `arms` is empty, when called outside an actor, or if an fd
|
||||
/// arm fails to register (EBADF, EMFILE, a second waiter on one fd —
|
||||
/// see [`try_select`] for the fallible form; channel-only selects cannot
|
||||
/// fail).
|
||||
pub fn select(arms: &[&dyn Selectable]) -> usize {
|
||||
try_select(arms).expect("select(): fd arm failed to register (use try_select)")
|
||||
}
|
||||
|
||||
/// [`select`], fallible: `Err` when an arm fails to register (only fd
|
||||
/// arms can — EBADF, EMFILE on the epoll set, or a second waiter on an
|
||||
/// fd that already has one). On `Err` the wait is fully retired and no
|
||||
/// registration is left behind: every arm registered before the failing
|
||||
/// one has been unregistered.
|
||||
pub fn try_select(arms: &[&dyn Selectable]) -> std::io::Result<usize> {
|
||||
assert!(!arms.is_empty(), "select() on an empty arm list");
|
||||
let me = crate::actor::current_pid().expect("select() called outside an actor");
|
||||
loop {
|
||||
let epoch = crate::scheduler::begin_wait();
|
||||
if let Some(i) = register_arms(me, epoch, arms)? {
|
||||
return Ok(i);
|
||||
}
|
||||
|
||||
// Stale fd registrations are not harmless (a losing fd arm's
|
||||
// waiters entry poisons the fd with AlreadyExists and its
|
||||
// kernel-side ONESHOT registration can fire arbitrarily late), so
|
||||
// selects containing fd arms run an eager cleanup pass after the
|
||||
// park — including when a terminal stop unwinds out of it, via
|
||||
// the guard. Channel-only selects skip all of it: `eager` is
|
||||
// false, the guard is disarmed, and the loser-arm self-cleaning
|
||||
// story is unchanged.
|
||||
let eager = arms.iter().any(|a| a.sel_eager_cleanup());
|
||||
let mut guard = UnregisterGuard { arms, me, epoch, armed: eager };
|
||||
|
||||
crate::scheduler::park_current();
|
||||
|
||||
if eager {
|
||||
unregister_arms(arms, me, epoch);
|
||||
}
|
||||
guard.armed = false;
|
||||
drop(guard);
|
||||
|
||||
// Woken precisely: an arm's send (message) or last-sender drop
|
||||
// (closure) consumed our epoch, and both leave their arm ready —
|
||||
// return the first one, in priority order (which may be a
|
||||
// different, higher-priority arm than the one that woke us; its
|
||||
// message stays queued and re-reports ready on the next call).
|
||||
// Fd arms classify by a fresh zero-timeout poll, so they too are
|
||||
// a pure function of state — independent of the registration the
|
||||
// cleanup pass just removed.
|
||||
for (i, arm) in arms.iter().enumerate() {
|
||||
if arm.sel_ready() {
|
||||
return Ok(i);
|
||||
}
|
||||
}
|
||||
// Unreachable by protocol (a stop wake unwinds out of
|
||||
// park_current). Defensive: re-open the wait and re-register —
|
||||
// stale own-registrations are overwritten (channels) or were
|
||||
// removed by the cleanup pass above (fds).
|
||||
}
|
||||
}
|
||||
|
||||
/// Eager-cleanup sweep: remove every fd arm's registration that is still
|
||||
/// ours. No-op per channel arm (one virtual call); one io-lock visit per
|
||||
/// fd arm.
|
||||
fn unregister_arms(arms: &[&dyn Selectable], me: Pid, epoch: u32) {
|
||||
for arm in arms {
|
||||
if arm.sel_eager_cleanup() {
|
||||
arm.sel_unregister(me, epoch);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Stop-unwind twin of the explicit cleanup pass: a terminal stop unwinds
|
||||
/// out of `park_current`, and a registered fd arm must not outlive its
|
||||
/// actor (the generalization of `wait_fd`'s `Dereg`). Disarmed on the
|
||||
/// normal path after the explicit pass runs; never armed when no fd arm
|
||||
/// registered, keeping the channel-only path guard-free in effect.
|
||||
struct UnregisterGuard<'a> {
|
||||
arms: &'a [&'a dyn Selectable],
|
||||
me: Pid,
|
||||
epoch: u32,
|
||||
armed: bool,
|
||||
}
|
||||
|
||||
impl Drop for UnregisterGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
if self.armed {
|
||||
unregister_arms(self.arms, self.me, self.epoch);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The registration pass shared by [`select`] and [`select_timeout`]:
|
||||
/// check-or-register each arm, in priority order, each atomically under its
|
||||
/// own lock. Cross-arm atomicity is unnecessary: an arm becoming ready
|
||||
/// right after its registration wakes the caller through the protocol (the
|
||||
/// prep-to-park window is closed by RunningNotified).
|
||||
///
|
||||
/// `Ok(Some(i))` = arm `i` was ready, the pass stopped, and the wait has
|
||||
/// been RETIRED (no park may follow): earlier arms hold live-epoch
|
||||
/// registrations, so earlier *fd* arms are unregistered eagerly, then the
|
||||
/// epoch is bumped, a landed notification eaten, and a pending stop
|
||||
/// re-observed — without which a stale arm wake could fault a later
|
||||
/// one-shot park. `Err` = an arm failed to register; identical unwind
|
||||
/// (earlier fd arms unregistered, wait retired). `Ok(None)` = every arm
|
||||
/// registered; the caller parks.
|
||||
fn register_arms(
|
||||
me: Pid,
|
||||
epoch: u32,
|
||||
arms: &[&dyn Selectable],
|
||||
) -> std::io::Result<Option<usize>> {
|
||||
for (i, arm) in arms.iter().enumerate() {
|
||||
let registered = match arm.sel_register(me, epoch) {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
unregister_arms(&arms[..i], me, epoch);
|
||||
crate::scheduler::retire_wait();
|
||||
return Err(e);
|
||||
}
|
||||
};
|
||||
if !registered {
|
||||
unregister_arms(&arms[..i], me, epoch);
|
||||
crate::scheduler::retire_wait();
|
||||
return Ok(Some(i));
|
||||
}
|
||||
}
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
/// The [`select_timeout`] timer target: stateless, because precise wakes
|
||||
/// make classification a pure function of channel state. The entry is
|
||||
/// stamped with the select's epoch; if an arm already won, this unpark dies
|
||||
/// at the word's epoch CAS (the no-cancellation convention in `timer.rs`).
|
||||
struct SelectTimeout;
|
||||
impl crate::timer::TimerTarget for SelectTimeout {
|
||||
fn on_timeout(&self, pid: Pid, epoch: u32) {
|
||||
crate::scheduler::unpark_at(pid, epoch);
|
||||
}
|
||||
}
|
||||
|
||||
/// [`select`] with a deadline: returns `Some(index)` like `select`, or
|
||||
/// `None` once `timeout` elapses with no arm ready.
|
||||
///
|
||||
/// All of `select`'s semantics carry over (priority order, closed arms
|
||||
/// permanently ready, no fairness promise). The timeout is one more stamped
|
||||
/// waker on the same wait epoch — nothing is registered in any arm for it,
|
||||
/// so there is nothing to cancel or leak: an arm winning leaves the timer
|
||||
/// entry to expire as a stale-epoch no-op; the timer winning leaves the
|
||||
/// arms' registrations to self-clean exactly as a `select` loser's would.
|
||||
///
|
||||
/// The wake is classified from state alone (wakes are precise): some arm
|
||||
/// ready → `Some` of the first, in priority order; none ready → the timer
|
||||
/// was the only remaining stamped waker → `None`. A message that races the
|
||||
/// deadline resolves message-first, as `recv_timeout` does.
|
||||
///
|
||||
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-due
|
||||
/// timer is drained, then reports `None` unless an arm was already ready.
|
||||
///
|
||||
/// Panics if `arms` is empty, when called outside an actor, or if an fd
|
||||
/// arm fails to register (see [`try_select_timeout`] for the fallible
|
||||
/// form; channel-only selects cannot fail).
|
||||
pub fn select_timeout(
|
||||
arms: &[&dyn Selectable],
|
||||
timeout: std::time::Duration,
|
||||
) -> Option<usize> {
|
||||
try_select_timeout(arms, timeout)
|
||||
.expect("select_timeout(): fd arm failed to register (use try_select_timeout)")
|
||||
}
|
||||
|
||||
/// [`select_timeout`], fallible: `Err` when an arm fails to register
|
||||
/// (only fd arms can). On `Err` the wait is fully retired and no
|
||||
/// registration — arm-side or kernel-side — is left behind.
|
||||
pub fn try_select_timeout(
|
||||
arms: &[&dyn Selectable],
|
||||
timeout: std::time::Duration,
|
||||
) -> std::io::Result<Option<usize>> {
|
||||
assert!(!arms.is_empty(), "select_timeout() on an empty arm list");
|
||||
let me = crate::actor::current_pid().expect("select_timeout() called outside an actor");
|
||||
let epoch = crate::scheduler::begin_wait();
|
||||
if let Some(i) = register_arms(me, epoch, arms)? {
|
||||
return Ok(Some(i)); // ready now: the timer was never armed
|
||||
}
|
||||
|
||||
// Arm the timer after the registration pass, outside every Channel
|
||||
// lock (insert takes the timers lock).
|
||||
let deadline = crate::timer::deadline_from_now(timeout);
|
||||
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = std::sync::Arc::new(SelectTimeout);
|
||||
crate::scheduler::insert_wait_timer(deadline, me, target, epoch);
|
||||
|
||||
// Same eager-cleanup story as `try_select`: the timer arm needs none
|
||||
// (stateless, stale entries die at the epoch CAS), channel arms need
|
||||
// none, fd arms do — and a timer win in particular leaves every fd
|
||||
// arm's registration behind, which without this pass would poison
|
||||
// those fds until a kernel event happened to fire.
|
||||
let eager = arms.iter().any(|a| a.sel_eager_cleanup());
|
||||
let mut guard = UnregisterGuard { arms, me, epoch, armed: eager };
|
||||
|
||||
crate::scheduler::park_current();
|
||||
|
||||
if eager {
|
||||
unregister_arms(arms, me, epoch);
|
||||
}
|
||||
guard.armed = false;
|
||||
drop(guard);
|
||||
|
||||
// Woken precisely: an arm (ready below) or the timer (nothing ready).
|
||||
Ok(arms.iter().position(|arm| arm.sel_ready()))
|
||||
}
|
||||
|
||||
+11
-111
@@ -1,114 +1,14 @@
|
||||
//! Cooperative context switching, x86-64.
|
||||
//! Cooperative context switching — public façade over the arch backend.
|
||||
//!
|
||||
//! Two naked-asm functions move execution between a scheduler thread and an
|
||||
//! actor running on its own mmap'd stack. The compiler cannot do this; the
|
||||
//! whole point of `#[unsafe(naked)]` is that we control every instruction.
|
||||
//! The real implementation lives in `crate::arch`, selected per `target_arch`.
|
||||
//! This module re-exports the stable surface (`init_actor_stack`, the two
|
||||
//! `switch_to_*` shims, and the `*_actor_sp` accessors) so the scheduler,
|
||||
//! `preempt`, and the `tests/context.rs` integration tests keep importing
|
||||
//! `smarm::context::{...}` unchanged regardless of which ISA is built.
|
||||
//!
|
||||
//! `SCHEDULER_SP` and `ACTOR_SP` are thread-locals holding each side's saved
|
||||
//! stack pointer. `init_actor_stack` builds the initial stack so that the
|
||||
//! first `switch_to_actor` lands inside the entry function with `rsp % 16 == 8`
|
||||
//! (the x86-64 ABI requirement at function entry).
|
||||
//! See `crate::arch` for the contract every backend implements, and
|
||||
//! `crate::arch::{x86_64, aarch64}` for the per-ISA register choreography.
|
||||
|
||||
use std::cell::Cell;
|
||||
|
||||
thread_local! {
|
||||
static SCHEDULER_SP: Cell<usize> = const { Cell::new(0) };
|
||||
static ACTOR_SP: Cell<usize> = const { Cell::new(0) };
|
||||
}
|
||||
|
||||
fn get_scheduler_sp() -> usize { SCHEDULER_SP.with(|c| c.get()) }
|
||||
fn set_scheduler_sp(v: usize) { SCHEDULER_SP.with(|c| c.set(v)) }
|
||||
pub fn get_actor_sp() -> usize { ACTOR_SP.with(|c| c.get()) }
|
||||
pub fn set_actor_sp(v: usize) { ACTOR_SP.with(|c| c.set(v)) }
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Initial stack layout
|
||||
//
|
||||
// We start from aligned_top = top & ~15, then bias by 8 and push seven 8-byte
|
||||
// slots (downward). The first `switch_to_actor` pops r15..rbx and `ret`s —
|
||||
// landing in `entry` with rsp % 16 == 8 (the x86-64 ABI state at the point
|
||||
// just after a `call`, which is what `entry` is compiled to expect).
|
||||
//
|
||||
// Layout (high → low), relative to aligned_top = top & ~15. Note the entry
|
||||
// slot is at aligned_top - 16, NOT aligned_top - 8: the function does
|
||||
// `(top & ~15) - 8` and *then* a `-= 8` before the first write, so the first
|
||||
// stored word lands at aligned_top - 16. Verified by single-stepping the
|
||||
// `ret` under llmdbg: entry sits at an address with %16 == 0, so the post-ret
|
||||
// rsp is %16 == 8.
|
||||
//
|
||||
// aligned_top - 8 : (unused padding; keeps entry's slot %16 == 0)
|
||||
// aligned_top - 16 : entry ptr ← `ret` target. Post-ret: rsp % 16 == 8.
|
||||
// aligned_top - 24 : rbx = 0
|
||||
// aligned_top - 32 : rbp = 0
|
||||
// aligned_top - 40 : r12 = 0
|
||||
// aligned_top - 48 : r13 = 0
|
||||
// aligned_top - 56 : r14 = 0
|
||||
// aligned_top - 64 : r15 = 0 ← initial rsp
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn init_actor_stack(top: *mut u8, entry: extern "C-unwind" fn()) -> usize {
|
||||
unsafe {
|
||||
let mut sp = (top as usize & !15) - 8;
|
||||
sp -= 8; (sp as *mut usize).write(entry as usize); // ret target
|
||||
sp -= 8; (sp as *mut usize).write(0); // rbx
|
||||
sp -= 8; (sp as *mut usize).write(0); // rbp
|
||||
sp -= 8; (sp as *mut usize).write(0); // r12
|
||||
sp -= 8; (sp as *mut usize).write(0); // r13
|
||||
sp -= 8; (sp as *mut usize).write(0); // r14
|
||||
sp -= 8; (sp as *mut usize).write(0); // r15
|
||||
sp
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Context switch shims
|
||||
//
|
||||
// Each shim:
|
||||
// 1. Pushes the six callee-saved integer registers.
|
||||
// 2. Snaps rsp into rdi and calls the Rust helper that stores it.
|
||||
// 3. Calls the Rust helper that returns the *other* side's saved rsp.
|
||||
// 4. Moves that into rsp.
|
||||
// 5. Pops the six registers and rets.
|
||||
//
|
||||
// XMM registers are NOT saved here. We rely on every yield happening through
|
||||
// a Rust call site, which means the compiler has spilled any live XMM state
|
||||
// to the stack before we get here. (This is the same argument the compiler
|
||||
// uses internally — callee-saved regs are what survive a `call`, and the
|
||||
// SysV AMD64 ABI says XMM0–15 are all caller-saved.) If we ever yield from
|
||||
// a place that isn't a Rust call boundary, this assumption breaks.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[unsafe(naked)]
|
||||
unsafe extern "C" fn switch_to_actor_asm() {
|
||||
core::arch::naked_asm!(
|
||||
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
|
||||
"mov rdi, rsp",
|
||||
"call {set_sched_sp}",
|
||||
"call {get_actor_sp}",
|
||||
"mov rsp, rax",
|
||||
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
|
||||
"ret",
|
||||
set_sched_sp = sym set_scheduler_sp,
|
||||
get_actor_sp = sym get_actor_sp,
|
||||
);
|
||||
}
|
||||
|
||||
/// Resume the actor whose sp is in `ACTOR_SP`. Returns when the actor yields.
|
||||
pub unsafe fn switch_to_actor() {
|
||||
unsafe { switch_to_actor_asm() };
|
||||
}
|
||||
|
||||
#[unsafe(naked)]
|
||||
pub unsafe extern "C" fn switch_to_scheduler() {
|
||||
core::arch::naked_asm!(
|
||||
"push rbx", "push rbp", "push r12", "push r13", "push r14", "push r15",
|
||||
"mov rdi, rsp",
|
||||
"call {set_actor_sp}",
|
||||
"call {get_sched_sp}",
|
||||
"mov rsp, rax",
|
||||
"pop r15", "pop r14", "pop r13", "pop r12", "pop rbp", "pop rbx",
|
||||
"ret",
|
||||
set_actor_sp = sym set_actor_sp,
|
||||
get_sched_sp = sym get_scheduler_sp,
|
||||
);
|
||||
}
|
||||
pub use crate::arch::{
|
||||
get_actor_sp, init_actor_stack, set_actor_sp, switch_to_actor, switch_to_scheduler,
|
||||
};
|
||||
|
||||
+35
-258
@@ -6,23 +6,14 @@
|
||||
//! [`call`](ServerRef::call) (synchronous, returns a reply) or
|
||||
//! [`cast`](ServerRef::cast) (fire-and-forget).
|
||||
//!
|
||||
//! ## One inbox, many arms
|
||||
//! ## Why a single inbox
|
||||
//!
|
||||
//! Every call and cast travels the *same* inbox channel as an [`Envelope`]
|
||||
//! (calls and casts are ordered relative to each other, like Erlang); the
|
||||
//! server loop dispatches by variant. A `call` carries a freshly made
|
||||
//! one-shot reply channel; the server sends the reply straight back down it.
|
||||
//!
|
||||
//! Out-of-band messages ride *separate* channels composed at the wait via
|
||||
//! [`select`](crate::channel::select): info channels handed over at start
|
||||
//! ([`ServerBuilder::with_info`]) are dispatched to
|
||||
//! [`handle_info`](GenServer::handle_info). Arm priority is
|
||||
//! **infos before inbox**, in declaration order — a hot inbox cannot starve
|
||||
//! an out-of-band message; conversely a hot info channel CAN starve the
|
||||
//! inbox, deliberately (system-message semantics). An info channel whose
|
||||
//! senders are all gone is silently dropped from the arm set (per the
|
||||
//! closed-arm-is-ready-forever rule on `select`); a closed *inbox* still
|
||||
//! means graceful shutdown.
|
||||
//! smarm has no `select` and no unified per-process mailbox (see the runtime
|
||||
//! gotchas in `task.md`), so a server cannot wait on a "call channel" and a
|
||||
//! "cast channel" simultaneously. Instead every message — call or cast —
|
||||
//! travels the *same* inbox channel as an [`Envelope`], and the server loop
|
||||
//! dispatches by variant. A `call` carries a freshly made one-shot reply
|
||||
//! channel; the server sends the reply straight back down it.
|
||||
//!
|
||||
//! ## Server death
|
||||
//!
|
||||
@@ -45,12 +36,13 @@
|
||||
//!
|
||||
//! ## Not here (yet)
|
||||
//!
|
||||
//! No dynamic info subscription: the info-channel set is fixed at start.
|
||||
//! Revisit against a real consumer. (Monitor `Down` forwarding is dynamic —
|
||||
//! see `handle_down` — because monitors are inherently created at runtime.)
|
||||
//! No `handle_info` and no call timeout. Both need machinery smarm currently
|
||||
//! defers: a call timeout needs a per-`recv` deadline (cf. the deferred
|
||||
//! `Signal::Timeout`), and `handle_info` needs the cross-channel mailbox merge
|
||||
//! that selective receive deliberately left unmade. They are slated to land
|
||||
//! together with timeouts.
|
||||
|
||||
use crate::channel::{channel, select, Receiver, Selectable, Sender};
|
||||
use crate::monitor::{Down, Monitor};
|
||||
use crate::channel::{channel, Receiver, Sender};
|
||||
use crate::pid::Pid;
|
||||
use crate::scheduler::{spawn, spawn_under};
|
||||
|
||||
@@ -66,17 +58,9 @@ pub trait GenServer: Send + 'static {
|
||||
type Reply: Send + 'static;
|
||||
/// Asynchronous request type (carried by [`ServerRef::cast`]).
|
||||
type Cast: Send + 'static;
|
||||
/// Out-of-band message type, delivered to [`handle_info`](Self::handle_info)
|
||||
/// from the info channels registered at start
|
||||
/// ([`ServerBuilder::with_info`]). Servers with several out-of-band
|
||||
/// sources enum them up into one `Info`. Use `()` if unused.
|
||||
type Info: Send + 'static;
|
||||
|
||||
/// Runs once inside the server actor before any message is handled. The
|
||||
/// [`ServerCtx`] is the loop's one runtime hook: clone its [`Watcher`]
|
||||
/// into the state here to be able to [`watch`](Watcher::watch) monitors
|
||||
/// from any later handler.
|
||||
fn init(&mut self, _ctx: &ServerCtx) {}
|
||||
/// Runs once inside the server actor before any message is handled.
|
||||
fn init(&mut self) {}
|
||||
|
||||
/// Handle a synchronous call and produce the reply sent back to the caller.
|
||||
fn handle_call(&mut self, request: Self::Call) -> Self::Reply;
|
||||
@@ -84,14 +68,6 @@ pub trait GenServer: Send + 'static {
|
||||
/// Handle a fire-and-forget cast.
|
||||
fn handle_cast(&mut self, request: Self::Cast);
|
||||
|
||||
/// Handle an out-of-band message from one of the info channels. Default:
|
||||
/// drop it.
|
||||
fn handle_info(&mut self, _info: Self::Info) {}
|
||||
|
||||
/// Handle a [`Down`] from a monitor handed to the loop via
|
||||
/// [`Watcher::watch`]. Default: drop it.
|
||||
fn handle_down(&mut self, _down: Down) {}
|
||||
|
||||
/// Runs as the server actor exits, on any exit path (see module docs).
|
||||
fn terminate(&mut self) {}
|
||||
}
|
||||
@@ -124,19 +100,6 @@ pub enum CallError {
|
||||
ServerDown,
|
||||
}
|
||||
|
||||
/// Returned by [`ServerRef::call_timeout`].
|
||||
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
|
||||
pub enum CallTimeoutError {
|
||||
/// The server was already gone, or died before replying.
|
||||
ServerDown,
|
||||
/// The deadline passed before a reply arrived. The request stays in the
|
||||
/// server's inbox: it will still be *handled*, but the reply is discarded
|
||||
/// (the abandoned reply channel's receiver is dropped, so the server's
|
||||
/// reply send fails harmlessly). Erlang behaves the same way; design
|
||||
/// idempotent calls accordingly.
|
||||
Timeout,
|
||||
}
|
||||
|
||||
/// Returned by [`ServerRef::cast`] when the server is no longer reachable.
|
||||
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
|
||||
pub enum CastError {
|
||||
@@ -161,33 +124,6 @@ impl<G: GenServer> ServerRef<G> {
|
||||
reply_rx.recv().map_err(|_| CallError::ServerDown)
|
||||
}
|
||||
|
||||
/// Bounded synchronous request-reply: like [`call`](Self::call), but
|
||||
/// gives up after `timeout`, returning [`CallTimeoutError::Timeout`].
|
||||
///
|
||||
/// The roadmap sketched this as monitor + wait-reply-or-Down + demonitor;
|
||||
/// that machinery is unnecessary here because server death is already
|
||||
/// observable on the reply channel itself — the reply sender is dropped
|
||||
/// as the server's stack unwinds, closing the channel and waking the
|
||||
/// parked caller (see the module docs). So a bounded call is exactly
|
||||
/// [`Receiver::recv_timeout`] on the reply channel: `Ok` on a reply,
|
||||
/// `Disconnected` → [`CallTimeoutError::ServerDown`], `Timeout` →
|
||||
/// [`CallTimeoutError::Timeout`]. Nothing is registered, so nothing can
|
||||
/// leak on the timeout path by construction.
|
||||
pub fn call_timeout(
|
||||
&self,
|
||||
request: G::Call,
|
||||
timeout: std::time::Duration,
|
||||
) -> Result<G::Reply, CallTimeoutError> {
|
||||
let (reply_tx, reply_rx) = channel::<G::Reply>();
|
||||
self.tx
|
||||
.send(Envelope::Call(request, reply_tx))
|
||||
.map_err(|_| CallTimeoutError::ServerDown)?;
|
||||
reply_rx.recv_timeout(timeout).map_err(|e| match e {
|
||||
crate::channel::RecvTimeoutError::Disconnected => CallTimeoutError::ServerDown,
|
||||
crate::channel::RecvTimeoutError::Timeout => CallTimeoutError::Timeout,
|
||||
})
|
||||
}
|
||||
|
||||
/// Fire-and-forget request. Returns once the message is enqueued; does not
|
||||
/// wait for the server to handle it. [`CastError::ServerDown`] if the inbox
|
||||
/// is already closed.
|
||||
@@ -198,109 +134,24 @@ impl<G: GenServer> ServerRef<G> {
|
||||
}
|
||||
}
|
||||
|
||||
/// The server loop's runtime hook, passed to [`GenServer::init`]. Currently
|
||||
/// carries only the [`Watcher`]; opaque so fields can grow without breaking.
|
||||
pub struct ServerCtx {
|
||||
watcher: Watcher,
|
||||
}
|
||||
|
||||
impl ServerCtx {
|
||||
/// A clonable handle to the loop's monitor intake. Store it in the state
|
||||
/// during `init` to watch monitors from later handlers.
|
||||
pub fn watcher(&self) -> Watcher {
|
||||
self.watcher.clone()
|
||||
}
|
||||
|
||||
/// Shorthand for `ctx.watcher().watch(m)` when watching during `init`.
|
||||
pub fn watch(&self, m: Monitor) {
|
||||
self.watcher.watch(m)
|
||||
}
|
||||
}
|
||||
|
||||
/// Hands [`Monitor`]s to a server loop; their [`Down`]s are dispatched to
|
||||
/// [`GenServer::handle_down`]. Down arms outrank every other arm (a death
|
||||
/// notice cannot be starved), and a delivered `Down` retires its arm —
|
||||
/// monitors are one-shot.
|
||||
#[derive(Clone)]
|
||||
pub struct Watcher {
|
||||
tx: Sender<Monitor>,
|
||||
}
|
||||
|
||||
impl Watcher {
|
||||
/// Transfer `m` to the server loop. If the server is already gone the
|
||||
/// monitor is silently dropped (its queued `Down`, if any, with it) —
|
||||
/// there is no loop left to care.
|
||||
pub fn watch(&self, m: Monitor) {
|
||||
let _ = self.tx.send(m);
|
||||
}
|
||||
}
|
||||
|
||||
/// Configure-then-start construction for servers that need more than a bare
|
||||
/// [`start`]: info channels, a supervisor. Consumed by [`start`](Self::start).
|
||||
///
|
||||
/// ```ignore
|
||||
/// let server = ServerBuilder::new(state)
|
||||
/// .with_info(events_rx)
|
||||
/// .under(supervisor_pid)
|
||||
/// .start();
|
||||
/// ```
|
||||
pub struct ServerBuilder<G: GenServer> {
|
||||
state: G,
|
||||
infos: Vec<Receiver<G::Info>>,
|
||||
supervisor: Option<Pid>,
|
||||
}
|
||||
|
||||
impl<G: GenServer> ServerBuilder<G> {
|
||||
pub fn new(state: G) -> Self {
|
||||
ServerBuilder { state, infos: Vec::new(), supervisor: None }
|
||||
}
|
||||
|
||||
/// Add an out-of-band channel; messages arriving on it are dispatched to
|
||||
/// [`GenServer::handle_info`]. Repeatable; arm priority is call order
|
||||
/// (earlier = higher), and every info channel outranks the inbox.
|
||||
pub fn with_info(mut self, rx: Receiver<G::Info>) -> Self {
|
||||
self.infos.push(rx);
|
||||
self
|
||||
}
|
||||
|
||||
/// Spawn the server under an explicit supervisor pid (via [`spawn_under`])
|
||||
/// so it slots into the supervision tree.
|
||||
pub fn under(mut self, supervisor: Pid) -> Self {
|
||||
self.supervisor = Some(supervisor);
|
||||
self
|
||||
}
|
||||
|
||||
/// Spawn the server actor and hand back its [`ServerRef`]. The server's
|
||||
/// lifetime is governed by its refs, not by joining, so the backing join
|
||||
/// handle is dropped.
|
||||
pub fn start(self) -> ServerRef<G> {
|
||||
let (tx, rx) = channel::<Envelope<G>>();
|
||||
let ServerBuilder { state, infos, supervisor } = self;
|
||||
let handle = match supervisor {
|
||||
Some(sup) => spawn_under(sup, move || server_loop::<G>(rx, state, infos)),
|
||||
None => spawn(move || server_loop::<G>(rx, state, infos)),
|
||||
};
|
||||
ServerRef { tx, pid: handle.pid() }
|
||||
}
|
||||
}
|
||||
|
||||
/// Spawn `state` as a server under the current actor (via [`spawn`]). Returns a
|
||||
/// [`ServerRef`]. Shorthand for `ServerBuilder::new(state).start()`.
|
||||
/// [`ServerRef`]; the server's lifetime is governed by its refs, not by
|
||||
/// joining, so the backing join handle is dropped.
|
||||
pub fn start<G: GenServer>(state: G) -> ServerRef<G> {
|
||||
ServerBuilder::new(state).start()
|
||||
let (tx, rx) = channel::<Envelope<G>>();
|
||||
let handle = spawn(move || server_loop::<G>(rx, state));
|
||||
ServerRef { tx, pid: handle.pid() }
|
||||
}
|
||||
|
||||
/// Like [`start`], but spawns the server under an explicit supervisor pid (via
|
||||
/// [`spawn_under`]) so it slots into the supervision tree.
|
||||
pub fn start_under<G: GenServer>(supervisor: Pid, state: G) -> ServerRef<G> {
|
||||
ServerBuilder::new(state).under(supervisor).start()
|
||||
let (tx, rx) = channel::<Envelope<G>>();
|
||||
let handle = spawn_under(supervisor, move || server_loop::<G>(rx, state));
|
||||
ServerRef { tx, pid: handle.pid() }
|
||||
}
|
||||
|
||||
fn server_loop<G: GenServer>(
|
||||
rx: Receiver<Envelope<G>>,
|
||||
state: G,
|
||||
mut infos: Vec<Receiver<G::Info>>,
|
||||
) {
|
||||
fn server_loop<G: GenServer>(rx: Receiver<Envelope<G>>, state: G) {
|
||||
// terminate() must run on every exit path (clean close, panic, stop), so it
|
||||
// lives in this guard's Drop rather than after the loop.
|
||||
struct Terminate<G: GenServer>(G);
|
||||
@@ -310,96 +161,22 @@ fn server_loop<G: GenServer>(
|
||||
}
|
||||
}
|
||||
|
||||
fn dispatch<G: GenServer>(state: &mut G, env: Envelope<G>) {
|
||||
match env {
|
||||
Envelope::Call(request, reply_tx) => {
|
||||
let reply = state.handle_call(request);
|
||||
let mut guard = Terminate(state);
|
||||
guard.0.init();
|
||||
|
||||
loop {
|
||||
match rx.recv() {
|
||||
Ok(Envelope::Call(request, reply_tx)) => {
|
||||
let reply = guard.0.handle_call(request);
|
||||
// The caller may have gone away (e.g. cancelled while parked);
|
||||
// a failed reply send is not the server's problem.
|
||||
let _ = reply_tx.send(reply);
|
||||
}
|
||||
Envelope::Cast(request) => state.handle_cast(request),
|
||||
Ok(Envelope::Cast(request)) => guard.0.handle_cast(request),
|
||||
// All ServerRefs dropped → inbox closed → graceful shutdown.
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
|
||||
let mut guard = Terminate(state);
|
||||
|
||||
// The control arm: Watchers feed Monitors to the loop through it. The
|
||||
// ctx (and with it the loop's own sender) drops right after init — a
|
||||
// state that didn't clone the Watcher closes the arm, the first select
|
||||
// observes the closure, and the loop falls back to the plain-inbox park.
|
||||
let (watch_tx, watch_rx) = channel::<Monitor>();
|
||||
guard.0.init(&ServerCtx { watcher: Watcher { tx: watch_tx } });
|
||||
|
||||
let mut monitors: Vec<Monitor> = Vec::new();
|
||||
let mut watch_open = true;
|
||||
|
||||
loop {
|
||||
if monitors.is_empty() && !watch_open && infos.is_empty() {
|
||||
// Nothing to select over: park on the inbox alone, exactly the
|
||||
// pre-v0.8 loop.
|
||||
match rx.recv() {
|
||||
Ok(env) => dispatch(&mut guard.0, env),
|
||||
// All ServerRefs dropped → inbox closed → graceful shutdown.
|
||||
Err(_) => break,
|
||||
}
|
||||
} else {
|
||||
// Arm priority: downs, then the control arm, then infos (each in
|
||||
// declaration order), then the inbox. The arm slice is rebuilt
|
||||
// per iteration because every set but the inbox shrinks or grows.
|
||||
let nd = monitors.len();
|
||||
let nw = watch_open as usize;
|
||||
let i = {
|
||||
let mut arms: Vec<&dyn Selectable> =
|
||||
Vec::with_capacity(nd + nw + infos.len() + 1);
|
||||
for m in &monitors {
|
||||
arms.push(&m.rx);
|
||||
}
|
||||
if watch_open {
|
||||
arms.push(&watch_rx);
|
||||
}
|
||||
for r in &infos {
|
||||
arms.push(r);
|
||||
}
|
||||
arms.push(&rx);
|
||||
select(&arms)
|
||||
};
|
||||
if i < nd {
|
||||
// A Down retires its arm either way: delivered (one-shot) or
|
||||
// closed without delivering (defensive; shouldn't happen).
|
||||
let m = monitors.remove(i);
|
||||
if let Ok(Some(down)) = m.rx.try_recv() {
|
||||
guard.0.handle_down(down);
|
||||
}
|
||||
} else if i < nd + nw {
|
||||
match watch_rx.try_recv() {
|
||||
Ok(Some(m)) => monitors.push(m),
|
||||
// Single-receiver: nothing can drain the arm between
|
||||
// select's ready and our try_recv.
|
||||
Ok(None) => debug_assert!(false, "ready control arm was empty"),
|
||||
// Every Watcher gone: stop selecting on the arm.
|
||||
Err(_) => watch_open = false,
|
||||
}
|
||||
} else if i < nd + nw + infos.len() {
|
||||
let j = i - nd - nw;
|
||||
match infos[j].try_recv() {
|
||||
Ok(Some(info)) => guard.0.handle_info(info),
|
||||
Ok(None) => debug_assert!(false, "ready info arm was empty"),
|
||||
// Senders all gone: drop the arm, keep serving. `remove`
|
||||
// (not swap_remove) — order is priority.
|
||||
Err(_) => {
|
||||
infos.remove(j);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
match rx.try_recv() {
|
||||
Ok(Some(env)) => dispatch(&mut guard.0, env),
|
||||
Ok(None) => debug_assert!(false, "ready inbox was empty"),
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
}
|
||||
// Observation point so a server whose arms are never empty stays
|
||||
// Observation point so a server whose inbox is never empty stays
|
||||
// preemptible and cancellable.
|
||||
crate::check!();
|
||||
}
|
||||
|
||||
@@ -44,13 +44,11 @@
|
||||
//!
|
||||
//! Fd hygiene
|
||||
//! ==========
|
||||
//! An actor stopped while waiting on an fd unwinds out of `wait_fd`'s park;
|
||||
//! a drop guard there (armed after a successful register, forgotten on a
|
||||
//! normal wake) removes the `waiters` entry iff it is still that wait's
|
||||
//! `(pid, epoch)` and only then `EPOLL_CTL_DEL`s the fd — an entry already
|
||||
//! consumed by a racing `FdReady` means the fd may carry someone else's
|
||||
//! fresh registration, which must be left alone. `epoll_register` keeps a
|
||||
//! defensive bare DEL before ADD as belt-and-braces.
|
||||
//! If an actor dies while waiting on an fd, the registration is leaked
|
||||
//! (the fd stays in the epollfd, armed). EPOLLONESHOT bounds the damage:
|
||||
//! at most one stale wakeup, after which the kernel disarms. The stale
|
||||
//! wakeup hits a dead pid in `waiters` and is dropped. Acceptable for v0.2;
|
||||
//! a future pass should DEL on actor death.
|
||||
//!
|
||||
//! Buffers used with `read`/`write` should be on fds opened with
|
||||
//! `O_NONBLOCK`. If they aren't, the syscall may block the scheduler
|
||||
@@ -86,9 +84,6 @@ use std::thread::JoinHandle as OsJoinHandle;
|
||||
pub type IoResult = Result<Box<dyn Any + Send>, Box<dyn Any + Send>>;
|
||||
|
||||
struct Request {
|
||||
/// The submitter's park-epoch — carried through to the `Blocking`
|
||||
/// completion so the wake is epoch-matched.
|
||||
epoch: u32,
|
||||
pid: Pid,
|
||||
/// The work to perform. Returns the wire-form result directly.
|
||||
work: Box<dyn FnOnce() -> IoResult + Send>,
|
||||
@@ -98,7 +93,7 @@ struct Request {
|
||||
pub enum Completion {
|
||||
/// A `block_on_io` closure has finished (Ok = return value, Err = panic
|
||||
/// payload).
|
||||
Blocking { pid: Pid, epoch: u32, result: IoResult },
|
||||
Blocking { pid: Pid, result: IoResult },
|
||||
/// An fd registered via `wait_readable`/`wait_writable` is ready. The
|
||||
/// scheduler looks up the parked pid in `waiters`, unparks it, and
|
||||
/// removes the entry. `pid` isn't in this variant because the epoll
|
||||
@@ -136,7 +131,7 @@ pub struct IoThread {
|
||||
/// One parked actor per registered fd. Populated by `wait_readable` /
|
||||
/// `wait_writable` and drained by the scheduler when a `FdReady`
|
||||
/// completion is processed.
|
||||
pub waiters: HashMap<RawFd, (Pid, u32)>,
|
||||
pub waiters: HashMap<RawFd, Pid>,
|
||||
|
||||
// ----- Threads -----
|
||||
|
||||
@@ -236,12 +231,12 @@ impl IoThread {
|
||||
}
|
||||
|
||||
/// Hand a request to the pool. Increments `outstanding`.
|
||||
pub fn submit(&mut self, pid: Pid, epoch: u32, work: Box<dyn FnOnce() -> IoResult + Send>) {
|
||||
pub fn submit(&mut self, pid: Pid, work: Box<dyn FnOnce() -> IoResult + Send>) {
|
||||
self.outstanding += 1;
|
||||
// Send can only fail if the pool has hung up, which only happens
|
||||
// on shutdown. submit during shutdown is a bug.
|
||||
self.tx
|
||||
.send(Request { pid, epoch, work })
|
||||
.send(Request { pid, work })
|
||||
.expect("io pool hung up unexpectedly");
|
||||
}
|
||||
|
||||
@@ -260,15 +255,6 @@ impl IoThread {
|
||||
self.wake_read
|
||||
}
|
||||
|
||||
/// Write the wake pipe directly: rouse every scheduler thread blocked in
|
||||
/// its idle `poll_wake`. Used by the terminal (AllDone) path — an idle
|
||||
/// sibling may be blocked on a snapshot that nothing will ever refresh
|
||||
/// (an orphaned timer deadline, or `io_outstanding` from a waiter that
|
||||
/// was stop-cancelled and so never produces a completion).
|
||||
pub fn wake(&self) {
|
||||
wake_scheduler(self.wake_write);
|
||||
}
|
||||
|
||||
/// Register interest in `fd` becoming readable/writable; record `pid`
|
||||
/// as the parked waiter. The epoll thread will push a `FdReady`
|
||||
/// completion when the kernel signals.
|
||||
@@ -279,7 +265,6 @@ impl IoThread {
|
||||
&mut self,
|
||||
fd: RawFd,
|
||||
pid: Pid,
|
||||
epoch: u32,
|
||||
readable: bool,
|
||||
writable: bool,
|
||||
) -> io::Result<()> {
|
||||
@@ -293,10 +278,10 @@ impl IoThread {
|
||||
));
|
||||
}
|
||||
|
||||
// Belt-and-braces: the unwind guard in `wait_fd` is responsible for
|
||||
// cleaning up a stopped waiter's registration, but a bare DEL is
|
||||
// harmless if the fd isn't registered (ENOENT) and removes any leak
|
||||
// a path we haven't thought of might leave behind.
|
||||
// Defensive cleanup: if a previous actor died while waiting on this
|
||||
// fd, the kernel-side registration was leaked (we don't walk all
|
||||
// waiters on actor death). A bare DEL is harmless if the fd isn't
|
||||
// registered (ENOENT), and removes any leak.
|
||||
unsafe {
|
||||
libc::epoll_ctl(self.epollfd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut());
|
||||
}
|
||||
@@ -318,7 +303,7 @@ impl IoThread {
|
||||
if r < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
self.waiters.insert(fd, (pid, epoch));
|
||||
self.waiters.insert(fd, pid);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -384,7 +369,7 @@ fn pool_loop(
|
||||
completions: Arc<Mutex<VecDeque<Completion>>>,
|
||||
wake_write: RawFd,
|
||||
) {
|
||||
while let Ok(Request { pid, epoch, work }) = rx.recv() {
|
||||
while let Ok(Request { pid, work }) = rx.recv() {
|
||||
let result: IoResult = match panic::catch_unwind(panic::AssertUnwindSafe(work)) {
|
||||
Ok(r) => r,
|
||||
Err(payload) => Err(payload),
|
||||
@@ -392,7 +377,7 @@ fn pool_loop(
|
||||
completions
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push_back(Completion::Blocking { pid, epoch, result });
|
||||
.push_back(Completion::Blocking { pid, result });
|
||||
wake_scheduler(wake_write);
|
||||
}
|
||||
}
|
||||
|
||||
+4
-14
@@ -11,6 +11,7 @@
|
||||
//!
|
||||
//! See `LOOM.md` for the design intent and the deferred-for-later list.
|
||||
|
||||
pub mod arch;
|
||||
pub mod stack;
|
||||
pub mod context;
|
||||
pub mod preempt;
|
||||
@@ -23,15 +24,9 @@ pub mod timer;
|
||||
pub mod io;
|
||||
pub mod mutex;
|
||||
pub mod monitor;
|
||||
pub mod registry;
|
||||
pub mod link;
|
||||
pub mod gen_server;
|
||||
pub mod runtime;
|
||||
pub(crate) mod raw_mutex;
|
||||
pub(crate) mod slot_state;
|
||||
pub(crate) mod sync_shim;
|
||||
#[doc(hidden)] // pub only so benches/rq_micro.rs can drive the raw structures
|
||||
pub mod run_queue;
|
||||
pub mod trace;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -45,21 +40,16 @@ static ALLOCATOR: preempt::PreemptingAllocator = preempt::PreemptingAllocator;
|
||||
// Public API re-exports
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub use channel::{
|
||||
channel, select, select_timeout, try_select, try_select_timeout, Receiver, RecvError,
|
||||
RecvTimeoutError, Selectable, Sender,
|
||||
};
|
||||
pub use gen_server::{CallError, CallTimeoutError, CastError, GenServer, ServerBuilder, ServerCtx, ServerRef, Watcher};
|
||||
pub use channel::{channel, Receiver, RecvError, Sender};
|
||||
pub use gen_server::{CallError, CastError, GenServer, ServerRef};
|
||||
pub use link::{link, trap_exit, unlink, ExitSignal};
|
||||
pub use monitor::{demonitor, monitor, Down, DownReason, Monitor, MonitorId};
|
||||
pub use mutex::{LockTimeout, Mutex, MutexGuard};
|
||||
pub use pid::Pid;
|
||||
pub use registry::{name_of, register, unregister, whereis, RegisterError};
|
||||
pub use runtime::{init, Config, Runtime};
|
||||
pub use scheduler::{
|
||||
block_on_io, request_stop, run, self_pid, sleep, spawn, spawn_under, wait_readable,
|
||||
wait_readable_timeout, wait_writable, wait_writable_timeout, yield_now, FdArm, JoinError,
|
||||
JoinHandle,
|
||||
wait_writable, yield_now, JoinError, JoinHandle,
|
||||
};
|
||||
pub use supervisor::{ChildSpec, OneForOne, Restart, Signal, Strategy};
|
||||
|
||||
|
||||
+44
-66
@@ -49,9 +49,10 @@
|
||||
//! [`Down`]: crate::monitor::Down
|
||||
//! [`request_stop`]: crate::scheduler::request_stop
|
||||
|
||||
use crate::channel::{channel, Receiver};
|
||||
use crate::channel::{channel, Receiver, Sender};
|
||||
use crate::monitor::DownReason;
|
||||
use crate::pid::Pid;
|
||||
use crate::runtime::State;
|
||||
use crate::scheduler::{request_stop, self_pid, with_runtime};
|
||||
|
||||
/// A linked peer's death, delivered to a trapping actor's inbox.
|
||||
@@ -77,13 +78,11 @@ pub fn trap_exit() -> Receiver<ExitSignal> {
|
||||
let (tx, rx) = channel::<ExitSignal>();
|
||||
let me = self_pid();
|
||||
with_runtime(|inner| {
|
||||
if let Some(slot) = inner.slot_at(me) {
|
||||
let mut cold = slot.cold.lock();
|
||||
// Own slot: generation is necessarily current (we're running).
|
||||
if let Some(actor) = cold.actor.as_mut() {
|
||||
inner.with_shared(|s| {
|
||||
if let Some(actor) = s.slot_mut(me).and_then(|slot| slot.actor.as_mut()) {
|
||||
actor.trap = Some(tx);
|
||||
}
|
||||
}
|
||||
})
|
||||
});
|
||||
rx
|
||||
}
|
||||
@@ -101,61 +100,45 @@ pub fn link(target: Pid) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Cold locks are leaves: never hold two at once. The link is recorded one
|
||||
// side at a time, TARGET FIRST — that ordering is what makes the race
|
||||
// window sound:
|
||||
//
|
||||
// - Once `me` is in `target.links`, the target's death always reaches us
|
||||
// (its finalize cascade walks that list). So after step 1 succeeds, the
|
||||
// link semantics are already live.
|
||||
// - If the target dies between step 1 and step 2, its cascade removes
|
||||
// `target` from OUR links (a no-op, we haven't added it yet) and
|
||||
// delivers the exit signal — correct, the link was established. Our
|
||||
// subsequent step-2 insert leaves a stale `target` entry in `me.links`;
|
||||
// stale entries are benign by construction (every cascade walk
|
||||
// re-verifies the peer's word; `unlink` removes them like any other).
|
||||
//
|
||||
// The reverse order would be unsound: target dying in the window would
|
||||
// walk its links WITHOUT us — a silently dead link that we believe is live.
|
||||
let registered_on_target = with_runtime(|inner| match inner.slot_at(target) {
|
||||
Some(slot) => {
|
||||
let mut cold = slot.cold.lock();
|
||||
if slot.is_live_for(target) && cold.actor.is_some() {
|
||||
if !cold.links.contains(&me) {
|
||||
cold.links.push(me);
|
||||
// Under the lock: if the target is live, record the link both ways and
|
||||
// return `None`. If it is gone, return the caller's trap sender (if any)
|
||||
// so we can deliver the NoProc signal after releasing the lock.
|
||||
let dead_action: Option<Option<Sender<ExitSignal>>> = with_runtime(|inner| {
|
||||
inner.with_shared(|s| {
|
||||
let target_live = matches!(
|
||||
s.slot(target),
|
||||
Some(slot) if slot.actor.is_some() && !matches!(slot.state, State::Done)
|
||||
);
|
||||
if target_live {
|
||||
if let Some(slot) = s.slot_mut(me) {
|
||||
if !slot.links.contains(&target) {
|
||||
slot.links.push(target);
|
||||
}
|
||||
}
|
||||
true
|
||||
if let Some(slot) = s.slot_mut(target) {
|
||||
if !slot.links.contains(&me) {
|
||||
slot.links.push(me);
|
||||
}
|
||||
}
|
||||
None
|
||||
} else {
|
||||
false
|
||||
// Grab our own trap sender so the NoProc delivery (below)
|
||||
// doesn't need a second lock acquisition.
|
||||
Some(
|
||||
s.slot(me)
|
||||
.and_then(|slot| slot.actor.as_ref())
|
||||
.and_then(|a| a.trap.clone()),
|
||||
)
|
||||
}
|
||||
}
|
||||
None => false,
|
||||
});
|
||||
|
||||
if registered_on_target {
|
||||
with_runtime(|inner| {
|
||||
let slot = inner.slot_at(me).expect("link: own slot vanished");
|
||||
let mut cold = slot.cold.lock();
|
||||
if !cold.links.contains(&target) {
|
||||
cold.links.push(target);
|
||||
}
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Target already gone: deliver NoProc to ourselves — as a message if
|
||||
// trapping, otherwise as a cooperative stop.
|
||||
let my_trap = with_runtime(|inner| {
|
||||
inner.slot_at(me).and_then(|slot| {
|
||||
let cold = slot.cold.lock();
|
||||
cold.actor.as_ref().and_then(|a| a.trap.clone())
|
||||
})
|
||||
});
|
||||
match my_trap {
|
||||
Some(tx) => {
|
||||
|
||||
match dead_action {
|
||||
None => {} // linked successfully
|
||||
Some(Some(tx)) => {
|
||||
let _ = tx.send(ExitSignal { from: target, reason: DownReason::NoProc });
|
||||
}
|
||||
None => request_stop(me),
|
||||
Some(None) => request_stop(me),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -169,18 +152,13 @@ pub fn unlink(target: Pid) {
|
||||
return;
|
||||
}
|
||||
with_runtime(|inner| {
|
||||
// One cold lock at a time (leaf rule). Order is immaterial here:
|
||||
// a half-removed link is just a stale entry on one side, and stale
|
||||
// entries are benign (re-verified on every cascade walk).
|
||||
if let Some(slot) = inner.slot_at(me) {
|
||||
let mut cold = slot.cold.lock();
|
||||
cold.links.retain(|p| *p != target);
|
||||
}
|
||||
if let Some(slot) = inner.slot_at(target) {
|
||||
let mut cold = slot.cold.lock();
|
||||
if slot.generation() == target.generation() {
|
||||
cold.links.retain(|p| *p != me);
|
||||
inner.with_shared(|s| {
|
||||
if let Some(slot) = s.slot_mut(me) {
|
||||
slot.links.retain(|p| *p != target);
|
||||
}
|
||||
}
|
||||
if let Some(slot) = s.slot_mut(target) {
|
||||
slot.links.retain(|p| *p != me);
|
||||
}
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
+25
-28
@@ -47,6 +47,7 @@
|
||||
|
||||
use crate::channel::{channel, Receiver, Sender};
|
||||
use crate::pid::Pid;
|
||||
use crate::runtime::State;
|
||||
use crate::scheduler::with_runtime;
|
||||
|
||||
/// Why a monitored actor went down.
|
||||
@@ -109,26 +110,24 @@ pub struct Monitor {
|
||||
pub fn monitor(target: Pid) -> Monitor {
|
||||
let (tx, rx) = channel::<Down>();
|
||||
|
||||
// Register under the target's cold lock. `tx.clone()` takes the channel's
|
||||
// own lock — a Channel-class RawMutex, explicitly permitted *under* a Leaf
|
||||
// (cold) lock by the lock order (see raw_mutex.rs). We must still not
|
||||
// *send* under the lock, as `Sender::send` can unpark a parked receiver,
|
||||
// and there's no reason to nest that.
|
||||
// Register under the shared lock. We allocate the id and (if the target is
|
||||
// live) clone the sender into its monitor list, keeping the original `tx`
|
||||
// for the NoProc fallback. `tx.clone()` only touches the channel's own
|
||||
// mutex, never the shared runtime mutex, so it is safe under the lock — but
|
||||
// we must not *send* here, as `Sender::send` can call back in to unpark a
|
||||
// parked receiver and the shared mutex is not reentrant.
|
||||
let (id, registered) = with_runtime(|inner| {
|
||||
let id = inner.alloc_monitor_id();
|
||||
let registered = match inner.slot_at(target) {
|
||||
Some(slot) => {
|
||||
let mut cold = slot.cold.lock();
|
||||
if slot.is_live_for(target) {
|
||||
cold.monitors.push((id, tx.clone()));
|
||||
inner.with_shared(|s| {
|
||||
let id = s.alloc_monitor_id();
|
||||
let registered = match s.slot_mut(target) {
|
||||
Some(slot) if !matches!(slot.state, State::Done) => {
|
||||
slot.monitors.push((id, tx.clone()));
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
None => false,
|
||||
};
|
||||
(id, registered)
|
||||
_ => false,
|
||||
};
|
||||
(id, registered)
|
||||
})
|
||||
});
|
||||
|
||||
if !registered {
|
||||
@@ -148,18 +147,16 @@ pub fn monitor(target: Pid) -> Monitor {
|
||||
/// dropping the [`Monitor`] closes its receiver and the queued notice goes with
|
||||
/// it — the analogue of Erlang's `demonitor(Ref, [flush])`.
|
||||
pub fn demonitor(m: &Monitor) -> Option<MonitorId> {
|
||||
// Remove the registration under the target's cold lock, but move the
|
||||
// `Sender` *out* and let it drop only after the lock is released:
|
||||
// dropping the last sender runs `Sender::drop`, which may unpark a parked
|
||||
// receiver — legal under a cold lock, but pointless to nest.
|
||||
// Remove the registration under the lock, but move the `Sender` *out* and
|
||||
// let it drop only after the lock is released: dropping the last sender
|
||||
// runs `Sender::drop`, which may unpark a parked receiver → `with_shared`,
|
||||
// and the shared mutex is not reentrant.
|
||||
let removed: Option<(MonitorId, Sender<Down>)> = with_runtime(|inner| {
|
||||
let slot = inner.slot_at(m.target)?;
|
||||
let mut cold = slot.cold.lock();
|
||||
if slot.generation() != m.target.generation() {
|
||||
return None; // slot reused; the Down already fired
|
||||
}
|
||||
let pos = cold.monitors.iter().position(|(mid, _)| *mid == m.id)?;
|
||||
Some(cold.monitors.remove(pos))
|
||||
inner.with_shared(|s| {
|
||||
let slot = s.slot_mut(m.target)?;
|
||||
let pos = slot.monitors.iter().position(|(mid, _)| *mid == m.id)?;
|
||||
Some(slot.monitors.remove(pos))
|
||||
})
|
||||
});
|
||||
// `removed`'s sender drops here, outside the lock.
|
||||
removed.map(|(id, _sender)| id)
|
||||
|
||||
+18
-24
@@ -33,15 +33,13 @@ impl std::error::Error for LockTimeout {}
|
||||
|
||||
struct Wait {
|
||||
pid: Pid,
|
||||
/// The wait's park-epoch (slot-word wait identity, see slot_state.rs).
|
||||
/// Grants and timeouts wake via `unpark_at(pid, epoch)`; a stale entry
|
||||
/// can neither be granted by mistake nor wake the wrong wait.
|
||||
epoch: u32,
|
||||
seq: u64,
|
||||
}
|
||||
|
||||
struct MutexState {
|
||||
holder: Option<Pid>,
|
||||
waiters: VecDeque<Wait>,
|
||||
next_seq: u64,
|
||||
default_timeout: Duration,
|
||||
}
|
||||
|
||||
@@ -55,6 +53,7 @@ impl MutexCore {
|
||||
state: StdMutex::new(MutexState {
|
||||
holder: None,
|
||||
waiters: VecDeque::new(),
|
||||
next_seq: 0,
|
||||
default_timeout,
|
||||
}),
|
||||
}
|
||||
@@ -62,17 +61,17 @@ impl MutexCore {
|
||||
}
|
||||
|
||||
impl TimerTarget for MutexCore {
|
||||
fn on_timeout(&self, pid: Pid, epoch: u32) {
|
||||
fn on_timeout(&self, pid: Pid, wait_seq: u64) {
|
||||
let unpark = {
|
||||
let mut st = self.state.lock().unwrap();
|
||||
// Remove from waiters only if still there with matching epoch.
|
||||
// Remove from waiters only if still there with matching seq.
|
||||
// If the lock was already granted (holder == Some(pid)), the
|
||||
// timer fired after the grant — treat as no-op; the actor
|
||||
// will see `is_holder == true` and return Ok.
|
||||
if st.holder == Some(pid) {
|
||||
return;
|
||||
}
|
||||
let pos = st.waiters.iter().position(|w| w.pid == pid && w.epoch == epoch);
|
||||
let pos = st.waiters.iter().position(|w| w.pid == pid && w.seq == wait_seq);
|
||||
if pos.is_some() {
|
||||
st.waiters.remove(pos.unwrap());
|
||||
true
|
||||
@@ -81,7 +80,7 @@ impl TimerTarget for MutexCore {
|
||||
}
|
||||
};
|
||||
if unpark {
|
||||
scheduler::unpark_at(pid, epoch);
|
||||
scheduler::unpark(pid);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -134,25 +133,20 @@ impl<T> Mutex<T> {
|
||||
|
||||
// Slow path: register as a waiter, set timeout, park.
|
||||
let _np = scheduler::NoPreempt::enter();
|
||||
let epoch = {
|
||||
let seq = {
|
||||
let mut st = self.core.state.lock().unwrap();
|
||||
// begin_wait is lock-free — legal under the state lock; this
|
||||
// makes the epoch atomic with the registration's visibility to
|
||||
// grants and timeouts.
|
||||
let epoch = scheduler::begin_wait();
|
||||
st.waiters.push_back(Wait { pid: me, epoch });
|
||||
epoch
|
||||
let seq = st.next_seq;
|
||||
st.next_seq = st.next_seq.wrapping_add(1);
|
||||
st.waiters.push_back(Wait { pid: me, seq });
|
||||
seq
|
||||
};
|
||||
|
||||
let target: Arc<dyn TimerTarget> = self.core.clone();
|
||||
let deadline = timer::deadline_from_now(timeout);
|
||||
scheduler::insert_wait_timer(deadline, me, target, epoch);
|
||||
scheduler::insert_wait_timer(deadline, me, target, seq);
|
||||
scheduler::park_current();
|
||||
|
||||
// Resumed — precisely: only our grant or our timer can wake this
|
||||
// wait (both epoch-stamped; a stop wake unwinds out of
|
||||
// park_current). The one-shot interpretation below is therefore
|
||||
// exhaustive. Are we the holder?
|
||||
// Resumed. Are we the holder?
|
||||
let is_holder = self.core.state.lock().unwrap().holder == Some(me);
|
||||
if is_holder {
|
||||
let value = self.value.lock().unwrap().take()
|
||||
@@ -234,12 +228,12 @@ impl<T> Drop for MutexGuard<'_, T> {
|
||||
let v = self.value.take().expect("MutexGuard: double drop");
|
||||
*self.mutex.value.lock().unwrap() = Some(v);
|
||||
|
||||
let next = {
|
||||
let next_pid = {
|
||||
let mut st = self.mutex.core.state.lock().unwrap();
|
||||
match st.waiters.pop_front() {
|
||||
Some(w) => {
|
||||
st.holder = Some(w.pid);
|
||||
Some((w.pid, w.epoch))
|
||||
Some(w.pid)
|
||||
}
|
||||
None => {
|
||||
st.holder = None;
|
||||
@@ -247,8 +241,8 @@ impl<T> Drop for MutexGuard<'_, T> {
|
||||
}
|
||||
}
|
||||
};
|
||||
if let Some((pid, epoch)) = next {
|
||||
scheduler::unpark_at(pid, epoch);
|
||||
if let Some(pid) = next_pid {
|
||||
scheduler::unpark(pid);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+18
-21
@@ -30,7 +30,17 @@ use std::cell::Cell;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
pub const DEFAULT_ALLOC_INTERVAL: u32 = 128;
|
||||
|
||||
// The timeslice is measured in `rdtsc()` ticks, and the tick *unit* differs by
|
||||
// ISA (see `crate::arch::read_cycle_counter`). Both constants target ~100µs.
|
||||
// x86-64 : TSC ≈ CPU base clock; 300_000 ticks ≈ 100µs on a 3 GHz core.
|
||||
// aarch64: CNTVCT runs at CNTFRQ_EL0, commonly ~24 MHz; 2_400 ticks ≈ 100µs.
|
||||
// A real implementation would read the frequency at startup and compute this;
|
||||
// for now the constant is calibrated per-arch. Tune on-device if needed.
|
||||
#[cfg(target_arch = "x86_64")]
|
||||
pub const DEFAULT_TIMESLICE_CYCLES: u64 = 300_000; // ≈ 100µs on a 3 GHz CPU
|
||||
#[cfg(target_arch = "aarch64")]
|
||||
pub const DEFAULT_TIMESLICE_CYCLES: u64 = 2_400; // ≈ 100µs at a 24 MHz CNTVCT
|
||||
|
||||
thread_local! {
|
||||
/// While `false`, the allocator hook is a no-op.
|
||||
@@ -116,14 +126,12 @@ pub fn reset_timeslice() {
|
||||
TIMESLICE_START.with(|c| c.set(rdtsc()));
|
||||
}
|
||||
|
||||
/// Per-core cycle counter for the timeslice clock. Delegates to the active
|
||||
/// arch backend (`rdtsc` on x86-64, `CNTVCT_EL0` on aarch64). The tick unit
|
||||
/// is ISA-defined, so `DEFAULT_TIMESLICE_CYCLES` is calibrated per-arch below.
|
||||
#[inline(always)]
|
||||
pub fn rdtsc() -> u64 {
|
||||
unsafe {
|
||||
// SAFETY: x86-64 only. `lfence` serialises the instruction stream so
|
||||
// we don't measure time before prior instructions retire.
|
||||
core::arch::asm!("lfence", options(nostack, nomem, preserves_flags));
|
||||
core::arch::x86_64::_rdtsc()
|
||||
}
|
||||
crate::arch::read_cycle_counter()
|
||||
}
|
||||
|
||||
pub struct PreemptingAllocator;
|
||||
@@ -171,22 +179,11 @@ pub fn maybe_preempt() {
|
||||
let n = c.get();
|
||||
if n == 0 {
|
||||
c.set(CONFIGURED_ALLOC_INTERVAL.with(|i| i.get()));
|
||||
// Cooperative cancellation shares the amortised cadence with the
|
||||
// timeslice check. Observe a pending stop first: if we are being
|
||||
// cancelled there is no point yielding, we unwind instead.
|
||||
check_cancelled();
|
||||
if PREEMPTION_ENABLED.with(|e| e.get()) {
|
||||
// Cooperative cancellation shares the amortised cadence with
|
||||
// the timeslice check, and shares its gate: while preemption
|
||||
// is disabled (`NoPreempt`, `with_shared`, channel critical
|
||||
// sections) the stop sentinel must NOT be raised, because an
|
||||
// allocation-triggered unwind inside a region holding a
|
||||
// `std::sync::Mutex` would poison it — one `request_stop` at
|
||||
// the wrong moment would then cascade `lock().unwrap()`
|
||||
// panics through every later user of that lock. Observation
|
||||
// is merely deferred to the next enabled allocation or the
|
||||
// wakeup side of the next park/yield, both of which are
|
||||
// lock-free points by construction.
|
||||
//
|
||||
// Observe a pending stop first: if we are being cancelled
|
||||
// there is no point yielding, we unwind instead.
|
||||
check_cancelled();
|
||||
let start = TIMESLICE_START.with(|s| s.get());
|
||||
if rdtsc().saturating_sub(start) > CONFIGURED_TIMESLICE_CYCLES.with(|t| t.get()) {
|
||||
// SAFETY: reachable only inside an actor (the scheduler
|
||||
|
||||
@@ -1,360 +0,0 @@
|
||||
//! A minimal futex-based mutex that cannot poison.
|
||||
//!
|
||||
//! `std::sync::Mutex` poisons on unwind, turning one panic into a cascade of
|
||||
//! `lock().unwrap()` panics in every later user. The runtime's internal
|
||||
//! critical sections must never unwind anyway (the stop sentinel is gated
|
||||
//! behind `PREEMPTION_ENABLED`, and the guard below disables preemption), so
|
||||
//! poisoning buys nothing and costs a failure mode. This mutex has no poison
|
||||
//! state by construction.
|
||||
//!
|
||||
//! Two further properties the runtime wants:
|
||||
//!
|
||||
//! - **The guard enters `NoPreempt`.** A timeslice switch while holding an OS
|
||||
//! mutex would suspend the actor with the lock held, stalling every other
|
||||
//! OS thread that touches it until the actor is resumed. Disabling
|
||||
//! preemption for the (short) critical section keeps lock hold times
|
||||
//! bounded. It also closes the unwind hole structurally: with
|
||||
//! `PREEMPTION_ENABLED` false, `maybe_preempt` neither yields nor raises
|
||||
//! the stop sentinel, so no allocation inside the critical section can
|
||||
//! unwind it.
|
||||
//! - **No std machinery.** One `AtomicU32` and two futex syscalls; friendlier
|
||||
//! to an eventual embedded port than `std::sync::Mutex` (swap the futex for
|
||||
//! a spin or WFE backend).
|
||||
//!
|
||||
//! Algorithm: the classic three-state futex mutex (Drepper, "Futexes Are
|
||||
//! Tricky", mutex3). 0 = unlocked, 1 = locked, 2 = locked with (possible)
|
||||
//! waiters. Uncontended lock/unlock is one CAS / one swap, no syscall.
|
||||
//!
|
||||
//! Lock-order position — two classes (see [`LockClass`]):
|
||||
//!
|
||||
//! - **Leaf**: slot cold locks, the free list, the stack pool, the name
|
||||
//! registry. Mutual leaves — never hold two at once.
|
||||
//! - **Channel**: a channel's internal lock. May be acquired *under* a Leaf
|
||||
//! (finalize clones the supervisor/trap senders, and `monitor()` clones the
|
||||
//! Down sender, all under a cold lock — structural, the sender lives in the
|
||||
//! slot), but nothing may be acquired under a Channel lock: channel
|
||||
//! critical sections call only the lock-free unpark protocol.
|
||||
//!
|
||||
//! So the total order is Leaf → Channel, one of each at most. Holding either
|
||||
//! while pushing to the run queue is permitted (unpark from inside a cold or
|
||||
//! channel section); the reverse — taking any `RawMutex` from inside a
|
||||
//! run-queue op — cannot arise (queue ops call nothing).
|
||||
|
||||
use std::cell::UnsafeCell;
|
||||
use std::ops::{Deref, DerefMut};
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
|
||||
const UNLOCKED: u32 = 0;
|
||||
const LOCKED: u32 = 1;
|
||||
const CONTENDED: u32 = 2;
|
||||
|
||||
/// How many `pause` spins to burn before falling back to the futex. Critical
|
||||
/// sections under this lock are tens of nanoseconds (push to a Vec, clone a
|
||||
/// sender), so a short spin almost always avoids the syscall.
|
||||
const SPIN_LIMIT: u32 = 64;
|
||||
|
||||
/// Which rung of the two-rung lock order a `RawMutex` occupies. Debug builds
|
||||
/// enforce the order mechanically (see the module docs); release builds carry
|
||||
/// no state and no checks.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(crate) enum LockClass {
|
||||
/// Runtime cold data: slot cold locks, free list, stack pool, registry.
|
||||
/// Mutual leaves among themselves; a Channel lock may be taken under one.
|
||||
Leaf,
|
||||
/// A channel's internal lock. One at a time, nothing acquired under it;
|
||||
/// may itself be acquired under a Leaf.
|
||||
Channel,
|
||||
}
|
||||
|
||||
// The ordering rules, mechanically enforced (debug builds): a deadlock from a
|
||||
// violated order is a hang waiting for the right interleaving, so fail at the
|
||||
// acquisition that violates it, not in the eventual hang.
|
||||
#[cfg(debug_assertions)]
|
||||
thread_local! {
|
||||
static LEAVES_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
|
||||
static CHANNELS_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn order_check_acquire(class: LockClass) {
|
||||
#[cfg(debug_assertions)]
|
||||
match class {
|
||||
LockClass::Leaf => LEAVES_HELD.with(|l| {
|
||||
debug_assert_eq!(
|
||||
l.get(),
|
||||
0,
|
||||
"lock order violated: acquiring a Leaf RawMutex while already \
|
||||
holding one (cold locks / free list / stack pool / registry \
|
||||
are mutual leaves)"
|
||||
);
|
||||
CHANNELS_HELD.with(|c| {
|
||||
debug_assert_eq!(
|
||||
c.get(),
|
||||
0,
|
||||
"lock order violated: acquiring a Leaf RawMutex under a \
|
||||
channel lock (order is Leaf -> Channel, never the reverse)"
|
||||
);
|
||||
});
|
||||
l.set(l.get() + 1);
|
||||
}),
|
||||
LockClass::Channel => CHANNELS_HELD.with(|c| {
|
||||
debug_assert_eq!(
|
||||
c.get(),
|
||||
0,
|
||||
"lock order violated: acquiring a channel lock while already \
|
||||
holding one (channel locks are mutual leaves)"
|
||||
);
|
||||
c.set(c.get() + 1);
|
||||
}),
|
||||
}
|
||||
#[cfg(not(debug_assertions))]
|
||||
let _ = class;
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn order_check_release(class: LockClass) {
|
||||
#[cfg(debug_assertions)]
|
||||
match class {
|
||||
LockClass::Leaf => LEAVES_HELD.with(|c| c.set(c.get() - 1)),
|
||||
LockClass::Channel => CHANNELS_HELD.with(|c| c.set(c.get() - 1)),
|
||||
}
|
||||
#[cfg(not(debug_assertions))]
|
||||
let _ = class;
|
||||
}
|
||||
|
||||
pub(crate) struct RawMutex<T> {
|
||||
state: AtomicU32,
|
||||
class: LockClass,
|
||||
data: UnsafeCell<T>,
|
||||
}
|
||||
|
||||
// SAFETY: standard mutex argument — exclusive access to `data` is mediated by
|
||||
// `state`; `T: Send` suffices for both because `&RawMutex` only ever hands out
|
||||
// access to one thread at a time.
|
||||
unsafe impl<T: Send> Send for RawMutex<T> {}
|
||||
unsafe impl<T: Send> Sync for RawMutex<T> {}
|
||||
|
||||
impl<T> RawMutex<T> {
|
||||
/// A Leaf-class mutex — the default for runtime cold data.
|
||||
pub(crate) const fn new(data: T) -> Self {
|
||||
Self::with_class(data, LockClass::Leaf)
|
||||
}
|
||||
|
||||
/// A Channel-class mutex — for channel internals only.
|
||||
pub(crate) const fn new_channel(data: T) -> Self {
|
||||
Self::with_class(data, LockClass::Channel)
|
||||
}
|
||||
|
||||
pub(crate) const fn with_class(data: T, class: LockClass) -> Self {
|
||||
Self {
|
||||
state: AtomicU32::new(UNLOCKED),
|
||||
class,
|
||||
data: UnsafeCell::new(data),
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub(crate) fn lock(&self) -> RawMutexGuard<'_, T> {
|
||||
// Enter NoPreempt *before* acquiring, so a preemption can't fire
|
||||
// between acquisition and guard construction.
|
||||
let prev_preempt = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
|
||||
order_check_acquire(self.class);
|
||||
if self
|
||||
.state
|
||||
.compare_exchange(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
|
||||
.is_err()
|
||||
{
|
||||
self.lock_slow();
|
||||
}
|
||||
RawMutexGuard { m: self, prev_preempt }
|
||||
}
|
||||
|
||||
#[cold]
|
||||
fn lock_slow(&self) {
|
||||
// Bounded spin first: the expected hold time is far below the cost of
|
||||
// a futex round trip.
|
||||
let mut spins = 0;
|
||||
loop {
|
||||
let s = self.state.load(Ordering::Relaxed);
|
||||
if s == UNLOCKED
|
||||
&& self
|
||||
.state
|
||||
.compare_exchange_weak(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
|
||||
.is_ok()
|
||||
{
|
||||
return;
|
||||
}
|
||||
spins += 1;
|
||||
if spins >= SPIN_LIMIT {
|
||||
break;
|
||||
}
|
||||
std::hint::spin_loop();
|
||||
}
|
||||
// Futex path. Mark contended and sleep until woken; on wake, retake
|
||||
// by swapping to CONTENDED (we cannot know whether other waiters
|
||||
// remain, so we must conservatively keep the contended marker).
|
||||
while self.state.swap(CONTENDED, Ordering::Acquire) != UNLOCKED {
|
||||
futex_wait(&self.state, CONTENDED);
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn unlock(&self) {
|
||||
if self.state.swap(UNLOCKED, Ordering::Release) == CONTENDED {
|
||||
futex_wake(&self.state, 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) struct RawMutexGuard<'a, T> {
|
||||
m: &'a RawMutex<T>,
|
||||
prev_preempt: bool,
|
||||
}
|
||||
|
||||
impl<T> Deref for RawMutexGuard<'_, T> {
|
||||
type Target = T;
|
||||
#[inline]
|
||||
fn deref(&self) -> &T {
|
||||
// SAFETY: guard existence implies exclusive ownership of the lock.
|
||||
unsafe { &*self.m.data.get() }
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> DerefMut for RawMutexGuard<'_, T> {
|
||||
#[inline]
|
||||
fn deref_mut(&mut self) -> &mut T {
|
||||
// SAFETY: as above, plus &mut self.
|
||||
unsafe { &mut *self.m.data.get() }
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> Drop for RawMutexGuard<'_, T> {
|
||||
#[inline]
|
||||
fn drop(&mut self) {
|
||||
self.m.unlock();
|
||||
order_check_release(self.m.class);
|
||||
// Restore preemption only after the lock is released.
|
||||
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(self.prev_preempt));
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// futex (x86-64 Linux; master is x86-only, see arm-port branch)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub(crate) fn futex_wait(state: &AtomicU32, expected: u32) {
|
||||
// SAFETY: `state` is a valid, aligned u32 for the duration of the call.
|
||||
// Spurious wakeups and EAGAIN (value already changed) are both handled by
|
||||
// the caller's retry loop.
|
||||
unsafe {
|
||||
libc::syscall(
|
||||
libc::SYS_futex,
|
||||
state.as_ptr(),
|
||||
libc::FUTEX_WAIT | libc::FUTEX_PRIVATE_FLAG,
|
||||
expected,
|
||||
std::ptr::null::<libc::timespec>(),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn futex_wake(state: &AtomicU32, n: i32) {
|
||||
// SAFETY: as above.
|
||||
unsafe {
|
||||
libc::syscall(
|
||||
libc::SYS_futex,
|
||||
state.as_ptr(),
|
||||
libc::FUTEX_WAKE | libc::FUTEX_PRIVATE_FLAG,
|
||||
n,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::Arc;
|
||||
|
||||
#[test]
|
||||
fn uncontended_lock_unlock() {
|
||||
let m = RawMutex::new(0u64);
|
||||
for _ in 0..1000 {
|
||||
*m.lock() += 1;
|
||||
}
|
||||
assert_eq!(*m.lock(), 1000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn contended_counter_is_exact() {
|
||||
const THREADS: usize = 8;
|
||||
const PER: u64 = 50_000;
|
||||
let m = Arc::new(RawMutex::new(0u64));
|
||||
let hs: Vec<_> = (0..THREADS)
|
||||
.map(|_| {
|
||||
let m = m.clone();
|
||||
std::thread::spawn(move || {
|
||||
for _ in 0..PER {
|
||||
*m.lock() += 1;
|
||||
}
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
assert_eq!(*m.lock(), THREADS as u64 * PER);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_poison_on_unwind() {
|
||||
let m = Arc::new(RawMutex::new(0u64));
|
||||
let m2 = m.clone();
|
||||
let _ = std::thread::spawn(move || {
|
||||
let _g = m2.lock();
|
||||
panic!("unwind while holding");
|
||||
})
|
||||
.join();
|
||||
// A std Mutex would now be poisoned; this one just works.
|
||||
*m.lock() += 1;
|
||||
assert_eq!(*m.lock(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn channel_lock_nests_under_leaf() {
|
||||
// The permitted ordering: Leaf -> Channel (finalize/monitor clone a
|
||||
// sender under a cold lock). Must not trip the order check.
|
||||
let leaf = RawMutex::new(0u64);
|
||||
let chan = RawMutex::new_channel(0u64);
|
||||
let _l = leaf.lock();
|
||||
let _c = chan.lock();
|
||||
}
|
||||
|
||||
#[cfg(debug_assertions)]
|
||||
#[test]
|
||||
#[should_panic(expected = "lock order violated")]
|
||||
fn leaf_under_channel_is_rejected() {
|
||||
let leaf = RawMutex::new(0u64);
|
||||
let chan = RawMutex::new_channel(0u64);
|
||||
let _c = chan.lock();
|
||||
let _l = leaf.lock(); // Channel -> Leaf: forbidden
|
||||
}
|
||||
|
||||
#[cfg(debug_assertions)]
|
||||
#[test]
|
||||
#[should_panic(expected = "lock order violated")]
|
||||
fn two_leaves_are_rejected() {
|
||||
let a = RawMutex::new(0u64);
|
||||
let b = RawMutex::new(0u64);
|
||||
let _ga = a.lock();
|
||||
let _gb = b.lock(); // leaves are mutual: forbidden
|
||||
}
|
||||
|
||||
#[cfg(debug_assertions)]
|
||||
#[test]
|
||||
#[should_panic(expected = "lock order violated")]
|
||||
fn two_channel_locks_are_rejected() {
|
||||
let a = RawMutex::new_channel(0u64);
|
||||
let b = RawMutex::new_channel(0u64);
|
||||
let _ga = a.lock();
|
||||
let _gb = b.lock(); // channel locks are mutual leaves: forbidden
|
||||
}
|
||||
}
|
||||
-169
@@ -1,169 +0,0 @@
|
||||
//! Named pid registry.
|
||||
//!
|
||||
//! `register(name, pid)` binds a name to a live actor; `whereis(name)` looks
|
||||
//! it up; `name_of(pid)` is the inverse. Erlang semantics: at most one name
|
||||
//! per pid, at most one pid per name, registering over a live binding is an
|
||||
//! error. The registry is a *bimap* (two `HashMap`s kept in exact inverse
|
||||
//! under one lock) so both lookup directions are O(1) — `name_of` exists so
|
||||
//! supervisors, tracing, and diagnostics can label a pid without scanning.
|
||||
//!
|
||||
//! ## Cleanup is lazy
|
||||
//!
|
||||
//! There is no hook in `finalize_actor` and no name field in `Slot` (which
|
||||
//! would buy into the reset-in-three-places slot invariant). Instead every
|
||||
//! operation that touches a binding checks the bound pid's liveness via the
|
||||
//! generation-checked slot word — a `Pid` is `(index, generation)` and a
|
||||
//! generation is never reused, so a stale binding is *detectable*, never
|
||||
//! misdirected. Bindings to dead actors behave as absent and are pruned on
|
||||
//! contact: `whereis`/`name_of` return `None`, `register` treats the name as
|
||||
//! free. The cost is that a dead binding lingers until something looks at it;
|
||||
//! the payoff is zero coupling to the actor lifecycle.
|
||||
//!
|
||||
//! ## Locking
|
||||
//!
|
||||
//! One `RawMutex` (Leaf class) in `RuntimeInner`. Liveness checks under it
|
||||
//! read only the atomic slot state word — no second lock is ever taken, so
|
||||
//! the leaf rule holds trivially.
|
||||
|
||||
use crate::pid::Pid;
|
||||
use crate::scheduler::with_runtime;
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Why a [`register`] call was rejected.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum RegisterError {
|
||||
/// The name is bound to a different, still-live actor.
|
||||
NameTaken { holder: Pid },
|
||||
/// The pid already has a (live) registered name; one name per pid.
|
||||
PidAlreadyRegistered { name: String },
|
||||
/// The pid does not refer to a live actor.
|
||||
NoProc,
|
||||
}
|
||||
|
||||
impl std::fmt::Display for RegisterError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
RegisterError::NameTaken { holder } => {
|
||||
write!(f, "name is already registered to live actor {holder}")
|
||||
}
|
||||
RegisterError::PidAlreadyRegistered { name } => {
|
||||
write!(f, "pid is already registered as {name:?}")
|
||||
}
|
||||
RegisterError::NoProc => write!(f, "pid does not refer to a live actor"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for RegisterError {}
|
||||
|
||||
/// The bimap. Invariant (held under the registry lock): `by_name` and
|
||||
/// `by_pid` are exact inverses of each other at all times.
|
||||
pub(crate) struct Registry {
|
||||
by_name: HashMap<String, Pid>,
|
||||
by_pid: HashMap<Pid, String>,
|
||||
}
|
||||
|
||||
impl Registry {
|
||||
pub(crate) fn new() -> Self {
|
||||
Self { by_name: HashMap::new(), by_pid: HashMap::new() }
|
||||
}
|
||||
|
||||
fn remove_binding(&mut self, name: &str, pid: Pid) {
|
||||
self.by_name.remove(name);
|
||||
self.by_pid.remove(&pid);
|
||||
debug_assert_eq!(self.by_name.len(), self.by_pid.len(), "bimap out of sync");
|
||||
}
|
||||
|
||||
fn insert_binding(&mut self, name: String, pid: Pid) {
|
||||
self.by_pid.insert(pid, name.clone());
|
||||
self.by_name.insert(name, pid);
|
||||
debug_assert_eq!(self.by_name.len(), self.by_pid.len(), "bimap out of sync");
|
||||
}
|
||||
}
|
||||
|
||||
/// Is `pid` a live actor right now? Atomic slot-word read; no lock.
|
||||
fn live(inner: &crate::runtime::RuntimeInner, pid: Pid) -> bool {
|
||||
inner.slot_at(pid).is_some_and(|s| s.is_live_for(pid))
|
||||
}
|
||||
|
||||
/// Bind `name` to the live actor `pid`.
|
||||
///
|
||||
/// Fails with [`RegisterError::NoProc`] if `pid` is not live,
|
||||
/// [`RegisterError::NameTaken`] if the name is bound to a *live* actor
|
||||
/// (a binding to a dead actor is evicted and the name treated as free), and
|
||||
/// [`RegisterError::PidAlreadyRegistered`] if `pid` already has a name.
|
||||
///
|
||||
/// Panics if called outside `Runtime::run()`.
|
||||
pub fn register(name: impl Into<String>, pid: Pid) -> Result<(), RegisterError> {
|
||||
let name = name.into();
|
||||
with_runtime(|inner| {
|
||||
let mut reg = inner.registry.lock();
|
||||
if !live(inner, pid) {
|
||||
return Err(RegisterError::NoProc);
|
||||
}
|
||||
let prior = reg.by_name.get(&name).copied();
|
||||
if let Some(holder) = prior {
|
||||
if holder == pid {
|
||||
return Ok(()); // already exactly this binding: idempotent
|
||||
}
|
||||
if live(inner, holder) {
|
||||
return Err(RegisterError::NameTaken { holder });
|
||||
}
|
||||
// Stale binding: the holder died. Evict and treat the name as free.
|
||||
reg.remove_binding(&name, holder);
|
||||
}
|
||||
if let Some(existing) = reg.by_pid.get(&pid) {
|
||||
// `pid` is live (checked above) and pids are never reused, so this
|
||||
// binding is necessarily current — one name per pid.
|
||||
return Err(RegisterError::PidAlreadyRegistered { name: existing.clone() });
|
||||
}
|
||||
reg.insert_binding(name, pid);
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
/// Look up the pid bound to `name`. `None` if unbound, or if the bound actor
|
||||
/// is no longer live (the stale binding is pruned on the way out).
|
||||
pub fn whereis(name: &str) -> Option<Pid> {
|
||||
with_runtime(|inner| {
|
||||
let mut reg = inner.registry.lock();
|
||||
let pid = *reg.by_name.get(name)?;
|
||||
if live(inner, pid) {
|
||||
Some(pid)
|
||||
} else {
|
||||
reg.remove_binding(name, pid);
|
||||
None
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// The inverse lookup: the name `pid` is registered under, if any. `None` if
|
||||
/// unregistered or no longer live (pruning the stale binding).
|
||||
pub fn name_of(pid: Pid) -> Option<String> {
|
||||
with_runtime(|inner| {
|
||||
let mut reg = inner.registry.lock();
|
||||
let name = reg.by_pid.get(&pid)?.clone();
|
||||
if live(inner, pid) {
|
||||
Some(name)
|
||||
} else {
|
||||
reg.remove_binding(&name, pid);
|
||||
None
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Remove the binding for `name`, returning the pid it was bound to if that
|
||||
/// actor is still live. A binding to a dead actor is pruned and reported as
|
||||
/// `None`, consistent with [`whereis`].
|
||||
pub fn unregister(name: &str) -> Option<Pid> {
|
||||
with_runtime(|inner| {
|
||||
let mut reg = inner.registry.lock();
|
||||
let pid = *reg.by_name.get(name)?;
|
||||
reg.remove_binding(name, pid);
|
||||
if live(inner, pid) {
|
||||
Some(pid)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -1,562 +0,0 @@
|
||||
//! The run queue, selected at COMPILE TIME by mutually-exclusive cargo
|
||||
//! features (no runtime dispatch — the scheduler's pop loop is the hottest
|
||||
//! code in the runtime):
|
||||
//!
|
||||
//! - `rq-mutex` (default) — `Mutex<VecDeque>`. The control/baseline:
|
||||
//! strictly FIFO, trivially correct, one global lock.
|
||||
//! - `rq-mpmc` — a single hand-rolled Vyukov bounded MPMC ring (per-cell
|
||||
//! sequence numbers). Strict FIFO, lock-free, one hot
|
||||
//! enqueue/dequeue cache-line pair.
|
||||
//! - `rq-striped` — M Vyukov rings with fetch-add ticket distribution.
|
||||
//! *Relaxed* FIFO: ordering across stripes is bounded-skewed
|
||||
//! (≈ one ring's worth of reordering per stripe), in exchange
|
||||
//! for spreading the hot line M ways. Predicted winner at
|
||||
//! high core counts; phase 4's shootout decides.
|
||||
//!
|
||||
//! Select non-default variants with `--no-default-features --features rq-…`
|
||||
//! (cargo features are additive, so the default must be switched off).
|
||||
//!
|
||||
//! All variants are compiled unconditionally (so every build runs every
|
||||
//! variant's unit tests); the feature only picks which one the runtime uses
|
||||
//! via the [`RunQueue`] alias.
|
||||
//!
|
||||
//! # Contract (shared by all variants)
|
||||
//!
|
||||
//! - **Occupancy is bounded by `max_actors`.** A pid is in the queue at most
|
||||
//! once (pushes pair 1:1 with transitions into `Queued`; only the
|
||||
//! scheduler transitions `Queued → Running` — see the state-machine docs
|
||||
//! in `runtime.rs`), and at most `max_actors` actors exist. With the RFC
|
||||
//! 005 wake slot enabled the invariant reads "in (slot ⊕ shared queue) at
|
||||
//! most once" — a slot push *replaces* the queue push at the same
|
||||
//! protocol point, and a displacement moves the occupant, never copies
|
||||
//! it — so the bound holds verbatim. The bounded rings are sized ≥
|
||||
//! `max_actors`, so **`push` is infallible**; a full
|
||||
//! ring is an invariant violation and panics loudly rather than spinning.
|
||||
//! - **Preemption must be disabled around every push/pop** (debug-asserted).
|
||||
//! For the mutex variant this is the usual no-switch/no-unwind-under-lock
|
||||
//! rule. For the rings it is *load-bearing in a sharper way*: a producer
|
||||
//! suspended between claiming a cell and publishing its sequence number
|
||||
//! stalls every consumer behind that cell — on a busy runtime that is a
|
||||
//! livelock, since the suspended actor's own resume entry sits behind the
|
||||
//! hole. Callers get this for free: every queue op happens inside
|
||||
//! `with_runtime`/`try_with_runtime` (NoPreempt for their span since
|
||||
//! phase 2) or on a scheduler thread between resumes (preemption off).
|
||||
//! - **`pop() == None` is a snapshot, not a fence.** A push that is mid-
|
||||
//! publish (or in a stripe the probe already passed) may be missed; the
|
||||
//! caller's idle path sleeps ≤ 100µs and retries, so the cost is a bounded
|
||||
//! latency blip, never a lost entry. Termination does not lean on this:
|
||||
//! the all-clear is `live_actors == 0` (+ io quiescent), and `live == 0`
|
||||
//! already implies the queue holds nothing actionable — see the argument
|
||||
//! in `schedule_loop`.
|
||||
//! - `len()` is approximate (stats only).
|
||||
|
||||
use crate::pid::Pid;
|
||||
use crate::sync_shim::{AtomicUsize, Ordering, UnsafeCell};
|
||||
use std::mem::MaybeUninit;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Feature selection
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[cfg(not(any(feature = "rq-mutex", feature = "rq-mpmc", feature = "rq-striped")))]
|
||||
compile_error!(
|
||||
"smarm: no run queue selected. Enable exactly one of the features \
|
||||
`rq-mutex` (default), `rq-mpmc`, `rq-striped`."
|
||||
);
|
||||
#[cfg(all(feature = "rq-mutex", feature = "rq-mpmc"))]
|
||||
compile_error!(
|
||||
"smarm: features `rq-mutex` and `rq-mpmc` are mutually exclusive \
|
||||
(use --no-default-features to drop the default `rq-mutex`)."
|
||||
);
|
||||
#[cfg(all(feature = "rq-mutex", feature = "rq-striped"))]
|
||||
compile_error!(
|
||||
"smarm: features `rq-mutex` and `rq-striped` are mutually exclusive \
|
||||
(use --no-default-features to drop the default `rq-mutex`)."
|
||||
);
|
||||
#[cfg(all(feature = "rq-mpmc", feature = "rq-striped"))]
|
||||
compile_error!("smarm: features `rq-mpmc` and `rq-striped` are mutually exclusive.");
|
||||
|
||||
#[cfg(feature = "rq-mutex")]
|
||||
pub(crate) type RunQueue = MutexQueue;
|
||||
#[cfg(feature = "rq-mpmc")]
|
||||
pub(crate) type RunQueue = MpmcRing;
|
||||
#[cfg(feature = "rq-striped")]
|
||||
pub(crate) type RunQueue = StripedRing;
|
||||
|
||||
#[inline]
|
||||
fn assert_no_preempt() {
|
||||
debug_assert!(
|
||||
!crate::preempt::PREEMPTION_ENABLED.with(|c| c.get()),
|
||||
"run-queue op with preemption enabled — a switch mid-op stalls or \
|
||||
corrupts the queue; route through with_runtime or scheduler context"
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// rq-mutex — the baseline
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub struct MutexQueue {
|
||||
q: std::sync::Mutex<std::collections::VecDeque<Pid>>,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
impl MutexQueue {
|
||||
pub fn new(_threads: usize, max_actors: usize) -> Self {
|
||||
Self {
|
||||
// Pre-size: the queue can never outgrow the slab, and one
|
||||
// allocation at init beats reallocating under the lock later.
|
||||
q: std::sync::Mutex::new(std::collections::VecDeque::with_capacity(max_actors)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn push(&self, pid: Pid) {
|
||||
assert_no_preempt();
|
||||
self.q.lock().unwrap().push_back(pid);
|
||||
}
|
||||
|
||||
pub fn pop(&self) -> Option<Pid> {
|
||||
assert_no_preempt();
|
||||
self.q.lock().unwrap().pop_front()
|
||||
}
|
||||
|
||||
pub fn len(&self) -> u64 {
|
||||
self.q.lock().unwrap().len() as u64
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// rq-mpmc — Vyukov bounded MPMC ring
|
||||
// ---------------------------------------------------------------------------
|
||||
//
|
||||
// Dmitry Vyukov's bounded MPMC queue: each cell carries a sequence number.
|
||||
// A producer may write cell `i` when `seq == pos` (its turn); it publishes
|
||||
// with `seq = pos + 1`. A consumer may read when `seq == pos + 1`; it
|
||||
// releases the cell to the next lap with `seq = pos + capacity`. Producers
|
||||
// and consumers each contend on one counter; cell handoff is a per-cell
|
||||
// Acquire/Release pair, so unrelated push/pop pairs don't serialize.
|
||||
|
||||
/// Pad to a cache-line pair so the producer and consumer counters (and the
|
||||
/// cells) don't false-share.
|
||||
#[repr(align(128))]
|
||||
struct CachePadded<T>(T);
|
||||
|
||||
struct Cell {
|
||||
seq: AtomicUsize,
|
||||
pid: UnsafeCell<MaybeUninit<Pid>>,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub struct MpmcRing {
|
||||
buf: Box<[Cell]>,
|
||||
mask: usize,
|
||||
enqueue_pos: CachePadded<AtomicUsize>,
|
||||
dequeue_pos: CachePadded<AtomicUsize>,
|
||||
}
|
||||
|
||||
// SAFETY: cells are handed off between threads via the per-cell seq
|
||||
// (Release on publish, Acquire on claim); Pid is Copy + Send.
|
||||
unsafe impl Send for MpmcRing {}
|
||||
unsafe impl Sync for MpmcRing {}
|
||||
|
||||
#[allow(dead_code)]
|
||||
impl MpmcRing {
|
||||
pub fn new(_threads: usize, max_actors: usize) -> Self {
|
||||
Self::with_capacity(max_actors)
|
||||
}
|
||||
|
||||
/// Pub for the raw-structure microbench (sized to the op count so the
|
||||
/// occupancy contract is trivially met there). Runtime code uses `new`.
|
||||
pub fn with_capacity(min_cap: usize) -> Self {
|
||||
// Occupancy ≤ max_actors (queue contract), so capacity = the next
|
||||
// power of two ≥ max_actors can never overflow. (≥ 2 so mask works.)
|
||||
let cap = min_cap.next_power_of_two().max(2);
|
||||
let buf: Box<[Cell]> = (0..cap)
|
||||
.map(|i| Cell {
|
||||
seq: AtomicUsize::new(i),
|
||||
pid: UnsafeCell::new(MaybeUninit::uninit()),
|
||||
})
|
||||
.collect();
|
||||
Self {
|
||||
buf,
|
||||
mask: cap - 1,
|
||||
enqueue_pos: CachePadded(AtomicUsize::new(0)),
|
||||
dequeue_pos: CachePadded(AtomicUsize::new(0)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn push(&self, pid: Pid) {
|
||||
assert_no_preempt();
|
||||
assert!(
|
||||
self.try_push(pid),
|
||||
"smarm: run queue overflow — occupancy exceeded the slab bound, \
|
||||
which the at-most-once-enqueued invariant forbids. This is a \
|
||||
runtime bug (double enqueue), not a capacity tuning problem."
|
||||
);
|
||||
}
|
||||
|
||||
/// One full claim attempt; `false` only if the ring is full.
|
||||
fn try_push(&self, pid: Pid) -> bool {
|
||||
let mut pos = self.enqueue_pos.0.load(Ordering::Relaxed);
|
||||
loop {
|
||||
let cell = &self.buf[pos & self.mask];
|
||||
let seq = cell.seq.load(Ordering::Acquire);
|
||||
let diff = seq as isize - pos as isize;
|
||||
if diff == 0 {
|
||||
// Our turn: claim the position.
|
||||
match self.enqueue_pos.0.compare_exchange_weak(
|
||||
pos, pos + 1, Ordering::Relaxed, Ordering::Relaxed,
|
||||
) {
|
||||
Ok(_) => {
|
||||
// SAFETY: the claim gives us exclusive write access
|
||||
// to this cell until we publish below.
|
||||
cell.pid.with_mut(|p| unsafe { (*p).write(pid) });
|
||||
cell.seq.store(pos + 1, Ordering::Release);
|
||||
return true;
|
||||
}
|
||||
Err(actual) => pos = actual,
|
||||
}
|
||||
} else if diff < 0 {
|
||||
return false; // full — a whole lap behind
|
||||
} else {
|
||||
pos = self.enqueue_pos.0.load(Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn pop(&self) -> Option<Pid> {
|
||||
assert_no_preempt();
|
||||
let mut pos = self.dequeue_pos.0.load(Ordering::Relaxed);
|
||||
loop {
|
||||
let cell = &self.buf[pos & self.mask];
|
||||
let seq = cell.seq.load(Ordering::Acquire);
|
||||
let diff = seq as isize - (pos + 1) as isize;
|
||||
if diff == 0 {
|
||||
match self.dequeue_pos.0.compare_exchange_weak(
|
||||
pos, pos + 1, Ordering::Relaxed, Ordering::Relaxed,
|
||||
) {
|
||||
Ok(_) => {
|
||||
// SAFETY: the claim gives us exclusive read access;
|
||||
// the producer's Release publish made `pid` visible
|
||||
// to our Acquire load of `seq`.
|
||||
let pid = cell.pid.with(|p| unsafe { (*p).assume_init_read() });
|
||||
// Release the cell for the next lap.
|
||||
cell.seq.store(pos + self.mask + 1, Ordering::Release);
|
||||
return Some(pid);
|
||||
}
|
||||
Err(actual) => pos = actual,
|
||||
}
|
||||
} else if diff < 0 {
|
||||
// Empty (or the producer at this cell hasn't published yet —
|
||||
// a snapshot miss the caller's idle-retry loop absorbs).
|
||||
return None;
|
||||
} else {
|
||||
pos = self.dequeue_pos.0.load(Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn len(&self) -> u64 {
|
||||
let e = self.enqueue_pos.0.load(Ordering::Relaxed);
|
||||
let d = self.dequeue_pos.0.load(Ordering::Relaxed);
|
||||
e.saturating_sub(d) as u64
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// rq-striped — M Vyukov rings, ticket-distributed
|
||||
// ---------------------------------------------------------------------------
|
||||
//
|
||||
// Producers fetch-add a ticket and start probing at stripe `ticket % M`;
|
||||
// consumers do the same with their own ticket. Under symmetric load the
|
||||
// tickets spread producers and consumers uniformly, so each stripe sees
|
||||
// ~1/M of the traffic and the single hot cache-line pair becomes M cooler
|
||||
// ones. FIFO is relaxed: two pushes that land in different stripes can be
|
||||
// popped in either order, with skew bounded by stripe occupancy imbalance.
|
||||
//
|
||||
// Push probes forward from its home stripe until a `try_push` succeeds.
|
||||
// Σ stripe capacity ≥ 2 × max_actors while occupancy ≤ max_actors, so at
|
||||
// every instant at least half the total capacity is free and the probe
|
||||
// terminates (in practice on the first stripe).
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub struct StripedRing {
|
||||
stripes: Box<[MpmcRing]>,
|
||||
/// Stripe count minus one (count is a power of two).
|
||||
stripe_mask: usize,
|
||||
push_ticket: CachePadded<AtomicUsize>,
|
||||
pop_ticket: CachePadded<AtomicUsize>,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
impl StripedRing {
|
||||
pub fn new(threads: usize, max_actors: usize) -> Self {
|
||||
// One stripe per scheduler thread, rounded up to a power of two —
|
||||
// more stripes than threads buys nothing (at most `threads` ops are
|
||||
// in flight) and costs pop-probe latency when mostly empty.
|
||||
let n = threads.max(1).next_power_of_two();
|
||||
// Per-stripe capacity: 2 × max_actors / n in total, and never below
|
||||
// a floor that keeps degenerate configs (tiny slab, many threads)
|
||||
// trivially correct.
|
||||
let per = ((2 * max_actors) / n).next_power_of_two().max(8);
|
||||
let stripes: Box<[MpmcRing]> = (0..n).map(|_| MpmcRing::with_capacity(per)).collect();
|
||||
Self {
|
||||
stripes,
|
||||
stripe_mask: n - 1,
|
||||
push_ticket: CachePadded(AtomicUsize::new(0)),
|
||||
pop_ticket: CachePadded(AtomicUsize::new(0)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn push(&self, pid: Pid) {
|
||||
assert_no_preempt();
|
||||
let home = self.push_ticket.0.fetch_add(1, Ordering::Relaxed);
|
||||
// Probe from the home stripe; capacity headroom (Σ ≥ 2×occupancy)
|
||||
// guarantees a free stripe exists, so the outer loop terminates.
|
||||
// The retry-from-home lap handles the racy case where every stripe
|
||||
// momentarily refused us.
|
||||
loop {
|
||||
for i in 0..=self.stripe_mask {
|
||||
let s = &self.stripes[(home + i) & self.stripe_mask];
|
||||
if s.try_push(pid) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
std::hint::spin_loop();
|
||||
}
|
||||
}
|
||||
|
||||
pub fn pop(&self) -> Option<Pid> {
|
||||
assert_no_preempt();
|
||||
let home = self.pop_ticket.0.fetch_add(1, Ordering::Relaxed);
|
||||
for i in 0..=self.stripe_mask {
|
||||
if let Some(pid) = self.stripes[(home + i) & self.stripe_mask].pop() {
|
||||
return Some(pid);
|
||||
}
|
||||
}
|
||||
None // snapshot miss possible across stripes; idle-retry absorbs it
|
||||
}
|
||||
|
||||
pub fn len(&self) -> u64 {
|
||||
self.stripes.iter().map(|s| s.len()).sum()
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tests — all variants, in every build (the feature only picks the alias)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[cfg(all(test, not(loom)))]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::collections::HashSet;
|
||||
use std::sync::Arc;
|
||||
|
||||
fn pid(i: u32) -> Pid {
|
||||
Pid::new(i, 0)
|
||||
}
|
||||
|
||||
fn fifo_smoke<Q>(q: &Q, push: impl Fn(&Q, Pid), pop: impl Fn(&Q) -> Option<Pid>) {
|
||||
for i in 0..100 {
|
||||
push(q, pid(i));
|
||||
}
|
||||
for i in 0..100 {
|
||||
assert_eq!(pop(q), Some(pid(i)));
|
||||
}
|
||||
assert_eq!(pop(q), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mutex_fifo() {
|
||||
let q = MutexQueue::new(1, 1024);
|
||||
fifo_smoke(&q, |q, p| q.push(p), |q| q.pop());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mpmc_fifo_single_thread() {
|
||||
let q = MpmcRing::new(1, 1024);
|
||||
fifo_smoke(&q, |q, p| q.push(p), |q| q.pop());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mpmc_wraps_many_laps() {
|
||||
let q = MpmcRing::with_capacity(8);
|
||||
for lap in 0..1000u32 {
|
||||
for i in 0..8 {
|
||||
q.push(pid(lap * 8 + i));
|
||||
}
|
||||
for i in 0..8 {
|
||||
assert_eq!(q.pop(), Some(pid(lap * 8 + i)));
|
||||
}
|
||||
}
|
||||
assert_eq!(q.pop(), None);
|
||||
}
|
||||
|
||||
/// N producers, M consumers, every element exactly once. Run on plain OS
|
||||
/// threads (PREEMPTION_ENABLED defaults false, satisfying the contract).
|
||||
fn exactly_once<Q: Send + Sync + 'static>(
|
||||
q: Q,
|
||||
push: fn(&Q, Pid),
|
||||
pop: fn(&Q) -> Option<Pid>,
|
||||
producers: u32,
|
||||
consumers: u32,
|
||||
per_producer: u32,
|
||||
) {
|
||||
let q = Arc::new(q);
|
||||
let total = (producers * per_producer) as usize;
|
||||
let popped = Arc::new(std::sync::Mutex::new(Vec::with_capacity(total)));
|
||||
let remaining = Arc::new(AtomicUsize::new(total));
|
||||
|
||||
let mut hs = Vec::new();
|
||||
for p in 0..producers {
|
||||
let q = q.clone();
|
||||
hs.push(std::thread::spawn(move || {
|
||||
for i in 0..per_producer {
|
||||
push(&q, pid(p * per_producer + i));
|
||||
}
|
||||
}));
|
||||
}
|
||||
for _ in 0..consumers {
|
||||
let q = q.clone();
|
||||
let popped = popped.clone();
|
||||
let remaining = remaining.clone();
|
||||
hs.push(std::thread::spawn(move || {
|
||||
let mut local = Vec::new();
|
||||
while remaining.load(Ordering::Relaxed) > 0 {
|
||||
if let Some(pid) = pop(&q) {
|
||||
remaining.fetch_sub(1, Ordering::Relaxed);
|
||||
local.push(pid);
|
||||
} else {
|
||||
std::hint::spin_loop();
|
||||
}
|
||||
}
|
||||
popped.lock().unwrap().extend(local);
|
||||
}));
|
||||
}
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
|
||||
let popped = popped.lock().unwrap();
|
||||
assert_eq!(popped.len(), total, "count mismatch");
|
||||
let set: HashSet<u64> = popped.iter().map(|p| ((p.index() as u64) << 32) | p.generation() as u64).collect();
|
||||
assert_eq!(set.len(), total, "duplicate or lost element");
|
||||
assert_eq!(pop(&q), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mpmc_exactly_once_contended() {
|
||||
exactly_once(MpmcRing::new(8, 4096), |q, p| q.push(p), |q| q.pop(), 4, 4, 1000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn striped_exactly_once_contended() {
|
||||
exactly_once(StripedRing::new(8, 4096), |q, p| q.push(p), |q| q.pop(), 4, 4, 1000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn striped_drains_after_skewed_load() {
|
||||
// Hammer pushes from one thread (all tickets walk the stripes in
|
||||
// order) and verify a single consumer sees every element.
|
||||
let q = StripedRing::new(4, 64);
|
||||
let mut seen = HashSet::new();
|
||||
for i in 0..64 {
|
||||
q.push(pid(i));
|
||||
}
|
||||
while let Some(p) = q.pop() {
|
||||
assert!(seen.insert(p.index()));
|
||||
}
|
||||
assert_eq!(seen.len(), 64);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// loom model tests — RUSTFLAGS="--cfg loom" cargo test --lib --release
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[cfg(all(test, loom))]
|
||||
mod loom_tests {
|
||||
use super::*;
|
||||
use loom::sync::Arc;
|
||||
use loom::thread;
|
||||
|
||||
fn pid(i: u32) -> Pid {
|
||||
Pid::new(i, 0)
|
||||
}
|
||||
|
||||
/// Two producers, main-thread consumer: both elements arrive exactly
|
||||
/// once, across every interleaving — including through a lap wraparound
|
||||
/// (capacity 2 forces cell reuse).
|
||||
#[test]
|
||||
fn mpmc_two_producers_exactly_once() {
|
||||
loom::model(|| {
|
||||
let q = Arc::new(MpmcRing::with_capacity(2));
|
||||
let mut hs = Vec::new();
|
||||
for i in 0..2u32 {
|
||||
let q = q.clone();
|
||||
hs.push(thread::spawn(move || q.push(pid(i))));
|
||||
}
|
||||
let mut got = Vec::new();
|
||||
while got.len() < 2 {
|
||||
match q.pop() {
|
||||
Some(p) => got.push(p.index()),
|
||||
None => thread::yield_now(),
|
||||
}
|
||||
}
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
got.sort_unstable();
|
||||
assert_eq!(got, vec![0, 1]);
|
||||
assert!(q.pop().is_none());
|
||||
});
|
||||
}
|
||||
|
||||
/// Producer races a consumer on a single element: the consumer either
|
||||
/// gets it or sees a clean None — never a torn/duplicated element.
|
||||
#[test]
|
||||
fn mpmc_push_pop_race() {
|
||||
loom::model(|| {
|
||||
let q = Arc::new(MpmcRing::with_capacity(2));
|
||||
let q2 = q.clone();
|
||||
let prod = thread::spawn(move || q2.push(pid(7)));
|
||||
let seen = q.pop();
|
||||
prod.join().unwrap();
|
||||
match seen {
|
||||
Some(p) => {
|
||||
assert_eq!(p.index(), 7);
|
||||
assert!(q.pop().is_none());
|
||||
}
|
||||
None => assert_eq!(q.pop().map(|p| p.index()), Some(7)),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Striped: two producers landing in (potentially) different stripes,
|
||||
/// main-thread consumer drains both exactly once.
|
||||
#[test]
|
||||
fn striped_two_producers_exactly_once() {
|
||||
loom::model(|| {
|
||||
let q = Arc::new(StripedRing::new(2, 4));
|
||||
let mut hs = Vec::new();
|
||||
for i in 0..2u32 {
|
||||
let q = q.clone();
|
||||
hs.push(thread::spawn(move || q.push(pid(i))));
|
||||
}
|
||||
let mut got = Vec::new();
|
||||
while got.len() < 2 {
|
||||
match q.pop() {
|
||||
Some(p) => got.push(p.index()),
|
||||
None => thread::yield_now(),
|
||||
}
|
||||
}
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
got.sort_unstable();
|
||||
assert_eq!(got, vec![0, 1]);
|
||||
assert!(q.pop().is_none());
|
||||
});
|
||||
}
|
||||
}
|
||||
+438
-1045
File diff suppressed because it is too large
Load Diff
+134
-341
@@ -21,39 +21,18 @@ use std::sync::Arc;
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Borrow the current runtime. Panics if called outside `Runtime::run()`.
|
||||
///
|
||||
/// The whole span runs with preemption disabled. Two reasons, both load-
|
||||
/// bearing:
|
||||
///
|
||||
/// - The `RUNTIME` thread-local borrow is live across `f`. A preemption-
|
||||
/// driven context switch inside `f` can resume the actor on a DIFFERENT OS
|
||||
/// thread; the borrow guard would then increment this thread's RefCell but
|
||||
/// decrement the other's — a count underflow that leaves that thread's
|
||||
/// RefCell permanently "mutably borrowed". Holding a thread-local guard
|
||||
/// across a potential switch point is the one unforgivable sin of green
|
||||
/// threads; disabling preemption makes "no switch inside `f`" structural
|
||||
/// instead of an accident of which bodies happen to allocate.
|
||||
/// - `f` is runtime bookkeeping. Suspending an actor halfway through it (or
|
||||
/// unwinding via the stop sentinel, which shares the gate) is never wanted.
|
||||
pub(crate) fn with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> R {
|
||||
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
|
||||
let result = RUNTIME.with(|r| {
|
||||
RUNTIME.with(|r| {
|
||||
let b = r.borrow();
|
||||
let inner = b.as_ref().expect("smarm: not inside Runtime::run()");
|
||||
f(inner)
|
||||
});
|
||||
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
|
||||
result
|
||||
})
|
||||
}
|
||||
|
||||
/// Borrow the runtime if present; returns `None` otherwise.
|
||||
/// Used on cleanup paths (channel Drop during teardown).
|
||||
/// Same preemption gate as [`with_runtime`]; same reasons.
|
||||
pub(crate) fn try_with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> Option<R> {
|
||||
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
|
||||
let result = RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)));
|
||||
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
|
||||
result
|
||||
RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -75,34 +54,22 @@ impl JoinHandle {
|
||||
|
||||
pub fn join(mut self) -> Result<(), JoinError> {
|
||||
use crate::actor::Outcome;
|
||||
use crate::runtime::State; // need State visibility
|
||||
|
||||
let me = current_pid().expect("join() called outside an actor");
|
||||
|
||||
loop {
|
||||
// Check-Done-or-register-waiter is atomic under the target's cold
|
||||
// lock; finalize publishes Done and takes the waiter list under
|
||||
// the same lock, so we either see the outcome or are woken.
|
||||
let outcome = with_runtime(|inner| {
|
||||
let slot = inner.slot_at(self.pid)
|
||||
.expect("join: pid index out of range");
|
||||
let mut cold = slot.cold.lock();
|
||||
match slot.status_for(self.pid) {
|
||||
// Our outstanding handle pins the slot: it cannot be
|
||||
// reclaimed (generation cannot change) while we hold it.
|
||||
crate::slot_state::Status::Stale => {
|
||||
panic!("join: target slot has been reused")
|
||||
}
|
||||
crate::slot_state::Status::Done => {
|
||||
Some(cold.outcome.take().expect("Done slot must have outcome"))
|
||||
}
|
||||
crate::slot_state::Status::Live => {
|
||||
// begin_wait is lock-free, legal under the cold lock;
|
||||
// registering under it makes the epoch atomic with
|
||||
// the check-Done-or-register linearization point.
|
||||
cold.waiters.push((me, begin_wait()));
|
||||
inner.with_shared(|s| {
|
||||
let slot = s.slot_mut(self.pid)
|
||||
.expect("join: target slot has been reused");
|
||||
if matches!(slot.state, State::Done) {
|
||||
Some(slot.outcome.take().expect("Done slot must have outcome"))
|
||||
} else {
|
||||
slot.waiters.push(me);
|
||||
None
|
||||
}
|
||||
}
|
||||
})
|
||||
});
|
||||
|
||||
match outcome {
|
||||
@@ -128,25 +95,20 @@ impl JoinHandle {
|
||||
|
||||
fn decrement_handle_count(&mut self) {
|
||||
with_runtime(|inner| {
|
||||
let should_reclaim = match inner.slot_at(self.pid) {
|
||||
Some(slot) => {
|
||||
let mut cold = slot.cold.lock();
|
||||
match slot.status_for(self.pid) {
|
||||
crate::slot_state::Status::Stale => false,
|
||||
status => {
|
||||
cold.outstanding_handles =
|
||||
cold.outstanding_handles.saturating_sub(1);
|
||||
cold.outstanding_handles == 0
|
||||
&& status == crate::slot_state::Status::Done
|
||||
}
|
||||
inner.with_shared(|s| {
|
||||
let should_reclaim = match s.slot_mut(self.pid) {
|
||||
Some(slot) => {
|
||||
slot.outstanding_handles =
|
||||
slot.outstanding_handles.saturating_sub(1);
|
||||
matches!(slot.state, crate::runtime::State::Done)
|
||||
&& slot.outstanding_handles == 0
|
||||
}
|
||||
None => false,
|
||||
};
|
||||
if should_reclaim {
|
||||
crate::runtime::reclaim_slot(s, self.pid);
|
||||
}
|
||||
None => false,
|
||||
};
|
||||
if should_reclaim {
|
||||
// Re-verified inside; benign if finalize's reclaim won a race.
|
||||
crate::runtime::reclaim_slot(inner, self.pid);
|
||||
}
|
||||
})
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -167,28 +129,56 @@ impl Drop for JoinHandle {
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn spawn(f: impl FnOnce() + Send + 'static) -> JoinHandle {
|
||||
let parent = current_pid().unwrap_or_else(|| {
|
||||
// Outside an actor but inside run(): the initial spawn. with_runtime
|
||||
// panics with "not inside Runtime::run()" if there's no runtime at all.
|
||||
with_runtime(|_| crate::runtime::ROOT_PID)
|
||||
});
|
||||
let parent = current_pid()
|
||||
.or_else(|| with_runtime(|inner| inner.with_shared(|s| s.root_pid)))
|
||||
.expect("spawn() before run()");
|
||||
spawn_under(parent, f)
|
||||
}
|
||||
|
||||
pub fn spawn_under(supervisor: Pid, f: impl FnOnce() + Send + 'static) -> JoinHandle {
|
||||
// Stack + closure boxing happen before ANY runtime lock is taken: no
|
||||
// syscall and no allocation ever stalls another scheduler thread.
|
||||
let stack = with_runtime(|inner| inner.stack_pool.lock().pop())
|
||||
// Try to reuse a stack from the pool; fall back to a fresh mmap if empty.
|
||||
// Allocation happens before taking the shared lock so any syscall doesn't
|
||||
// stall other scheduler threads.
|
||||
let stack = with_runtime(|inner| inner.stack_pool.lock().unwrap().pop())
|
||||
.unwrap_or_else(|| {
|
||||
crate::stack::Stack::new(crate::runtime::ACTOR_STACK_SIZE)
|
||||
.expect("stack allocation failed")
|
||||
});
|
||||
let sp = init_actor_stack(stack.top(), crate::actor::trampoline);
|
||||
let closure: crate::runtime::Closure = Box::new(f);
|
||||
|
||||
let pid = with_runtime(|inner| {
|
||||
let idx = inner.allocate_slot(); // panics loudly on slab exhaustion
|
||||
crate::runtime::install_actor(inner, idx, sp, stack, supervisor, closure)
|
||||
inner.with_shared(|s| {
|
||||
let (idx, gen) = s.allocate_slot();
|
||||
let pid = Pid::new(idx, gen);
|
||||
let slot = &mut s.slots[idx as usize];
|
||||
slot.actor = Some(crate::actor::Actor {
|
||||
pid,
|
||||
stack,
|
||||
sp,
|
||||
supervisor,
|
||||
stop: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)),
|
||||
trap: None,
|
||||
});
|
||||
slot.state = crate::runtime::State::Runnable;
|
||||
slot.outstanding_handles = 1;
|
||||
slot.outcome = None;
|
||||
slot.waiters.clear();
|
||||
slot.supervisor_channel = None;
|
||||
slot.monitors.clear();
|
||||
slot.links.clear();
|
||||
slot.pending_unpark = false;
|
||||
slot.pending_io_result = None;
|
||||
s.run_queue.push_back(pid);
|
||||
// Grow the closures vec to cover this slot index, then store.
|
||||
let idx = idx as usize;
|
||||
if s.pending_closures.len() <= idx {
|
||||
s.pending_closures.resize_with(idx + 1, || None);
|
||||
}
|
||||
s.pending_closures[idx] = Some(Box::new(f) as crate::runtime::Closure);
|
||||
crate::te!(crate::trace::Event::Spawn { parent: supervisor, child: pid });
|
||||
crate::te!(crate::trace::Event::Enqueue(pid));
|
||||
pid
|
||||
})
|
||||
});
|
||||
|
||||
JoinHandle { pid, consumed: false }
|
||||
@@ -212,18 +202,6 @@ pub fn yield_now() {
|
||||
}
|
||||
|
||||
pub fn park_current() {
|
||||
// Entry-side observation point: a stop flagged while we were QUEUED is
|
||||
// otherwise lost — the stop's wildcard unpark no-ops on a Queued actor
|
||||
// (the pending run "is" the wake), so if our first action on resume is
|
||||
// this park, no wake is ever coming and the wake-side check below is
|
||||
// unreachable. Checking here closes that hole; a flag that lands after
|
||||
// this check is covered by the existing protocol (the stop's unpark
|
||||
// finds Running / the prep-to-park window, sets Notified, and the
|
||||
// park-return re-queues us into the wake-side check). Unwinding from
|
||||
// here is the same unwind path as the wake side: leftover wait
|
||||
// registrations are stale-epoch / dead-generation and self-clean at
|
||||
// their wakers' failed CAS.
|
||||
crate::preempt::check_cancelled();
|
||||
runtime::set_yield_intent(YieldIntent::Park);
|
||||
unsafe { crate::context::switch_to_scheduler() };
|
||||
// Observation point on the wakeup side of every blocking primitive
|
||||
@@ -234,54 +212,39 @@ pub fn park_current() {
|
||||
}
|
||||
|
||||
pub fn unpark(pid: Pid) {
|
||||
// The whole protocol lives on the slot's packed word (gen + epoch +
|
||||
// state in one CAS) — see slot_state.rs. No runtime lock unless we
|
||||
// enqueue. WILDCARD form: reserved for terminal wakes (request_stop);
|
||||
// every registration-based waker must use `unpark_at`.
|
||||
let _ = try_with_runtime(|inner| inner.unpark(pid));
|
||||
}
|
||||
|
||||
/// Epoch-matched unpark: wake `pid` only if its current wait is still the
|
||||
/// one this waker registered for. The form every registration-based waker
|
||||
/// (channel senders, mutex grants, wait-timers, io completions, joiner
|
||||
/// wakes) must use — see slot_state.rs for the consuming-wake rules.
|
||||
pub(crate) fn unpark_at(pid: Pid, epoch: u32) {
|
||||
let _ = try_with_runtime(|inner| inner.unpark_at(pid, epoch));
|
||||
}
|
||||
|
||||
/// Open a new wait for the CURRENT actor: bump its park-epoch and return
|
||||
/// it. Call once per wait, before registering `(pid, epoch)` with any
|
||||
/// waker. Lock-free (one CAS on the own slot word), so it is legal under
|
||||
/// any lock, including a Channel-class lock.
|
||||
pub(crate) fn begin_wait() -> u32 {
|
||||
let me = current_pid().expect("begin_wait() called outside an actor");
|
||||
with_runtime(|inner| inner.begin_wait(me))
|
||||
}
|
||||
|
||||
/// Close the current actor's wait WITHOUT parking on it. For the no-park
|
||||
/// exit of multi-registration waits (`select` finding an arm ready at
|
||||
/// registration time): bumps the epoch so in-flight wakes die at their CAS,
|
||||
/// eats a notification that already landed, then observes a pending stop —
|
||||
/// in that order. After this, leftover registrations are stale-epoch and
|
||||
/// self-clean at their wakers' failed CAS; nothing can fault the actor's
|
||||
/// next one-shot park.
|
||||
pub(crate) fn retire_wait() {
|
||||
let me = current_pid().expect("retire_wait() called outside an actor");
|
||||
with_runtime(|inner| inner.retire_wait(me));
|
||||
// A request_stop that fired before the clear had its notification
|
||||
// eaten, but it set the stop flag first — observe it here and unwind.
|
||||
// One that fires after re-notifies a Running word as usual.
|
||||
crate::preempt::check_cancelled();
|
||||
let result = try_with_runtime(|inner| {
|
||||
inner.with_shared(|s| {
|
||||
if let Some(slot) = s.slot_mut(pid) {
|
||||
match slot.state {
|
||||
crate::runtime::State::Parked => {
|
||||
// Actor is suspended — safe to re-queue immediately.
|
||||
slot.state = crate::runtime::State::Runnable;
|
||||
s.run_queue.push_back(pid);
|
||||
crate::te!(crate::trace::Event::UnparkDirect(pid));
|
||||
crate::te!(crate::trace::Event::Enqueue(pid));
|
||||
}
|
||||
crate::runtime::State::Runnable => {
|
||||
// Actor is still running (between registering its
|
||||
// parked_receiver and calling park_current). Set the
|
||||
// flag; the scheduler will re-queue after the Park
|
||||
// yield instead of sleeping.
|
||||
slot.pending_unpark = true;
|
||||
crate::te!(crate::trace::Event::UnparkDeferred(pid));
|
||||
}
|
||||
crate::runtime::State::Done => {}
|
||||
}
|
||||
}
|
||||
})
|
||||
});
|
||||
let _ = result;
|
||||
}
|
||||
|
||||
/// Request cooperative cancellation of `pid`.
|
||||
///
|
||||
/// Sets the actor's stop flag and wakes it so it observes the flag promptly:
|
||||
/// a parked target is re-queued; a running target is marked notified, so its
|
||||
/// next park returns immediately to an observation point. The actor realises
|
||||
/// the stop as a controlled unwind at its next observation point
|
||||
/// (`check!()`/allocation, or the wakeup side of a blocking park),
|
||||
/// terminating with `Outcome::Stopped`.
|
||||
/// Sets the actor's stop flag and, if it is parked, wakes it so it observes
|
||||
/// the flag promptly. The actor realises the stop as a controlled unwind at
|
||||
/// its next observation point (`check!()`/allocation, or the wakeup side of a
|
||||
/// blocking park), terminating with `Outcome::Stopped`.
|
||||
///
|
||||
/// This is best-effort and cooperative: an actor that never reaches an
|
||||
/// observation point — a tight loop with no `check!()`, no allocation, and no
|
||||
@@ -289,19 +252,21 @@ pub(crate) fn retire_wait() {
|
||||
/// if `pid` is already gone.
|
||||
pub fn request_stop(pid: Pid) {
|
||||
let _ = try_with_runtime(|inner| {
|
||||
if let Some(slot) = inner.slot_at(pid) {
|
||||
{
|
||||
let cold = slot.cold.lock();
|
||||
// Verify under the cold lock: generation can't change while
|
||||
// we hold it (reclaim takes the same lock).
|
||||
if slot.generation() == pid.generation() {
|
||||
if let Some(actor) = cold.actor.as_ref() {
|
||||
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
inner.with_shared(|s| {
|
||||
if let Some(slot) = s.slot_mut(pid) {
|
||||
if let Some(actor) = slot.actor.as_ref() {
|
||||
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
// Wake a parked target so it reaches its post-park observation
|
||||
// point now. A Runnable target will observe on its next yield;
|
||||
// a Done target has nothing to stop.
|
||||
if matches!(slot.state, crate::runtime::State::Parked) {
|
||||
slot.state = crate::runtime::State::Runnable;
|
||||
s.run_queue.push_back(pid);
|
||||
crate::te!(crate::trace::Event::Enqueue(pid));
|
||||
}
|
||||
}
|
||||
inner.unpark(pid);
|
||||
}
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
@@ -331,9 +296,8 @@ impl Drop for NoPreempt {
|
||||
pub fn sleep(duration: std::time::Duration) {
|
||||
let me = current_pid().expect("sleep() called outside an actor");
|
||||
let _np = NoPreempt::enter();
|
||||
let epoch = begin_wait();
|
||||
let deadline = crate::timer::deadline_from_now(duration);
|
||||
with_runtime(|inner| inner.timers.lock().unwrap().insert_sleep(deadline, me, epoch));
|
||||
with_runtime(|inner| inner.with_shared(|s| s.timers.insert_sleep(deadline, me)));
|
||||
park_current();
|
||||
}
|
||||
|
||||
@@ -341,14 +305,16 @@ pub fn insert_wait_timer(
|
||||
deadline: std::time::Instant,
|
||||
pid: Pid,
|
||||
target: std::sync::Arc<dyn crate::timer::TimerTarget>,
|
||||
epoch: u32,
|
||||
wait_seq: u64,
|
||||
) {
|
||||
with_runtime(|inner| {
|
||||
inner.timers.lock().unwrap().insert(
|
||||
deadline,
|
||||
pid,
|
||||
crate::timer::Reason::WaitTimeout { target, epoch },
|
||||
);
|
||||
inner.with_shared(|s| {
|
||||
s.timers.insert(
|
||||
deadline,
|
||||
pid,
|
||||
crate::timer::Reason::WaitTimeout { target, wait_seq },
|
||||
);
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
@@ -368,24 +334,19 @@ where
|
||||
});
|
||||
{
|
||||
let _np = NoPreempt::enter();
|
||||
let epoch = begin_wait();
|
||||
with_runtime(|inner| {
|
||||
let mut io = inner.io.lock().unwrap();
|
||||
io.as_mut().expect("io thread not started").submit(me, epoch, work);
|
||||
});
|
||||
with_runtime(|inner| inner.with_shared(|s| {
|
||||
let io = s.io.as_mut().expect("io thread not started");
|
||||
io.submit(me, work);
|
||||
}));
|
||||
park_current();
|
||||
}
|
||||
let result = with_runtime(|inner| {
|
||||
let slot = inner.slot_at(me).expect("block_on_io: own slot vanished");
|
||||
let mut cold = slot.cold.lock();
|
||||
debug_assert_eq!(
|
||||
slot.generation(), me.generation(),
|
||||
"block_on_io: own slot reused mid-park"
|
||||
);
|
||||
cold.pending_io_result
|
||||
let result = with_runtime(|inner| inner.with_shared(|s| {
|
||||
s.slot_mut(me)
|
||||
.expect("block_on_io: own slot vanished")
|
||||
.pending_io_result
|
||||
.take()
|
||||
.expect("block_on_io: resumed without a result")
|
||||
});
|
||||
}));
|
||||
match result {
|
||||
Ok(any) => *any.downcast::<T>().expect("block_on_io: type mismatch"),
|
||||
Err(payload) => std::panic::resume_unwind(payload),
|
||||
@@ -403,179 +364,14 @@ pub fn wait_writable(fd: std::os::fd::RawFd) -> std::io::Result<()> {
|
||||
fn wait_fd(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::Result<()> {
|
||||
let me = current_pid().expect("wait_*() called outside an actor");
|
||||
let _np = NoPreempt::enter();
|
||||
let epoch = begin_wait();
|
||||
with_runtime(|inner| {
|
||||
let mut io = inner.io.lock().unwrap();
|
||||
io.as_mut().expect("io thread not started").epoll_register(fd, me, epoch, readable, writable)
|
||||
})?;
|
||||
|
||||
// If a terminal stop unwinds us out of the park below, the registration
|
||||
// must not outlive us: a stale `waiters` entry fails every future
|
||||
// `wait_*` on this fd with AlreadyExists, and the kernel-side ADD leaks
|
||||
// until the fd happens to be reused. Clean up iff the entry is still
|
||||
// ours — a `FdReady` racing the stop may have consumed it already (it
|
||||
// removes + DELs under the io lock), after which the fd may even carry
|
||||
// ANOTHER actor's fresh registration; in that case touch nothing.
|
||||
struct Dereg {
|
||||
fd: std::os::fd::RawFd,
|
||||
me: Pid,
|
||||
epoch: u32,
|
||||
}
|
||||
impl Drop for Dereg {
|
||||
fn drop(&mut self) {
|
||||
with_runtime(|inner| {
|
||||
let mut io = inner.io.lock().unwrap();
|
||||
if let Some(io) = io.as_mut() {
|
||||
if io.waiters.get(&self.fd) == Some(&(self.me, self.epoch)) {
|
||||
io.waiters.remove(&self.fd);
|
||||
io.epoll_deregister(self.fd);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
let guard = Dereg { fd, me, epoch };
|
||||
with_runtime(|inner| inner.with_shared(|s| {
|
||||
let io = s.io.as_mut().expect("io thread not started");
|
||||
io.epoll_register(fd, me, readable, writable)
|
||||
}))?;
|
||||
park_current();
|
||||
// Normal wake: the FdReady path removed the entry and DEL'd the fd
|
||||
// before the unpark, so the guard's check would be a guaranteed no-op —
|
||||
// skip the io lock on the hot path. (Dereg owns nothing; forget leaks
|
||||
// no resource.)
|
||||
std::mem::forget(guard);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// FdArm — fd readiness as a select arm (RFC 008)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// An fd-readiness arm for [`crate::select`] / [`crate::select_timeout`]:
|
||||
/// ready when the fd is readable (resp. writable), composable with channel
|
||||
/// receivers on one wait epoch. Phase-1 rules apply: one waiter per fd at a
|
||||
/// time, one direction per arm (duplex on a single fd needs `dup`; epoll
|
||||
/// registrations key on the open file description, so dup'd fds register
|
||||
/// independently).
|
||||
pub struct FdArm {
|
||||
fd: std::os::fd::RawFd,
|
||||
readable: bool,
|
||||
writable: bool,
|
||||
}
|
||||
|
||||
impl FdArm {
|
||||
pub fn readable(fd: std::os::fd::RawFd) -> Self {
|
||||
FdArm { fd, readable: true, writable: false }
|
||||
}
|
||||
|
||||
pub fn writable(fd: std::os::fd::RawFd) -> Self {
|
||||
FdArm { fd, readable: false, writable: true }
|
||||
}
|
||||
}
|
||||
|
||||
impl crate::channel::sealed::Sealed for FdArm {}
|
||||
|
||||
impl crate::channel::Selectable for FdArm {
|
||||
/// Ready-now check is a zero-timeout `poll(2)`; if the requested events
|
||||
/// are pending the wait is retired without registering (`Ok(false)`,
|
||||
/// the channel-arm contract). Otherwise register with the io thread —
|
||||
/// every failure surfaces as `Err` (EBADF including a closed-fd
|
||||
/// POLLNVAL, EMFILE on the epoll set, AlreadyExists for a second
|
||||
/// waiter on the fd): the fallible-out, nothing-left-behind rule, in
|
||||
/// deviation from RFC 008's permanently-ready lean, which would spin a
|
||||
/// consumer whose fd is healthy but unregistrable (EMFILE).
|
||||
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool> {
|
||||
if poll_events(self.fd, self.readable, self.writable)? {
|
||||
return Ok(false);
|
||||
}
|
||||
with_runtime(|inner| {
|
||||
let mut io = inner.io.lock().unwrap();
|
||||
io.as_mut()
|
||||
.expect("io thread not started")
|
||||
.epoll_register(self.fd, pid, epoch, self.readable, self.writable)
|
||||
})?;
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Classification is the same zero-timeout poll: a pure function of fd
|
||||
/// state, independent of the registration the cleanup pass removed. An
|
||||
/// error here (EBADF: fd closed mid-wait) reports READY — the
|
||||
/// consumer's read/write surfaces the errno; a dead arm is an event,
|
||||
/// not a hang.
|
||||
fn sel_ready(&self) -> bool {
|
||||
poll_events(self.fd, self.readable, self.writable).unwrap_or(true)
|
||||
}
|
||||
|
||||
/// `wait_fd`'s `Dereg` compare, verbatim: remove the waiters entry and
|
||||
/// kernel-side registration iff the entry is still `(pid, epoch)`-ours.
|
||||
/// A `FdReady` racing the wake may have consumed it (it removes + DELs
|
||||
/// under the io lock), after which the fd may even carry ANOTHER
|
||||
/// actor's fresh registration; in that case touch nothing.
|
||||
fn sel_unregister(&self, pid: Pid, epoch: u32) {
|
||||
with_runtime(|inner| {
|
||||
let mut io = inner.io.lock().unwrap();
|
||||
if let Some(io) = io.as_mut() {
|
||||
if io.waiters.get(&self.fd) == Some(&(pid, epoch)) {
|
||||
io.waiters.remove(&self.fd);
|
||||
io.epoll_deregister(self.fd);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn sel_eager_cleanup(&self) -> bool {
|
||||
true
|
||||
}
|
||||
}
|
||||
|
||||
/// Zero-timeout `poll(2)`: are any of the requested events (or ERR/HUP,
|
||||
/// which make the consumer's read/write fail loudly rather than park
|
||||
/// forever) pending on `fd`? POLLNVAL maps to `Err(EBADF)`.
|
||||
fn poll_events(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::Result<bool> {
|
||||
let mut events: libc::c_short = 0;
|
||||
if readable {
|
||||
events |= libc::POLLIN;
|
||||
}
|
||||
if writable {
|
||||
events |= libc::POLLOUT;
|
||||
}
|
||||
let mut pfd = libc::pollfd { fd, events, revents: 0 };
|
||||
loop {
|
||||
let r = unsafe { libc::poll(&mut pfd, 1, 0) };
|
||||
if r < 0 {
|
||||
let e = std::io::Error::last_os_error();
|
||||
if e.kind() == std::io::ErrorKind::Interrupted {
|
||||
continue;
|
||||
}
|
||||
return Err(e);
|
||||
}
|
||||
if r == 0 {
|
||||
return Ok(false);
|
||||
}
|
||||
if pfd.revents & libc::POLLNVAL != 0 {
|
||||
return Err(std::io::Error::from_raw_os_error(libc::EBADF));
|
||||
}
|
||||
return Ok(pfd.revents & (events | libc::POLLERR | libc::POLLHUP) != 0);
|
||||
}
|
||||
}
|
||||
|
||||
/// Wait until `fd` is readable or `timeout` elapses: `Ok(true)` = ready,
|
||||
/// `Ok(false)` = timed out. A one-arm [`crate::try_select_timeout`].
|
||||
pub fn wait_readable_timeout(
|
||||
fd: std::os::fd::RawFd,
|
||||
timeout: std::time::Duration,
|
||||
) -> std::io::Result<bool> {
|
||||
let arm = FdArm::readable(fd);
|
||||
Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some())
|
||||
}
|
||||
|
||||
/// Wait until `fd` is writable or `timeout` elapses: `Ok(true)` = ready,
|
||||
/// `Ok(false)` = timed out.
|
||||
pub fn wait_writable_timeout(
|
||||
fd: std::os::fd::RawFd,
|
||||
timeout: std::time::Duration,
|
||||
) -> std::io::Result<bool> {
|
||||
let arm = FdArm::writable(fd);
|
||||
Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some())
|
||||
}
|
||||
|
||||
pub fn read(fd: std::os::fd::RawFd, buf: &mut [u8]) -> std::io::Result<usize> {
|
||||
wait_readable(fd)?;
|
||||
let n = unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) };
|
||||
@@ -593,16 +389,13 @@ pub fn write(fd: std::os::fd::RawFd, buf: &[u8]) -> std::io::Result<usize> {
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn register_supervisor_channel(pid: Pid, sender: Sender<Signal>) {
|
||||
with_runtime(|inner| {
|
||||
let slot = inner.slot_at(pid)
|
||||
.unwrap_or_else(|| panic!("register_supervisor_channel: pid {:?} not found", pid));
|
||||
let mut cold = slot.cold.lock();
|
||||
assert_eq!(
|
||||
slot.generation(), pid.generation(),
|
||||
"register_supervisor_channel: pid {:?} not found", pid
|
||||
);
|
||||
cold.supervisor_channel = Some(sender);
|
||||
});
|
||||
with_runtime(|inner| inner.with_shared(|s| {
|
||||
if let Some(slot) = s.slot_mut(pid) {
|
||||
slot.supervisor_channel = Some(sender);
|
||||
} else {
|
||||
panic!("register_supervisor_channel: pid {:?} not found", pid);
|
||||
}
|
||||
}));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,624 +0,0 @@
|
||||
//! The per-slot scheduling state machine, as a standalone unit.
|
||||
//!
|
||||
//! One atomic word packs `(generation << 32) | (epoch << 8) | state`; every
|
||||
//! transition is a CAS on the packed word, so the generation check is atomic
|
||||
//! with the transition — no ABA, no acting on a recycled slot. The diagram
|
||||
//! and the full protocol rationale live in `runtime.rs`; this module is the
|
||||
//! mechanism, factored out so that:
|
||||
//!
|
||||
//! - loom can model-check the production transitions directly (see the
|
||||
//! `loom_tests` module; built with `RUSTFLAGS="--cfg loom"`), and
|
||||
//! - every method asserts the precondition it relies on (`debug_assert!` —
|
||||
//! these are hot paths), per the assert-the-invariants house rule.
|
||||
//!
|
||||
//! ## The park-epoch (wait identity)
|
||||
//!
|
||||
//! The middle 24 bits carry the slot's *park-epoch*: the identity of the
|
||||
//! actor's current (or most recent) wait. The rules:
|
||||
//!
|
||||
//! - [`begin_wait`](StateWord::begin_wait) bumps the epoch and returns it;
|
||||
//! the actor calls it once per wait, *before* registering itself with any
|
||||
//! waker. Registrations carry `(pid, epoch)`.
|
||||
//! - A wake may be **epoch-matched** (`unpark(gen, Some(epoch))`): it lands
|
||||
//! only if the word still carries that epoch. Wakers whose registration
|
||||
//! handle can outlive the wait it was created for (channel senders, mutex
|
||||
//! grants, wait-timers) MUST use this form.
|
||||
//! - Every successful wake **consumes** the epoch — `Parked(e) → Queued(e+1)`,
|
||||
//! `Running(e) → RunningNotified(e+1)` — so at most one wake can ever land
|
||||
//! per wait, by construction. A loser in a multi-waker race (e.g. the
|
||||
//! non-winning arms of a `select`) fails the epoch check and no-ops; it can
|
||||
//! neither steal a future wait's wake nor leave a pending notification that
|
||||
//! would fault a later one-shot park (`Mutex::lock_timeout`, `sleep`,
|
||||
//! `block_on_io`, `wait_fd` all rely on wakes being *meaningful*).
|
||||
//! - The wildcard form (`unpark(gen, None)`) also consumes, and is reserved
|
||||
//! for terminal wakes — `request_stop` — which never return control to the
|
||||
//! code that parked.
|
||||
//!
|
||||
//! Epoch wrap (24 bits = 16.7M waits) is harmless: a collision would require
|
||||
//! a taken registration to stay in flight across a full wrap of the *same
|
||||
//! actor's* waits, and registrations are consumed at take-time under their
|
||||
//! primitive's lock — the exposure is the taker's instruction window.
|
||||
//!
|
||||
//! Atomics come from `sync_shim` (std normally, `loom::sync` under
|
||||
//! `cfg(loom)`).
|
||||
|
||||
use crate::sync_shim::{AtomicU64, Ordering};
|
||||
|
||||
pub(crate) const ST_VACANT: u64 = 0;
|
||||
pub(crate) const ST_QUEUED: u64 = 1;
|
||||
pub(crate) const ST_RUNNING: u64 = 2;
|
||||
pub(crate) const ST_RUNNING_NOTIFIED: u64 = 3;
|
||||
pub(crate) const ST_PARKED: u64 = 4;
|
||||
pub(crate) const ST_DONE: u64 = 5;
|
||||
|
||||
/// Park-epoch width: 24 bits, packed at word bits 8..32.
|
||||
pub(crate) const EPOCH_MASK: u32 = 0x00FF_FFFF;
|
||||
|
||||
#[inline]
|
||||
pub(crate) const fn pack(gen: u32, epoch: u32, st: u64) -> u64 {
|
||||
debug_assert!(epoch & !EPOCH_MASK == 0);
|
||||
((gen as u64) << 32) | ((epoch as u64) << 8) | st
|
||||
}
|
||||
#[inline]
|
||||
pub(crate) const fn word_gen(w: u64) -> u32 {
|
||||
(w >> 32) as u32
|
||||
}
|
||||
#[inline]
|
||||
pub(crate) const fn word_epoch(w: u64) -> u32 {
|
||||
((w >> 8) as u32) & EPOCH_MASK
|
||||
}
|
||||
#[inline]
|
||||
pub(crate) const fn word_state(w: u64) -> u64 {
|
||||
w & 0xFF
|
||||
}
|
||||
|
||||
/// What an unpark amounted to. The caller owns the side effects (enqueue,
|
||||
/// trace events) — this module is pure state.
|
||||
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
|
||||
pub(crate) enum Unpark {
|
||||
/// Parked → Queued: the caller must enqueue the pid.
|
||||
Enqueue,
|
||||
/// Running → RunningNotified: the scheduler's park-return will re-queue.
|
||||
Notified,
|
||||
/// Stale generation, stale epoch, already queued/notified, done, or
|
||||
/// vacant.
|
||||
Noop,
|
||||
}
|
||||
|
||||
/// A pid's-eye view of the slot, for cold paths that hold the slot lock.
|
||||
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
|
||||
pub(crate) enum Status {
|
||||
/// The generation no longer matches: the slot was reclaimed (and possibly
|
||||
/// reused) — the pid is stale.
|
||||
Stale,
|
||||
/// The actor terminated; its outcome is (or was) in the slot.
|
||||
Done,
|
||||
/// Alive in some scheduling state (Queued / Running / Notified / Parked).
|
||||
Live,
|
||||
}
|
||||
|
||||
pub(crate) struct StateWord(AtomicU64);
|
||||
|
||||
impl StateWord {
|
||||
pub(crate) fn new() -> Self {
|
||||
Self(AtomicU64::new(pack(0, 0, ST_VACANT)))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub(crate) fn load(&self) -> u64 {
|
||||
self.0.load(Ordering::Acquire)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub(crate) fn generation(&self) -> u32 {
|
||||
word_gen(self.load())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub(crate) fn status_for(&self, gen: u32) -> Status {
|
||||
let w = self.load();
|
||||
if word_gen(w) != gen {
|
||||
return Status::Stale;
|
||||
}
|
||||
match word_state(w) {
|
||||
ST_DONE => Status::Done,
|
||||
// A matching generation on a Vacant slot is unreachable for any
|
||||
// ISSUED pid — reclaim bumps the generation in the very store
|
||||
// that vacates, and install publishes Queued before the pid
|
||||
// escapes. But `Pid::new` is public, so a forged / never-issued
|
||||
// pid (e.g. `Pid::new(5, 0)` against a fresh slab) can land
|
||||
// here; for those, "no such actor" is the correct total answer.
|
||||
ST_VACANT => Status::Stale,
|
||||
_ => Status::Live,
|
||||
}
|
||||
}
|
||||
|
||||
/// Spawn-side publish: Vacant → Queued. The caller owns the vacant slot
|
||||
/// exclusively (it popped the index from the free list), so this is a
|
||||
/// plain Release store; it is the moment the actor becomes visible to
|
||||
/// pops, unparks, and stops. The epoch starts at 0 for each occupancy
|
||||
/// (`set_done` zeroes it; wait identity never crosses a lifetime).
|
||||
pub(crate) fn publish_queued(&self, gen: u32) {
|
||||
debug_assert_eq!(
|
||||
self.load(),
|
||||
pack(gen, 0, ST_VACANT),
|
||||
"publish over a non-vacant slot"
|
||||
);
|
||||
self.0.store(pack(gen, 0, ST_QUEUED), Ordering::Release);
|
||||
}
|
||||
|
||||
/// Scheduler pop-side claim: Queued → Running, epoch preserved. `false`
|
||||
/// means the popped pid is stale — by the at-most-once-enqueued
|
||||
/// invariant, a generation mismatch is the only possible failure
|
||||
/// (asserted). Nothing can move a matching-gen word off Queued (wakes
|
||||
/// no-op on Queued), so the CAS loop is single-shot in practice.
|
||||
#[must_use]
|
||||
pub(crate) fn try_claim(&self, gen: u32) -> bool {
|
||||
loop {
|
||||
let w = self.load();
|
||||
if word_gen(w) != gen {
|
||||
return false;
|
||||
}
|
||||
debug_assert_eq!(
|
||||
word_state(w),
|
||||
ST_QUEUED,
|
||||
"queued pid found in unexpected state {} — double enqueue?",
|
||||
word_state(w)
|
||||
);
|
||||
if self
|
||||
.0
|
||||
.compare_exchange(
|
||||
w,
|
||||
pack(gen, word_epoch(w), ST_RUNNING),
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Yield return path: Running | RunningNotified → Queued, epoch
|
||||
/// preserved. A notification that arrived mid-run coalesces into the
|
||||
/// re-queue. Caller must enqueue. CAS loop because a notify can bump the
|
||||
/// epoch between the read and the exchange.
|
||||
pub(crate) fn yield_return(&self, gen: u32) {
|
||||
loop {
|
||||
let w = self.load();
|
||||
debug_assert!(
|
||||
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
|
||||
&& word_gen(w) == gen,
|
||||
"yield return from invalid word {w:#x}"
|
||||
);
|
||||
if self
|
||||
.0
|
||||
.compare_exchange(
|
||||
w,
|
||||
pack(gen, word_epoch(w), ST_QUEUED),
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Park return path. `true` = actually parked. `false` = an unpark landed
|
||||
/// in the prep-to-park window (RunningNotified); the word is already back
|
||||
/// to Queued and the caller must enqueue — the lost-wakeup window,
|
||||
/// closed. Epoch preserved on both paths (the notify already consumed
|
||||
/// it).
|
||||
#[must_use]
|
||||
pub(crate) fn park_return(&self, gen: u32) -> bool {
|
||||
loop {
|
||||
let w = self.load();
|
||||
debug_assert_eq!(word_gen(w), gen, "park return with stale gen");
|
||||
let target = match word_state(w) {
|
||||
ST_RUNNING => ST_PARKED,
|
||||
ST_RUNNING_NOTIFIED => ST_QUEUED,
|
||||
st => unreachable!("park return from invalid state {st}"),
|
||||
};
|
||||
if self
|
||||
.0
|
||||
.compare_exchange(
|
||||
w,
|
||||
pack(gen, word_epoch(w), target),
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
return target == ST_PARKED;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Open a new wait: bump the park-epoch and return it. Called by the
|
||||
/// waiting actor itself (so the state is Running, or RunningNotified if
|
||||
/// a terminal wake is already pending — the bump preserves the pending
|
||||
/// notification), once per wait, BEFORE registering `(pid, epoch)` with
|
||||
/// any waker.
|
||||
#[must_use]
|
||||
pub(crate) fn begin_wait(&self, gen: u32) -> u32 {
|
||||
loop {
|
||||
let w = self.load();
|
||||
debug_assert!(
|
||||
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
|
||||
&& word_gen(w) == gen,
|
||||
"begin_wait from invalid word {w:#x}"
|
||||
);
|
||||
let next = word_epoch(w).wrapping_add(1) & EPOCH_MASK;
|
||||
if self
|
||||
.0
|
||||
.compare_exchange(
|
||||
w,
|
||||
pack(gen, next, word_state(w)),
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
return next;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The unpark protocol — the one way anything outside the scheduler makes
|
||||
/// an actor runnable. See [`Unpark`] for the caller's obligations.
|
||||
///
|
||||
/// `want = Some(epoch)` is the epoch-matched form: lands only if the word
|
||||
/// still carries that epoch (i.e. the wait it was registered for is still
|
||||
/// the current, un-woken wait). `want = None` is the wildcard, reserved
|
||||
/// for terminal wakes. Both forms CONSUME the epoch on success.
|
||||
#[must_use]
|
||||
pub(crate) fn unpark(&self, gen: u32, want: Option<u32>) -> Unpark {
|
||||
loop {
|
||||
let w = self.load();
|
||||
if word_gen(w) != gen {
|
||||
return Unpark::Noop;
|
||||
}
|
||||
if let Some(e) = want {
|
||||
if word_epoch(w) != e {
|
||||
return Unpark::Noop;
|
||||
}
|
||||
}
|
||||
let bumped = word_epoch(w).wrapping_add(1) & EPOCH_MASK;
|
||||
match word_state(w) {
|
||||
ST_PARKED => {
|
||||
if self
|
||||
.0
|
||||
.compare_exchange(
|
||||
w,
|
||||
pack(gen, bumped, ST_QUEUED),
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
return Unpark::Enqueue;
|
||||
}
|
||||
}
|
||||
ST_RUNNING => {
|
||||
if self
|
||||
.0
|
||||
.compare_exchange(
|
||||
w,
|
||||
pack(gen, bumped, ST_RUNNING_NOTIFIED),
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
return Unpark::Notified;
|
||||
}
|
||||
}
|
||||
_ => return Unpark::Noop, // Queued | Notified | Done | Vacant
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Eat a pending notification: RunningNotified → Running, epoch
|
||||
/// preserved; no-op on Running. Called by the RUNNING actor itself, on
|
||||
/// the no-park exit of a wait it registered for but never parked on
|
||||
/// (`select` returning a ready arm at registration time), AFTER bumping
|
||||
/// the epoch and BEFORE re-checking its stop flag:
|
||||
///
|
||||
/// - post-bump, the only wakers that can have set RunningNotified are
|
||||
/// ones stamped with the just-retired epoch (a select arm) or a
|
||||
/// terminal wildcard (`request_stop`);
|
||||
/// - the caller's stop-flag check AFTER the clear catches the terminal
|
||||
/// case (the flag is set before the wake fires), so eating its
|
||||
/// notification loses nothing — and a stop arriving later re-notifies
|
||||
/// a Running word as usual;
|
||||
/// - what remains eaten is exactly the stale arm wake that would
|
||||
/// otherwise fault the actor's next one-shot park.
|
||||
///
|
||||
/// Returns whether a notification was eaten.
|
||||
pub(crate) fn clear_notify(&self, gen: u32) -> bool {
|
||||
loop {
|
||||
let w = self.load();
|
||||
debug_assert!(
|
||||
matches!(word_state(w), ST_RUNNING | ST_RUNNING_NOTIFIED)
|
||||
&& word_gen(w) == gen,
|
||||
"clear_notify from invalid word {w:#x}"
|
||||
);
|
||||
if word_state(w) != ST_RUNNING_NOTIFIED {
|
||||
return false;
|
||||
}
|
||||
if self
|
||||
.0
|
||||
.compare_exchange(
|
||||
w,
|
||||
pack(gen, word_epoch(w), ST_RUNNING),
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Finalize: Running | RunningNotified → Done, epoch zeroed (wait
|
||||
/// identity never crosses an occupancy). Called by the scheduler that
|
||||
/// just ran the actor to completion (so those are the only legal prior
|
||||
/// states), under the slot's cold lock so join's check-or-register is
|
||||
/// linearized against it.
|
||||
pub(crate) fn set_done(&self, gen: u32) {
|
||||
let prev = self.0.swap(pack(gen, 0, ST_DONE), Ordering::AcqRel);
|
||||
debug_assert!(
|
||||
matches!(word_state(prev), ST_RUNNING | ST_RUNNING_NOTIFIED)
|
||||
&& word_gen(prev) == gen,
|
||||
"finalize from invalid word {prev:#x}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Reclaim: Done → Vacant(gen + 1). The generation bump IS the reclaim:
|
||||
/// every stale pid is dead from this store onwards. Caller holds the cold
|
||||
/// lock and has verified eligibility (asserted).
|
||||
pub(crate) fn reclaim(&self, gen: u32) {
|
||||
debug_assert_eq!(
|
||||
self.load(),
|
||||
pack(gen, 0, ST_DONE),
|
||||
"reclaim of a non-Done slot"
|
||||
);
|
||||
self.0
|
||||
.store(pack(gen.wrapping_add(1), 0, ST_VACANT), Ordering::Release);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// loom model tests — RUSTFLAGS="--cfg loom" cargo test --lib --release
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[cfg(all(test, loom))]
|
||||
mod loom_tests {
|
||||
use super::*;
|
||||
use loom::sync::atomic::{AtomicBool, AtomicUsize};
|
||||
use loom::sync::Arc;
|
||||
use loom::thread;
|
||||
use std::sync::atomic::Ordering as O;
|
||||
|
||||
/// THE lost-wakeup theorem. A waiter registers a condition check then
|
||||
/// parks (as every parking site does); a waker sets the condition then
|
||||
/// unparks. In every interleaving the waiter must end up runnable —
|
||||
/// parked-forever-with-condition-set must be unreachable.
|
||||
#[test]
|
||||
fn no_lost_wakeup_park_vs_unpark() {
|
||||
loom::model(|| {
|
||||
let word = Arc::new(StateWord::new());
|
||||
word.publish_queued(0);
|
||||
assert!(word.try_claim(0)); // scheduler claimed: actor Running
|
||||
let epoch = word.begin_wait(0); // actor opens the wait
|
||||
|
||||
let ready = Arc::new(AtomicBool::new(false));
|
||||
let enqueues = Arc::new(AtomicUsize::new(0));
|
||||
|
||||
// Waker: make the condition true, then wake the registered wait.
|
||||
let w = word.clone();
|
||||
let r = ready.clone();
|
||||
let e = enqueues.clone();
|
||||
let waker = thread::spawn(move || {
|
||||
r.store(true, O::SeqCst);
|
||||
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
|
||||
e.fetch_add(1, O::SeqCst);
|
||||
}
|
||||
});
|
||||
|
||||
// Waiter (as the scheduler executes it): re-check the condition,
|
||||
// park only if still false; a Notified park-return re-queues.
|
||||
let parked = if ready.load(O::SeqCst) {
|
||||
false // condition already visible: doesn't park at all
|
||||
} else if word.park_return(0) {
|
||||
true
|
||||
} else {
|
||||
enqueues.fetch_add(1, O::SeqCst); // notified → re-queued
|
||||
false
|
||||
};
|
||||
|
||||
waker.join().unwrap();
|
||||
|
||||
let w = word.load();
|
||||
if parked {
|
||||
// Parked is only a FINAL state if the waker's unpark moved it
|
||||
// back to Queued (+ one enqueue). Parked-and-stays-parked
|
||||
// would be the lost wakeup.
|
||||
assert_eq!(word_state(w), ST_QUEUED, "lost wakeup: parked forever");
|
||||
assert_eq!(enqueues.load(O::SeqCst), 1);
|
||||
} else {
|
||||
// Never more than one enqueue (at-most-once-enqueued).
|
||||
assert!(enqueues.load(O::SeqCst) <= 1);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Two concurrent unparkers, one parked actor: exactly one wins the
|
||||
/// enqueue (at-most-once), regardless of interleaving. Both stamped with
|
||||
/// the live epoch — the consuming bump is what serializes them.
|
||||
#[test]
|
||||
fn two_unparkers_one_enqueue() {
|
||||
loom::model(|| {
|
||||
let word = Arc::new(StateWord::new());
|
||||
word.publish_queued(0);
|
||||
assert!(word.try_claim(0));
|
||||
let epoch = word.begin_wait(0);
|
||||
assert!(word.park_return(0)); // actor parked
|
||||
|
||||
let enqueues = Arc::new(AtomicUsize::new(0));
|
||||
let mut hs = Vec::new();
|
||||
for _ in 0..2 {
|
||||
let w = word.clone();
|
||||
let e = enqueues.clone();
|
||||
hs.push(thread::spawn(move || {
|
||||
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
|
||||
e.fetch_add(1, O::SeqCst);
|
||||
}
|
||||
}));
|
||||
}
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
assert_eq!(enqueues.load(O::SeqCst), 1);
|
||||
assert_eq!(word_state(word.load()), ST_QUEUED);
|
||||
});
|
||||
}
|
||||
|
||||
/// The stale-epoch theorem — what `select`'s loser arms lean on. An
|
||||
/// actor opens a wait, two registered wakers race it (against the park
|
||||
/// itself, covering the prep-to-park window); afterwards the actor is
|
||||
/// runnable exactly once, and a LATE waker still stamped with the
|
||||
/// consumed epoch can neither enqueue nor notify — in every
|
||||
/// interleaving. (Under wildcard semantics the late waker would corrupt
|
||||
/// the actor's NEXT one-shot park; this is the theorem that buys
|
||||
/// `Mutex::lock_timeout`/`sleep`/`block_on_io` their unchanged code.)
|
||||
#[test]
|
||||
fn consumed_epoch_unpark_never_lands() {
|
||||
loom::model(|| {
|
||||
let word = Arc::new(StateWord::new());
|
||||
word.publish_queued(0);
|
||||
assert!(word.try_claim(0));
|
||||
let epoch = word.begin_wait(0);
|
||||
|
||||
// Two arms race the wake, concurrent with the park itself.
|
||||
let enqueues = Arc::new(AtomicUsize::new(0));
|
||||
let mut hs = Vec::new();
|
||||
for _ in 0..2 {
|
||||
let w = word.clone();
|
||||
let e = enqueues.clone();
|
||||
hs.push(thread::spawn(move || {
|
||||
if w.unpark(0, Some(epoch)) == Unpark::Enqueue {
|
||||
e.fetch_add(1, O::SeqCst);
|
||||
}
|
||||
}));
|
||||
}
|
||||
let mut runnable_via_notify = false;
|
||||
if !word.park_return(0) {
|
||||
runnable_via_notify = true; // notified in prep-to-park
|
||||
}
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
|
||||
// Exactly one path made the actor runnable.
|
||||
let direct = enqueues.load(O::SeqCst);
|
||||
if runnable_via_notify {
|
||||
assert_eq!(direct, 0, "woken twice: notify AND enqueue");
|
||||
} else {
|
||||
assert_eq!(direct, 1, "parked forever, or woken twice");
|
||||
}
|
||||
assert_eq!(word_state(word.load()), ST_QUEUED);
|
||||
|
||||
// The actor runs again. A waker still holding the OLD epoch —
|
||||
// a select loser arm firing later — must be a strict no-op,
|
||||
// not a pending notification.
|
||||
assert!(word.try_claim(0));
|
||||
assert_eq!(word.unpark(0, Some(epoch)), Unpark::Noop);
|
||||
assert_eq!(word_state(word.load()), ST_RUNNING, "stale epoch notified a live run");
|
||||
});
|
||||
}
|
||||
|
||||
/// The retire theorem — `select`'s no-park exit. An actor opens a wait
|
||||
/// and registers, then finds an arm ready and returns WITHOUT parking;
|
||||
/// a loser arm's waker fires concurrently, stamped with the live epoch.
|
||||
/// The exit retires the wait (bump, then eat): in every interleaving
|
||||
/// the run ends on a clean Running word — no pending notification
|
||||
/// survives to fault the actor's next one-shot park — and the waker
|
||||
/// never enqueues.
|
||||
#[test]
|
||||
fn retire_eats_late_arm_notification() {
|
||||
loom::model(|| {
|
||||
let word = Arc::new(StateWord::new());
|
||||
word.publish_queued(0);
|
||||
assert!(word.try_claim(0));
|
||||
let epoch = word.begin_wait(0); // select opens + registers
|
||||
|
||||
let w = word.clone();
|
||||
let waker = thread::spawn(move || w.unpark(0, Some(epoch)));
|
||||
|
||||
// No-park exit: bump (invalidates in-flight wakes), then eat
|
||||
// (consumes one that already landed).
|
||||
let _ = word.begin_wait(0);
|
||||
word.clear_notify(0);
|
||||
|
||||
assert_ne!(waker.join().unwrap(), Unpark::Enqueue);
|
||||
assert_eq!(
|
||||
word_state(word.load()),
|
||||
ST_RUNNING,
|
||||
"stale arm wake survived the retire"
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/// The ABA theorem: a stale-generation unpark racing reclaim + reuse can
|
||||
/// never touch the slot's new occupant.
|
||||
#[test]
|
||||
fn stale_unpark_never_hits_reused_slot() {
|
||||
loom::model(|| {
|
||||
let word = Arc::new(StateWord::new());
|
||||
// Gen-0 actor runs to completion.
|
||||
word.publish_queued(0);
|
||||
assert!(word.try_claim(0));
|
||||
|
||||
let w = word.clone();
|
||||
let stale = thread::spawn(move || w.unpark(0, None));
|
||||
|
||||
// Scheduler: finalize, reclaim, and a new spawn reuses the slot.
|
||||
word.set_done(0);
|
||||
word.reclaim(0);
|
||||
word.publish_queued(1);
|
||||
|
||||
// The stale unpark may have squeezed in only while gen 0 was
|
||||
// still Running (→ Notified) — in which case set_done's swap
|
||||
// absorbed it — or it observed Done/Vacant/gen-1 and no-op'd.
|
||||
// Either way it must never claim an enqueue.
|
||||
assert_ne!(stale.join().unwrap(), Unpark::Enqueue);
|
||||
// And the new occupant is exactly where its spawn put it.
|
||||
assert_eq!(word.load(), pack(1, 0, ST_QUEUED));
|
||||
});
|
||||
}
|
||||
|
||||
/// Unpark racing the claim itself: whatever the interleaving, the actor
|
||||
/// is Running or RunningNotified afterwards and nobody enqueued (it was
|
||||
/// never parked).
|
||||
#[test]
|
||||
fn unpark_vs_claim_coalesces() {
|
||||
loom::model(|| {
|
||||
let word = Arc::new(StateWord::new());
|
||||
word.publish_queued(0);
|
||||
|
||||
let w = word.clone();
|
||||
let unparker = thread::spawn(move || w.unpark(0, None));
|
||||
|
||||
assert!(word.try_claim(0)); // the entry is ours; claim must win
|
||||
let r = unparker.join().unwrap();
|
||||
assert_ne!(r, Unpark::Enqueue);
|
||||
let st = word_state(word.load());
|
||||
assert!(matches!(st, ST_RUNNING | ST_RUNNING_NOTIFIED));
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -1,37 +0,0 @@
|
||||
//! std vs loom indirection for the modules that loom model-checks
|
||||
//! (`slot_state`, `run_queue`). Everything else uses std paths directly —
|
||||
//! the full runtime (context switches, futexes, real TLS) is not loom-able
|
||||
//! and is never executed under `cfg(loom)`.
|
||||
//!
|
||||
//! Build the loom models with: `RUSTFLAGS="--cfg loom" cargo test --lib --release`
|
||||
|
||||
#[cfg(loom)]
|
||||
pub(crate) use loom::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
|
||||
|
||||
#[cfg(not(loom))]
|
||||
pub(crate) use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
|
||||
|
||||
/// `UnsafeCell` with loom's `with`/`with_mut` access API; pass-through cost
|
||||
/// is zero in normal builds (`#[inline]`, newtype over std's cell).
|
||||
#[cfg(loom)]
|
||||
pub(crate) use loom::cell::UnsafeCell;
|
||||
|
||||
#[cfg(not(loom))]
|
||||
pub(crate) struct UnsafeCell<T>(std::cell::UnsafeCell<T>);
|
||||
|
||||
#[cfg(not(loom))]
|
||||
impl<T> UnsafeCell<T> {
|
||||
pub(crate) fn new(v: T) -> Self {
|
||||
Self(std::cell::UnsafeCell::new(v))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub(crate) fn with<R>(&self, f: impl FnOnce(*const T) -> R) -> R {
|
||||
f(self.0.get())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub(crate) fn with_mut<R>(&self, f: impl FnOnce(*mut T) -> R) -> R {
|
||||
f(self.0.get())
|
||||
}
|
||||
}
|
||||
+16
-19
@@ -18,9 +18,9 @@
|
||||
//! No cancellation. When a non-timer wakeup happens (e.g. lock granted
|
||||
//! before timeout), the timer entry is left in the heap. It will be popped
|
||||
//! eventually and the dispatch will observe "actor is no longer parked /
|
||||
//! the wait's epoch was consumed" and no-op. Cost is ~32 bytes per stale
|
||||
//! entry plus a few cycles on pop; acceptable given the upper bound is "one
|
||||
//! entry per parked actor".
|
||||
//! wait_seq is stale" and no-op. Cost is ~32 bytes per stale entry plus a
|
||||
//! few cycles on pop; acceptable given the upper bound is "one entry per
|
||||
//! parked actor".
|
||||
//!
|
||||
//! Stale pids (slot reused since the timer was inserted) are filtered on
|
||||
//! pop by the scheduler — same convention as the run queue.
|
||||
@@ -35,21 +35,18 @@ use std::time::{Duration, Instant};
|
||||
///
|
||||
/// Held inside `Entry`, dispatched by the scheduler in `pop_due`.
|
||||
pub enum Reason {
|
||||
/// `sleep(d)`. Wake `pid` via the epoch-matched unpark: if anything
|
||||
/// else (necessarily a terminal wake) already consumed the wait, the
|
||||
/// entry is stale and no-ops at the CAS.
|
||||
Sleep { epoch: u32 },
|
||||
/// A bounded wait (`Mutex::lock_timeout`, `Receiver::recv_timeout`,
|
||||
/// `select_timeout`). On expiry the scheduler calls
|
||||
/// `target.on_timeout(pid, epoch)`. The target then decides whether
|
||||
/// `pid` was actually still waiting (registration still present under
|
||||
/// its lock), and if so takes the registration and unparks via
|
||||
/// `unpark_at`. The epoch is the slot-word park-epoch — the runtime-wide
|
||||
/// wait identity — so a stale entry is doubly inert: the registration
|
||||
/// check misses, and even a racing unpark fails the word's epoch CAS.
|
||||
/// `loom::sleep(d)`. Unpark `pid` unconditionally (modulo the usual
|
||||
/// "still parked?" check the scheduler applies).
|
||||
Sleep,
|
||||
/// A bounded wait — currently only `Mutex::lock_timeout`. On expiry the
|
||||
/// scheduler calls `target.on_timeout(pid, wait_seq)`. The target then
|
||||
/// decides whether `pid` was actually still waiting, and if so unparks
|
||||
/// it with whatever error the wait was bounded for. `wait_seq` lets the
|
||||
/// target tell apart "this wait" from "a later wait by the same actor
|
||||
/// on the same target".
|
||||
WaitTimeout {
|
||||
target: Arc<dyn TimerTarget>,
|
||||
epoch: u32,
|
||||
wait_seq: u64,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -58,7 +55,7 @@ pub enum Reason {
|
||||
/// Implementors: do not touch `SchedulerState` other than via the public
|
||||
/// `unpark` / channel APIs. The scheduler is mid-iteration when this fires.
|
||||
pub trait TimerTarget: Send + Sync {
|
||||
fn on_timeout(&self, pid: Pid, epoch: u32);
|
||||
fn on_timeout(&self, pid: Pid, wait_seq: u64);
|
||||
}
|
||||
|
||||
pub struct Entry {
|
||||
@@ -107,8 +104,8 @@ impl Timers {
|
||||
}
|
||||
|
||||
/// Insert a `Sleep` timer. Convenience for the common case.
|
||||
pub fn insert_sleep(&mut self, deadline: Instant, pid: Pid, epoch: u32) {
|
||||
self.insert(deadline, pid, Reason::Sleep { epoch });
|
||||
pub fn insert_sleep(&mut self, deadline: Instant, pid: Pid) {
|
||||
self.insert(deadline, pid, Reason::Sleep);
|
||||
}
|
||||
|
||||
/// Insert an arbitrary timer entry.
|
||||
|
||||
@@ -58,9 +58,6 @@ mod inner {
|
||||
// Queue
|
||||
Enqueue(Pid),
|
||||
Dequeue(Pid),
|
||||
// RFC 005 wake slot
|
||||
SlotPush(Pid), // actor-context wake parked in the waking thread's slot
|
||||
SlotPop(Pid), // scheduler resumed a pid from its own slot
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
@@ -240,8 +237,6 @@ mod inner {
|
||||
Event::RecvWake(p) => ("recv_wake".into(), p.index()),
|
||||
Event::Enqueue(p) => ("enqueue".into(), p.index()),
|
||||
Event::Dequeue(p) => ("dequeue".into(), p.index()),
|
||||
Event::SlotPush(p) => ("slot_push".into(), p.index()),
|
||||
Event::SlotPop(p) => ("slot_pop".into(), p.index()),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
# smarm — task.md (next steps)
|
||||
|
||||
Handoff for a future session reusing this sandbox. Read top to bottom once
|
||||
before starting; the gotchas section is hard-won and will save you a faceplant.
|
||||
|
||||
## Resume the environment
|
||||
|
||||
- Repo: `smarm`. Two branches (the old single `arm-port` stack was split):
|
||||
- `master` — the mainline, and HEAD. Carries roadmap #1–#5: cooperative
|
||||
cancellation, supervisor strategies (one_for_one/all, rest_for_one) + the
|
||||
orphaned-timer shutdown fix, links/trap_exit, selective receive, gen_server,
|
||||
and demonitor/`MonitorId`. Tagged `v0.4.0`. x86-64 Linux only.
|
||||
- `arm-port` — `master` plus a single commit: `feat(arch): aarch64 context
|
||||
switch + cycle counter`. Extracts the x86-64 context-switch / stack-init /
|
||||
cycle-counter out of `context.rs` into a `target_arch`-gated `src/arch/`
|
||||
(x86_64 + aarch64 backends) and adds an AAPCS64 backend. ⚠️ UNTESTED: never
|
||||
built or run on real ARM hardware. The x86-64 path is unchanged
|
||||
(`arch/x86_64.rs` is the old `context.rs` body verbatim), so the x86 suite
|
||||
passing says nothing about the aarch64 backend. Build + test on-device
|
||||
before trusting it.
|
||||
- Toolchain is installed but NOT on PATH in a fresh shell. First line of every
|
||||
session: `. "$HOME/.cargo/env"` (rustc/cargo 1.96).
|
||||
- Build `cargo build` · all tests `cargo test` · one suite `cargo test --test monitor`.
|
||||
- Bench probe `cargo bench --bench general` (custom print-only harness; compiles
|
||||
tokio in release the first time — slow — but `target/` persists across git
|
||||
checkouts so it's paid once).
|
||||
- Perf regression check: `git checkout <pre-change-sha>` →
|
||||
`cargo bench --bench general | tee before.txt` → `git checkout arm-port` →
|
||||
run again → diff the `smarm 1-thread` medians for `chained_spawn` and
|
||||
`yield_many` (those exercise spawn/finalize/scheduler). Numbers are noisy on
|
||||
this shared CPU; treat as "regression beyond noise?" not a precise delta.
|
||||
|
||||
## Roadmap (dependency order)
|
||||
|
||||
### 1. Cooperative cancellation — the keystone ✅ DONE (`a8ddb4a`)
|
||||
Everything below (one_for_all/rest_for_one, links) needs a *safe* way to stop a
|
||||
running peer. Forcible teardown of another green thread's stack is unsound here
|
||||
(shared heap + Drop). So: cooperative stop the actor observes and unwinds itself.
|
||||
|
||||
Shipped as designed (sentinel unwind, not Result-threading). Notes for what
|
||||
came next / future readers:
|
||||
- Stop flag lives on `Actor` behind `Arc<AtomicBool>` (fresh per spawn), NOT a
|
||||
`Slot` field — sidesteps the three-place reset, at the cost of one small
|
||||
alloc per spawn. The scheduler hands the resume path a raw `*const AtomicBool`
|
||||
(no per-resume refcount traffic); `yield_many` bench stayed at baseline,
|
||||
`chained_spawn` ~+6% from that alloc (left as-is; move to a `Slot` field if it
|
||||
ever matters).
|
||||
- Observation points: amortised `maybe_preempt`/`check!()` path + the wakeup
|
||||
side of `park_current`/`yield_now`. Sentinel = `StopSentinel` (zero-size),
|
||||
recognised in the trampoline → `Outcome::Stopped`. `join()` on a stopped actor
|
||||
returns `Ok(())` (no payload to propagate; reason is on the monitor channel).
|
||||
- Documented gaps confirmed by tests: no-observation-point loop can't be stopped
|
||||
(same as preemption); a user `catch_unwind` can swallow the sentinel but the
|
||||
flag stays set so the next yield re-raises.
|
||||
|
||||
Original plan, for reference:
|
||||
- Add a per-actor stop flag (Slot field + atomic, or check via shared state).
|
||||
- `request_stop(pid)`: set the flag, unpark if parked.
|
||||
- Realize the stop as a **controlled unwind**: when the scheduler resumes a
|
||||
stop-requested actor, inject a sentinel panic (dedicated payload type) so the
|
||||
existing `trampoline` `catch_unwind` tears the stack down and runs Drop. The
|
||||
trampoline recognizes the sentinel and reports a new `Outcome::Stopped`
|
||||
(distinct from a user `Panic`). This avoids changing every blocking-op
|
||||
signature.
|
||||
- Alternative considered: thread `Result<_, Cancelled>` through recv/sleep/
|
||||
lock/io. Rejected — large API churn. Go with the sentinel unwind.
|
||||
- Caveat to document: user code with its own `catch_unwind` can swallow the
|
||||
sentinel (cf. Erlang `catch`); re-check the flag at the next yield and/or
|
||||
re-raise. And a tight no-alloc loop without `check!()` can't be stopped —
|
||||
same inherent limitation as preemption.
|
||||
- Observation points: `maybe_preempt()`/`check!()` (cheap flag check) and the
|
||||
blocking parks (recv/sleep/mutex/io) on the stop-driven unpark.
|
||||
- Tests: looping actor on `check!()` gets stopped → `Outcome::Stopped`, Drop
|
||||
guards ran; parked-on-recv actor gets stopped; no-check loop documents the gap.
|
||||
|
||||
### 2. one_for_all / rest_for_one + ordered shutdown ✅ DONE (`351dc9c`)
|
||||
Shipped. What landed vs the plan:
|
||||
- `Strategy::{OneForOne,OneForAll,RestForOne}` selected via `.strategy()`,
|
||||
default `OneForOne`. The struct keeps the `OneForOne` name (compat; existing
|
||||
tests untouched) — a rename to `Supervisor` is a deferred refactor.
|
||||
- The *triggering* child's `Restart` policy decides whether anything restarts;
|
||||
the strategy decides which live siblings are cycled (all / index-> after the
|
||||
failed one). Survivors are `request_stop`'d in reverse start order, awaited on
|
||||
the existing `supervisor_channel` funnel (no new channel, no `select`),
|
||||
restarted in start order. One failure = one intensity tick regardless of group
|
||||
size. Out-of-band signals during an await are stashed and replayed.
|
||||
- `Signal::Stopped(pid)` + `DownReason::Stopped` added (kept distinct from Exit,
|
||||
as planned). A `Stopped` signal counts as abnormal for the restart decision.
|
||||
- Ordered shutdown: on cap-trip / mailbox-close-with-survivors, stop remaining
|
||||
children in reverse start order and await them (no-op on the normal exit).
|
||||
- ⚠️ Surfaced + fixed a latent keystone bug (`e80334b`): a cancelled
|
||||
sleeping/timeout actor orphans its timer entry, and the scheduler's shutdown
|
||||
check counted pending timers → `run()` hung until the dead actor's deadline
|
||||
fired (a `sleep(30s)` sleeper hung shutdown 30s). Fix: timers no longer gate
|
||||
shutdown (`live == 0` already implies nothing a timer could wake); heap is
|
||||
cleared on exit. Independent of the supervisor work.
|
||||
- Tests: all-restart (sibling cycled despite clean exit), suffix-restart
|
||||
(prefix child left alone), reverse-order teardown.
|
||||
|
||||
Original plan, for reference:
|
||||
- one_for_all: on any child failure, `request_stop` all siblings, await their
|
||||
termination signals, restart all per spec.
|
||||
- rest_for_one: stop+restart the failed child and those started after it.
|
||||
- Supervisor shutdown: stop children in reverse start order.
|
||||
- Decide signal surface: add `Signal::Stopped(pid)` + `DownReason::Stopped`
|
||||
rather than folding into Exit (clearer for the supervisor's await logic).
|
||||
- Tests: all-restart, suffix-restart, reverse-order shutdown.
|
||||
|
||||
### 3. Links + trap_exit ✅ DONE (`6581484`)
|
||||
- `Slot.links: Vec<Pid>` (bidirectional); `link`/`unlink`; `trap_exit()` flag
|
||||
lives on `Actor` (fresh per spawn → a restarted child starts un-trapped, and
|
||||
no fourth slot-reset site).
|
||||
- On finalize, reverse links are cleared under the lock (always — keeps the
|
||||
cascade acyclic), then for each linked peer: abnormal death (`Panic`/
|
||||
`Stopped`) → `request_stop(peer)` unless peer traps, in which case deliver an
|
||||
`ExitSignal` *message* instead. Normal exit does NOT propagate. Linking an
|
||||
already-dead pid delivers an immediate `NoProc` signal (message if trapping,
|
||||
else `request_stop(self)` — not a silent no-op).
|
||||
- Resolved: the trap inbox is a **dedicated** channel (`trap_exit() ->
|
||||
Receiver<ExitSignal>`), distinct from the monitor `Down` channel; `ExitSignal`
|
||||
reuses `DownReason` and carries no panic payload (joiner-only, as with
|
||||
monitors). `spawn_link` deferred to #5.
|
||||
- Tests (`tests/link.rs`): linked pair one panics → other stopped (+Drop ran);
|
||||
trap_exit → other gets a message and survives; normal exit doesn't propagate;
|
||||
dead-pid link stops a non-trapper / messages a trapper; `unlink` prevents
|
||||
propagation.
|
||||
|
||||
### 4. Selective receive (independent track) ✅ DONE (`03f3875`)
|
||||
Shipped. What landed vs the plan:
|
||||
- `Receiver::recv_match(pred) -> Result<T, RecvError>` scans the queued
|
||||
`VecDeque` front-to-back, removes+returns the first match, leaves the rest in
|
||||
arrival order; parks and re-scans when nothing matches. `try_recv_match` (the
|
||||
non-blocking variant, mirroring `try_recv`) rolled in same commit.
|
||||
- Wakeup turned out cheaper than feared: `Sender::send` *already* took
|
||||
`parked_receiver` on every push, so "wake on ANY send" needed no send-side
|
||||
change. The one real edit was relaxing `Sender::drop` to unpark the parked
|
||||
receiver on the last-sender drop regardless of queue emptiness — a selective
|
||||
receiver can park on a non-empty no-match queue and must wake to observe
|
||||
closure. No-op for plain `recv` (only ever parks on an empty queue); stress
|
||||
suite stays green.
|
||||
- `pred` is `Fn(&T) -> bool` (not `FnMut`) on purpose: it's re-run from scratch
|
||||
on every scan, so a stateful predicate would re-count surprisingly. It runs
|
||||
under the channel lock — keep it cheap/pure, don't re-enter the channel.
|
||||
- Close semantics: `recv_match` returns `Err(RecvError)` only when closed AND no
|
||||
queued message matches; a match is still returned on a closed channel.
|
||||
Non-matches are left for a later `recv`.
|
||||
- Tests (`tests/selective_recv.rs`): out-of-order match pulled first; non-matches
|
||||
remain in order; park-on-non-empty then wake on a match; closed-with-only-
|
||||
non-matches → Err; closed-but-match-present → match; `try_recv_match` states.
|
||||
|
||||
Original plan, for reference:
|
||||
- Add `Receiver::recv_match(pred) -> T`: scan the queued `VecDeque`, remove+return
|
||||
first match, leave the rest in order; park and re-scan on new arrivals.
|
||||
- This changes channel wakeup: a parked selective receiver must wake on ANY send
|
||||
(not just empty→nonempty) and re-scan. Touches `channel.rs` carefully — the
|
||||
stress tests guard lost-wakeup invariants; keep them green.
|
||||
- Tests: messages arrive out of interest-order; match pulled first; non-matches
|
||||
remain for a later `recv`.
|
||||
|
||||
### 5. Grab-bag (each its own small commit)
|
||||
- `spawn_link`: spawn-and-link atomically (deferred from #3); thin wrapper over
|
||||
`spawn_under` + `link`, but do it under one lock so there's no window where
|
||||
the child dies before the link is recorded.
|
||||
- `demonitor`: needs a per-monitor id to remove a specific sender. Decide the
|
||||
monitor API NOW before more code depends on it — likely return a
|
||||
`Monitor { id, rx }` instead of a bare `Receiver<Down>`.
|
||||
✅ DONE (this commit). What landed vs the plan:
|
||||
- `monitor()` now returns `Monitor { id, target, rx }` (added `target` over
|
||||
the sketched `{id, rx}` so `demonitor` jumps straight to the slot instead of
|
||||
scanning every slot for the id). `MonitorId(u64)` is opaque, from a
|
||||
monotonic `next_monitor_id` counter on `SharedState`, bumped under the
|
||||
shared lock in `monitor()` — no atomics, deterministic, never reused.
|
||||
- `Slot.monitors: Vec<(MonitorId, Sender<Down>)>`. The three slot-reset sites
|
||||
were untouched — they `.clear()`/`Vec::new()`, which is element-type-
|
||||
agnostic, so no new reset obligation. `finalize_actor` just destructures
|
||||
`(_, m)` and sends as before.
|
||||
- `demonitor(&Monitor) -> Option<MonitorId>`: `Some(id)` when a live
|
||||
registration was found+removed, `None` when it had already fired (drained on
|
||||
finalize), was `NoProc`, or the slot was reclaimed. Chose `Option<MonitorId>`
|
||||
over a bare bool — names which registration went. Generation half of the pid
|
||||
makes a stale demonitor a clean no-op: a recycled slot index fails
|
||||
`slot_mut`'s generation check, so it can never strip a different actor's
|
||||
monitor.
|
||||
- ⚠️ Reentrancy: the removed `Sender` is `remove`d out of the Vec under the
|
||||
lock but **dropped after the lock is released** — `Sender::drop` can unpark a
|
||||
parked receiver → `with_shared`, and the shared mutex is non-reentrant. Same
|
||||
discipline as `finalize_actor`.
|
||||
- "Flush" (discard a `Down` the target already queued) falls out of dropping
|
||||
the `Monitor`: `demonitor(&m); drop(m)`. That's the cleanup the still-to-come
|
||||
gen_server **call timeout** wants — monitor the server, wait reply-or-Down-
|
||||
or-deadline, then demonitor+drop so a timed-out call leaks no registration
|
||||
and no stale `Down`.
|
||||
- Perf: touches `Slot` + `finalize_actor`, but `chained_spawn`/`yield_many`
|
||||
register no monitors, so the Vec stays empty (take-empty is identical cost,
|
||||
finalize loop runs zero times). before/after `general` probe medians within
|
||||
noise. Tests (`tests/monitor.rs`): demonitor-stops-delivery, one-of-many
|
||||
(siblings untouched), after-fire-is-None.
|
||||
- Named registry: `register(name,pid)`/`whereis`/`send_by_name`; a
|
||||
`HashMap<String,Pid>` in `SharedState`.
|
||||
- gen_server-style call/cast: request-reply correlation as a thin layer over
|
||||
channels (`call` sends `{req, reply_tx}`, awaits `reply_rx`); no runtime change.
|
||||
✅ DONE (`a4fcf6c`). What landed vs the plan:
|
||||
- `GenServer` trait on the state value: assoc `Call`/`Reply`/`Cast` types,
|
||||
required `handle_call`/`handle_cast`, optional `init`/`terminate` hooks.
|
||||
`ServerRef<G>` is a clonable inbox sender + `pid()`; `start` / `start_under`.
|
||||
- One inbox, not two: a single `Envelope { Call(req, reply_tx) | Cast }`
|
||||
channel, dispatched by variant. Forced by no-`select`/no-unified-mailbox —
|
||||
a server can't wait on a call channel and a cast channel at once.
|
||||
- Server-down falls out of channel closure (no monitor needed): `send` fails
|
||||
if the inbox is gone; the reply sender drops on the server's unwind so a
|
||||
parked caller wakes to `Err`. Both → `Call/CastError::ServerDown`.
|
||||
- `terminate` runs via a drop guard → fires on *every* exit path (clean inbox
|
||||
close, handler panic, `request_stop`), not just the clean one. Caveat: it
|
||||
may run mid-unwind, so keep it non-blocking (a panic inside it during an
|
||||
unwind double-panics → abort).
|
||||
- No `handle_info`, no call timeout — both deferred to land with timeouts
|
||||
(`handle_info` needs the still-unmade cross-channel mailbox merge; a call
|
||||
timeout needs a per-`recv` deadline / `Signal::Timeout`).
|
||||
- Pure additive layer (no Slot/scheduler/spawn/finalize change) → no perf
|
||||
check run. Tests (`tests/gen_server.rs`): cast→call roundtrip, init/terminate
|
||||
ordering, both server-down paths.
|
||||
- Docs: README now points at the experimental, untested aarch64 port on the
|
||||
`arm-port` branch. The module table still calls `context` x86-64-only — true
|
||||
for `master`, since the `src/arch/` split rides on `arm-port`. Fold the arch/
|
||||
split and ARM64-supported wording into the README module table + build section
|
||||
once `arm-port` is validated on hardware and merged.
|
||||
|
||||
## Gotchas / invariants (respect these)
|
||||
|
||||
- **Shared mutex is non-reentrant.** `Sender::send` can call `unpark` →
|
||||
`with_shared`. NEVER send on a channel while holding the shared lock. Pattern:
|
||||
`mem::take` the senders/data under the lock, send after releasing. See
|
||||
`finalize_actor` (supervisor signal + monitor Downs both sent post-lock).
|
||||
- **`finalize_actor` order:** take stack/waiters/monitors under lock + set
|
||||
Done/outcome → recycle stack (post-lock) → deliver supervisor Signal + monitor
|
||||
Downs (post-lock) → unpark joiners → reclaim slot iff `outstanding_handles==0`.
|
||||
Death notifications always precede reclamation, so a pid carried in a
|
||||
Signal/Down is still matchable even as its slot is about to be reused.
|
||||
- **Slot lifecycle is reset in THREE places** — `Slot::vacant()`,
|
||||
`reclaim_slot()` (runtime.rs), and the slot-init block in `spawn_under`
|
||||
(scheduler.rs). Any new Slot field must be reset in all three (monitors was).
|
||||
- **Pid = (index, generation);** stale handles caught by generation mismatch in
|
||||
`slot()/slot_mut()`. The monitor `NoProc` path relies on this.
|
||||
- **No `select`, no unified per-process mailbox.** Why the supervisor uses the
|
||||
single `supervisor_channel` funnel rather than N monitor channels. trap_exit
|
||||
resolved this by giving each trapping actor a dedicated `Receiver<ExitSignal>`
|
||||
inbox (see #3); selective receive (#4) stayed *per-channel* (`recv_match`
|
||||
scans one channel's queue) rather than introducing a cross-channel mailbox —
|
||||
if selective receive ever needs to span the monitor/trap inboxes too, that
|
||||
cross-channel merge is the still-unmade decision.
|
||||
- **Cooperative-only**: preemption and (future) cancellation both depend on the
|
||||
actor reaching `check!()`/yield/alloc/blocking points.
|
||||
- `run()` is single-thread (`Config::exact(1)`); tests rely on deterministic
|
||||
single-thread ordering (parent runs until it parks). Multi-thread via
|
||||
`runtime::init(Config…)`.
|
||||
|
||||
## Workflow expectations (from the human)
|
||||
|
||||
- TDD: write the failing test first, then implement.
|
||||
- Commit incrementally with conventional-commit messages; keep each commit a
|
||||
reviewable unit (they diff in their IDE and are the filter to the codebase).
|
||||
- Run the full suite before each commit; check perf when a change touches
|
||||
Slot/scheduler/spawn/finalize hot paths.
|
||||
-120
@@ -1,120 +0,0 @@
|
||||
# Tests
|
||||
|
||||
Integration tests for the runtime. Each file owns one feature area or one
|
||||
class of bug. Everything here runs under plain `cargo test`; the loom model
|
||||
tests are the exception — they live **in the library** (`src/slot_state.rs`,
|
||||
`src/run_queue.rs`), not in this directory, because loom must compile the
|
||||
production code with shimmed atomics (see "Loom" below).
|
||||
|
||||
## Running
|
||||
|
||||
```
|
||||
cargo test # debug build — RUN THIS ONE: all invariant asserts live
|
||||
cargo test --release # what users actually execute (LTO, no debug_asserts)
|
||||
```
|
||||
|
||||
Debug builds are not just "slower tests": the runtime self-checks its
|
||||
invariants only there — every `StateWord` transition asserts its
|
||||
precondition, `enqueue` asserts the exact `(gen, Queued)` word, `RawMutex`
|
||||
enforces the never-two-cold-locks leaf rule with a per-thread held-count,
|
||||
`live_actors` checks for double-finalize underflow. A green release run with
|
||||
a red debug run means an invariant broke without (yet) corrupting behavior —
|
||||
treat it as a real failure.
|
||||
|
||||
### The queue-variant matrix
|
||||
|
||||
The run queue is compile-time selected; the suite must pass under all three
|
||||
(features are additive, so drop the default first):
|
||||
|
||||
```
|
||||
cargo test # rq-mutex (default)
|
||||
cargo test --no-default-features --features rq-mpmc
|
||||
cargo test --no-default-features --features rq-striped
|
||||
```
|
||||
|
||||
### Loom (model checking)
|
||||
|
||||
```
|
||||
RUSTFLAGS="--cfg loom" cargo test --lib --release
|
||||
```
|
||||
|
||||
Exhaustively explores interleavings of the slot state machine
|
||||
(`src/slot_state.rs`: lost-wakeup, at-most-once-enqueue, the stale-pid ABA
|
||||
theorem, unpark-vs-claim) and the ring queues (`src/run_queue.rs`:
|
||||
exactly-once through lap wraparound, push/pop races). Models run the
|
||||
production transitions through `src/sync_shim.rs` — std atomics normally,
|
||||
`loom::sync` under `--cfg loom`. `RawMutex` is deliberately not modeled:
|
||||
futexes can't be, and it's the textbook Drepper mutex3 with stress and
|
||||
unwind-safety tests of its own.
|
||||
|
||||
### Trace feature
|
||||
|
||||
`cargo test --features smarm-trace` exists mainly to catch bit-rot in the
|
||||
`te!()` call sites; run it after touching scheduler paths.
|
||||
|
||||
### Before a runtime-core PR
|
||||
|
||||
The full matrix, in rough order of bug-finding power per minute:
|
||||
|
||||
1. `cargo test` (debug, default variant)
|
||||
2. debug under `rq-mpmc` and `rq-striped`
|
||||
3. `cargo test --release`
|
||||
4. loom
|
||||
5. `cargo build --features smarm-trace`
|
||||
|
||||
## Catalog
|
||||
|
||||
**Low-level units (no scheduler)**
|
||||
| file | covers |
|
||||
|---|---|
|
||||
| `context.rs` | `init_actor_stack` + the naked-asm context-switch shims, poked directly |
|
||||
| `stack.rs` | the mmap'd stack allocator |
|
||||
| `pid.rs` | pid packing/equality |
|
||||
|
||||
**Feature areas (run under a real runtime)**
|
||||
| file | covers |
|
||||
|---|---|
|
||||
| `runtime.rs` | `Config`, `Runtime::run`, re-running a runtime, correctness under genuine parallelism |
|
||||
| `scheduler.rs` | spawn / join / panic delivery / `yield_now` / `self_pid` |
|
||||
| `channel.rs` | send/recv (recv parks, so these need the runtime) |
|
||||
| `selective_recv.rs` | `recv_match` / `try_recv_match` |
|
||||
| `mutex.rs` | the actor-blocking `Mutex<T>` (lock parks) |
|
||||
| `timer.rs` | `sleep` ordering — time-sensitive, generous tolerances by design |
|
||||
| `io.rs` | `block_on_io`: blocking closures on the pool while the actor parks |
|
||||
| `io_epoll.rs` | `wait_readable` / `wait_writable` + the `read`/`write` sugar |
|
||||
| `preempt.rs` | explicit preemption via `smarm::check!()` |
|
||||
| `cancel.rs` | cooperative cancellation (`request_stop`) — the keystone semantics |
|
||||
| `monitor.rs` | `monitor` delivers exactly one `Down`; `demonitor` |
|
||||
| `link.rs` | bidirectional links + `trap_exit` |
|
||||
| `supervisor.rs` | one-for-one supervision |
|
||||
| `gen_server.rs` | call/cast round-trips, lifecycle callbacks, server-down detection |
|
||||
|
||||
**Regression & stress**
|
||||
| file | covers |
|
||||
|---|---|
|
||||
| `stress.rs` | lost wakeups, pid-table pressure, thundering herds, panic isolation under concurrency. Where the phase-2 RefCell-migration bug was caught. |
|
||||
| `poison_stop.rs` | `request_stop` racing an alloc-under-lock must not poison/abort. See its header for the full story. |
|
||||
| `many_timers_multi_thread.rs` | multi-thread sleep-timer lost-wakeup regression |
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Each test owns its runtime.** `init(Config::exact(N))` + `rt.run(...)`;
|
||||
never share a `Runtime` between tests. Oversubscription (`exact(4)` on one
|
||||
core) is deliberate — forced interleaving at yield points is how
|
||||
single-core CI finds races at all.
|
||||
- **Regression tests must be validated against the bug.** A regression test
|
||||
that passes with the bug reintroduced is documentation, not a test.
|
||||
Reintroduce the fix's inverse locally and watch it fail before trusting it
|
||||
(`poison_stop.rs` went through exactly this: its first version never fired
|
||||
the sentinel under a lock, and was rewritten until it SIGABRT'd pre-fix).
|
||||
- **Stochastic tests get the odds stacked.** Use `Config::alloc_interval(1)`
|
||||
to make every allocation an observation point, many actors, and both
|
||||
phases of any every-other-allocation cadence (see
|
||||
`poison_stop::self_stop_during_spawn...`).
|
||||
- **Time-based assertions use ordering, not durations.** Assert
|
||||
"didn't return instantly" / "A woke before B", with generous tolerances;
|
||||
CI machines are slow and noisy.
|
||||
- New invariants added to the runtime should come with the assert at the
|
||||
point of reliance (debug_assert on hot paths) *and*, where the invariant is
|
||||
a protocol, a loom model in the owning module — that combination is what
|
||||
made phases 2–5 land without a single post-merge race so far.
|
||||
@@ -130,61 +130,3 @@ fn join_on_stopped_actor_returns_ok() {
|
||||
assert!(h.join().is_ok(), "join on a stopped actor returns Ok(())");
|
||||
});
|
||||
}
|
||||
|
||||
/// Regression: `request_stop` against a QUEUED actor must not be lossy.
|
||||
///
|
||||
/// The stop flag is set, but the wildcard unpark no-ops on a Queued actor
|
||||
/// (the pending run "is" the wake). If the actor's first action on resume
|
||||
/// is a blocking park — no allocation, no `check!()` on the way — a
|
||||
/// wake-side-only check in `park_current` never runs: the actor parks with
|
||||
/// the stop flag already set, and nothing will ever wake it. The runtime
|
||||
/// then idles forever (the root is parked on the monitor channel).
|
||||
///
|
||||
/// Fix: an entry-side `check_cancelled` in `park_current`. The remaining
|
||||
/// window (flag set after the entry check, before the park lands) is closed
|
||||
/// by the existing protocol: the stop's unpark then finds Running /
|
||||
/// the prep-to-park window, sets Notified, and the park-return re-queues
|
||||
/// into the wake-side check.
|
||||
///
|
||||
/// Watchdog harness: without the fix this deadlocks, so the runtime runs on
|
||||
/// a side thread and the test fails on a timeout instead of hanging cargo.
|
||||
#[test]
|
||||
fn stop_flagged_while_queued_lands_at_first_park() {
|
||||
use std::sync::mpsc;
|
||||
use std::time::Duration;
|
||||
|
||||
let dropped = Arc::new(AtomicBool::new(false));
|
||||
let saw_stopped = Arc::new(AtomicBool::new(false));
|
||||
let (d, s) = (dropped.clone(), saw_stopped.clone());
|
||||
|
||||
let (done_tx, done_rx) = mpsc::channel::<()>();
|
||||
std::thread::spawn(move || {
|
||||
let rt = smarm::init(smarm::Config::exact(1));
|
||||
rt.run(move || {
|
||||
let h = spawn(move || {
|
||||
let _g = DropFlag(d);
|
||||
let (tx, rx) = channel::<u8>();
|
||||
let _keep = tx; // keep the channel open: recv() parks
|
||||
let _ = rx.recv(); // first observation point is this park
|
||||
});
|
||||
let pid = h.pid();
|
||||
let down = monitor(pid);
|
||||
// Stop while the child is still QUEUED — before it ever runs.
|
||||
// The unpark no-ops; only the flag is left behind.
|
||||
request_stop(pid);
|
||||
let dn = down.rx.recv().expect("monitor channel closed before Down");
|
||||
assert_eq!(dn.pid, pid);
|
||||
if matches!(dn.reason, DownReason::Stopped) {
|
||||
s.store(true, Ordering::SeqCst);
|
||||
}
|
||||
let _ = h.join();
|
||||
});
|
||||
let _ = done_tx.send(());
|
||||
});
|
||||
done_rx
|
||||
.recv_timeout(Duration::from_secs(10))
|
||||
.expect("runtime deadlocked: stop against a QUEUED actor was lost at its first park");
|
||||
|
||||
assert!(saw_stopped.load(Ordering::SeqCst), "expected DownReason::Stopped");
|
||||
assert!(dropped.load(Ordering::SeqCst), "Drop guard must run during the cancellation unwind");
|
||||
}
|
||||
|
||||
@@ -108,189 +108,3 @@ fn recv_returns_err_when_all_senders_dropped() {
|
||||
|
||||
assert!(saw_err.load(std::sync::atomic::Ordering::SeqCst));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn channel_ops_interleaved_with_monitor_churn_multi_thread() {
|
||||
// Regression for the RawMutex migration: monitor registration clones the
|
||||
// Down sender under the target's cold (Leaf) lock, which now nests a
|
||||
// Channel-class lock under it. Debug builds enforce the Leaf -> Channel
|
||||
// ordering on every acquisition, so driving channels, monitors, and actor
|
||||
// death concurrently across schedulers makes any ordering regression
|
||||
// panic here rather than deadlock in the field.
|
||||
use std::sync::atomic::{AtomicI64, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
let total = Arc::new(AtomicI64::new(0));
|
||||
let total2 = total.clone();
|
||||
smarm::init(smarm::Config::exact(4)).run(move || {
|
||||
let (tx, rx) = channel::<i64>();
|
||||
let consumer = spawn(move || {
|
||||
let mut sum = 0;
|
||||
while let Ok(v) = rx.recv() {
|
||||
sum += v;
|
||||
}
|
||||
OUT.with(|c| c.set(sum)); // not asserted cross-thread; see total
|
||||
total2.fetch_add(sum, Ordering::Relaxed);
|
||||
});
|
||||
|
||||
let mut handles = Vec::new();
|
||||
for i in 0..32i64 {
|
||||
let tx = tx.clone();
|
||||
handles.push(spawn(move || {
|
||||
// Short-lived target whose death fires the monitor below.
|
||||
let t = spawn(move || {
|
||||
tx.send(i).unwrap();
|
||||
});
|
||||
let m = smarm::monitor(t.pid());
|
||||
t.join().unwrap();
|
||||
// Down delivery exercises send-from-finalize.
|
||||
let d = m.rx.recv().unwrap();
|
||||
assert_eq!(d.reason, smarm::DownReason::Exit);
|
||||
}));
|
||||
}
|
||||
drop(tx);
|
||||
for h in handles {
|
||||
h.join().unwrap();
|
||||
}
|
||||
consumer.join().unwrap();
|
||||
});
|
||||
assert_eq!(total.load(std::sync::atomic::Ordering::Relaxed), (0..32).sum::<i64>());
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// recv_timeout
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
use smarm::RecvTimeoutError;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
#[test]
|
||||
fn recv_timeout_returns_queued_message_immediately() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<i64>();
|
||||
tx.send(5).unwrap();
|
||||
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(5));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recv_timeout_times_out_on_silent_channel() {
|
||||
run(|| {
|
||||
let (_tx, rx) = channel::<i64>();
|
||||
let start = Instant::now();
|
||||
let r = rx.recv_timeout(Duration::from_millis(50));
|
||||
assert_eq!(r, Err(RecvTimeoutError::Timeout));
|
||||
assert!(start.elapsed() >= Duration::from_millis(50));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recv_timeout_wakes_promptly_on_send() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<i64>();
|
||||
let h = spawn(move || {
|
||||
let start = Instant::now();
|
||||
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(9));
|
||||
// Far below the timeout: the send woke us, not the deadline.
|
||||
assert!(start.elapsed() < Duration::from_secs(1));
|
||||
});
|
||||
smarm::yield_now();
|
||||
tx.send(9).unwrap();
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recv_timeout_reports_disconnected_on_close() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<i64>();
|
||||
let h = spawn(move || {
|
||||
assert_eq!(
|
||||
rx.recv_timeout(Duration::from_secs(10)),
|
||||
Err(RecvTimeoutError::Disconnected)
|
||||
);
|
||||
});
|
||||
smarm::yield_now();
|
||||
drop(tx);
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recv_timeout_zero_duration_is_a_bounded_poll() {
|
||||
run(|| {
|
||||
let (_tx, rx) = channel::<i64>();
|
||||
assert_eq!(rx.recv_timeout(Duration::ZERO), Err(RecvTimeoutError::Timeout));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn channel_remains_usable_after_a_timeout() {
|
||||
// The stale timer entry from the first (timed-out) wait must not cancel
|
||||
// or corrupt later waits — seq isolation.
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<i64>();
|
||||
assert_eq!(
|
||||
rx.recv_timeout(Duration::from_millis(10)),
|
||||
Err(RecvTimeoutError::Timeout)
|
||||
);
|
||||
// Plain recv still works...
|
||||
tx.send(1).unwrap();
|
||||
assert_eq!(rx.recv(), Ok(1));
|
||||
// ...and so does a second bounded wait, woken by a send.
|
||||
let h = spawn(move || {
|
||||
tx.send(2).unwrap();
|
||||
});
|
||||
assert_eq!(rx.recv_timeout(Duration::from_secs(10)), Ok(2));
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recv_timeout_many_waiters_multi_thread() {
|
||||
// Mixed outcomes under real parallelism: half the channels get fed,
|
||||
// half time out; every actor must resolve correctly.
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
let got = Arc::new(AtomicU32::new(0));
|
||||
let timed_out = Arc::new(AtomicU32::new(0));
|
||||
let (got2, timed_out2) = (got.clone(), timed_out.clone());
|
||||
smarm::init(smarm::Config::exact(4)).run(move || {
|
||||
let mut handles = Vec::new();
|
||||
for i in 0..24i64 {
|
||||
let (tx, rx) = channel::<i64>();
|
||||
let got = got2.clone();
|
||||
let timed_out = timed_out2.clone();
|
||||
handles.push(spawn(move || match rx.recv_timeout(Duration::from_millis(100)) {
|
||||
Ok(v) => {
|
||||
assert_eq!(v, i);
|
||||
got.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
Err(RecvTimeoutError::Timeout) => {
|
||||
timed_out.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
Err(e) => panic!("unexpected: {e}"),
|
||||
}));
|
||||
if i % 2 == 0 {
|
||||
handles.push(spawn(move || {
|
||||
tx.send(i).unwrap();
|
||||
}));
|
||||
}
|
||||
// odd i: tx drops here -> Disconnected, not Timeout! Keep it alive
|
||||
// instead by leaking the sender into a holder actor that outlives
|
||||
// the deadline.
|
||||
else {
|
||||
handles.push(spawn(move || {
|
||||
smarm::sleep(Duration::from_millis(200));
|
||||
drop(tx);
|
||||
}));
|
||||
}
|
||||
}
|
||||
for h in handles {
|
||||
h.join().unwrap();
|
||||
}
|
||||
});
|
||||
assert_eq!(got.load(std::sync::atomic::Ordering::Relaxed), 12);
|
||||
assert_eq!(timed_out.load(std::sync::atomic::Ordering::Relaxed), 12);
|
||||
}
|
||||
|
||||
@@ -58,6 +58,7 @@ use std::sync::OnceLock;
|
||||
static REG_BEFORE: OnceLock<[u64; 4]> = OnceLock::new();
|
||||
static REG_AFTER: OnceLock<[u64; 4]> = OnceLock::new();
|
||||
|
||||
#[cfg(target_arch = "x86_64")]
|
||||
extern "C-unwind" fn actor_reg_check() {
|
||||
unsafe {
|
||||
let s0: u64 = 0xAAAA_BBBB_0000_0001;
|
||||
@@ -83,6 +84,39 @@ extern "C-unwind" fn actor_reg_check() {
|
||||
}
|
||||
}
|
||||
|
||||
// aarch64 sibling: same intent, AAPCS64 callee-saved integer registers.
|
||||
// LLVM reserves x19 (and uses x29/x30 as fp/lr), so those can't be named as
|
||||
// asm operands. We use x23–x26 instead — callee-saved and bindable — moving
|
||||
// known values into them, yielding, then reading them back, exactly mirroring
|
||||
// the x86 test's move-into-r12..r15 pattern. The shim saves x23–x26 in its
|
||||
// `stp x23,x24` / `stp x25,x26` pairs, so a clean round-trip proves they
|
||||
// survive the switch.
|
||||
#[cfg(target_arch = "aarch64")]
|
||||
extern "C-unwind" fn actor_reg_check() {
|
||||
unsafe {
|
||||
let s0: u64 = 0xAAAA_BBBB_0000_0001;
|
||||
let s1: u64 = 0xCCCC_DDDD_0000_0002;
|
||||
let s2: u64 = 0xEEEE_FFFF_0000_0003;
|
||||
let s3: u64 = 0x1111_2222_0000_0004;
|
||||
|
||||
core::arch::asm!(
|
||||
"mov x23, {s0}", "mov x24, {s1}", "mov x25, {s2}", "mov x26, {s3}",
|
||||
s0 = in(reg) s0, s1 = in(reg) s1, s2 = in(reg) s2, s3 = in(reg) s3,
|
||||
out("x23") _, out("x24") _, out("x25") _, out("x26") _,
|
||||
);
|
||||
REG_BEFORE.set([s0, s1, s2, s3]).ok();
|
||||
switch_to_scheduler();
|
||||
|
||||
let a0: u64; let a1: u64; let a2: u64; let a3: u64;
|
||||
core::arch::asm!(
|
||||
"mov {a0}, x23", "mov {a1}, x24", "mov {a2}, x25", "mov {a3}, x26",
|
||||
a0 = out(reg) a0, a1 = out(reg) a1, a2 = out(reg) a2, a3 = out(reg) a3,
|
||||
);
|
||||
REG_AFTER.set([a0, a1, a2, a3]).ok();
|
||||
switch_to_scheduler();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn callee_saved_registers_survive_yield() {
|
||||
let stack = Stack::new(64 * 1024).unwrap();
|
||||
|
||||
@@ -1,403 +0,0 @@
|
||||
//! RFC 008 — fd arms in select. Beyond the functional cases, the
|
||||
//! *_stays_usable tests are the soundness probes for the one asymmetry the
|
||||
//! RFC must close: a losing CHANNEL arm's stale registration is inert, but
|
||||
//! a losing FD arm's registration (waiters entry + kernel ONESHOT) poisons
|
||||
//! the fd with AlreadyExists until the eager cleanup pass removes it. Every
|
||||
//! "loser" scenario therefore re-waits on the same fd afterwards and must
|
||||
//! succeed — pre-cleanup, each of those re-waits errors or hangs.
|
||||
//!
|
||||
//! House pattern: actor panics are trampoline-caught and `run` returns
|
||||
//! normally, so every test funnels its result into an outcome flag asserted
|
||||
//! OUTSIDE `run` — an in-actor assertion alone passes vacuously.
|
||||
|
||||
use smarm::{
|
||||
channel, run, select, select_timeout, spawn, try_select, wait_readable,
|
||||
wait_readable_timeout, wait_writable_timeout, yield_now, FdArm,
|
||||
};
|
||||
use std::os::fd::RawFd;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pipe helper (as in io_epoll.rs)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
struct Pipe {
|
||||
read: RawFd,
|
||||
write: RawFd,
|
||||
}
|
||||
|
||||
impl Pipe {
|
||||
fn new() -> Self {
|
||||
let mut fds: [libc::c_int; 2] = [0; 2];
|
||||
let r = unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_CLOEXEC | libc::O_NONBLOCK) };
|
||||
assert_eq!(r, 0, "pipe2 failed");
|
||||
Pipe { read: fds[0], write: fds[1] }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Pipe {
|
||||
fn drop(&mut self) {
|
||||
unsafe {
|
||||
libc::close(self.read);
|
||||
libc::close(self.write);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn raw_write(fd: RawFd, buf: &[u8]) -> isize {
|
||||
unsafe { libc::write(fd, buf.as_ptr() as *const _, buf.len()) }
|
||||
}
|
||||
|
||||
fn raw_read(fd: RawFd, buf: &mut [u8]) -> isize {
|
||||
unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) }
|
||||
}
|
||||
|
||||
fn flag() -> (Arc<AtomicBool>, Arc<AtomicBool>) {
|
||||
let f = Arc::new(AtomicBool::new(false));
|
||||
(f.clone(), f)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Ready-now: data already pending retires the wait without parking.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn fd_arm_ready_now_returns_without_parking() {
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
assert_eq!(raw_write(p.write, b"x"), 1);
|
||||
let (_tx, rx) = channel::<i64>();
|
||||
let fd_arm = FdArm::readable(p.read);
|
||||
// fd arm at index 1: the ready-now path must also work for a
|
||||
// non-first arm (and clean nothing — channel arms are inert).
|
||||
let i = select(&[&rx, &fd_arm]);
|
||||
assert_eq!(i, 1);
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(p.read, &mut buf), 1);
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Park-then-wake: fd arm wins against an idle channel arm.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn fd_arm_parks_until_data_then_wins() {
|
||||
let got = Arc::new(AtomicU32::new(0));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
let (_tx_keepalive, rx) = channel::<i64>();
|
||||
|
||||
let h = spawn(move || {
|
||||
let fd_arm = FdArm::readable(rfd);
|
||||
let i = select(&[&fd_arm, &rx]);
|
||||
assert_eq!(i, 0);
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
got2.store(buf[0] as u32, Ordering::SeqCst);
|
||||
});
|
||||
yield_now(); // let it park
|
||||
assert_eq!(raw_write(wfd, b"y"), 1);
|
||||
let _ = h.join();
|
||||
});
|
||||
assert_eq!(got.load(Ordering::SeqCst), b'y' as u32);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// THE asymmetry probe: channel arm wins, losing fd arm must be cleaned —
|
||||
// the same actor (and the io thread) must be able to wait that fd again.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn losing_fd_arm_is_cleaned_up_and_fd_stays_usable() {
|
||||
let got = Arc::new(AtomicU32::new(0));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
let (tx, rx) = channel::<i64>();
|
||||
|
||||
let h = spawn(move || {
|
||||
let fd_arm = FdArm::readable(rfd);
|
||||
// Channel wins; the fd arm registered and lost.
|
||||
let i = select(&[&fd_arm, &rx]);
|
||||
assert_eq!(i, 1);
|
||||
assert_eq!(rx.try_recv().unwrap(), Some(7));
|
||||
|
||||
// Pre-cleanup this wait_readable fails AlreadyExists (the
|
||||
// waiters entry is stale-ours) — the cleanup pass must have
|
||||
// removed it.
|
||||
wait_readable(rfd).unwrap();
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
got2.store(buf[0] as u32, Ordering::SeqCst);
|
||||
});
|
||||
yield_now(); // let it park in the select
|
||||
tx.send(7).unwrap();
|
||||
yield_now(); // let it reach the second wait
|
||||
assert_eq!(raw_write(wfd, b"z"), 1);
|
||||
let _ = h.join();
|
||||
});
|
||||
assert_eq!(got.load(Ordering::SeqCst), b'z' as u32);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Ready-now on a LATER arm must unregister an earlier fd arm (the
|
||||
// register_arms prefix-cleanup path: no park ever happens).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn ready_now_later_arm_cleans_earlier_fd_arm() {
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
let (tx, rx) = channel::<i64>();
|
||||
tx.send(1).unwrap(); // arm 1 ready before the select
|
||||
|
||||
let fd_arm = FdArm::readable(rfd);
|
||||
let i = select(&[&fd_arm, &rx]);
|
||||
assert_eq!(i, 1);
|
||||
assert_eq!(rx.try_recv().unwrap(), Some(1));
|
||||
|
||||
// The fd arm registered (idle pipe), then arm 1 retired the wait.
|
||||
// Its registration must have been removed in the same pass.
|
||||
assert_eq!(raw_write(wfd, b"a"), 1);
|
||||
wait_readable(rfd).unwrap();
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Two fd arms in one select (phase-1: distinct fds, nothing special).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn two_fd_arms_second_fires_first_stays_usable() {
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let pa = Pipe::new();
|
||||
let pb = Pipe::new();
|
||||
let (rfd_a, wfd_a) = (pa.read, pa.write);
|
||||
let (rfd_b, wfd_b) = (pb.read, pb.write);
|
||||
|
||||
let h = spawn(move || {
|
||||
let a = FdArm::readable(rfd_a);
|
||||
let b = FdArm::readable(rfd_b);
|
||||
let i = select(&[&a, &b]);
|
||||
assert_eq!(i, 1);
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd_b, &mut buf), 1);
|
||||
|
||||
// Arm a lost; its fd must be immediately re-waitable.
|
||||
assert_eq!(raw_write(wfd_a, b"q"), 1);
|
||||
wait_readable(rfd_a).unwrap();
|
||||
assert_eq!(raw_read(rfd_a, &mut buf), 1);
|
||||
assert_eq!(buf[0], b'q');
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
yield_now();
|
||||
assert_eq!(raw_write(wfd_b, b"b"), 1);
|
||||
let _ = h.join();
|
||||
});
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// select_timeout: timer wins over an idle fd arm; the fd is left clean.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn select_timeout_timer_beats_idle_fd_arm_and_fd_stays_usable() {
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
|
||||
let fd_arm = FdArm::readable(rfd);
|
||||
let start = Instant::now();
|
||||
let r = select_timeout(&[&fd_arm], Duration::from_millis(30));
|
||||
assert!(r.is_none(), "idle fd must time out");
|
||||
assert!(start.elapsed() >= Duration::from_millis(30));
|
||||
|
||||
// Timer win is exactly the case where the fd arm's registration is
|
||||
// left behind without an eager pass.
|
||||
assert_eq!(raw_write(wfd, b"c"), 1);
|
||||
wait_readable(rfd).unwrap();
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Derived wrappers.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn wait_readable_timeout_times_out_then_succeeds_with_data() {
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
|
||||
let start = Instant::now();
|
||||
assert_eq!(wait_readable_timeout(rfd, Duration::from_millis(30)).unwrap(), false);
|
||||
assert!(start.elapsed() >= Duration::from_millis(30));
|
||||
|
||||
// Timed-out wait must leave the fd clean; ready path returns true.
|
||||
assert_eq!(raw_write(wfd, b"d"), 1);
|
||||
assert_eq!(wait_readable_timeout(rfd, Duration::from_secs(5)).unwrap(), true);
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_readable_timeout_wakes_on_late_data() {
|
||||
let got = Arc::new(AtomicU32::new(0));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
let h = spawn(move || {
|
||||
assert_eq!(wait_readable_timeout(rfd, Duration::from_secs(5)).unwrap(), true);
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
got2.store(buf[0] as u32, Ordering::SeqCst);
|
||||
});
|
||||
yield_now();
|
||||
assert_eq!(raw_write(wfd, b"e"), 1);
|
||||
let _ = h.join();
|
||||
});
|
||||
assert_eq!(got.load(Ordering::SeqCst), b'e' as u32);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_writable_timeout_ready_now_on_empty_pipe() {
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
// An empty pipe's write end is writable: ready-now path, no park.
|
||||
assert_eq!(wait_writable_timeout(p.write, Duration::from_secs(5)).unwrap(), true);
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Error surface: registration failure is an Err from try_select, with the
|
||||
// wait retired (the actor can immediately wait on something else).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn try_select_surfaces_registration_error_and_retires_the_wait() {
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let bad: RawFd = {
|
||||
let p = Pipe::new();
|
||||
p.read
|
||||
}; // both ends closed by Drop: EBADF on registration
|
||||
|
||||
let fd_arm = FdArm::readable(bad);
|
||||
let err = try_select(&[&fd_arm]).unwrap_err();
|
||||
// EBADF, whether the pre-poll or epoll_ctl ADD reports it.
|
||||
assert_eq!(err.raw_os_error(), Some(libc::EBADF));
|
||||
|
||||
// The wait was retired: a normal select right after works.
|
||||
let (tx, rx) = channel::<i64>();
|
||||
tx.send(9).unwrap();
|
||||
let i = select(&[&rx]);
|
||||
assert_eq!(i, 0);
|
||||
assert_eq!(rx.try_recv().unwrap(), Some(9));
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Stop-unwind: an actor stopped while parked in an fd-arm select must not
|
||||
// poison the fd (the UnregisterGuard generalization of wait_fd's Dereg).
|
||||
// Mirrors io_epoll.rs::stopped_waiter_does_not_poison_the_fd.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn stopped_selector_does_not_poison_the_fd() {
|
||||
let seen = Arc::new(AtomicU32::new(0));
|
||||
let seen_outer = seen.clone();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
let (_tx_keepalive, rx) = channel::<i64>();
|
||||
|
||||
let h = spawn(move || {
|
||||
let fd_arm = FdArm::readable(rfd);
|
||||
select(&[&fd_arm, &rx]);
|
||||
unreachable!("neither arm ever fires while this actor lives");
|
||||
});
|
||||
yield_now(); // let it reach the park
|
||||
smarm::request_stop(h.pid());
|
||||
let _ = h.join(); // Ok(()): stopped, not panicked
|
||||
|
||||
// Second waiter on the SAME fd must register and be woken.
|
||||
let seen2 = seen.clone();
|
||||
let h2 = spawn(move || {
|
||||
wait_readable(rfd).unwrap();
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
seen2.store(buf[0] as u32, Ordering::SeqCst);
|
||||
});
|
||||
yield_now();
|
||||
assert_eq!(raw_write(wfd, b"x"), 1);
|
||||
let _ = h2.join();
|
||||
});
|
||||
assert_eq!(seen_outer.load(Ordering::SeqCst), b'x' as u32);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase-1 misuse: a second waiter on an fd that already has one is an Err
|
||||
// (AlreadyExists), not a hang — and does not disturb the first waiter.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn second_waiter_on_same_fd_errs_without_disturbing_the_first() {
|
||||
let got = Arc::new(AtomicU32::new(0));
|
||||
let got2 = got.clone();
|
||||
let (ok, ok2) = flag();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
|
||||
let h = spawn(move || {
|
||||
wait_readable(rfd).unwrap();
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
got2.store(buf[0] as u32, Ordering::SeqCst);
|
||||
});
|
||||
yield_now(); // first waiter parked
|
||||
|
||||
let fd_arm = FdArm::readable(rfd);
|
||||
let err = try_select(&[&fd_arm]).unwrap_err();
|
||||
assert_eq!(err.kind(), std::io::ErrorKind::AlreadyExists);
|
||||
|
||||
// First waiter still wakes normally.
|
||||
assert_eq!(raw_write(wfd, b"w"), 1);
|
||||
let _ = h.join();
|
||||
ok2.store(true, Ordering::SeqCst);
|
||||
});
|
||||
assert_eq!(got.load(Ordering::SeqCst), b'w' as u32);
|
||||
assert!(ok.load(Ordering::SeqCst));
|
||||
}
|
||||
+2
-304
@@ -1,7 +1,7 @@
|
||||
//! gen_server tests: call round-trip, cast, lifecycle callbacks, and the two
|
||||
//! server-down detection paths (reply-channel close vs. inbox-send failure).
|
||||
|
||||
use smarm::gen_server::{start, CallError, GenServer, ServerBuilder};
|
||||
use smarm::gen_server::{start, CallError, GenServer};
|
||||
use smarm::run;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
@@ -26,7 +26,6 @@ impl GenServer for Counter {
|
||||
type Call = Req;
|
||||
type Reply = i64;
|
||||
type Cast = Op;
|
||||
type Info = ();
|
||||
|
||||
fn handle_call(&mut self, req: Req) -> i64 {
|
||||
match req {
|
||||
@@ -69,9 +68,8 @@ impl GenServer for Lifecycle {
|
||||
type Call = ();
|
||||
type Reply = ();
|
||||
type Cast = ();
|
||||
type Info = ();
|
||||
|
||||
fn init(&mut self, _ctx: &smarm::gen_server::ServerCtx) {
|
||||
fn init(&mut self) {
|
||||
self.log.lock().unwrap().push("init");
|
||||
}
|
||||
|
||||
@@ -137,303 +135,3 @@ fn call_after_server_gone_is_server_down() {
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), Some(Err(CallError::ServerDown)));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// call_timeout
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
use smarm::gen_server::CallTimeoutError;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
/// Replies after sleeping `delay_ms` (parking the server actor, not the OS
|
||||
/// thread), so callers can race a deadline against the reply.
|
||||
struct Slow;
|
||||
|
||||
impl GenServer for Slow {
|
||||
type Call = u64; // delay in ms
|
||||
type Reply = u64;
|
||||
type Cast = ();
|
||||
type Info = ();
|
||||
|
||||
fn handle_call(&mut self, delay_ms: u64) -> u64 {
|
||||
if delay_ms > 0 {
|
||||
smarm::sleep(Duration::from_millis(delay_ms));
|
||||
}
|
||||
delay_ms
|
||||
}
|
||||
|
||||
fn handle_cast(&mut self, _: ()) {}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn call_timeout_returns_reply_within_deadline() {
|
||||
run(|| {
|
||||
let srv = start(Slow);
|
||||
assert_eq!(srv.call_timeout(0, Duration::from_secs(10)), Ok(0));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn call_timeout_times_out_on_slow_handler() {
|
||||
run(|| {
|
||||
let srv = start(Slow);
|
||||
let start_t = Instant::now();
|
||||
let r = srv.call_timeout(500, Duration::from_millis(50));
|
||||
assert_eq!(r, Err(CallTimeoutError::Timeout));
|
||||
let elapsed = start_t.elapsed();
|
||||
// Gave up at the deadline, not at the reply.
|
||||
assert!(elapsed >= Duration::from_millis(50));
|
||||
assert!(elapsed < Duration::from_millis(500));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn server_survives_an_abandoned_call_and_late_reply_is_discarded() {
|
||||
run(|| {
|
||||
let srv = start(Slow);
|
||||
assert_eq!(
|
||||
srv.call_timeout(100, Duration::from_millis(20)),
|
||||
Err(CallTimeoutError::Timeout)
|
||||
);
|
||||
// The timed-out request is still handled; its reply send fails
|
||||
// harmlessly (receiver dropped). The server must keep serving, and
|
||||
// the late reply must not leak into THIS call's reply channel.
|
||||
assert_eq!(srv.call_timeout(0, Duration::from_secs(10)), Ok(0));
|
||||
// Plain unbounded call still fine too.
|
||||
assert_eq!(srv.call(0), Ok(0));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn call_timeout_to_dead_server_is_server_down_not_timeout() {
|
||||
struct Bomb;
|
||||
impl GenServer for Bomb {
|
||||
type Call = ();
|
||||
type Info = ();
|
||||
type Reply = ();
|
||||
type Cast = ();
|
||||
fn handle_call(&mut self, _: ()) {
|
||||
panic!("kaboom");
|
||||
}
|
||||
fn handle_cast(&mut self, _: ()) {}
|
||||
}
|
||||
|
||||
run(|| {
|
||||
let srv = start(Bomb);
|
||||
// Dies mid-call: reply channel closes -> ServerDown (even though the
|
||||
// generous deadline never fires).
|
||||
assert_eq!(
|
||||
srv.call_timeout((), Duration::from_secs(10)),
|
||||
Err(CallTimeoutError::ServerDown)
|
||||
);
|
||||
// Already gone: inbox send fails -> ServerDown.
|
||||
assert_eq!(
|
||||
srv.call_timeout((), Duration::from_secs(10)),
|
||||
Err(CallTimeoutError::ServerDown)
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// handle_info: out-of-band channels selected alongside the inbox (v0.8)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Logs every message it handles, in order; a call reads the log back.
|
||||
struct Logger {
|
||||
log: Vec<&'static str>,
|
||||
}
|
||||
|
||||
impl GenServer for Logger {
|
||||
type Call = ();
|
||||
type Reply = Vec<&'static str>;
|
||||
type Cast = ();
|
||||
type Info = &'static str;
|
||||
|
||||
fn handle_call(&mut self, _: ()) -> Vec<&'static str> {
|
||||
self.log.clone()
|
||||
}
|
||||
|
||||
fn handle_cast(&mut self, _: ()) {
|
||||
self.log.push("cast");
|
||||
}
|
||||
|
||||
fn handle_info(&mut self, info: &'static str) {
|
||||
self.log.push(info);
|
||||
}
|
||||
}
|
||||
|
||||
// An info message is dispatched to handle_info, interleaved with normal
|
||||
// service.
|
||||
#[test]
|
||||
fn info_is_dispatched() {
|
||||
let got = Arc::new(Mutex::new(Vec::new()));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let (info_tx, info_rx) = smarm::channel::<&'static str>();
|
||||
let server = ServerBuilder::new(Logger { log: Vec::new() })
|
||||
.with_info(info_rx)
|
||||
.start();
|
||||
info_tx.send("info").unwrap();
|
||||
*got2.lock().unwrap() = server.call(()).unwrap();
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), vec!["info"]);
|
||||
}
|
||||
|
||||
// Arm priority: with a cast AND an info both queued before the server first
|
||||
// runs, the info is handled first — info arms outrank the inbox. Relies on
|
||||
// run()'s deterministic single-thread ordering.
|
||||
#[test]
|
||||
fn info_outranks_inbox() {
|
||||
let got = Arc::new(Mutex::new(Vec::new()));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let (info_tx, info_rx) = smarm::channel::<&'static str>();
|
||||
let server = ServerBuilder::new(Logger { log: Vec::new() })
|
||||
.with_info(info_rx)
|
||||
.start();
|
||||
// The server actor hasn't run yet: both messages are queued before
|
||||
// its first select. Inbox first in *send* order, info first in *arm*
|
||||
// order — arm order must win.
|
||||
server.cast(()).unwrap();
|
||||
info_tx.send("info").unwrap();
|
||||
*got2.lock().unwrap() = server.call(()).unwrap();
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), vec!["info", "cast"]);
|
||||
}
|
||||
|
||||
// Two info channels: declaration order is priority order.
|
||||
#[test]
|
||||
fn info_arms_keep_declaration_priority() {
|
||||
let got = Arc::new(Mutex::new(Vec::new()));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let (hi_tx, hi_rx) = smarm::channel::<&'static str>();
|
||||
let (lo_tx, lo_rx) = smarm::channel::<&'static str>();
|
||||
let server = ServerBuilder::new(Logger { log: Vec::new() })
|
||||
.with_info(hi_rx)
|
||||
.with_info(lo_rx)
|
||||
.start();
|
||||
// Sent low-priority first; handled high-priority first.
|
||||
lo_tx.send("lo").unwrap();
|
||||
hi_tx.send("hi").unwrap();
|
||||
*got2.lock().unwrap() = server.call(()).unwrap();
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), vec!["hi", "lo"]);
|
||||
}
|
||||
|
||||
// A closed info arm is silently dropped and the server keeps serving; the
|
||||
// closure does NOT reach handle_info and does NOT starve the inbox (the
|
||||
// closed-arm-is-ready-forever gotcha).
|
||||
#[test]
|
||||
fn closed_info_arm_is_dropped_silently() {
|
||||
let got = Arc::new(Mutex::new(Vec::new()));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let (info_tx, info_rx) = smarm::channel::<&'static str>();
|
||||
let server = ServerBuilder::new(Logger { log: Vec::new() })
|
||||
.with_info(info_rx)
|
||||
.start();
|
||||
drop(info_tx); // closed before the server's first select
|
||||
server.cast(()).unwrap();
|
||||
*got2.lock().unwrap() = server.call(()).unwrap();
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), vec!["cast"]);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// handle_down: monitors handed to the loop via Watcher (v0.8)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
use smarm::gen_server::Watcher;
|
||||
use smarm::{monitor, spawn, DownReason, Pid};
|
||||
|
||||
/// The motivating pattern: a server that spawns workers from a handler,
|
||||
/// watches them, and logs their deaths.
|
||||
struct Pool {
|
||||
watcher: Option<Watcher>,
|
||||
log: Vec<DownReason>,
|
||||
}
|
||||
|
||||
enum PoolCast {
|
||||
SpawnDoomedWorker,
|
||||
Watch(Pid),
|
||||
}
|
||||
|
||||
impl GenServer for Pool {
|
||||
type Call = ();
|
||||
type Reply = Vec<DownReason>;
|
||||
type Cast = PoolCast;
|
||||
type Info = ();
|
||||
|
||||
fn init(&mut self, ctx: &smarm::gen_server::ServerCtx) {
|
||||
self.watcher = Some(ctx.watcher());
|
||||
}
|
||||
|
||||
fn handle_call(&mut self, _: ()) -> Vec<DownReason> {
|
||||
self.log.clone()
|
||||
}
|
||||
|
||||
fn handle_cast(&mut self, cast: PoolCast) {
|
||||
let watcher = self.watcher.as_ref().expect("init ran first");
|
||||
match cast {
|
||||
PoolCast::SpawnDoomedWorker => {
|
||||
let h = spawn(|| panic!("worker died"));
|
||||
watcher.watch(monitor(h.pid()));
|
||||
}
|
||||
PoolCast::Watch(pid) => watcher.watch(monitor(pid)),
|
||||
}
|
||||
}
|
||||
|
||||
fn handle_down(&mut self, down: smarm::Down) {
|
||||
self.log.push(down.reason);
|
||||
}
|
||||
}
|
||||
|
||||
// A worker spawned and watched from inside a handler delivers its Down to
|
||||
// handle_down. Down arms outrank the inbox, so the death is in the log by
|
||||
// the time the follow-up call is answered.
|
||||
#[test]
|
||||
fn worker_pool_down_reaches_handle_down() {
|
||||
let got = Arc::new(Mutex::new(Vec::new()));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let server = start(Pool { watcher: None, log: Vec::new() });
|
||||
server.cast(PoolCast::SpawnDoomedWorker).unwrap();
|
||||
let _ = server.call(()).unwrap(); // sync point: cast handled, worker live
|
||||
*got2.lock().unwrap() = server.call(()).unwrap();
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), vec![DownReason::Panic]);
|
||||
}
|
||||
|
||||
// Watching an already-dead pid yields an immediate NoProc Down, and the down
|
||||
// arm outranks the inbox: the call cast *after* the watch still observes it.
|
||||
#[test]
|
||||
fn watch_dead_pid_is_noproc_down() {
|
||||
let got = Arc::new(Mutex::new(Vec::new()));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let h = spawn(|| {});
|
||||
let dead = h.pid();
|
||||
h.join().unwrap();
|
||||
let server = start(Pool { watcher: None, log: Vec::new() });
|
||||
server.cast(PoolCast::Watch(dead)).unwrap();
|
||||
*got2.lock().unwrap() = server.call(()).unwrap();
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), vec![DownReason::NoProc]);
|
||||
}
|
||||
|
||||
// A state that never clones the Watcher closes the control arm; the loop
|
||||
// falls back to the plain-inbox park and keeps serving. (Every pre-v0.8 test
|
||||
// in this file also exercises this path.)
|
||||
#[test]
|
||||
fn unused_ctx_closes_control_arm_silently() {
|
||||
let got = Arc::new(Mutex::new(0i64));
|
||||
let got2 = got.clone();
|
||||
run(move || {
|
||||
let server = start(Counter { n: 0 });
|
||||
server.cast(Op::Add(2)).unwrap();
|
||||
server.cast(Op::Add(40)).unwrap();
|
||||
*got2.lock().unwrap() = server.call(Req::Get).unwrap();
|
||||
});
|
||||
assert_eq!(*got.lock().unwrap(), 42);
|
||||
}
|
||||
|
||||
@@ -322,46 +322,3 @@ fn wait_writable_on_empty_pipe_returns_quickly() {
|
||||
elapsed
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fd hygiene on actor death (v0.8)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// An actor stopped while parked on an fd must not leak its `waiters` entry:
|
||||
// before the unwind-path guard in wait_fd, the stale entry made every
|
||||
// future wait_*() on that fd fail with AlreadyExists, forever.
|
||||
#[test]
|
||||
fn stopped_waiter_does_not_poison_the_fd() {
|
||||
let outcome = Arc::new(StdMutex::new(None::<u8>));
|
||||
let outcome2 = outcome.clone();
|
||||
run(move || {
|
||||
let p = Pipe::new();
|
||||
let (rfd, wfd) = (p.read, p.write);
|
||||
|
||||
// First waiter parks on the (empty) pipe and is stopped in place.
|
||||
let h = smarm::spawn(move || {
|
||||
wait_readable(rfd).unwrap();
|
||||
unreachable!("the pipe is never written while this actor lives");
|
||||
});
|
||||
yield_now(); // let it reach the park
|
||||
smarm::request_stop(h.pid());
|
||||
h.join().unwrap(); // Ok(()): stopped, not panicked
|
||||
|
||||
// Second waiter on the SAME fd must be able to register...
|
||||
let seen = Arc::new(AtomicU32::new(0));
|
||||
let seen2 = seen.clone();
|
||||
let h2 = smarm::spawn(move || {
|
||||
wait_readable(rfd).unwrap();
|
||||
let mut buf = [0u8; 1];
|
||||
assert_eq!(raw_read(rfd, &mut buf), 1);
|
||||
seen2.store(buf[0] as u32, Ordering::SeqCst);
|
||||
});
|
||||
yield_now(); // ...and park (a failed register would panic the unwrap)
|
||||
|
||||
// ...and actually be woken by readiness.
|
||||
assert_eq!(raw_write(wfd, b"x"), 1);
|
||||
h2.join().unwrap();
|
||||
*outcome2.lock().unwrap() = Some(seen.load(Ordering::SeqCst) as u8);
|
||||
});
|
||||
assert_eq!(*outcome.lock().unwrap(), Some(b'x'));
|
||||
}
|
||||
|
||||
+1
-2
@@ -1,5 +1,4 @@
|
||||
//! `smarm::Mutex<T>` (the actor-blocking mutex) tests. All run under the
|
||||
//! scheduler because `lock()`
|
||||
//! `loom::Mutex<T>` tests. All run under the scheduler because `lock()`
|
||||
//! needs to be able to park.
|
||||
|
||||
use smarm::{run, spawn, yield_now, LockTimeout, Mutex};
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
//! Regression: request_stop racing an alloc-under-lock must not poison any
|
||||
//! runtime mutex. Before the fix, check_cancelled() fired regardless of
|
||||
//! PREEMPTION_ENABLED, so a sentinel unwind could trigger inside with_shared
|
||||
//! (which allocates: run_queue.push_back, waiters.push, etc), poisoning the
|
||||
//! shared mutex and cascading lock().unwrap() panics.
|
||||
|
||||
use smarm::runtime::{init, Config};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
#[test]
|
||||
fn stop_storm_does_not_poison_runtime() {
|
||||
// alloc_interval(1): every allocation is an observation point, so pre-fix
|
||||
// the sentinel fires inside with_shared's alloc sites (run_queue.push_back,
|
||||
// waiters.push, ...) with near-certainty rather than 1-in-128.
|
||||
let rt = init(Config::exact(4).alloc_interval(1));
|
||||
let completed = Arc::new(AtomicUsize::new(0));
|
||||
let c = completed.clone();
|
||||
rt.run(move || {
|
||||
// Spawn many short actors that allocate + yield heavily (hammering
|
||||
// with_shared), and request_stop each one mid-flight from siblings.
|
||||
let mut handles = Vec::new();
|
||||
for _ in 0..200 {
|
||||
let h = smarm::spawn(|| {
|
||||
for _ in 0..50 {
|
||||
let _v: Vec<u8> = Vec::with_capacity(64); // alloc → maybe_preempt
|
||||
smarm::yield_now();
|
||||
}
|
||||
});
|
||||
handles.push(h);
|
||||
}
|
||||
// Cancel half of them; the others run to completion. If any lock got
|
||||
// poisoned the runtime would panic on a subsequent lock().unwrap().
|
||||
for (i, h) in handles.iter().enumerate() {
|
||||
if i % 2 == 0 {
|
||||
smarm::request_stop(h.pid());
|
||||
}
|
||||
}
|
||||
for h in handles {
|
||||
let _ = h.join();
|
||||
}
|
||||
c.fetch_add(1, Ordering::SeqCst);
|
||||
});
|
||||
assert_eq!(completed.load(Ordering::SeqCst), 1, "root completed cleanly");
|
||||
}
|
||||
|
||||
/// The sharper repro: a stop-flagged actor whose *next allocation* is the
|
||||
/// `Box::new(closure)` inside `spawn`'s `with_shared` critical section.
|
||||
/// Pre-fix, the ungated `check_cancelled` in `maybe_preempt` raises the
|
||||
/// sentinel right there, unwinding while the global shared mutex is held →
|
||||
/// poisoned → every later `with_shared` panics on every scheduler thread and
|
||||
/// the whole runtime collapses. Post-fix, observation is deferred to the next
|
||||
/// lock-free point and each actor dies a clean `Outcome::Stopped`.
|
||||
///
|
||||
/// `request_stop(self_pid())` sets the flag without parking (we're running),
|
||||
/// and the capturing closure makes `Box::new(f)` a real allocation. With
|
||||
/// `alloc_interval(1)` the check fires every other allocation, so each actor
|
||||
/// has ~50% odds of the check landing on the under-lock alloc; 64 actors make
|
||||
/// a pre-fix escape astronomically unlikely.
|
||||
#[test]
|
||||
fn self_stop_during_spawn_does_not_poison_shared_mutex() {
|
||||
let rt = init(Config::exact(4).alloc_interval(1));
|
||||
let completed = Arc::new(AtomicUsize::new(0));
|
||||
let c = completed.clone();
|
||||
rt.run(move || {
|
||||
let mut handles = Vec::new();
|
||||
for i in 0..64usize {
|
||||
let h = smarm::spawn(move || {
|
||||
// Vary pre-stop allocation count to cover both phases of the
|
||||
// every-other-allocation check cadence.
|
||||
for _ in 0..(i % 2) {
|
||||
let _phase: Vec<u8> = Vec::with_capacity(8);
|
||||
}
|
||||
let payload = vec![0u8; 64]; // captured → Box::new(f) allocates
|
||||
smarm::request_stop(smarm::self_pid());
|
||||
let grandchild = smarm::spawn(move || drop(payload));
|
||||
let _ = grandchild.join();
|
||||
});
|
||||
handles.push(h);
|
||||
}
|
||||
for h in handles {
|
||||
// Stopped reports Ok from join; the point is that join itself
|
||||
// (with_shared) doesn't panic on a poisoned mutex.
|
||||
let _ = h.join();
|
||||
}
|
||||
c.fetch_add(1, Ordering::SeqCst);
|
||||
});
|
||||
assert_eq!(completed.load(Ordering::SeqCst), 1, "root completed cleanly");
|
||||
}
|
||||
@@ -1,221 +0,0 @@
|
||||
//! Named registry tests. Run under the scheduler: registration requires a
|
||||
//! live runtime and live actors.
|
||||
|
||||
use smarm::{channel, name_of, register, run, spawn, unregister, whereis, RegisterError};
|
||||
|
||||
#[test]
|
||||
fn register_whereis_roundtrip() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<()>();
|
||||
let h = spawn(move || {
|
||||
rx.recv().unwrap();
|
||||
});
|
||||
register("worker", h.pid()).unwrap();
|
||||
assert_eq!(whereis("worker"), Some(h.pid()));
|
||||
assert_eq!(name_of(h.pid()).as_deref(), Some("worker"));
|
||||
tx.send(()).unwrap();
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn register_is_idempotent_for_same_binding() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<()>();
|
||||
let h = spawn(move || {
|
||||
rx.recv().unwrap();
|
||||
});
|
||||
register("svc", h.pid()).unwrap();
|
||||
// Same name, same pid: a no-op Ok, not NameTaken.
|
||||
register("svc", h.pid()).unwrap();
|
||||
tx.send(()).unwrap();
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn duplicate_name_on_live_holder_is_rejected() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<()>();
|
||||
let (tx2, rx2) = channel::<()>();
|
||||
let a = spawn(move || {
|
||||
rx.recv().unwrap();
|
||||
});
|
||||
let b = spawn(move || {
|
||||
rx2.recv().unwrap();
|
||||
});
|
||||
register("svc", a.pid()).unwrap();
|
||||
assert_eq!(
|
||||
register("svc", b.pid()),
|
||||
Err(RegisterError::NameTaken { holder: a.pid() })
|
||||
);
|
||||
tx.send(()).unwrap();
|
||||
tx2.send(()).unwrap();
|
||||
a.join().unwrap();
|
||||
b.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn one_name_per_pid() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<()>();
|
||||
let h = spawn(move || {
|
||||
rx.recv().unwrap();
|
||||
});
|
||||
register("first", h.pid()).unwrap();
|
||||
assert_eq!(
|
||||
register("second", h.pid()),
|
||||
Err(RegisterError::PidAlreadyRegistered { name: "first".into() })
|
||||
);
|
||||
tx.send(()).unwrap();
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn registering_a_dead_pid_is_noproc() {
|
||||
run(|| {
|
||||
let h = spawn(|| {});
|
||||
let pid = h.pid();
|
||||
h.join().unwrap();
|
||||
assert_eq!(register("ghost", pid), Err(RegisterError::NoProc));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn binding_evaporates_on_death_and_name_is_reusable() {
|
||||
run(|| {
|
||||
let a = spawn(|| {});
|
||||
let pid_a = a.pid();
|
||||
register("svc", pid_a).unwrap();
|
||||
a.join().unwrap();
|
||||
|
||||
// Dead holder: both lookup directions report unbound.
|
||||
assert_eq!(whereis("svc"), None);
|
||||
assert_eq!(name_of(pid_a), None);
|
||||
|
||||
// And the name is free for a successor.
|
||||
let (tx, rx) = channel::<()>();
|
||||
let b = spawn(move || {
|
||||
rx.recv().unwrap();
|
||||
});
|
||||
register("svc", b.pid()).unwrap();
|
||||
assert_eq!(whereis("svc"), Some(b.pid()));
|
||||
tx.send(()).unwrap();
|
||||
b.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn register_over_a_dead_holder_succeeds_without_lookup_in_between() {
|
||||
// The eviction path inside register() itself (not via whereis pruning).
|
||||
run(|| {
|
||||
let a = spawn(|| {});
|
||||
let pid_a = a.pid();
|
||||
register("svc", pid_a).unwrap();
|
||||
a.join().unwrap();
|
||||
|
||||
let (tx, rx) = channel::<()>();
|
||||
let b = spawn(move || {
|
||||
rx.recv().unwrap();
|
||||
});
|
||||
register("svc", b.pid()).unwrap();
|
||||
assert_eq!(whereis("svc"), Some(b.pid()));
|
||||
assert_eq!(name_of(pid_a), None);
|
||||
tx.send(()).unwrap();
|
||||
b.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unregister_frees_both_directions() {
|
||||
run(|| {
|
||||
let (tx, rx) = channel::<()>();
|
||||
let h = spawn(move || {
|
||||
rx.recv().unwrap();
|
||||
});
|
||||
register("svc", h.pid()).unwrap();
|
||||
assert_eq!(unregister("svc"), Some(h.pid()));
|
||||
assert_eq!(whereis("svc"), None);
|
||||
assert_eq!(name_of(h.pid()), None);
|
||||
assert_eq!(unregister("svc"), None);
|
||||
|
||||
// The pid may take a new name afterwards.
|
||||
register("svc2", h.pid()).unwrap();
|
||||
assert_eq!(name_of(h.pid()).as_deref(), Some("svc2"));
|
||||
tx.send(()).unwrap();
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn whereis_from_another_actor_and_usable_with_runtime_apis() {
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
let stopped = Arc::new(AtomicBool::new(false));
|
||||
let stopped2 = stopped.clone();
|
||||
run(move || {
|
||||
let (tx, rx) = channel::<()>();
|
||||
let svc = spawn(move || {
|
||||
// Parks forever; only a request_stop ends it.
|
||||
let _ = rx.recv();
|
||||
});
|
||||
register("stoppable", svc.pid()).unwrap();
|
||||
|
||||
let stopped3 = stopped2.clone();
|
||||
let client = spawn(move || {
|
||||
let pid = whereis("stoppable").expect("name must resolve cross-actor");
|
||||
// The registry's pids plug into the rest of the runtime API.
|
||||
smarm::request_stop(pid);
|
||||
stopped3.store(true, Ordering::Relaxed);
|
||||
});
|
||||
client.join().unwrap();
|
||||
svc.join().unwrap(); // a stopped actor joins Ok
|
||||
drop(tx);
|
||||
});
|
||||
assert!(stopped.load(Ordering::Relaxed));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn registry_under_concurrent_churn_multi_thread() {
|
||||
// Many actors racing to claim the same names while holders die: the
|
||||
// bimap invariant (debug-asserted internally) and Erlang semantics must
|
||||
// hold under real parallelism.
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
let wins = Arc::new(AtomicU32::new(0));
|
||||
let wins2 = wins.clone();
|
||||
smarm::init(smarm::Config::exact(4)).run(move || {
|
||||
let mut handles = Vec::new();
|
||||
for round in 0..8 {
|
||||
let name = format!("contested-{}", round % 2);
|
||||
for _ in 0..8 {
|
||||
let name = name.clone();
|
||||
let wins = wins2.clone();
|
||||
handles.push(spawn(move || {
|
||||
let me = smarm::self_pid();
|
||||
match register(&name, me) {
|
||||
Ok(()) => {
|
||||
wins.fetch_add(1, Ordering::Relaxed);
|
||||
smarm::yield_now();
|
||||
// May have been pruned-by-contact never; we are
|
||||
// alive, so our binding must still resolve to us.
|
||||
assert_eq!(whereis(&name), Some(me));
|
||||
assert_eq!(unregister(&name), Some(me));
|
||||
}
|
||||
Err(RegisterError::NameTaken { .. }) => {}
|
||||
Err(e) => panic!("unexpected register error: {e}"),
|
||||
}
|
||||
}));
|
||||
}
|
||||
}
|
||||
for h in handles {
|
||||
h.join().unwrap();
|
||||
}
|
||||
});
|
||||
// At least one registration per name must have succeeded.
|
||||
assert!(wins.load(Ordering::Relaxed) >= 2);
|
||||
}
|
||||
-342
@@ -1,342 +0,0 @@
|
||||
//! select tests. Beyond the functional cases, the *_stays_precise tests are
|
||||
//! soundness probes for the consuming-wake protocol: they fire stale loser-
|
||||
//! arm wakes at an actor and then run a one-shot park (`sleep`, whose early
|
||||
//! return would be observable as a short elapsed time) to prove the stale
|
||||
//! wake died at its epoch CAS instead of corrupting the next wait.
|
||||
|
||||
use smarm::{channel, run, select, spawn};
|
||||
use std::sync::atomic::{AtomicI64, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
#[test]
|
||||
fn ready_arm_returns_immediately_without_parking() {
|
||||
let out = Arc::new(AtomicI64::new(0));
|
||||
let out2 = out.clone();
|
||||
run(move || {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (_txb, rxb) = channel::<i64>();
|
||||
txa.send(42).unwrap();
|
||||
let i = select(&[&rxb, &rxa]);
|
||||
assert_eq!(i, 1);
|
||||
out2.store(rxa.try_recv().unwrap().expect("ready arm must hold a message"), Ordering::SeqCst);
|
||||
});
|
||||
assert_eq!(out.load(Ordering::SeqCst), 42);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lower_index_wins_when_several_arms_are_ready() {
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
txa.send(1).unwrap();
|
||||
txb.send(2).unwrap();
|
||||
// Documented priority order: index 0 first, regardless of send order.
|
||||
assert_eq!(select(&[&rxa, &rxb]), 0);
|
||||
assert_eq!(select(&[&rxb, &rxa]), 0);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parks_until_any_arm_fires() {
|
||||
let out = Arc::new(AtomicI64::new(0));
|
||||
let out2 = out.clone();
|
||||
run(move || {
|
||||
let (txa, _rxa_keepalive) = (channel::<i64>().0, ());
|
||||
let _hold = txa; // arm a: sender alive, never sends
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
let _keep_a_open = txa;
|
||||
let h = spawn(move || {
|
||||
smarm::sleep(Duration::from_millis(10));
|
||||
txb.send(7).unwrap();
|
||||
});
|
||||
let i = select(&[&rxa, &rxb]);
|
||||
assert_eq!(i, 1);
|
||||
out2.store(rxb.try_recv().unwrap().unwrap(), Ordering::SeqCst);
|
||||
h.join().unwrap();
|
||||
});
|
||||
assert_eq!(out.load(Ordering::SeqCst), 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn closed_arm_counts_as_ready() {
|
||||
run(|| {
|
||||
let (_keep, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
drop(txb);
|
||||
let i = select(&[&rxa, &rxb]);
|
||||
assert_eq!(i, 1);
|
||||
// The caller observes the closure on the arm itself.
|
||||
assert!(rxb.try_recv().is_err());
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn closure_while_parked_wakes_the_select() {
|
||||
run(|| {
|
||||
let (_keep, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
let h = spawn(move || {
|
||||
smarm::sleep(Duration::from_millis(10));
|
||||
drop(txb);
|
||||
});
|
||||
assert_eq!(select(&[&rxa, &rxb]), 1);
|
||||
assert!(rxb.try_recv().is_err());
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn loser_arm_wake_after_parked_select_stays_precise() {
|
||||
// Both arms registered; arm 0 wins the wake; arm 1's sender then fires
|
||||
// a wake stamped with the consumed epoch. The subsequent sleep is a
|
||||
// one-shot park: an early return would mean the stale wake landed.
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
let h = spawn(move || {
|
||||
smarm::sleep(Duration::from_millis(10));
|
||||
txa.send(1).unwrap(); // wins
|
||||
txb.send(2).unwrap(); // loser arm: stale-epoch wake
|
||||
});
|
||||
assert_eq!(select(&[&rxa, &rxb]), 0);
|
||||
assert_eq!(rxa.try_recv().unwrap(), Some(1));
|
||||
let t0 = Instant::now();
|
||||
smarm::sleep(Duration::from_millis(40));
|
||||
assert!(
|
||||
t0.elapsed() >= Duration::from_millis(40),
|
||||
"one-shot park returned early: a stale loser-arm wake landed"
|
||||
);
|
||||
// The loser's message was never lost.
|
||||
assert_eq!(select(&[&rxa, &rxb]), 1);
|
||||
assert_eq!(rxb.try_recv().unwrap(), Some(2));
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_park_exit_retires_the_wait() {
|
||||
// Arm 1 is ready at registration time, so select returns WITHOUT
|
||||
// parking while arm 0 holds a live-epoch registration. Arm 0's sender
|
||||
// then fires. retire_wait must have invalidated/eaten that wake;
|
||||
// the sleep proves it.
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
txb.send(2).unwrap();
|
||||
assert_eq!(select(&[&rxa, &rxb]), 1);
|
||||
assert_eq!(rxb.try_recv().unwrap(), Some(2));
|
||||
|
||||
let h = spawn(move || {
|
||||
txa.send(1).unwrap(); // fires at the (retired) select epoch
|
||||
});
|
||||
h.join().unwrap();
|
||||
|
||||
let t0 = Instant::now();
|
||||
smarm::sleep(Duration::from_millis(40));
|
||||
assert!(
|
||||
t0.elapsed() >= Duration::from_millis(40),
|
||||
"one-shot park returned early: the no-park exit leaked a wake"
|
||||
);
|
||||
assert_eq!(rxa.try_recv().unwrap(), Some(1));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn select_then_plain_recv_on_a_loser_arm() {
|
||||
// A leftover own-registration from a select must not trip the next
|
||||
// direct wait on that channel (it is overwritten, not asserted away).
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
txa.send(1).unwrap();
|
||||
assert_eq!(select(&[&rxa, &rxb]), 0); // rxb may keep a registration
|
||||
assert_eq!(rxa.try_recv().unwrap(), Some(1));
|
||||
let h = spawn(move || {
|
||||
smarm::sleep(Duration::from_millis(5));
|
||||
txb.send(9).unwrap();
|
||||
});
|
||||
assert_eq!(rxb.recv().unwrap(), 9);
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn select_loop_drains_two_producers_completely() {
|
||||
const N: i64 = 200;
|
||||
let out = Arc::new(AtomicI64::new(0));
|
||||
let out2 = out.clone();
|
||||
run(move || {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
let ha = spawn(move || {
|
||||
for v in 0..N {
|
||||
txa.send(v).unwrap();
|
||||
if v % 7 == 0 {
|
||||
smarm::yield_now();
|
||||
}
|
||||
}
|
||||
});
|
||||
let hb = spawn(move || {
|
||||
for v in 0..N {
|
||||
txb.send(v).unwrap();
|
||||
if v % 5 == 0 {
|
||||
smarm::yield_now();
|
||||
}
|
||||
}
|
||||
});
|
||||
// A closed arm stays permanently ready, so it must leave the arm
|
||||
// set once observed (keeping it would starve the other arm at
|
||||
// priority order) — select until the FIRST closure, then drain the
|
||||
// survivor with plain recv.
|
||||
let mut sum = 0i64;
|
||||
let survivor: &smarm::Receiver<i64> = loop {
|
||||
let i = select(&[&rxa, &rxb]);
|
||||
let arm: &smarm::Receiver<i64> = if i == 0 { &rxa } else { &rxb };
|
||||
match arm.try_recv() {
|
||||
Ok(Some(v)) => sum += v,
|
||||
Ok(None) => {} // defensive: ready arm already drained
|
||||
Err(_) => break if i == 0 { &rxb } else { &rxa },
|
||||
}
|
||||
};
|
||||
loop {
|
||||
match survivor.recv() {
|
||||
Ok(v) => sum += v,
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
out2.store(sum, Ordering::SeqCst);
|
||||
ha.join().unwrap();
|
||||
hb.join().unwrap();
|
||||
});
|
||||
assert_eq!(out.load(Ordering::SeqCst), 2 * (0..200i64).sum::<i64>());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn motivating_pattern_inbox_plus_monitor_down() {
|
||||
// The gen_server/handle_info shape select unlocks: a server waiting on
|
||||
// its request inbox AND a monitor Down channel in one park.
|
||||
use smarm::monitor;
|
||||
run(|| {
|
||||
let (req_tx, req_rx) = channel::<i64>();
|
||||
let worker = spawn(|| {
|
||||
smarm::sleep(Duration::from_millis(10));
|
||||
// worker exits -> Down fires
|
||||
});
|
||||
let m = monitor(worker.pid());
|
||||
let _keep_inbox_open = req_tx;
|
||||
|
||||
let mut served = 0;
|
||||
loop {
|
||||
match select(&[&m.rx, &req_rx]) {
|
||||
0 => {
|
||||
let _down = m.rx.try_recv().unwrap().expect("Down message");
|
||||
break;
|
||||
}
|
||||
_ => {
|
||||
if let Ok(Some(_)) = req_rx.try_recv() {
|
||||
served += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
assert_eq!(served, 0);
|
||||
worker.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// select_timeout
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
use smarm::select_timeout;
|
||||
|
||||
#[test]
|
||||
fn select_timeout_returns_none_after_the_deadline() {
|
||||
run(|| {
|
||||
let (_keep_a, rxa) = channel::<i64>();
|
||||
let (_keep_b, rxb) = channel::<i64>();
|
||||
let t0 = Instant::now();
|
||||
let r = select_timeout(&[&rxa, &rxb], Duration::from_millis(30));
|
||||
assert_eq!(r, None);
|
||||
assert!(t0.elapsed() >= Duration::from_millis(30));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn select_timeout_ready_arm_wins_without_arming_a_timer() {
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (_keep_b, rxb) = channel::<i64>();
|
||||
txa.send(5).unwrap();
|
||||
assert_eq!(select_timeout(&[&rxb, &rxa], Duration::from_millis(500)), Some(1));
|
||||
assert_eq!(rxa.try_recv().unwrap(), Some(5));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn select_timeout_arm_beats_timer_and_stale_entry_stays_inert() {
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (_keep_b, rxb) = channel::<i64>();
|
||||
let h = spawn(move || {
|
||||
smarm::sleep(Duration::from_millis(5));
|
||||
txa.send(1).unwrap();
|
||||
});
|
||||
let t0 = Instant::now();
|
||||
let r = select_timeout(&[&rxa, &rxb], Duration::from_millis(200));
|
||||
assert_eq!(r, Some(0));
|
||||
assert!(t0.elapsed() < Duration::from_millis(200));
|
||||
assert_eq!(rxa.try_recv().unwrap(), Some(1));
|
||||
h.join().unwrap();
|
||||
// The abandoned timer entry expires mid-sleep; a stale-epoch wake
|
||||
// landing would return this one-shot park early.
|
||||
let t1 = Instant::now();
|
||||
smarm::sleep(Duration::from_millis(250));
|
||||
assert!(
|
||||
t1.elapsed() >= Duration::from_millis(250),
|
||||
"one-shot park returned early: the stale select timer landed"
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn select_timeout_timer_first_message_still_delivered_later() {
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let h = spawn(move || {
|
||||
smarm::sleep(Duration::from_millis(60));
|
||||
txa.send(9).unwrap();
|
||||
});
|
||||
assert_eq!(select_timeout(&[&rxa], Duration::from_millis(10)), None);
|
||||
// The wait is over but the channel is intact: the late message
|
||||
// arrives and a fresh select sees it.
|
||||
assert_eq!(select(&[&rxa]), 0);
|
||||
assert_eq!(rxa.try_recv().unwrap(), Some(9));
|
||||
h.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn select_timeout_zero_duration_polls() {
|
||||
run(|| {
|
||||
let (txa, rxa) = channel::<i64>();
|
||||
let (_keep_b, rxb) = channel::<i64>();
|
||||
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::ZERO), None);
|
||||
txa.send(3).unwrap();
|
||||
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::ZERO), Some(0));
|
||||
assert_eq!(rxa.try_recv().unwrap(), Some(3));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn select_timeout_closed_arm_is_ready_not_a_timeout() {
|
||||
run(|| {
|
||||
let (_keep_a, rxa) = channel::<i64>();
|
||||
let (txb, rxb) = channel::<i64>();
|
||||
drop(txb);
|
||||
assert_eq!(select_timeout(&[&rxa, &rxb], Duration::from_millis(200)), Some(1));
|
||||
assert!(rxb.try_recv().is_err());
|
||||
});
|
||||
}
|
||||
@@ -1,260 +0,0 @@
|
||||
//! RFC 004 spinning-worker tests.
|
||||
//!
|
||||
//! The futex scheduler-park path is NOT loom-modelled (sync_shim.rs: the full
|
||||
//! runtime — context switches, futexes, TLS — is never run under cfg(loom)),
|
||||
//! so the Dekker submit/park ordering is guarded here instead, by driving the
|
||||
//! one true deadlock state and catching it as a timeout rather than a hung
|
||||
//! binary: every scheduler parked on the futex while runnable work exists.
|
||||
//!
|
||||
//! All tests use a NO-IO runtime (pure message passing) so the idle path is
|
||||
//! the bare `_` arm RFC 004 replaces — the futex park — not the timer/io arms,
|
||||
//! which keep their own wake sources. A lost wakeup therefore manifests as a
|
||||
//! permanent stall: a `recv` that never returns, an `rt.run` that never
|
||||
//! reaches AllDone. The watchdog turns that into a failed assertion.
|
||||
|
||||
use smarm::channel::channel;
|
||||
use smarm::runtime::{init, Config};
|
||||
use smarm::spawn;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
/// Run `body` on a side thread; fail (rather than hang the test binary) if it
|
||||
/// does not finish within `secs`. A hang here means a scheduler parked on the
|
||||
/// futex with work outstanding and nothing woke it — exactly the RFC 004
|
||||
/// failure mode. The leaked thread is acceptable: the process exits non-zero.
|
||||
fn run_with_timeout(secs: u64, body: impl FnOnce() + Send + 'static) {
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
std::thread::spawn(move || {
|
||||
body();
|
||||
let _ = tx.send(());
|
||||
});
|
||||
if rx.recv_timeout(Duration::from_secs(secs)).is_err() {
|
||||
panic!(
|
||||
"RFC 004 deadlock: workload did not terminate within {secs}s \
|
||||
— a scheduler wakeup was lost (work stranded with all schedulers \
|
||||
parked on the futex)"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A single token ping-ponging between two actors for `rounds` rounds, on a
|
||||
/// `threads`-wide no-io runtime with the given spin budget. With more threads
|
||||
/// than active actors, the surplus schedulers go idle and park; every send to
|
||||
/// the (parked) peer is a cross-thread wakeup. A lost wake stalls the bounce.
|
||||
/// The post-run count confirms no message was silently dropped.
|
||||
fn bounce(threads: usize, rounds: u64, budget: u64) {
|
||||
run_with_timeout(30, move || {
|
||||
let rt = init(Config::exact(threads).spin_budget_cycles(budget));
|
||||
let echoed = Arc::new(AtomicU64::new(0));
|
||||
let echoed_actor = echoed.clone();
|
||||
rt.run(move || {
|
||||
let (tx_ab, rx_ab) = channel::<u64>();
|
||||
let (tx_ba, rx_ba) = channel::<u64>();
|
||||
let echo = spawn(move || {
|
||||
for _ in 0..rounds {
|
||||
let v = rx_ab.recv().expect("echo recv");
|
||||
echoed_actor.fetch_add(1, Ordering::Relaxed);
|
||||
tx_ba.send(v + 1).expect("echo send");
|
||||
}
|
||||
});
|
||||
for i in 0..rounds {
|
||||
tx_ab.send(i).expect("ping send");
|
||||
let r = rx_ba.recv().expect("ping recv");
|
||||
assert_eq!(r, i + 1, "token corrupted in flight");
|
||||
}
|
||||
echo.join().expect("echo join");
|
||||
});
|
||||
assert_eq!(
|
||||
echoed.load(Ordering::Relaxed),
|
||||
rounds,
|
||||
"messages were lost, not just delayed"
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/// Tiny budget: schedulers park almost immediately, so essentially every
|
||||
/// cross-thread handoff exercises the futex wake. Sharpest stress on the
|
||||
/// submit-rule / lost-wakeup guards.
|
||||
#[test]
|
||||
fn bounce_forces_park_and_wake() {
|
||||
bounce(4, 20_000, 2_000);
|
||||
}
|
||||
|
||||
/// Large budget: idle schedulers stay hot spinners, so the submit rule mostly
|
||||
/// takes the `n_spinning > 0` skip and the last-spinner handoff (rule 4) does
|
||||
/// the waking. Confirms that path also makes progress and stays correct.
|
||||
#[test]
|
||||
fn bounce_with_hot_spinners() {
|
||||
bounce(4, 20_000, 2_000_000);
|
||||
}
|
||||
|
||||
/// Oversubscribed: more active actors than schedulers, so the run queue is
|
||||
/// rarely empty and the park/spin/submit machinery is hammered from every
|
||||
/// thread at once.
|
||||
#[test]
|
||||
fn bounce_oversubscribed() {
|
||||
bounce(2, 30_000, 5_000);
|
||||
}
|
||||
|
||||
/// Bursty fork: a parent makes many children runnable at once (the motivating
|
||||
/// fan-out), each sends one reply, the parent collects all of them. Exercises
|
||||
/// the submit rule under a burst — the first job eats one wake, the woken
|
||||
/// worker becomes a spinner and absorbs the rest — and confirms none are lost.
|
||||
fn fanout(threads: usize, children: u64, budget: u64) {
|
||||
run_with_timeout(30, move || {
|
||||
let rt = init(Config::exact(threads).spin_budget_cycles(budget));
|
||||
rt.run(move || {
|
||||
let (tx, rx) = channel::<u64>();
|
||||
for c in 0..children {
|
||||
let tx = tx.clone();
|
||||
spawn(move || {
|
||||
tx.send(c).expect("child send");
|
||||
});
|
||||
}
|
||||
drop(tx);
|
||||
let mut sum = 0u64;
|
||||
for _ in 0..children {
|
||||
sum += rx.recv().expect("collector recv");
|
||||
}
|
||||
assert_eq!(sum, (0..children).sum(), "a child reply was lost");
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fanout_burst_parks() {
|
||||
fanout(4, 256, 2_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fanout_burst_hot() {
|
||||
fanout(4, 256, 2_000_000);
|
||||
}
|
||||
|
||||
/// Shutdown guard: a mostly-idle spinning runtime must still terminate. Most
|
||||
/// schedulers park on the futex with no work ever coming; only the AllDone
|
||||
/// broadcast (`unpark_all_schedulers`) can release them. If that broadcast is
|
||||
/// missing, `rt.run` never returns and the watchdog fires.
|
||||
#[test]
|
||||
fn idle_runtime_shuts_down() {
|
||||
run_with_timeout(15, || {
|
||||
let rt = init(Config::exact(8).spin_budget_cycles(2_000));
|
||||
rt.run(|| {
|
||||
// One trivial cross-thread handoff, then everyone goes idle and the
|
||||
// runtime must drain to AllDone through parked schedulers.
|
||||
let (tx, rx) = channel::<u64>();
|
||||
let h = spawn(move || {
|
||||
tx.send(42).expect("send");
|
||||
});
|
||||
assert_eq!(rx.recv().expect("recv"), 42);
|
||||
h.join().expect("join");
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/// Repeated lifecycles: init→run→AllDone several times in one process, to
|
||||
/// catch any park/broadcast state that fails to reset between runtimes.
|
||||
#[test]
|
||||
fn repeated_spinning_lifecycles() {
|
||||
run_with_timeout(30, || {
|
||||
for _ in 0..20 {
|
||||
let rt = init(Config::exact(4).spin_budget_cycles(2_000));
|
||||
let echoed = Arc::new(AtomicU64::new(0));
|
||||
let e = echoed.clone();
|
||||
rt.run(move || {
|
||||
let (tx, rx) = channel::<u64>();
|
||||
let worker = spawn(move || {
|
||||
for _ in 0..500 {
|
||||
let v = rx.recv().expect("recv");
|
||||
e.fetch_add(v, Ordering::Relaxed);
|
||||
}
|
||||
});
|
||||
for i in 0..500u64 {
|
||||
tx.send(i).expect("send");
|
||||
}
|
||||
worker.join().expect("worker join");
|
||||
});
|
||||
assert_eq!(echoed.load(Ordering::Relaxed), (0..500u64).sum());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Shutdown with siblings *pinned* in the futex park — the test that actually
|
||||
/// exercises the AllDone broadcast. A single long-running actor holds `live`
|
||||
/// at 1 (CPU-bound, never sending or spawning) long enough for every other
|
||||
/// scheduler to idle, spin out its budget, and commit to `futex_wait`. Because
|
||||
/// nothing is ever enqueued during that window, the submit-rule wakes never
|
||||
/// fire and those siblings stay parked; the final finalize enqueues no one, so
|
||||
/// the live-check self-termination cannot save them either. Only
|
||||
/// `unpark_all_schedulers` at AllDone can. Gut that broadcast and this hangs.
|
||||
#[test]
|
||||
fn shutdown_releases_pinned_parked_siblings() {
|
||||
run_with_timeout(15, || {
|
||||
let rt = init(Config::exact(8).spin_budget_cycles(2_000));
|
||||
rt.run(|| {
|
||||
// ~50ms of pure CPU: far longer than the µs it takes the other
|
||||
// seven schedulers to park, so they are provably in futex_wait
|
||||
// before this returns and live drops to 0.
|
||||
let start = std::time::Instant::now();
|
||||
let mut x = 0u64;
|
||||
while start.elapsed() < Duration::from_millis(50) {
|
||||
x = x.wrapping_mul(6364136223846793005).wrapping_add(1);
|
||||
std::hint::black_box(x);
|
||||
}
|
||||
std::hint::black_box(x);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/// cross-thread handoff — but the futex park path must still be inert and
|
||||
/// correct (the lone thread is never parked while it has work). Pins the
|
||||
/// degenerate cap=0 case.
|
||||
#[test]
|
||||
fn single_thread_spinning_is_inert() {
|
||||
run_with_timeout(15, || {
|
||||
let rt = init(Config::exact(1).spin_budget_cycles(2_000));
|
||||
let echoed = Arc::new(AtomicU64::new(0));
|
||||
let e = echoed.clone();
|
||||
rt.run(move || {
|
||||
let (tx, rx) = channel::<u64>();
|
||||
let worker = spawn(move || {
|
||||
for _ in 0..1_000 {
|
||||
let v = rx.recv().expect("recv");
|
||||
e.fetch_add(v, Ordering::Relaxed);
|
||||
}
|
||||
});
|
||||
for i in 0..1_000u64 {
|
||||
tx.send(i).expect("send");
|
||||
}
|
||||
worker.join().expect("join");
|
||||
});
|
||||
assert_eq!(echoed.load(Ordering::Relaxed), (0..1_000u64).sum());
|
||||
});
|
||||
}
|
||||
|
||||
/// `spin_budget_cycles == 0` is the opt-out: the feature is fully off and the
|
||||
/// runtime uses the historical `thread::sleep` idle path. Must still run a
|
||||
/// cross-thread workload to completion (it always did) — this just guards that
|
||||
/// the gating left the off-path intact.
|
||||
#[test]
|
||||
fn budget_zero_keeps_old_behaviour() {
|
||||
run_with_timeout(20, || {
|
||||
let rt = init(Config::exact(4).spin_budget_cycles(0));
|
||||
let sum = Arc::new(AtomicU64::new(0));
|
||||
let s = sum.clone();
|
||||
rt.run(move || {
|
||||
let (tx, rx) = channel::<u64>();
|
||||
let worker = spawn(move || {
|
||||
for _ in 0..1_000 {
|
||||
s.fetch_add(rx.recv().expect("recv"), Ordering::Relaxed);
|
||||
}
|
||||
});
|
||||
for i in 0..1_000u64 {
|
||||
tx.send(i).expect("send");
|
||||
}
|
||||
worker.join().expect("join");
|
||||
});
|
||||
assert_eq!(sum.load(Ordering::Relaxed), (0..1_000u64).sum());
|
||||
});
|
||||
}
|
||||
@@ -1,115 +0,0 @@
|
||||
//! Terminal wake: a scheduler thread blocked in the idle wait must be woken
|
||||
//! when a sibling reaches the all-done verdict, or `rt.run` stalls in its
|
||||
//! worker join.
|
||||
//!
|
||||
//! Mechanism (runtime.rs, `Pop::Idle`): an idle scheduler snapshots
|
||||
//! `peek_deadline()` / `io_outstanding` and blocks in `poll_wake` (or
|
||||
//! `thread::sleep`) on that snapshot. `enqueue` does not write the wake
|
||||
//! pipe — only IO completions do — so the snapshot can go terminally stale:
|
||||
//!
|
||||
//! - An actor parked in `wait_readable` that is `request_stop`ped produces
|
||||
//! NO completion (cancellation deregisters the waiter); a sibling that
|
||||
//! blocked on `io_outstanding > 0` with no timers is in `poll(-1)` forever.
|
||||
//! - An actor cancelled out of a long `sleep` leaves its timer entry
|
||||
//! orphaned; a sibling that blocked on that deadline sleeps it out in full.
|
||||
//!
|
||||
//! In both cases the remaining work completes on the *other* scheduler
|
||||
//! thread, which hits AllDone and returns — and nothing wakes the blocked
|
||||
//! one. `Runtime::run` joins it: a permanent hang in the first case, a
|
||||
//! full-deadline stall in the second.
|
||||
//!
|
||||
//! Both tests force the window deterministically: the root busy-spins
|
||||
//! (creating no timer entries and occupying one scheduler thread) so the
|
||||
//! other thread settles into the stale idle wait before the stop is issued.
|
||||
|
||||
use std::sync::mpsc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
struct PipePair {
|
||||
read: libc::c_int,
|
||||
write: libc::c_int,
|
||||
}
|
||||
|
||||
impl PipePair {
|
||||
fn new() -> Self {
|
||||
let mut fds: [libc::c_int; 2] = [0; 2];
|
||||
let r = unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_CLOEXEC | libc::O_NONBLOCK) };
|
||||
assert_eq!(r, 0, "pipe2 failed");
|
||||
PipePair { read: fds[0], write: fds[1] }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for PipePair {
|
||||
fn drop(&mut self) {
|
||||
unsafe {
|
||||
libc::close(self.read);
|
||||
libc::close(self.write);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Occupy the current scheduler thread without creating timer entries or
|
||||
/// parking. A plain spin has no observation points, so the root stays
|
||||
/// on-CPU and the sibling scheduler is left alone with the idle branch.
|
||||
fn spin_for(d: Duration) {
|
||||
let t0 = Instant::now();
|
||||
while t0.elapsed() < d {
|
||||
std::hint::spin_loop();
|
||||
}
|
||||
}
|
||||
|
||||
/// Run `body` under a 2-scheduler runtime on a watchdog thread; fail if
|
||||
/// `Runtime::run` has not returned within `limit`.
|
||||
fn run_with_watchdog(limit: Duration, body: impl FnOnce() + Send + 'static) {
|
||||
let (done_tx, done_rx) = mpsc::channel::<()>();
|
||||
std::thread::spawn(move || {
|
||||
let rt = smarm::init(smarm::Config::exact(2));
|
||||
rt.run(body);
|
||||
let _ = done_tx.send(());
|
||||
});
|
||||
done_rx
|
||||
.recv_timeout(limit)
|
||||
.expect("Runtime::run did not return: idle scheduler thread was never woken at termination");
|
||||
}
|
||||
|
||||
/// Permanent-hang variant: sibling blocked in `poll_wake(wake_fd, None)`
|
||||
/// because `io_outstanding > 0` (one actor parked in `wait_readable`) and no
|
||||
/// timers are pending. The waiter is then stop-cancelled — no IO completion
|
||||
/// ever writes the wake pipe — and everything else finishes on the root's
|
||||
/// thread. Without a terminal wake, `rt.run` never returns.
|
||||
#[test]
|
||||
fn run_returns_after_io_waiter_is_stop_cancelled() {
|
||||
run_with_watchdog(Duration::from_secs(10), || {
|
||||
let pipe = PipePair::new();
|
||||
let rfd = pipe.read;
|
||||
let h = smarm::spawn(move || {
|
||||
// Never-readable fd (write end open, nothing written).
|
||||
let _ = smarm::wait_readable(rfd);
|
||||
});
|
||||
// Let the waiter park and the sibling scheduler settle into the
|
||||
// io_outstanding>0 / no-timers idle wait: poll(wake_fd, -1).
|
||||
spin_for(Duration::from_millis(200));
|
||||
smarm::request_stop(h.pid());
|
||||
let _ = h.join();
|
||||
drop(pipe);
|
||||
});
|
||||
}
|
||||
|
||||
/// Finite-stall variant: sibling blocked on an orphaned long timer
|
||||
/// deadline. An actor cancelled out of `sleep(60s)` leaves its timer entry
|
||||
/// behind (documented as harmless at AllDone — but a sibling that already
|
||||
/// blocked on that deadline sleeps it out in full, stalling `rt.run` for
|
||||
/// the better part of a minute).
|
||||
#[test]
|
||||
fn run_returns_after_long_sleeper_is_stop_cancelled() {
|
||||
run_with_watchdog(Duration::from_secs(10), || {
|
||||
let h = smarm::spawn(|| {
|
||||
smarm::sleep(Duration::from_secs(60));
|
||||
});
|
||||
// Let the sleeper park and the sibling scheduler block on the 60s
|
||||
// deadline: poll(wake_fd, ~60_000ms).
|
||||
spin_for(Duration::from_millis(200));
|
||||
smarm::request_stop(h.pid());
|
||||
let _ = h.join();
|
||||
});
|
||||
}
|
||||
+15
-15
@@ -124,11 +124,11 @@ use smarm::pid::Pid;
|
||||
use smarm::timer::{Reason, TimerTarget, Timers};
|
||||
|
||||
struct RecordingTarget {
|
||||
calls: Mutex<Vec<(Pid, u32)>>,
|
||||
calls: Mutex<Vec<(Pid, u64)>>,
|
||||
}
|
||||
impl TimerTarget for RecordingTarget {
|
||||
fn on_timeout(&self, pid: Pid, epoch: u32) {
|
||||
self.calls.lock().unwrap().push((pid, epoch));
|
||||
fn on_timeout(&self, pid: Pid, seq: u64) {
|
||||
self.calls.lock().unwrap().push((pid, seq));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -137,9 +137,9 @@ fn timers_pop_due_returns_entries_in_deadline_order() {
|
||||
let mut t = Timers::new();
|
||||
let now = Instant::now();
|
||||
// Insert out of order; pop_due should hand them back sorted by deadline.
|
||||
t.insert_sleep(now + Duration::from_millis(30), Pid::new(0, 0), 1);
|
||||
t.insert_sleep(now + Duration::from_millis(10), Pid::new(1, 0), 1);
|
||||
t.insert_sleep(now + Duration::from_millis(20), Pid::new(2, 0), 1);
|
||||
t.insert_sleep(now + Duration::from_millis(30), Pid::new(0, 0));
|
||||
t.insert_sleep(now + Duration::from_millis(10), Pid::new(1, 0));
|
||||
t.insert_sleep(now + Duration::from_millis(20), Pid::new(2, 0));
|
||||
|
||||
// Advance past all of them.
|
||||
let due = t.pop_due(now + Duration::from_millis(50));
|
||||
@@ -152,8 +152,8 @@ fn timers_pop_due_returns_entries_in_deadline_order() {
|
||||
fn timers_only_pop_entries_whose_deadline_has_passed() {
|
||||
let mut t = Timers::new();
|
||||
let now = Instant::now();
|
||||
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0), 1);
|
||||
t.insert_sleep(now + Duration::from_millis(100), Pid::new(1, 0), 1);
|
||||
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0));
|
||||
t.insert_sleep(now + Duration::from_millis(100), Pid::new(1, 0));
|
||||
|
||||
let due = t.pop_due(now + Duration::from_millis(20));
|
||||
assert_eq!(due.len(), 1);
|
||||
@@ -169,11 +169,11 @@ fn timers_mix_sleep_and_wait_timeout_reasons() {
|
||||
let target = Arc::new(RecordingTarget { calls: Mutex::new(Vec::new()) });
|
||||
let now = Instant::now();
|
||||
|
||||
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0), 1);
|
||||
t.insert_sleep(now + Duration::from_millis(5), Pid::new(0, 0));
|
||||
t.insert(
|
||||
now + Duration::from_millis(10),
|
||||
Pid::new(1, 0),
|
||||
Reason::WaitTimeout { target: target.clone(), epoch: 42 },
|
||||
Reason::WaitTimeout { target: target.clone(), wait_seq: 42 },
|
||||
);
|
||||
|
||||
let due = t.pop_due(now + Duration::from_millis(20));
|
||||
@@ -181,11 +181,11 @@ fn timers_mix_sleep_and_wait_timeout_reasons() {
|
||||
|
||||
// Order: Sleep (5ms) first, WaitTimeout (10ms) second.
|
||||
match &due[0].reason {
|
||||
Reason::Sleep { .. } => {}
|
||||
Reason::Sleep => {}
|
||||
_ => panic!("first entry should be a Sleep"),
|
||||
}
|
||||
match &due[1].reason {
|
||||
Reason::WaitTimeout { epoch, .. } => assert_eq!(*epoch, 42),
|
||||
Reason::WaitTimeout { wait_seq, .. } => assert_eq!(*wait_seq, 42),
|
||||
_ => panic!("second entry should be a WaitTimeout"),
|
||||
}
|
||||
}
|
||||
@@ -197,9 +197,9 @@ fn same_deadline_entries_pop_in_insertion_order() {
|
||||
let mut t = Timers::new();
|
||||
let now = Instant::now();
|
||||
let d = now + Duration::from_millis(10);
|
||||
t.insert_sleep(d, Pid::new(0, 0), 1);
|
||||
t.insert_sleep(d, Pid::new(1, 0), 1);
|
||||
t.insert_sleep(d, Pid::new(2, 0), 1);
|
||||
t.insert_sleep(d, Pid::new(0, 0));
|
||||
t.insert_sleep(d, Pid::new(1, 0));
|
||||
t.insert_sleep(d, Pid::new(2, 0));
|
||||
|
||||
let due = t.pop_due(now + Duration::from_millis(20));
|
||||
let pids: Vec<u32> = due.iter().map(|e| e.pid.index()).collect();
|
||||
|
||||
@@ -1,176 +0,0 @@
|
||||
//! RFC 005 wake-slot tests: correctness with the slot on, push-policy
|
||||
//! discrimination (actor vs scheduler context, spawns), displacement, the
|
||||
//! default-off contract, and per-run counter reset.
|
||||
//!
|
||||
//! The slot is same-thread plain ops behind the existing queue-op contract,
|
||||
//! so there is nothing new to model-check (RFC 005 §Summary); these tests
|
||||
//! pin the *policy* — who lands in the slot, who never does — which is
|
||||
//! observable through the `slot_hits` / `slot_displacements` counters.
|
||||
|
||||
use smarm::channel::channel;
|
||||
use smarm::runtime::{init, Config};
|
||||
use smarm::spawn;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Ping-pong over channels: the slot's home pattern. Returns total roundtrips.
|
||||
fn ping_pong(pairs: usize, roundtrips: u64) -> impl FnOnce() + Send + 'static {
|
||||
move || {
|
||||
let handles: Vec<_> = (0..pairs)
|
||||
.map(|_| {
|
||||
spawn(move || {
|
||||
let (tx_ab, rx_ab) = channel::<u64>();
|
||||
let (tx_ba, rx_ba) = channel::<u64>();
|
||||
let echo = spawn(move || {
|
||||
for _ in 0..roundtrips {
|
||||
let v = rx_ab.recv().expect("echo recv");
|
||||
tx_ba.send(v + 1).expect("echo send");
|
||||
}
|
||||
});
|
||||
for i in 0..roundtrips {
|
||||
tx_ab.send(i).expect("ping send");
|
||||
assert_eq!(rx_ba.recv().expect("ping recv"), i + 1);
|
||||
}
|
||||
let _ = echo.join();
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
for h in handles {
|
||||
h.join().expect("pair");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Correctness: messaging workloads complete with the slot on, 1 and N threads
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn ping_pong_correct_with_slot_on_single_thread() {
|
||||
let rt = init(Config::exact(1).wake_slot(true));
|
||||
rt.run(ping_pong(4, 200));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ping_pong_correct_with_slot_on_multi_thread() {
|
||||
let rt = init(Config::exact(4).wake_slot(true));
|
||||
rt.run(ping_pong(8, 200));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Push policy: actor-context wakes hit the slot; the default is off
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn messaging_workload_exercises_the_slot() {
|
||||
let rt = init(Config::exact(1).wake_slot(true));
|
||||
rt.run(ping_pong(2, 100));
|
||||
let stats = rt.stats();
|
||||
assert!(
|
||||
stats.slot_hits() > 0,
|
||||
"send-wakes are actor-context unparks — the slot must see hits, got 0"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn slot_off_means_zero_slot_traffic() {
|
||||
// Off explicitly…
|
||||
let rt = init(Config::exact(1).wake_slot(false));
|
||||
rt.run(ping_pong(2, 100));
|
||||
assert_eq!(rt.stats().slot_hits(), 0);
|
||||
assert_eq!(rt.stats().slot_displacements(), 0);
|
||||
|
||||
// …and off by default (RFC 005: default off until the shootout accepts).
|
||||
let rt = init(Config::exact(1));
|
||||
rt.run(ping_pong(2, 100));
|
||||
assert_eq!(rt.stats().slot_hits(), 0, "wake_slot must default to OFF");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Push policy: spawns and join-wakes (finalize = scheduler context) bypass
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn spawn_join_workload_bypasses_the_slot() {
|
||||
let rt = init(Config::exact(1).wake_slot(true));
|
||||
rt.run(|| {
|
||||
for _ in 0..20 {
|
||||
let handles: Vec<_> = (0..50).map(|_| spawn(|| {})).collect();
|
||||
for h in handles {
|
||||
h.join().expect("trivial actor");
|
||||
}
|
||||
}
|
||||
});
|
||||
let stats = rt.stats();
|
||||
assert_eq!(
|
||||
stats.slot_hits(),
|
||||
0,
|
||||
"spawns go shared by policy and joiner wakes fire from finalize \
|
||||
(scheduler context) — a pure spawn/join workload must never touch \
|
||||
the slot"
|
||||
);
|
||||
assert_eq!(stats.slot_displacements(), 0);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Displacement: newest wake takes the slot, occupant goes shared — and runs
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn displaced_occupant_reaches_the_shared_queue_and_runs() {
|
||||
// Single thread, strict FIFO (rq-mutex default): rx1 and rx2 park on
|
||||
// their recvs before the sender runs; the sender's back-to-back sends
|
||||
// then produce two actor-context wakes — the second displaces the first.
|
||||
let rt = init(Config::exact(1).wake_slot(true));
|
||||
let ran = Arc::new(AtomicU64::new(0));
|
||||
let (r1, r2) = (ran.clone(), ran.clone());
|
||||
rt.run(move || {
|
||||
let (tx1, rx1) = channel::<u32>();
|
||||
let (tx2, rx2) = channel::<u32>();
|
||||
let h1 = spawn(move || {
|
||||
assert_eq!(rx1.recv().expect("rx1"), 1);
|
||||
r1.fetch_add(1, Ordering::Relaxed);
|
||||
});
|
||||
let h2 = spawn(move || {
|
||||
assert_eq!(rx2.recv().expect("rx2"), 2);
|
||||
r2.fetch_add(1, Ordering::Relaxed);
|
||||
});
|
||||
let sender = spawn(move || {
|
||||
tx1.send(1).expect("send 1"); // rx1 → slot
|
||||
tx2.send(2).expect("send 2"); // rx2 → slot, rx1 displaced → shared
|
||||
});
|
||||
sender.join().expect("sender");
|
||||
h1.join().expect("h1");
|
||||
h2.join().expect("h2");
|
||||
});
|
||||
assert_eq!(ran.load(Ordering::Relaxed), 2, "both receivers must run");
|
||||
let stats = rt.stats();
|
||||
assert!(
|
||||
stats.slot_displacements() >= 1,
|
||||
"back-to-back wakes of parked receivers must displace at least once"
|
||||
);
|
||||
assert!(stats.slot_hits() >= 1, "the displacing wake is slot-popped");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Counters reset at the start of each run() on a reused Runtime
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn slot_counters_reset_per_run() {
|
||||
let rt = init(Config::exact(1).wake_slot(true));
|
||||
rt.run(ping_pong(2, 100));
|
||||
let first = rt.stats().slot_hits();
|
||||
assert!(first > 0);
|
||||
|
||||
// A slot-bypassing run on the same handle must read 0, not `first`.
|
||||
rt.run(|| {
|
||||
let h = spawn(|| {});
|
||||
h.join().expect("trivial");
|
||||
});
|
||||
assert_eq!(
|
||||
rt.stats().slot_hits(),
|
||||
0,
|
||||
"counters are reset at run() start; the second run had no slot traffic"
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user