From a7ca6646d726afbb903acc57a140f05d3165ec8c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 9 Jun 2026 21:35:08 +0000 Subject: [PATCH] docs(bench,test): READMEs for benches/ and tests/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit benches/README.md: catalog of the two bench families (cross-runtime comparisons + the v0.5 run-queue shootout), how to run the shootout (scripts/bench_rq.sh, SMARM_BENCH_* knobs, the --no-default-features dance, summary.csv/RQCSV format), honest-numbers caveats (core count is the experiment; striped losing at low thread counts is expected), and the conventions for adding a bench. tests/README.md: catalog grouped by layer (low-level units / feature areas / regression+stress), the run matrix (debug-first — that's where the invariant asserts live — three queue variants, release, loom, trace), why loom tests live in src/ rather than tests/, and the house conventions: one runtime per test, oversubscription on purpose, regression tests validated against the reintroduced bug, odds-stacking for stochastic tests, ordering-not-duration time assertions, and new-invariant = assert + loom model. Also fixes a stale header in tests/mutex.rs that claimed 'loom::Mutex' — it tests smarm::Mutex, and the old name is actively confusing now that loom is real in this repo. --- benches/README.md | 89 ++++++++++++++++++++++++++++++++++ tests/README.md | 120 ++++++++++++++++++++++++++++++++++++++++++++++ tests/mutex.rs | 3 +- 3 files changed, 211 insertions(+), 1 deletion(-) create mode 100644 benches/README.md create mode 100644 tests/README.md diff --git a/benches/README.md b/benches/README.md new file mode 100644 index 0000000..0e0f3f1 --- /dev/null +++ b/benches/README.md @@ -0,0 +1,89 @@ +# Benches + +Two families live here: **comparison benches** (smarm vs tokio, predating +v0.5) and the **run-queue shootout** (v0.5 phase 4). All are plain binaries +(`harness = false` in `Cargo.toml`), so `cargo bench` just builds in release +and runs `main()` — no criterion, no magic. + +``` +cargo bench --bench # one bench +cargo bench # all of them (slow; rarely what you want) +``` + +## Catalog + +| file | what it measures | +|---|---| +| `primes.rs` | Compute fan-out/fan-in: counts primes across W workers. Pure compute throughput + spawn/join/channel cost. | +| `multi_scheduler.rs` | The original cross-runtime matrix: smarm (1 thread / N threads) vs tokio (current_thread / multi_thread) on compute, ping-pong, and spawn throughput. | +| `general.rs` | Workloads where neither runtime has a structural edge. Large gaps here mean real per-task/per-yield overhead differences — watch these for regressions. | +| `smarm_favored.rs` | Workloads the stackful green-thread model is built for. Single-thread numbers isolate per-switch cost from contention. | +| `tokio_favored.rs` | Workloads tokio's model is built for. Expect to lose; the value is knowing *by how much* and catching the gap widening. | +| `rq_micro.rs` | Run-queue **structures** in isolation (no runtime, no actors): push/pop throughput sweeping thread count × producer:consumer ratio. Covers all three queue types in one binary — the types compile in every build; only the runtime's alias is feature-selected. | +| `rq_runtime.rs` | The **whole scheduler** with the compile-time-selected queue: yield-storm (pure queue churn), ping-pong-pairs (park/unpark latency), spawn-storm (slab + free list + queue churn), sweeping scheduler count. Comparing variants requires rebuilding per `rq-*` feature. | + +## The run-queue shootout + +One command; it rebuilds `rq_runtime` once per queue variant, runs `rq_micro` +once, and aggregates: + +``` +./scripts/bench_rq.sh +# on a big box: +SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh +``` + +Outputs land in `bench_results/` (gitignored): one full log per run, plus +`summary.csv` assembled from the machine-readable `RQCSV,...` lines every +config prints alongside the human table. + +Manual single-variant runs need the feature dance (features are additive, so +the default `rq-mutex` must be switched off): + +``` +cargo bench --bench rq_runtime --no-default-features --features rq-striped +``` + +### Knobs (env vars, all optional) + +| var | default | used by | +|---|---|---| +| `SMARM_BENCH_THREADS` | `"1 2 4"` | both — space-separated sweep | +| `SMARM_BENCH_RUNS` | `5` | both — repetitions; the **median** is reported | +| `SMARM_BENCH_ITEMS` | `200000` | `rq_micro` — items per measurement | +| `SMARM_BENCH_YIELD_ACTORS` / `_YIELDS` | `200` / `500` | `rq_runtime` yield-storm | +| `SMARM_BENCH_PAIRS` / `_ROUNDTRIPS` | `32` / `1000` | `rq_runtime` ping-pong | +| `SMARM_BENCH_SPAWNS` | `5000` | `rq_runtime` spawn-storm | + +## Reading the numbers honestly + +- **Core count is the experiment.** On a 1-core machine (CI, sandboxes) the + sweep only validates the harness and catches gross pathologies — + oversubscribed schedulers measure context-switch noise, not contention. + Variant decisions come from a many-core box. +- The striped queue *should lose* at low thread counts (ticket overhead with + no contention to amortize) — that's expected, not a bug. +- Medians over `SMARM_BENCH_RUNS` absorb scheduling noise but not thermal / + turbo drift; for publishable numbers, pin the CPU governor and run a warmup + pass first. +- `spawn-storm` batches joins (1024 at a time) to stay well under the slab + cap; if you raise `SMARM_BENCH_SPAWNS` massively, that batching is why it + still works. + +## Adding a bench + +1. `benches/.rs` with a plain `main()`; print the house table (see any + existing bench) and, if it belongs to a sweep, a greppable CSV line with a + distinctive prefix (`RQCSV,` for the shootout family). +2. Register it in `Cargo.toml`: + ```toml + [[bench]] + name = "" + harness = false + ``` +3. Take parameters from `SMARM_BENCH_*` env vars with modest defaults — the + defaults must finish in seconds on one core, the env scales them up on + real hardware. +4. Report **medians**, and keep one measurement = one fresh runtime + (`init(Config::exact(t))` inside the measured closure constructor, the + `run()` inside the timed region) so runs don't contaminate each other. diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..f742fd4 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,120 @@ +# Tests + +Integration tests for the runtime. Each file owns one feature area or one +class of bug. Everything here runs under plain `cargo test`; the loom model +tests are the exception — they live **in the library** (`src/slot_state.rs`, +`src/run_queue.rs`), not in this directory, because loom must compile the +production code with shimmed atomics (see "Loom" below). + +## Running + +``` +cargo test # debug build — RUN THIS ONE: all invariant asserts live +cargo test --release # what users actually execute (LTO, no debug_asserts) +``` + +Debug builds are not just "slower tests": the runtime self-checks its +invariants only there — every `StateWord` transition asserts its +precondition, `enqueue` asserts the exact `(gen, Queued)` word, `RawMutex` +enforces the never-two-cold-locks leaf rule with a per-thread held-count, +`live_actors` checks for double-finalize underflow. A green release run with +a red debug run means an invariant broke without (yet) corrupting behavior — +treat it as a real failure. + +### The queue-variant matrix + +The run queue is compile-time selected; the suite must pass under all three +(features are additive, so drop the default first): + +``` +cargo test # rq-mutex (default) +cargo test --no-default-features --features rq-mpmc +cargo test --no-default-features --features rq-striped +``` + +### Loom (model checking) + +``` +RUSTFLAGS="--cfg loom" cargo test --lib --release +``` + +Exhaustively explores interleavings of the slot state machine +(`src/slot_state.rs`: lost-wakeup, at-most-once-enqueue, the stale-pid ABA +theorem, unpark-vs-claim) and the ring queues (`src/run_queue.rs`: +exactly-once through lap wraparound, push/pop races). Models run the +production transitions through `src/sync_shim.rs` — std atomics normally, +`loom::sync` under `--cfg loom`. `RawMutex` is deliberately not modeled: +futexes can't be, and it's the textbook Drepper mutex3 with stress and +unwind-safety tests of its own. + +### Trace feature + +`cargo test --features smarm-trace` exists mainly to catch bit-rot in the +`te!()` call sites; run it after touching scheduler paths. + +### Before a runtime-core PR + +The full matrix, in rough order of bug-finding power per minute: + +1. `cargo test` (debug, default variant) +2. debug under `rq-mpmc` and `rq-striped` +3. `cargo test --release` +4. loom +5. `cargo build --features smarm-trace` + +## Catalog + +**Low-level units (no scheduler)** +| file | covers | +|---|---| +| `context.rs` | `init_actor_stack` + the naked-asm context-switch shims, poked directly | +| `stack.rs` | the mmap'd stack allocator | +| `pid.rs` | pid packing/equality | + +**Feature areas (run under a real runtime)** +| file | covers | +|---|---| +| `runtime.rs` | `Config`, `Runtime::run`, re-running a runtime, correctness under genuine parallelism | +| `scheduler.rs` | spawn / join / panic delivery / `yield_now` / `self_pid` | +| `channel.rs` | send/recv (recv parks, so these need the runtime) | +| `selective_recv.rs` | `recv_match` / `try_recv_match` | +| `mutex.rs` | the actor-blocking `Mutex` (lock parks) | +| `timer.rs` | `sleep` ordering — time-sensitive, generous tolerances by design | +| `io.rs` | `block_on_io`: blocking closures on the pool while the actor parks | +| `io_epoll.rs` | `wait_readable` / `wait_writable` + the `read`/`write` sugar | +| `preempt.rs` | explicit preemption via `smarm::check!()` | +| `cancel.rs` | cooperative cancellation (`request_stop`) — the keystone semantics | +| `monitor.rs` | `monitor` delivers exactly one `Down`; `demonitor` | +| `link.rs` | bidirectional links + `trap_exit` | +| `supervisor.rs` | one-for-one supervision | +| `gen_server.rs` | call/cast round-trips, lifecycle callbacks, server-down detection | + +**Regression & stress** +| file | covers | +|---|---| +| `stress.rs` | lost wakeups, pid-table pressure, thundering herds, panic isolation under concurrency. Where the phase-2 RefCell-migration bug was caught. | +| `poison_stop.rs` | `request_stop` racing an alloc-under-lock must not poison/abort. See its header for the full story. | +| `many_timers_multi_thread.rs` | multi-thread sleep-timer lost-wakeup regression | + +## Conventions + +- **Each test owns its runtime.** `init(Config::exact(N))` + `rt.run(...)`; + never share a `Runtime` between tests. Oversubscription (`exact(4)` on one + core) is deliberate — forced interleaving at yield points is how + single-core CI finds races at all. +- **Regression tests must be validated against the bug.** A regression test + that passes with the bug reintroduced is documentation, not a test. + Reintroduce the fix's inverse locally and watch it fail before trusting it + (`poison_stop.rs` went through exactly this: its first version never fired + the sentinel under a lock, and was rewritten until it SIGABRT'd pre-fix). +- **Stochastic tests get the odds stacked.** Use `Config::alloc_interval(1)` + to make every allocation an observation point, many actors, and both + phases of any every-other-allocation cadence (see + `poison_stop::self_stop_during_spawn...`). +- **Time-based assertions use ordering, not durations.** Assert + "didn't return instantly" / "A woke before B", with generous tolerances; + CI machines are slow and noisy. +- New invariants added to the runtime should come with the assert at the + point of reliance (debug_assert on hot paths) *and*, where the invariant is + a protocol, a loom model in the owning module — that combination is what + made phases 2–5 land without a single post-merge race so far. diff --git a/tests/mutex.rs b/tests/mutex.rs index 6647c22..ac24c1b 100644 --- a/tests/mutex.rs +++ b/tests/mutex.rs @@ -1,4 +1,5 @@ -//! `loom::Mutex` tests. All run under the scheduler because `lock()` +//! `smarm::Mutex` (the actor-blocking mutex) tests. All run under the +//! scheduler because `lock()` //! needs to be able to park. use smarm::{run, spawn, yield_now, LockTimeout, Mutex};