docs(bench,test): READMEs for benches/ and tests/

benches/README.md: catalog of the two bench families (cross-runtime
comparisons + the v0.5 run-queue shootout), how to run the shootout
(scripts/bench_rq.sh, SMARM_BENCH_* knobs, the --no-default-features dance,
summary.csv/RQCSV format), honest-numbers caveats (core count is the
experiment; striped losing at low thread counts is expected), and the
conventions for adding a bench.

tests/README.md: catalog grouped by layer (low-level units / feature areas /
regression+stress), the run matrix (debug-first — that's where the invariant
asserts live — three queue variants, release, loom, trace), why loom tests
live in src/ rather than tests/, and the house conventions: one runtime per
test, oversubscription on purpose, regression tests validated against the
reintroduced bug, odds-stacking for stochastic tests, ordering-not-duration
time assertions, and new-invariant = assert + loom model.

Also fixes a stale header in tests/mutex.rs that claimed 'loom::Mutex' —
it tests smarm::Mutex, and the old name is actively confusing now that loom
is real in this repo.
This commit is contained in:
Claude
2026-06-09 21:35:08 +00:00
parent 039703dbeb
commit a7ca6646d7
3 changed files with 211 additions and 1 deletions
+89
View File
@@ -0,0 +1,89 @@
# Benches
Two families live here: **comparison benches** (smarm vs tokio, predating
v0.5) and the **run-queue shootout** (v0.5 phase 4). All are plain binaries
(`harness = false` in `Cargo.toml`), so `cargo bench` just builds in release
and runs `main()` — no criterion, no magic.
```
cargo bench --bench <name> # one bench
cargo bench # all of them (slow; rarely what you want)
```
## Catalog
| file | what it measures |
|---|---|
| `primes.rs` | Compute fan-out/fan-in: counts primes across W workers. Pure compute throughput + spawn/join/channel cost. |
| `multi_scheduler.rs` | The original cross-runtime matrix: smarm (1 thread / N threads) vs tokio (current_thread / multi_thread) on compute, ping-pong, and spawn throughput. |
| `general.rs` | Workloads where neither runtime has a structural edge. Large gaps here mean real per-task/per-yield overhead differences — watch these for regressions. |
| `smarm_favored.rs` | Workloads the stackful green-thread model is built for. Single-thread numbers isolate per-switch cost from contention. |
| `tokio_favored.rs` | Workloads tokio's model is built for. Expect to lose; the value is knowing *by how much* and catching the gap widening. |
| `rq_micro.rs` | Run-queue **structures** in isolation (no runtime, no actors): push/pop throughput sweeping thread count × producer:consumer ratio. Covers all three queue types in one binary — the types compile in every build; only the runtime's alias is feature-selected. |
| `rq_runtime.rs` | The **whole scheduler** with the compile-time-selected queue: yield-storm (pure queue churn), ping-pong-pairs (park/unpark latency), spawn-storm (slab + free list + queue churn), sweeping scheduler count. Comparing variants requires rebuilding per `rq-*` feature. |
## The run-queue shootout
One command; it rebuilds `rq_runtime` once per queue variant, runs `rq_micro`
once, and aggregates:
```
./scripts/bench_rq.sh
# on a big box:
SMARM_BENCH_THREADS="1 2 4 8 16 20" ./scripts/bench_rq.sh
```
Outputs land in `bench_results/` (gitignored): one full log per run, plus
`summary.csv` assembled from the machine-readable `RQCSV,...` lines every
config prints alongside the human table.
Manual single-variant runs need the feature dance (features are additive, so
the default `rq-mutex` must be switched off):
```
cargo bench --bench rq_runtime --no-default-features --features rq-striped
```
### Knobs (env vars, all optional)
| var | default | used by |
|---|---|---|
| `SMARM_BENCH_THREADS` | `"1 2 4"` | both — space-separated sweep |
| `SMARM_BENCH_RUNS` | `5` | both — repetitions; the **median** is reported |
| `SMARM_BENCH_ITEMS` | `200000` | `rq_micro` — items per measurement |
| `SMARM_BENCH_YIELD_ACTORS` / `_YIELDS` | `200` / `500` | `rq_runtime` yield-storm |
| `SMARM_BENCH_PAIRS` / `_ROUNDTRIPS` | `32` / `1000` | `rq_runtime` ping-pong |
| `SMARM_BENCH_SPAWNS` | `5000` | `rq_runtime` spawn-storm |
## Reading the numbers honestly
- **Core count is the experiment.** On a 1-core machine (CI, sandboxes) the
sweep only validates the harness and catches gross pathologies —
oversubscribed schedulers measure context-switch noise, not contention.
Variant decisions come from a many-core box.
- The striped queue *should lose* at low thread counts (ticket overhead with
no contention to amortize) — that's expected, not a bug.
- Medians over `SMARM_BENCH_RUNS` absorb scheduling noise but not thermal /
turbo drift; for publishable numbers, pin the CPU governor and run a warmup
pass first.
- `spawn-storm` batches joins (1024 at a time) to stay well under the slab
cap; if you raise `SMARM_BENCH_SPAWNS` massively, that batching is why it
still works.
## Adding a bench
1. `benches/<name>.rs` with a plain `main()`; print the house table (see any
existing bench) and, if it belongs to a sweep, a greppable CSV line with a
distinctive prefix (`RQCSV,` for the shootout family).
2. Register it in `Cargo.toml`:
```toml
[[bench]]
name = "<name>"
harness = false
```
3. Take parameters from `SMARM_BENCH_*` env vars with modest defaults — the
defaults must finish in seconds on one core, the env scales them up on
real hardware.
4. Report **medians**, and keep one measurement = one fresh runtime
(`init(Config::exact(t))` inside the measured closure constructor, the
`run()` inside the timed region) so runs don't contaminate each other.