Compare commits

..
4 Commits
Author SHA1 Message Date
smarm-agent 2f011f6ce4 fix(rfc-004): gate spinning on max_spinners>0, killing wake-nobody futex_wake at N=1
`spinning` was `spin_budget_cycles > 0` alone. At N=1 the pool-size cap is 0, so
spinning was enabled while no scheduler could ever enlist: n_spinning pinned at 0,
so the enqueue submit-rule guard (n_spinning==0) fired a wake-nobody futex_wake on
every push. Now gated on `budget > 0 && max_spinners > 0`; a debug_assert guards
the invariant against regression. No behavioural change (a 0-cap runtime never
parks on the futex either way) — single_thread_spinning_is_inert and the AllDone
liveness test (shutdown_releases_pinned_parked_siblings) still pass.

switch_cost N=1 (1-core sandbox): p50 ~466ns -> ~186ns. The wake-nobody syscall
was ~60% of the round-trip in wall-clock terms; the handoff's "~12%" was perf
self-time, not latency share. ROADMAP per-switch section updated to fold in the
N=1 spike correction (shim hypothesis is many-core only). Dropped the throwaway
budget_probe bench (switch_cost reproduces every state via .spin_budget_cycles/
.max_spinners and with percentiles).
2026-06-14 20:01:02 +00:00
smarm-agent debcb4e33b docs: per-switch N=1 local profile — shims are ~1% at N=1; cost is schedule_loop + run-queue + a hot-path futex_wake (revises handoff shim hypothesis to a many-core one) 2026-06-14 18:58:26 +00:00
smarm-agent 683f2ed02d bench: add switch_cost local-mode per-switch microbench (rdtsc+wall, RFC per-switch spike) 2026-06-14 18:56:16 +00:00
smarm-agent 4c1e5f19f1 RFC 004: default spin_budget_cycles=2000 (DEFAULT_SPIN_BUDGET_CYCLES, ~0.5us @3.7GHz), gate spinner cap on pool size (n<=1:0, 2..=4:n, n>=5:0) replacing N/2. From spin_sweep fork-join knee: ~4x p50 win at n<=4, pure cost at n>=8. Budget inert on large pools via 0 cap; explicit max_spinners() still overrides. 2026-06-14 18:27:39 +00:00
5 changed files with 417 additions and 17 deletions
+4
View File
@@ -65,3 +65,7 @@ harness = false
[[bench]] [[bench]]
name = "spin_sweep" name = "spin_sweep"
harness = false harness = false
[[bench]]
name = "switch_cost"
harness = false
+28 -5
View File
@@ -120,11 +120,34 @@ Needs an RFC.
#### Per-switch cost (context shims, epoch protocol) #### Per-switch cost (context shims, epoch protocol)
The shootout's residual: per-wake latency is 0.160.18 µs at N=1 and The shootout's residual: per-wake latency is 0.160.18 µs at N=1 and
0.81.2 µs at N=8+, dominated by the context-switch shims and the epoch 0.81.2 µs at N=8+. The N=1 profiling spike is **done** (`docs/perswitch-profile-n1.md`):
protocol, not the queue. On current evidence this is the larger constant — it **revises the old shim-first framing**. At N=1 the context shims + TLS sp
"the whole game" alongside the v0.9 work — but there is no spec yet. Needs a accessors are ~1% of self-time, not the dominant cost — the shim hypothesis was
profiling spike (where do the cycles actually go per park/unpark round-trip) always a *many-core* one (its only evidence was the N=1→N=8 jump, blamed on TLS
and then an RFC before it can be scheduled. access mode and cross-core coherency on the sp/epoch words), and nothing at N=1
on one core can confirm or refute it. The N=1 round-trip is instead dominated by
the scheduler core (`schedule_loop` + run-queue, ~33%) plus a removable
wake-nobody `futex_wake`.
That `futex_wake` is now **fixed**: it was an interaction between the RFC 004
spin defaults and the N=1 pool-size cap of 0 — `spinning` was gated on
`budget > 0` alone, so at N=1 spinning was enabled while the cap forbade any
spinner from enlisting, pinning `n_spinning` at 0 and firing a wake-nobody
`futex_wake` on every enqueue. `spinning` is now gated on
`budget > 0 && max_spinners > 0` (a `debug_assert` guards the invariant). On the
1-core sandbox this cut N=1 p50 from ~466 ns to ~186 ns — the syscall was ~60%
of the round-trip in wall-clock terms (the handoff's "~12%" was perf *self-time*,
not latency share).
Two separable targets remain: **(a)** the N=1 floor — `schedule_loop` + run-queue,
the irreducible scheduler core, no obvious cheap win left; **(b)** the N→8 slope —
shim / TLS / coherency, which needs the 5900X (HW-counter profiling; this sandbox
has no PMU). Still "the whole game" alongside v0.9. Next code chunk is a `remote`
mode for `benches/switch_cost.rs` (wake straddling two schedulers) — writable in
the sandbox, only measurable on the box. Do **not** optimise the shim until the
N=8 HW-counter data confirms it is the cost. Then an RFC; the
`spinning`-gating fix above may also warrant a small RFC since it touches the
RFC-004-settled futex path.
#### send_after / cancel_timer #### send_after / cancel_timer
Message-delivery timer on the existing min-heap (`timer.rs`): deliver a value to a Message-delivery timer on the existing min-heap (`timer.rs`): deliver a value to a
+256
View File
@@ -0,0 +1,256 @@
//! Per-switch (context-switch) cost microbench — the profiling-spike harness
//! for the ROADMAP "Per-switch cost (context shims, epoch protocol)" item.
//!
//! The spin work (RFC 004) is closed; the next perf target is the per-switch
//! cost itself. Shootout evidence: per-wake latency is ~0.160.18µs at N=1 but
//! ~0.81.2µs at N=8+, and the residual is attributed to the context-switch
//! shims (`src/context.rs`) and the epoch protocol — NOT the queue. This binary
//! isolates that round-trip so the cycles can be attributed under `perf` and an
//! rdtsc bracket, feeding the RFC.
//!
//! WHAT THE ROUND-TRIP IS
//!
//! `yield_now()` from inside an actor does exactly one park/unpark round-trip
//! with nothing else attached:
//!
//! actor: switch_to_scheduler ──► scheduler re-queues the actor (slot-word
//! (context.rs shim) epoch/state transition, run_queue push),
//! pops it straight back, switch_to_actor
//! actor resumes ◄──────────────────────────────────────────────────────
//!
//! No IO thread traffic, no channel, no timer, no cross-thread wake. On a
//! single-scheduler runtime the re-queue+repop never leaves this core, so the
//! sample is the *pure* shim + epoch + queue-op cost with zero coherency
//! traffic. That is the `local` baseline; a `remote` mode (wake straddling two
//! schedulers, to expose the N=1→N=8 coherency/TLS-mode jump) is a deliberate
//! follow-up and is NOT in this file yet — local first, per the spike plan.
//!
//! TWO LENSES ON THE SAME LOOP
//!
//! wall — `Instant` bracket per round-trip. Source of truth for µs, directly
//! comparable to the shootout's per-wake latency numbers.
//! cycles — `rdtsc` bracket per round-trip. Source of truth for the cycle
//! budget the RFC will reason in (the spin budget is in cycles too).
//!
//! Reporting both lets us *derive* the effective TSC frequency (cycles/ns) from
//! the same samples instead of hardcoding a nominal 3.7GHz — the spin_sweep
//! lesson was that nominal-vs-actual TSC drift is exactly what produces
//! red-herring numbers. If the derived freq matches the box's known base clock,
//! the two lenses corroborate; if not, that mismatch is itself a finding.
//!
//! Knobs (env):
//! SMARM_SWITCH_ROUNDS round-trips timed per run default 200000
//! SMARM_SWITCH_WARMUP untimed warmup round-trips default 10000
//! SMARM_SWITCH_RUNS runs (pooled latency, median) default 5
//!
//! Output: house table + one greppable line per run-set:
//! SWITCHCSV,<variant>,<mode>,<rounds>,<runs>,<n>,<p50_ns>,<p90_ns>,<p99_ns>,
//! <min_ns>,<max_ns>,<mean_ns>,<mean_cyc>,<derived_ghz>
//!
//! NOTE: a single yielding actor is the cooperative-scheduling tightest loop —
//! it never parks on a futex (the work is always immediately re-queued), so
//! this measures the switch+epoch+queue path, NOT the futex park. That is
//! intentional: the futex park is the spin work's territory (RFC 004), already
//! characterised. The unattributed constant the shootout flagged lives in the
//! switch itself, which is what this loop hammers.
use smarm::runtime::{init, Config};
use smarm::{run, spawn, yield_now};
use std::sync::{Arc, Mutex};
// --------------------------------------------------------------------------
// env helpers (house style, matching spin_sweep.rs / rq_runtime.rs)
// --------------------------------------------------------------------------
fn variant() -> &'static str {
if cfg!(feature = "rq-mpmc") {
"rq-mpmc"
} else if cfg!(feature = "rq-striped") {
"rq-striped"
} else {
"rq-mutex"
}
}
fn env_usize(key: &str, default: usize) -> usize {
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}
// --------------------------------------------------------------------------
// rdtsc — serialised so the bracket actually fences the round-trip.
//
// Plain `rdtsc` can be reordered around the work by an out-of-order core, which
// would smear the bracket. `rdtscp` retires prior instructions before reading
// the counter, and the trailing `lfence` blocks later instructions from
// climbing above the second read. Pair = (rdtscp; lfence) … work … (rdtscp;
// lfence): a standard cycle-accurate bracket. We read TSC_AUX too but ignore
// it; the point is the ordering guarantee, not the core id.
// --------------------------------------------------------------------------
#[inline(always)]
fn rdtsc_serialised() -> u64 {
#[cfg(target_arch = "x86_64")]
unsafe {
let mut aux = 0u32;
let t = core::arch::x86_64::__rdtscp(&mut aux);
core::arch::x86_64::_mm_lfence();
t
}
#[cfg(not(target_arch = "x86_64"))]
{
// Non-x86 fallback: nanosecond clock standing in for cycles. The derived
// "GHz" column then reads ~1.0 and is meaningless, but the wall lens and
// the harness still work. The spike target box is x86-64.
use std::time::Instant;
thread_local! { static T0: Instant = Instant::now(); }
T0.with(|t0| t0.elapsed().as_nanos() as u64)
}
}
// --------------------------------------------------------------------------
// percentile / median helpers (verbatim house idiom from spin_sweep.rs)
// --------------------------------------------------------------------------
/// Nearest-rank percentile over an already-sorted slice. `p` in [0, 100].
fn pct(sorted: &[u64], p: f64) -> u64 {
if sorted.is_empty() {
return 0;
}
let idx = ((p / 100.0) * (sorted.len() - 1) as f64).round() as usize;
sorted[idx.min(sorted.len() - 1)]
}
// --------------------------------------------------------------------------
// one run: a single actor yields ROUNDS times; we bracket each yield from
// inside the actor (the only vantage point — the actor is suspended during the
// scheduler half, so an external timer can't see a single round-trip).
//
// Per iteration we capture BOTH a wall-ns delta and a TSC-cycle delta around
// the same `yield_now()`. The loop overhead (two clock reads + a Vec push +
// the branch) rides along in every sample equally; we subtract an empty-loop
// self-calibration below so the reported number is the round-trip, not the
// instrumentation.
// --------------------------------------------------------------------------
struct RunSample {
lat_ns: Vec<u64>,
cyc: Vec<u64>,
}
fn one_run(threads: usize, rounds: usize, warmup: usize) -> RunSample {
let out: Arc<Mutex<Option<RunSample>>> = Arc::new(Mutex::new(None));
let out2 = out.clone();
let cfg = Config::exact(threads);
init(cfg);
run(move || {
let h = spawn(move || {
// Warmup: let the actor's stack/queue slot go hot, JIT-free but
// cache-warm, before any sample is kept.
for _ in 0..warmup {
yield_now();
}
let mut lat_ns = Vec::with_capacity(rounds);
let mut cyc = Vec::with_capacity(rounds);
for _ in 0..rounds {
let w0 = std::time::Instant::now();
let c0 = rdtsc_serialised();
yield_now();
let c1 = rdtsc_serialised();
let w1 = w0.elapsed();
cyc.push(c1.saturating_sub(c0));
lat_ns.push(w1.as_nanos() as u64);
}
*out2.lock().unwrap() = Some(RunSample { lat_ns, cyc });
});
let _ = h.join();
});
let sample = out.lock().unwrap().take().expect("actor stored a sample");
sample
}
/// Empty-loop self-calibration: the same bracket with the `yield_now()` removed,
/// run inline (no runtime). Gives the floor cost of two serialised clock reads +
/// the push, in both lenses, to subtract from the round-trip samples.
fn calibrate(rounds: usize) -> (u64, u64) {
let mut lat_ns = Vec::with_capacity(rounds);
let mut cyc = Vec::with_capacity(rounds);
let mut sink = 0u64;
for _ in 0..rounds {
let w0 = std::time::Instant::now();
let c0 = rdtsc_serialised();
// no yield — measure the bracket itself
let c1 = rdtsc_serialised();
let w1 = w0.elapsed();
sink ^= c1;
cyc.push(c1.saturating_sub(c0));
lat_ns.push(w1.as_nanos() as u64);
}
std::hint::black_box(sink);
cyc.sort_unstable();
lat_ns.sort_unstable();
// Use the medians as the floor — robust to the occasional interrupt.
(pct(&lat_ns, 50.0), pct(&cyc, 50.0))
}
fn main() {
let rounds = env_usize("SMARM_SWITCH_ROUNDS", 200_000);
let warmup = env_usize("SMARM_SWITCH_WARMUP", 10_000);
let runs = env_usize("SMARM_SWITCH_RUNS", 5);
let mode = "local";
// Calibrate the instrumentation floor once, with a healthy sample.
let (floor_ns, floor_cyc) = calibrate(rounds.min(50_000).max(10_000));
let mut pooled_ns: Vec<u64> = Vec::new();
let mut pooled_cyc: Vec<u64> = Vec::new();
for _ in 0..runs {
let s = one_run(1, rounds, warmup);
// Subtract the instrumentation floor; saturating so a sub-floor outlier
// (clock granularity) clamps to 0 rather than wrapping.
pooled_ns.extend(s.lat_ns.iter().map(|&v| v.saturating_sub(floor_ns)));
pooled_cyc.extend(s.cyc.iter().map(|&v| v.saturating_sub(floor_cyc)));
}
pooled_ns.sort_unstable();
pooled_cyc.sort_unstable();
let n = pooled_ns.len();
let mean_ns = pooled_ns.iter().map(|&v| v as f64).sum::<f64>() / n.max(1) as f64;
let mean_cyc = pooled_cyc.iter().map(|&v| v as f64).sum::<f64>() / n.max(1) as f64;
// Derived effective frequency: cycles per ns = GHz. Cross-checks the two
// lenses against the box's known base clock.
let derived_ghz = if mean_ns > 0.0 { mean_cyc / mean_ns } else { 0.0 };
let p50 = pct(&pooled_ns, 50.0);
let p90 = pct(&pooled_ns, 90.0);
let p99 = pct(&pooled_ns, 99.0);
let lo = *pooled_ns.first().unwrap_or(&0);
let hi = *pooled_ns.last().unwrap_or(&0);
// House table.
println!();
println!("per-switch cost — {} mode, variant={}", mode, variant());
println!(
" rounds={} warmup={} runs={} (instrumentation floor: {} ns / {} cyc, subtracted)",
rounds, warmup, runs, floor_ns, floor_cyc
);
println!(" {:<10} {:<10} {:<10} {:<10} {:<10}", "p50 ns", "p90 ns", "p99 ns", "min ns", "max ns");
println!(" {:<10} {:<10} {:<10} {:<10} {:<10}", p50, p90, p99, lo, hi);
println!(
" mean {:.1} ns | mean {:.0} cyc | derived {:.3} GHz",
mean_ns, mean_cyc, derived_ghz
);
// Greppable line — same spirit as SPINCSV.
println!(
"SWITCHCSV,{},{},{},{},{},{},{},{},{},{},{:.1},{:.0},{:.3}",
variant(), mode, rounds, runs, n, p50, p90, p99, lo, hi, mean_ns, mean_cyc, derived_ghz
);
}
+77
View File
@@ -0,0 +1,77 @@
# Per-switch cost — N=1 local profile (spike findings)
Measured with `benches/switch_cost.rs` (local mode: one actor, one scheduler,
tight `yield_now()` loop = one park/unpark round-trip with no IO/channel/timer
and no cross-core traffic). Sandbox: 1 core, kernel 6.18, **no PMU** (hardware
counters unavailable), so attribution is from `perf record -e task-clock`
(software timer sampling) plus the bench's own rdtsc/wall brackets.
## Numbers (stable across runs)
- Round-trip p50 ≈ **303 ns ≈ 828 cyc** (instrumentation floor subtracted).
- Derived effective clock ≈ **2.73 GHz** (rdtsc cyc / wall ns — the two lenses
corroborate, so the cycle counts are trustworthy).
- p90 316 ns, p99 472 ns; max is a multi-ms OS-deschedule outlier (1 shared
core) — ignore the max, trust the percentiles (the harness pools them).
## Attribution (perf task-clock, self-time, 28.6k samples / 12M round-trips)
| share | symbol | bucket |
|------:|--------|--------|
| 24.8% | `runtime::schedule_loop` | scheduler logic (slot-word/epoch + dispatch) |
| 8.6% | `MutexQueue::push`/`pop`/`len` | run-queue ops |
| ~12% | `do_syscall_64`+`syscall`+`futex_*` | **futex_wake on the hot path** |
| 3.5% | `IoThread::drain_completions` | the always-on IO thread (`run()` starts one) |
| ~30% | `main` + `clock_gettime`/Timespec + `quicksort` | **instrumentation** (timing + percentile sort) |
| ~1% | `switch_to_scheduler`+`switch_to_actor_asm`+ sp accessors | **the context shims + TLS** |
## Headline finding — revises the handoff hypothesis
The handoff named the **context shims** (`context.rs`: two `call`s into the
TLS sp accessors per switch) as the prime suspect for the per-switch cost.
**At N=1 that is not where the time goes — the shims + TLS are ~1% of
self-time.** The N=1 cost is dominated by:
1. **`schedule_loop` + run-queue ops (~33%)** — the epoch/slot-word transition
and the mutex run-queue push/pop on every re-queue.
2. **A `futex_wake` syscall (~12%)** firing on the hot path even though nothing
is parked. This is the **submit-rule wake** the handoff itself flagged (RFC
004 finding #1: "a parallelism/latency optimisation, NOT a liveness guard").
In a single-scheduler always-runnable loop it is pure cost — no one is ever
parked to wake. **First cheap-win candidate: suppress the submit wake when
there is no parked waiter** (a parked-count / n_spinning check before the
syscall). Needs care — finding #2's AllDone broadcast is the liveness-critical
one and must NOT be touched.
## What this does and does NOT show
- The handoff's shim hypothesis was a **many-core** hypothesis: its evidence was
the N=1→N=8 jump (0.18→1.2µs), attributed to TLS access mode (`__tls_get_addr`
vs `#[thread_local]`) and cross-core coherency on the sp/epoch words. **None of
that is observable at N=1 on one core.** This profile does NOT refute it; it
establishes that the shim is cheap *until cores contend*.
- So the spike question sharpens into two separable costs:
- **N=1 floor:** scheduler logic + a likely-removable futex_wake. Actionable now.
- **N→8 slope:** the shim/TLS/coherency cost. Needs the many-core box + a
`remote` bench mode (wake straddling two schedulers) + hardware PMU counters
(cache-misses, `MEM_LOAD…HITM` for coherency) — none available in this sandbox.
## Reproduce
```sh
. "$HOME/.cargo/env"
cargo build --release --bench switch_cost
BIN=$(ls -t target/release/deps/switch_cost-* | grep -v '\.d$' | head -1)
PERF=/usr/lib/linux-tools-6.8.0-124/perf # 6.8 perf on 6.18 kernel; sw events only here
# bench alone (numbers):
SMARM_SWITCH_ROUNDS=3000000 SMARM_SWITCH_WARMUP=50000 SMARM_SWITCH_RUNS=4 "$BIN"
# attribution (sw task-clock; HW counters need a real PMU / the 5900X):
SMARM_SWITCH_ROUNDS=3000000 "$PERF" record -F 4000 -g --call-graph fp -o /tmp/switch.data -- "$BIN"
"$PERF" report -i /tmp/switch.data --stdio --no-children
```
On the 5900X with a real PMU, drop `-e task-clock` for `-e cycles,instructions,
cache-misses,mem_load_retired.l3_miss` to get the coherency picture the N=8 case
needs.
+52 -12
View File
@@ -131,6 +131,13 @@ use std::thread;
/// Default capacity of the actor slot table. Slots are ~256 bytes, so the /// Default capacity of the actor slot table. Slots are ~256 bytes, so the
/// default costs ~4 MiB, allocated once at `init`. See [`Config::max_actors`]. /// default costs ~4 MiB, allocated once at `init`. See [`Config::max_actors`].
pub const DEFAULT_MAX_ACTORS: usize = 16_384; pub const DEFAULT_MAX_ACTORS: usize = 16_384;
/// RFC 004: default scheduler spin budget before parking, in TSC cycles.
/// ≈0.5µs at a 3.7 GHz nominal TSC. Sized from the spin_sweep fork-join knee:
/// the fork→join handoff is sub-µs, so ~0.5µs of spin keeps a worker hot across
/// it for a ~4× p50 latency win, and budgets past ~5k cycles buy nothing but
/// idle CPU. Only effective where the default spinner cap permits spinning
/// (small pools, n≤4); see the cap resolution in `init`.
pub const DEFAULT_SPIN_BUDGET_CYCLES: u64 = 2_000;
/// Runtime configuration. /// Runtime configuration.
/// ///
@@ -174,7 +181,7 @@ impl Config {
stack_pool_cap: n * 4, stack_pool_cap: n * 4,
max_actors: DEFAULT_MAX_ACTORS, max_actors: DEFAULT_MAX_ACTORS,
wake_slot: false, wake_slot: false,
spin_budget_cycles: 0, spin_budget_cycles: DEFAULT_SPIN_BUDGET_CYCLES,
max_spinners: None, max_spinners: None,
} }
} }
@@ -193,7 +200,7 @@ impl Config {
stack_pool_cap: max * 4, stack_pool_cap: max * 4,
max_actors: DEFAULT_MAX_ACTORS, max_actors: DEFAULT_MAX_ACTORS,
wake_slot: false, wake_slot: false,
spin_budget_cycles: 0, spin_budget_cycles: DEFAULT_SPIN_BUDGET_CYCLES,
max_spinners: None, max_spinners: None,
} }
} }
@@ -294,7 +301,7 @@ impl Default for Config {
stack_pool_cap: avail * 4, stack_pool_cap: avail * 4,
max_actors: DEFAULT_MAX_ACTORS, max_actors: DEFAULT_MAX_ACTORS,
wake_slot: false, wake_slot: false,
spin_budget_cycles: 0, spin_budget_cycles: DEFAULT_SPIN_BUDGET_CYCLES,
max_spinners: None, max_spinners: None,
} }
} }
@@ -533,10 +540,13 @@ pub(crate) struct RuntimeInner {
/// wait immediately (EAGAIN) instead of sleeping — the lost-wakeup guard. /// wait immediately (EAGAIN) instead of sleeping — the lost-wakeup guard.
/// One shared word; `futex_wake(1)` wakes exactly one parked scheduler. /// One shared word; `futex_wake(1)` wakes exactly one parked scheduler.
pub(crate) park_seq: AtomicU32, pub(crate) park_seq: AtomicU32,
/// RFC 004: `spin_budget_cycles > 0`. When false the feature is fully off — /// RFC 004: `spin_budget_cycles > 0 && max_spinners > 0`. When false the
/// the idle path is the historical `thread::sleep` park, `n_spinning` and /// feature is fully off — the idle path is the historical `thread::sleep`
/// `park_seq` are never touched, and the submit rule is a no-op. This is /// park, `n_spinning` and `park_seq` are never touched, and the submit rule
/// what makes `spin_budget_cycles = 0` recover today's behaviour exactly. /// is a no-op. Note the cap term: a budget with a 0 cap (e.g. the N=1
/// pool-size gate) can enlist no spinner, so it is treated as off — this is
/// what makes `spin_budget_cycles = 0` AND the 0-cap case recover today's
/// behaviour exactly.
pub(crate) spinning: bool, pub(crate) spinning: bool,
/// RFC 004: TSC cycles a scheduler searches before parking (`Config`). /// RFC 004: TSC cycles a scheduler searches before parking (`Config`).
pub(crate) spin_budget_cycles: u64, pub(crate) spin_budget_cycles: u64,
@@ -586,6 +596,24 @@ impl RuntimeInner {
let slots: Box<[Slot]> = (0..max_actors).map(|_| Slot::vacant()).collect(); let slots: Box<[Slot]> = (0..max_actors).map(|_| Slot::vacant()).collect();
// Low indices on top of the stack so early spawns get low pids. // Low indices on top of the stack so early spawns get low pids.
let free: Vec<u32> = (0..max_actors as u32).rev().collect(); let free: Vec<u32> = (0..max_actors as u32).rev().collect();
// RFC 004: the spin/futex machinery is only live when BOTH a budget is
// set AND the cap permits at least one spinner. A budget with a 0 cap
// (e.g. the N=1 pool-size gate) can never enlist a spinner, so
// `n_spinning` is pinned at 0 forever — which would make the enqueue
// submit-rule guard (`n_spinning == 0`) fire a wake-nobody futex_wake on
// every push. Gating on the cap too keeps that path inert exactly when
// it can do no work. Equivalent to budget=0 here, with no behavioural
// change (a 0-cap runtime never parks on the futex either way).
let spinning = spin_budget_cycles > 0 && max_spinners > 0;
// Guardrail: catch any future config path that re-enables spinning while
// leaving the cap at 0 — the pathology this gate exists to prevent. If
// we are spinning, at least one spinner must be enlistable.
debug_assert!(
!spinning || max_spinners > 0,
"spinning enabled (budget={spin_budget_cycles}) with max_spinners=0: \
no scheduler can ever enlist, so every enqueue would fire a \
wake-nobody futex_wake"
);
Arc::new(Self { Arc::new(Self {
run_queue: crate::run_queue::RunQueue::new(thread_count, max_actors), run_queue: crate::run_queue::RunQueue::new(thread_count, max_actors),
slots, slots,
@@ -594,7 +622,7 @@ impl RuntimeInner {
queue_len: AtomicU64::new(0), queue_len: AtomicU64::new(0),
n_spinning: AtomicU32::new(0), n_spinning: AtomicU32::new(0),
park_seq: AtomicU32::new(0), park_seq: AtomicU32::new(0),
spinning: spin_budget_cycles > 0, spinning,
spin_budget_cycles, spin_budget_cycles,
max_spinners, max_spinners,
timers: Mutex::new(Timers::new()), timers: Mutex::new(Timers::new()),
@@ -888,10 +916,22 @@ pub struct Runtime {
/// Initialise the runtime with the given config. Returns a reusable handle. /// Initialise the runtime with the given config. Returns a reusable handle.
pub fn init(config: Config) -> Runtime { pub fn init(config: Config) -> Runtime {
let n = config.resolved_thread_count(); let n = config.resolved_thread_count();
// RFC 004: default spinner cap is N/2 (capping at half keeps half the pool // RFC 004: default spinner cap is gated on pool size, from the spin_sweep
// hot, parks the rest). With N=1 this is 0 — a lone thread has no // fork-join data. Spinning is a ~4× p50 latency win at n≤4 workers but pure
// cross-thread handoff to hide, so it never spins. // cost at n≥8 (large pools stay hot on their own, so a spin budget only
let max_spinners = config.max_spinners.unwrap_or((n / 2) as u32); // steals CPU from workers doing real drain). So:
// n == 1 → 0 : a lone thread has no cross-thread handoff to hide.
// 2..=4 → n : the win regime; let every worker spin its budget.
// n >= 5 → 0 : park immediately, exactly the historical behaviour —
// the non-zero DEFAULT_SPIN_BUDGET_CYCLES is inert here
// because no worker is permitted to enter the spin path.
// An explicit Config::max_spinners(_) overrides this (e.g. a future autotune
// utility that picks budget+cap from measured load).
let max_spinners = config.max_spinners.unwrap_or(match n {
0 | 1 => 0,
2..=4 => n as u32,
_ => 0,
});
Runtime { Runtime {
inner: RuntimeInner::new( inner: RuntimeInner::new(
n, n,