Per-switch cost — N=1 local profile (spike findings)

Measured with benches/switch_cost.rs (local mode: one actor, one scheduler, tight yield_now() loop = one park/unpark round-trip with no IO/channel/timer and no cross-core traffic). Sandbox: 1 core, kernel 6.18, no PMU (hardware counters unavailable), so attribution is from perf record -e task-clock (software timer sampling) plus the bench's own rdtsc/wall brackets.

Numbers (stable across runs)

Round-trip p50 ≈ 303 ns ≈ 828 cyc (instrumentation floor subtracted).
Derived effective clock ≈ 2.73 GHz (rdtsc cyc / wall ns — the two lenses corroborate, so the cycle counts are trustworthy).
p90 316 ns, p99 472 ns; max is a multi-ms OS-deschedule outlier (1 shared core) — ignore the max, trust the percentiles (the harness pools them).

Attribution (perf task-clock, self-time, 28.6k samples / 12M round-trips)

share	symbol	bucket
24.8%	`runtime::schedule_loop`	scheduler logic (slot-word/epoch + dispatch)
8.6%	`MutexQueue::push`/`pop`/`len`	run-queue ops
~12%	`do_syscall_64`+`syscall`+`futex_*`	futex_wake on the hot path
3.5%	`IoThread::drain_completions`	the always-on IO thread (`run()` starts one)
~30%	`main` + `clock_gettime`/Timespec + `quicksort`	instrumentation (timing + percentile sort)
~1%	`switch_to_scheduler`+`switch_to_actor_asm`+ sp accessors	the context shims + TLS

Headline finding — revises the handoff hypothesis

The handoff named the context shims (context.rs: two calls into the TLS sp accessors per switch) as the prime suspect for the per-switch cost. At N=1 that is not where the time goes — the shims + TLS are ~1% of self-time. The N=1 cost is dominated by:

schedule_loop + run-queue ops (~33%) — the epoch/slot-word transition and the mutex run-queue push/pop on every re-queue.
A futex_wake syscall (~12%) firing on the hot path even though nothing is parked. This is the submit-rule wake the handoff itself flagged (RFC 004 finding #1: "a parallelism/latency optimisation, NOT a liveness guard"). In a single-scheduler always-runnable loop it is pure cost — no one is ever parked to wake. First cheap-win candidate: suppress the submit wake when there is no parked waiter (a parked-count / n_spinning check before the syscall). Needs care — finding #2's AllDone broadcast is the liveness-critical one and must NOT be touched.

What this does and does NOT show

The handoff's shim hypothesis was a many-core hypothesis: its evidence was the N=1→N=8 jump (0.18→1.2µs), attributed to TLS access mode (__tls_get_addr vs #[thread_local]) and cross-core coherency on the sp/epoch words. None of that is observable at N=1 on one core. This profile does NOT refute it; it establishes that the shim is cheap until cores contend.
So the spike question sharpens into two separable costs:
- N=1 floor: scheduler logic + a likely-removable futex_wake. Actionable now.
- N→8 slope: the shim/TLS/coherency cost. Needs the many-core box + a remote bench mode (wake straddling two schedulers) + hardware PMU counters (cache-misses, MEM_LOAD…HITM for coherency) — none available in this sandbox.

Reproduce

. "$HOME/.cargo/env"
cargo build --release --bench switch_cost
BIN=$(ls -t target/release/deps/switch_cost-* | grep -v '\.d$' | head -1)
PERF=/usr/lib/linux-tools-6.8.0-124/perf   # 6.8 perf on 6.18 kernel; sw events only here

# bench alone (numbers):
SMARM_SWITCH_ROUNDS=3000000 SMARM_SWITCH_WARMUP=50000 SMARM_SWITCH_RUNS=4 "$BIN"

# attribution (sw task-clock; HW counters need a real PMU / the 5900X):
SMARM_SWITCH_ROUNDS=3000000 "$PERF" record -F 4000 -g --call-graph fp -o /tmp/switch.data -- "$BIN"
"$PERF" report -i /tmp/switch.data --stdio --no-children

On the 5900X with a real PMU, drop -e task-clock for -e cycles,instructions, cache-misses,mem_load_retired.l3_miss to get the coherency picture the N=8 case needs.

4.0 KiB Raw Blame History