smarm/docs/perswitch-profile-n1.md

Per-switch cost — N=1 local profile (spike findings)

Measured with benches/switch_cost.rs (local mode: one actor, one scheduler, tight yield_now() loop = one park/unpark round-trip with no IO/channel/timer and no cross-core traffic). Sandbox: 1 core, kernel 6.18, no PMU (hardware counters unavailable), so attribution is from perf record -e task-clock (software timer sampling) plus the bench's own rdtsc/wall brackets.

Provenance (post-excision). This profile was captured against the spin-enabled build (the pre-excision HEAD, with the RFC 004 spinning workers live in src/runtime.rs). The RFC 004 spinning experiment has since been excised from master; idle schedulers are back to the historical thread::sleep wait. The futex_wake attribution below therefore reflects spinning machinery that is no longer present on current master — see the per-row and per-finding notes. The shim, schedule_loop, and run-queue findings are spin-independent and remain valid.

Numbers (stable across runs)

• Round-trip p50 ≈ 303 ns ≈ 828 cyc (instrumentation floor subtracted).
• Derived effective clock ≈ 2.73 GHz (rdtsc cyc / wall ns — the two lenses corroborate, so the cycle counts are trustworthy).
• p90 316 ns, p99 472 ns; max is a multi-ms OS-deschedule outlier (1 shared core) — ignore the max, trust the percentiles (the harness pools them).

Attribution (perf task-clock, self-time, 28.6k samples / 12M round-trips)

share	symbol	bucket
24.8%	runtime::schedule_loop	scheduler logic (slot-word/epoch + dispatch)
8.6%	MutexQueue::push/pop/len	run-queue ops
~12%	do_syscall_64+syscall+futex_*	futex_wake on the hot path — spinning submit-rule wake; removed by the RFC 004 excision (not on current master)
3.5%	IoThread::drain_completions	the always-on IO thread (run() starts one)
~30%	main + clock_gettime/Timespec + quicksort	instrumentation (timing + percentile sort)
~1%	switch_to_scheduler+switch_to_actor_asm+ sp accessors	the context shims + TLS

Headline finding — revises the handoff hypothesis

The handoff named the context shims (context.rs: two calls into the TLS sp accessors per switch) as the prime suspect for the per-switch cost. At N=1 that is not where the time goes — the shims + TLS are ~1% of self-time. The N=1 cost is dominated by:

1. schedule_loop + run-queue ops (~33%) — the epoch/slot-word transition and the mutex run-queue push/pop on every re-queue.
2. A futex_wake syscall (~12%) fired on the hot path even though nothing was parked. This was the spinning submit-rule wake introduced by the RFC 004 experiment — a parallelism/latency optimisation, not a liveness guard. In a single-scheduler always-runnable loop it was pure cost (no one was ever parked to wake). The RFC 004 excision removed this wake with the rest of the spinning machinery: on current master idle schedulers use thread::sleep again, so the N=1 hot path no longer makes this syscall.

What this does and does NOT show

• The handoff's shim hypothesis was a many-core hypothesis: its evidence was the N=1→N=8 jump (0.18→1.2µs), attributed to TLS access mode (__tls_get_addr vs #[thread_local]) and cross-core coherency on the sp/epoch words. None of that is observable at N=1 on one core. This profile does NOT refute it; it establishes that the shim is cheap until cores contend.
• So the spike question sharpens into two separable costs:
  • N=1 floor: scheduler logic (schedule_loop + run-queue ops). The futex_wake component was spinning machinery and is gone post-excision, so the remaining N=1 floor is the scheduler core itself.
  • N→8 slope: the shim/TLS/coherency cost. Needs the many-core box + a remote bench mode (wake straddling two schedulers) + hardware PMU counters (cache-misses, MEM_LOAD…HITM for coherency) — none available in this sandbox.

Reproduce

. "$HOME/.cargo/env"
cargo build --release --bench switch_cost
BIN=$(ls -t target/release/deps/switch_cost-* | grep -v '\.d$' | head -1)
PERF=/usr/lib/linux-tools-6.8.0-124/perf   # 6.8 perf on 6.18 kernel; sw events only here

# bench alone (numbers):
SMARM_SWITCH_ROUNDS=3000000 SMARM_SWITCH_WARMUP=50000 SMARM_SWITCH_RUNS=4 "$BIN"

# attribution (sw task-clock; HW counters need a real PMU / the 5900X):
SMARM_SWITCH_ROUNDS=3000000 "$PERF" record -F 4000 -g --call-graph fp -o /tmp/switch.data -- "$BIN"
"$PERF" report -i /tmp/switch.data --stdio --no-children

On the 5900X with a real PMU, drop -e task-clock for -e cycles,instructions, cache-misses,mem_load_retired.l3_miss to get the coherency picture the N=8 case needs.