From 371915ad07f5a3622129d13b3e42b2f2629d14c9 Mon Sep 17 00:00:00 2001
From: smarm-agent <agent@smarm.sandbox>
Date: Sun, 14 Jun 2026 18:58:26 +0000
Subject: [PATCH] =?UTF-8?q?docs:=20per-switch=20N=3D1=20local=20profile=20?=
 =?UTF-8?q?=E2=80=94=20shims=20are=20~1%=20at=20N=3D1;=20cost=20is=20sched?=
 =?UTF-8?q?ule=5Floop=20+=20run-queue=20+=20a=20hot-path=20futex=5Fwake=20?=
 =?UTF-8?q?(revises=20handoff=20shim=20hypothesis=20to=20a=20many-core=20o?=
 =?UTF-8?q?ne)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

---
 docs/perswitch-profile-n1.md | 77 ++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100644 docs/perswitch-profile-n1.md

diff --git a/docs/perswitch-profile-n1.md b/docs/perswitch-profile-n1.md
new file mode 100644
index 0000000..ab1527b
--- /dev/null
+++ b/docs/perswitch-profile-n1.md
@@ -0,0 +1,77 @@
+# Per-switch cost — N=1 local profile (spike findings)
+
+Measured with `benches/switch_cost.rs` (local mode: one actor, one scheduler,
+tight `yield_now()` loop = one park/unpark round-trip with no IO/channel/timer
+and no cross-core traffic). Sandbox: 1 core, kernel 6.18, **no PMU** (hardware
+counters unavailable), so attribution is from `perf record -e task-clock`
+(software timer sampling) plus the bench's own rdtsc/wall brackets.
+
+## Numbers (stable across runs)
+
+- Round-trip p50 ≈ **303 ns ≈ 828 cyc** (instrumentation floor subtracted).
+- Derived effective clock ≈ **2.73 GHz** (rdtsc cyc / wall ns — the two lenses
+  corroborate, so the cycle counts are trustworthy).
+- p90 316 ns, p99 472 ns; max is a multi-ms OS-deschedule outlier (1 shared
+  core) — ignore the max, trust the percentiles (the harness pools them).
+
+## Attribution (perf task-clock, self-time, 28.6k samples / 12M round-trips)
+
+| share | symbol | bucket |
+|------:|--------|--------|
+| 24.8% | `runtime::schedule_loop` | scheduler logic (slot-word/epoch + dispatch) |
+|  8.6% | `MutexQueue::push`/`pop`/`len` | run-queue ops |
+| ~12%  | `do_syscall_64`+`syscall`+`futex_*` | **futex_wake on the hot path** |
+|  3.5% | `IoThread::drain_completions` | the always-on IO thread (`run()` starts one) |
+| ~30%  | `main` + `clock_gettime`/Timespec + `quicksort` | **instrumentation** (timing + percentile sort) |
+|  ~1%  | `switch_to_scheduler`+`switch_to_actor_asm`+ sp accessors | **the context shims + TLS** |
+
+## Headline finding — revises the handoff hypothesis
+
+The handoff named the **context shims** (`context.rs`: two `call`s into the
+TLS sp accessors per switch) as the prime suspect for the per-switch cost.
+**At N=1 that is not where the time goes — the shims + TLS are ~1% of
+self-time.** The N=1 cost is dominated by:
+
+1. **`schedule_loop` + run-queue ops (~33%)** — the epoch/slot-word transition
+   and the mutex run-queue push/pop on every re-queue.
+2. **A `futex_wake` syscall (~12%)** firing on the hot path even though nothing
+   is parked. This is the **submit-rule wake** the handoff itself flagged (RFC
+   004 finding #1: "a parallelism/latency optimisation, NOT a liveness guard").
+   In a single-scheduler always-runnable loop it is pure cost — no one is ever
+   parked to wake. **First cheap-win candidate: suppress the submit wake when
+   there is no parked waiter** (a parked-count / n_spinning check before the
+   syscall). Needs care — finding #2's AllDone broadcast is the liveness-critical
+   one and must NOT be touched.
+
+## What this does and does NOT show
+
+- The handoff's shim hypothesis was a **many-core** hypothesis: its evidence was
+  the N=1→N=8 jump (0.18→1.2µs), attributed to TLS access mode (`__tls_get_addr`
+  vs `#[thread_local]`) and cross-core coherency on the sp/epoch words. **None of
+  that is observable at N=1 on one core.** This profile does NOT refute it; it
+  establishes that the shim is cheap *until cores contend*.
+- So the spike question sharpens into two separable costs:
+  - **N=1 floor:** scheduler logic + a likely-removable futex_wake. Actionable now.
+  - **N→8 slope:** the shim/TLS/coherency cost. Needs the many-core box + a
+    `remote` bench mode (wake straddling two schedulers) + hardware PMU counters
+    (cache-misses, `MEM_LOAD…HITM` for coherency) — none available in this sandbox.
+
+## Reproduce
+
+```sh
+. "$HOME/.cargo/env"
+cargo build --release --bench switch_cost
+BIN=$(ls -t target/release/deps/switch_cost-* | grep -v '\.d$' | head -1)
+PERF=/usr/lib/linux-tools-6.8.0-124/perf   # 6.8 perf on 6.18 kernel; sw events only here
+
+# bench alone (numbers):
+SMARM_SWITCH_ROUNDS=3000000 SMARM_SWITCH_WARMUP=50000 SMARM_SWITCH_RUNS=4 "$BIN"
+
+# attribution (sw task-clock; HW counters need a real PMU / the 5900X):
+SMARM_SWITCH_ROUNDS=3000000 "$PERF" record -F 4000 -g --call-graph fp -o /tmp/switch.data -- "$BIN"
+"$PERF" report -i /tmp/switch.data --stdio --no-children
+```
+
+On the 5900X with a real PMU, drop `-e task-clock` for `-e cycles,instructions,
+cache-misses,mem_load_retired.l3_miss` to get the coherency picture the N=8 case
+needs.