smarm-agent
64bd7e81a5
The profile was captured against the spin-enabled build. With the RFC 004 spinning experiment excised from master, the ~12% hot-path futex_wake it attributed to the spinning submit-rule no longer exists (idle wait is back to thread::sleep). Add a provenance note, mark the futex_wake row/finding as excised spin machinery, and drop the now-moot 'gate the submit wake' lead. Spin-independent findings (shims ~1%, schedule_loop + run-queue ~33%) unchanged.
88 lines
4.7 KiB
Markdown
88 lines
4.7 KiB
Markdown
# Per-switch cost — N=1 local profile (spike findings)
|
|
|
|
Measured with `benches/switch_cost.rs` (local mode: one actor, one scheduler,
|
|
tight `yield_now()` loop = one park/unpark round-trip with no IO/channel/timer
|
|
and no cross-core traffic). Sandbox: 1 core, kernel 6.18, **no PMU** (hardware
|
|
counters unavailable), so attribution is from `perf record -e task-clock`
|
|
(software timer sampling) plus the bench's own rdtsc/wall brackets.
|
|
|
|
> **Provenance (post-excision).** This profile was captured against the
|
|
> spin-enabled build (the pre-excision HEAD, with the RFC 004 spinning workers
|
|
> live in `src/runtime.rs`). The RFC 004 spinning experiment has since been
|
|
> excised from `master`; idle schedulers are back to the historical
|
|
> `thread::sleep` wait. The `futex_wake` attribution below therefore reflects
|
|
> spinning machinery that is **no longer present on current master** — see the
|
|
> per-row and per-finding notes. The shim, `schedule_loop`, and run-queue
|
|
> findings are spin-independent and remain valid.
|
|
|
|
## Numbers (stable across runs)
|
|
|
|
- Round-trip p50 ≈ **303 ns ≈ 828 cyc** (instrumentation floor subtracted).
|
|
- Derived effective clock ≈ **2.73 GHz** (rdtsc cyc / wall ns — the two lenses
|
|
corroborate, so the cycle counts are trustworthy).
|
|
- p90 316 ns, p99 472 ns; max is a multi-ms OS-deschedule outlier (1 shared
|
|
core) — ignore the max, trust the percentiles (the harness pools them).
|
|
|
|
## Attribution (perf task-clock, self-time, 28.6k samples / 12M round-trips)
|
|
|
|
| share | symbol | bucket |
|
|
|------:|--------|--------|
|
|
| 24.8% | `runtime::schedule_loop` | scheduler logic (slot-word/epoch + dispatch) |
|
|
| 8.6% | `MutexQueue::push`/`pop`/`len` | run-queue ops |
|
|
| ~12% | `do_syscall_64`+`syscall`+`futex_*` | **futex_wake on the hot path** — spinning submit-rule wake; removed by the RFC 004 excision (not on current master) |
|
|
| 3.5% | `IoThread::drain_completions` | the always-on IO thread (`run()` starts one) |
|
|
| ~30% | `main` + `clock_gettime`/Timespec + `quicksort` | **instrumentation** (timing + percentile sort) |
|
|
| ~1% | `switch_to_scheduler`+`switch_to_actor_asm`+ sp accessors | **the context shims + TLS** |
|
|
|
|
## Headline finding — revises the handoff hypothesis
|
|
|
|
The handoff named the **context shims** (`context.rs`: two `call`s into the
|
|
TLS sp accessors per switch) as the prime suspect for the per-switch cost.
|
|
**At N=1 that is not where the time goes — the shims + TLS are ~1% of
|
|
self-time.** The N=1 cost is dominated by:
|
|
|
|
1. **`schedule_loop` + run-queue ops (~33%)** — the epoch/slot-word transition
|
|
and the mutex run-queue push/pop on every re-queue.
|
|
2. **A `futex_wake` syscall (~12%)** fired on the hot path even though nothing
|
|
was parked. This was the spinning **submit-rule wake** introduced by the RFC
|
|
004 experiment — a parallelism/latency optimisation, not a liveness guard. In
|
|
a single-scheduler always-runnable loop it was pure cost (no one was ever
|
|
parked to wake). The RFC 004 excision removed this wake with the rest of the
|
|
spinning machinery: on current master idle schedulers use `thread::sleep`
|
|
again, so the N=1 hot path no longer makes this syscall.
|
|
|
|
## What this does and does NOT show
|
|
|
|
- The handoff's shim hypothesis was a **many-core** hypothesis: its evidence was
|
|
the N=1→N=8 jump (0.18→1.2µs), attributed to TLS access mode (`__tls_get_addr`
|
|
vs `#[thread_local]`) and cross-core coherency on the sp/epoch words. **None of
|
|
that is observable at N=1 on one core.** This profile does NOT refute it; it
|
|
establishes that the shim is cheap *until cores contend*.
|
|
- So the spike question sharpens into two separable costs:
|
|
- **N=1 floor:** scheduler logic (`schedule_loop` + run-queue ops). The
|
|
futex_wake component was spinning machinery and is gone post-excision, so the
|
|
remaining N=1 floor is the scheduler core itself.
|
|
- **N→8 slope:** the shim/TLS/coherency cost. Needs the many-core box + a
|
|
`remote` bench mode (wake straddling two schedulers) + hardware PMU counters
|
|
(cache-misses, `MEM_LOAD…HITM` for coherency) — none available in this sandbox.
|
|
|
|
## Reproduce
|
|
|
|
```sh
|
|
. "$HOME/.cargo/env"
|
|
cargo build --release --bench switch_cost
|
|
BIN=$(ls -t target/release/deps/switch_cost-* | grep -v '\.d$' | head -1)
|
|
PERF=/usr/lib/linux-tools-6.8.0-124/perf # 6.8 perf on 6.18 kernel; sw events only here
|
|
|
|
# bench alone (numbers):
|
|
SMARM_SWITCH_ROUNDS=3000000 SMARM_SWITCH_WARMUP=50000 SMARM_SWITCH_RUNS=4 "$BIN"
|
|
|
|
# attribution (sw task-clock; HW counters need a real PMU / the 5900X):
|
|
SMARM_SWITCH_ROUNDS=3000000 "$PERF" record -F 4000 -g --call-graph fp -o /tmp/switch.data -- "$BIN"
|
|
"$PERF" report -i /tmp/switch.data --stdio --no-children
|
|
```
|
|
|
|
On the 5900X with a real PMU, drop `-e task-clock` for `-e cycles,instructions,
|
|
cache-misses,mem_load_retired.l3_miss` to get the coherency picture the N=8 case
|
|
needs.
|