The first multi-core A/B was void: ~100% non-2xx on every row, so 'sustained
rps' was error-path throughput, not store work -- the spin patch's park/wake
path was never exercised. The harness reported it cheerfully anyway.
- spin_ab_urus.sh: per-run output now computes non-2xx vs approx total and
prints 'VOID: N% non-2xx' instead of a meaningless rps when >2% error.
- loadgen/_diag_status.lua: single-thread wrk script that tallies exact status
codes (response() + done()) and names the likely cause (401=auth not
reaching server / 404=store not pre-populated / 503=store actor down).
The aggregate 'Non-2xx' count can't distinguish these; this can.
Root cause still open: locally the store survives heavy keep-alive load (no
panic), prepop populates, auth+happy-path serve 200s -- so the storm isn't
reproducible in this 1-core sandbox. s3 at ~100% (not ~80%) rules out a plain
404 storm and points at auth or store-call erroring on every request; needs
the diag run on the real box to confirm.
The old 1/2/4/8 sweep with an 8-core loadgen plateaued ~102k rps at 4 and 8
server cores -- loadgen-bound, ~85% server CPU, so the 4-core (gate-active)
point was never actually saturated. Retune:
- CORES 1 2 4 8 -> 1 2 4 6. The gate is active at 2-4, inert at 1 and >=5, so
6 is now the saturating inert control and 4-vs-6 brackets the gate boundary
under real load (8 couldn't be saturated).
- LOADGEN_CPUS 12-19 (8c) -> 8-23 (16c); WRK_THREADS 8 -> 16; WRK_CONNS
256 -> 512. Gives >=2x loadgen-cores-per-server-core at every sweep point
(16:1/8:1/4:1/2.7:1) and leaves cores 6-7 for the OS.
- analyzer: gate label now follows the real 2..=4 rule (6 reads inert), and
the reading guide notes to confirm the 6-core control is CPU- not
loadgen-bound before trusting it.
Note: at 22/24 logical CPUs in use, loadgen will share SMT siblings with the
server; sibling-disjoint isn't achievable at 16 loadgen cores. Override
LOADGEN_CPUS after checking lscpu -e if that matters.
- store.rs: MemStore now implements GenServer (one inbox, one hop/request),
replacing the AnyReq + two-forwarder fan-in that cost two extra park/unpark
hops per request and contaminated the spin signal. StoreHandle is now an
enum {Memory(ServerRef<MemStore>) | Sqlite{reads,writes}} with one call().
Response status/body semantics unchanged (S3 stays comparable).
- SQLite store left on raw actors (single-writer + reader pool); not ported
(single-inbox would serialise S4 reads) and not part of the spin A/B. Kept
behind the handle so --store=sqlite still builds.
- handlers.rs: 6 handlers collapse to state.store().call(StoreReq::..); the
per-handler channel+send+recv boilerplate is gone. main.rs untouched.
- spin_ab_urus.sh + analyze_spin_ab_urus.py: A/B the spin policy on urus by
checking out smarm b0c9685 (pre-spin) vs HEAD, S2/S3 on urus-mem, server
cores 1/2/4/8 (2/4 gate-active, 1/8 inert), printing RPS speedup + latency
deltas. Verified: builds release; mem+sqlite smoke pass; scripts dry-run +
analyzer fixture-checked. Real sweep needs the multi-core box.
sandbox-agent