Root cause of the ~100% non-2xx benchmark: the load scripts never actually
ran. Two stacked failures, each of which makes wrk silently fall back to its
built-in default request (GET / with no auth -> 401 -> 100% non-2xx):
1. require('common') failed: wrk runs with cwd at the repo root, where
common.lua (in loadgen/) is not on package.path. Now each script prepends
its own directory (via debug.getinfo source) before requiring.
2. os.getpid() is not in luajit's os library (wrk doesn't add it), so the
randomseed line aborted the chunk once the require was fixed. Guarded with
(os.getpid and os.getpid() or 0). It was per-process anyway (same PID across
wrk threads), so it never varied the per-thread seed regardless.
Verified under luajit 2.1 (same interpreter wrk uses), loading each script as
wrk does from the repo-root cwd and via absolute path: all now load, define
request(), and set the Authorization header. The require always failed first,
so neither bug was ever caught -- the wrk path was never exercised end-to-end
(only manual curl, which worked). Not caused by the gen_server port or the
core/loadgen retune; those stand. Re-run _diag_status.lua to confirm real
status codes, then the A/B is finally measuring store work.
The first multi-core A/B was void: ~100% non-2xx on every row, so 'sustained
rps' was error-path throughput, not store work -- the spin patch's park/wake
path was never exercised. The harness reported it cheerfully anyway.
- spin_ab_urus.sh: per-run output now computes non-2xx vs approx total and
prints 'VOID: N% non-2xx' instead of a meaningless rps when >2% error.
- loadgen/_diag_status.lua: single-thread wrk script that tallies exact status
codes (response() + done()) and names the likely cause (401=auth not
reaching server / 404=store not pre-populated / 503=store actor down).
The aggregate 'Non-2xx' count can't distinguish these; this can.
Root cause still open: locally the store survives heavy keep-alive load (no
panic), prepop populates, auth+happy-path serve 200s -- so the storm isn't
reproducible in this 1-core sandbox. s3 at ~100% (not ~80%) rules out a plain
404 storm and points at auth or store-call erroring on every request; needs
the diag run on the real box to confirm.
The old 1/2/4/8 sweep with an 8-core loadgen plateaued ~102k rps at 4 and 8
server cores -- loadgen-bound, ~85% server CPU, so the 4-core (gate-active)
point was never actually saturated. Retune:
- CORES 1 2 4 8 -> 1 2 4 6. The gate is active at 2-4, inert at 1 and >=5, so
6 is now the saturating inert control and 4-vs-6 brackets the gate boundary
under real load (8 couldn't be saturated).
- LOADGEN_CPUS 12-19 (8c) -> 8-23 (16c); WRK_THREADS 8 -> 16; WRK_CONNS
256 -> 512. Gives >=2x loadgen-cores-per-server-core at every sweep point
(16:1/8:1/4:1/2.7:1) and leaves cores 6-7 for the OS.
- analyzer: gate label now follows the real 2..=4 rule (6 reads inert), and
the reading guide notes to confirm the 6-core control is CPU- not
loadgen-bound before trusting it.
Note: at 22/24 logical CPUs in use, loadgen will share SMT siblings with the
server; sibling-disjoint isn't achievable at 16 loadgen cores. Override
LOADGEN_CPUS after checking lscpu -e if that matters.
- store.rs: MemStore now implements GenServer (one inbox, one hop/request),
replacing the AnyReq + two-forwarder fan-in that cost two extra park/unpark
hops per request and contaminated the spin signal. StoreHandle is now an
enum {Memory(ServerRef<MemStore>) | Sqlite{reads,writes}} with one call().
Response status/body semantics unchanged (S3 stays comparable).
- SQLite store left on raw actors (single-writer + reader pool); not ported
(single-inbox would serialise S4 reads) and not part of the spin A/B. Kept
behind the handle so --store=sqlite still builds.
- handlers.rs: 6 handlers collapse to state.store().call(StoreReq::..); the
per-handler channel+send+recv boilerplate is gone. main.rs untouched.
- spin_ab_urus.sh + analyze_spin_ab_urus.py: A/B the spin policy on urus by
checking out smarm b0c9685 (pre-spin) vs HEAD, S2/S3 on urus-mem, server
cores 1/2/4/8 (2/4 gate-active, 1/8 inert), printing RPS speedup + latency
deltas. Verified: builds release; mem+sqlite smoke pass; scripts dry-run +
analyzer fixture-checked. Real sweep needs the multi-core box.
Markk116
sandbox-agent