Root cause of the ~100% non-2xx benchmark: the load scripts never actually
ran. Two stacked failures, each of which makes wrk silently fall back to its
built-in default request (GET / with no auth -> 401 -> 100% non-2xx):
1. require('common') failed: wrk runs with cwd at the repo root, where
common.lua (in loadgen/) is not on package.path. Now each script prepends
its own directory (via debug.getinfo source) before requiring.
2. os.getpid() is not in luajit's os library (wrk doesn't add it), so the
randomseed line aborted the chunk once the require was fixed. Guarded with
(os.getpid and os.getpid() or 0). It was per-process anyway (same PID across
wrk threads), so it never varied the per-thread seed regardless.
Verified under luajit 2.1 (same interpreter wrk uses), loading each script as
wrk does from the repo-root cwd and via absolute path: all now load, define
request(), and set the Authorization header. The require always failed first,
so neither bug was ever caught -- the wrk path was never exercised end-to-end
(only manual curl, which worked). Not caused by the gen_server port or the
core/loadgen retune; those stand. Re-run _diag_status.lua to confirm real
status codes, then the A/B is finally measuring store work.
The first multi-core A/B was void: ~100% non-2xx on every row, so 'sustained
rps' was error-path throughput, not store work -- the spin patch's park/wake
path was never exercised. The harness reported it cheerfully anyway.
- spin_ab_urus.sh: per-run output now computes non-2xx vs approx total and
prints 'VOID: N% non-2xx' instead of a meaningless rps when >2% error.
- loadgen/_diag_status.lua: single-thread wrk script that tallies exact status
codes (response() + done()) and names the likely cause (401=auth not
reaching server / 404=store not pre-populated / 503=store actor down).
The aggregate 'Non-2xx' count can't distinguish these; this can.
Root cause still open: locally the store survives heavy keep-alive load (no
panic), prepop populates, auth+happy-path serve 200s -- so the storm isn't
reproducible in this 1-core sandbox. s3 at ~100% (not ~80%) rules out a plain
404 storm and points at auth or store-call erroring on every request; needs
the diag run on the real box to confirm.
sandbox-agent