Files
smarm/tests
Claude 1282c3a08d feat(cluster): RFC 010 c7b — discovery Strategy, static seeds, connector dial loop
Phase 2 gate: 3-node mesh under the subprocess harness, repeatable (10/10).

Strategy (ratified): push-based, spawned as its own actor by the connector —
it emits Discovery events into a channel whenever it learns something and
may run forever; the connector owns all retry/backoff state. StaticSeeds
announces its list once and exits. Discovery is #[non_exhaustive] and
additive-only (candidates announced, never withdrawn) so expiry can land
later without breaking strategies.

One-viable correction to the ratified Discovery shape, flagged: a candidate
is a (name, addr) PAIR, not a bare address. The dial path and the D7
tie-break are keyed by peer name (the dial intent must be registered before
connecting so a crossing inbound Hello sees it), so an anonymous dial would
reintroduce exactly the simultaneous-connect flap D7 exists to prevent.
Discovery mechanisms know names — that is what they discover.

Connector: plain select-loop actor (the c6 shape) folding cmd inbox,
discovery stream, membership stream, and the earliest retry deadline into
one wait. It tracks who is up by SUBSCRIBING TO MEMBERSHIP like any
consumer — first consumer of c7a's snapshot-then-stream surface, no
privileged channel into the manager. Backoff: 250ms doubling to a 5s cap
(the c6c class of one-viable constants), reset on node_up; node_down
schedules a prompt redial with a fresh sequence. A candidate bearing the
local name is parked (that seed is us); every other failure retries — in
particular NameTaken can be our own ghost at the peer, not yet reaped by
its liveness timer, so it must not park. Dials run inline in the loop, the
acceptor's deliberate serialization (each attempt bounded by the connect +
handshake deadlines).

cluster::start(Config {node_name, meta, listen_addr, strategy}) is now the
integrated node start: supervised manager + acceptor + connector. It
completes the node identity: build_hash = cluster::BUILD_HASH (first
consumer, closing the c6d loose end) and incarnation = self_incarnation()
— unix-epoch MILLIS truncated to u32, not seconds: a supervised
crash-and-restart inside one second is routine, and seconds would collide
the ghost with its successor. Cluster handle: local_addr()/local()/
shutdown(); drop stops acceptor+connector loops, manager subtree detaches
(same split as AcceptorHandle alone).

Roadmap-binding, asserted in review: no consumer touches the connection
table — Manager.conns and ConnEntry stay private; the only exposures are
Call::Peers (sorted names, pre-existing) and the membership surface.

tests/cluster_mesh.rs 2/0, 10/10 flake runs: (1) 3-node mesh forms; kill
one (SIGKILL via Drop, per the retractable-state trap: roles park forever)
=> node_down at both survivors; restart same name => new incarnation at
every observer, distinguishable from the ghost; (2) seed unreachable at
start (pre-reserved closed port; accepted micro steal-window, documented)
then arriving later => edge forms via the retry path. All cluster suites
regression-clean (envelope 15, handshake 11, transport 11, lifecycle 1,
liveness 3, connect 9, two_node 3, membership 4); clippy --lib green both
configs; fmt clean; default build compiles.
2026-08-15 07:14:13 +00:00
..
2026-05-23 16:09:35 +00:00

Tests

Integration tests for the runtime. Each file owns one feature area or one class of bug. Everything here runs under plain cargo test; the loom model tests are the exception — they live in the library (src/slot_state.rs, src/run_queue.rs), not in this directory, because loom must compile the production code with shimmed atomics (see "Loom" below).

Running

cargo test                # debug build — RUN THIS ONE: all invariant asserts live
cargo test --release      # what users actually execute (LTO, no debug_asserts)

Debug builds are not just "slower tests": the runtime self-checks its invariants only there — every StateWord transition asserts its precondition, enqueue asserts the exact (gen, Queued) word, RawMutex enforces the never-two-cold-locks leaf rule with a per-thread held-count, live_actors checks for double-finalize underflow. A green release run with a red debug run means an invariant broke without (yet) corrupting behavior — treat it as a real failure.

The queue-variant matrix

The run queue is compile-time selected; the suite must pass under all three (features are additive, so drop the default first):

cargo test                                              # rq-mutex (default)
cargo test --no-default-features --features rq-mpmc
cargo test --no-default-features --features rq-striped

Loom (model checking)

RUSTFLAGS="--cfg loom" cargo test --lib --release

Exhaustively explores interleavings of the slot state machine (src/slot_state.rs: lost-wakeup, at-most-once-enqueue, the stale-pid ABA theorem, unpark-vs-claim) and the ring queues (src/run_queue.rs: exactly-once through lap wraparound, push/pop races). Models run the production transitions through src/sync_shim.rs — std atomics normally, loom::sync under --cfg loom. RawMutex is deliberately not modeled: futexes can't be, and it's the textbook Drepper mutex3 with stress and unwind-safety tests of its own.

Trace feature

cargo test --features smarm-trace exists mainly to catch bit-rot in the te!() call sites; run it after touching scheduler paths.

Before a runtime-core PR

The full matrix, in rough order of bug-finding power per minute:

  1. cargo test (debug, default variant)
  2. debug under rq-mpmc and rq-striped
  3. cargo test --release
  4. loom
  5. cargo build --features smarm-trace

Catalog

Low-level units (no scheduler)

file covers
context.rs init_actor_stack + the naked-asm context-switch shims, poked directly
stack.rs the mmap'd stack allocator
pid.rs pid packing/equality

Feature areas (run under a real runtime)

file covers
runtime.rs Config, Runtime::run, re-running a runtime, correctness under genuine parallelism
scheduler.rs spawn / join / panic delivery / yield_now / self_pid
channel.rs send/recv (recv parks, so these need the runtime)
selective_recv.rs recv_match / try_recv_match
mutex.rs the actor-blocking Mutex<T> (lock parks)
timer.rs sleep ordering — time-sensitive, generous tolerances by design
io.rs block_on_io: blocking closures on the pool while the actor parks
io_epoll.rs wait_readable / wait_writable + the read/write sugar
preempt.rs explicit preemption via smarm::check!()
cancel.rs cooperative cancellation (request_stop) — the keystone semantics
monitor.rs monitor delivers exactly one Down; demonitor
link.rs bidirectional links + trap_exit
supervisor.rs one-for-one supervision
gen_server.rs call/cast round-trips, lifecycle callbacks, server-down detection

Regression & stress

file covers
stress.rs lost wakeups, pid-table pressure, thundering herds, panic isolation under concurrency. Where the phase-2 RefCell-migration bug was caught.
poison_stop.rs request_stop racing an alloc-under-lock must not poison/abort. See its header for the full story.
many_timers_multi_thread.rs multi-thread sleep-timer lost-wakeup regression

Conventions

  • Each test owns its runtime. init(Config::exact(N)) + rt.run(...); never share a Runtime between tests. Oversubscription (exact(4) on one core) is deliberate — forced interleaving at yield points is how single-core CI finds races at all.
  • Regression tests must be validated against the bug. A regression test that passes with the bug reintroduced is documentation, not a test. Reintroduce the fix's inverse locally and watch it fail before trusting it (poison_stop.rs went through exactly this: its first version never fired the sentinel under a lock, and was rewritten until it SIGABRT'd pre-fix).
  • Stochastic tests get the odds stacked. Use Config::alloc_interval(1) to make every allocation an observation point, many actors, and both phases of any every-other-allocation cadence (see poison_stop::self_stop_during_spawn...).
  • Time-based assertions use ordering, not durations. Assert "didn't return instantly" / "A woke before B", with generous tolerances; CI machines are slow and noisy.
  • New invariants added to the runtime should come with the assert at the point of reliance (debug_assert on hot paths) and, where the invariant is a protocol, a loom model in the owning module — that combination is what made phases 2–5 land without a single post-merge race so far.