One unconditional footer line whenever there are results:
"note: impacts are lower bounds — undershoot grows with speedup pct;
rankings unaffected" — surfacing the RFC 007 Validation fidelity statement
where users actually look, instead of only in the RFC. Wording pinned by
the summary test.
send_after_wall / send_after_named_wall (+ Timers::insert_send_wall) arm a
message-delivery timer that opts out of the RFC 007 virtual-time shift and
fires at its raw deadline regardless of injected delay — the Send-reason
sibling of sleep_wall, closing the jar item whose substrate efbc254 landed.
For deadlines that reflect the outside world (protocol timeouts, wall-clock
schedules) rather than workload pacing. cancel_timer is anchor-agnostic and
unchanged; without the feature the API exists and is identical to send_after.
The gen_server timer layer (send_after_to, RFC 015 §5) deliberately stays
virtual-only — an opt-out there means new options on the gen_server/statem
timeout API, out of scope for now.
Tests: wall send fires at raw deadline while a virtual sibling shifts;
cancel on a wall send with debt outstanding; featureless delivery/cancel
smokes through the public named API. 15/15 causal, 34/34 binaries both
feature configs, lib clippy clean both.
New timer anchor: insert_sleep_wall / scheduler::sleep_wall (exported) opt a
Sleep entry out of the RFC 007 virtual-time shift, so it fires at its raw
deadline regardless of injected delay. Featureless config is unchanged (the
API exists but is identical to sleep).
The causal controller's window/cooldown sleeps and the tsc_hz calibration
sleep use it on the actor path (the OS-thread path was already wall). This
fixes the controller's own sleeps dilating under its own injection —
experiment windows stretched ~2x at 50% speedup (337ms -> 646ms injected/
window). Deltas were rate-normalized so results were unbiased; this fixes
sweep cost, not bias. The general wall-anchored-timer-semantics jar item
(user-facing opt-out) remains open; this lands the substrate.
Test: wall_timer_ignores_injected_delay — wall entry fires at raw deadline
while a virtual sibling in the same heap shifts. 13/13 causal, 34/34
binaries both feature configs.
Injected delays dilate virtual time for the workload, but timer deadlines
stayed wall-anchored: a sleep or receive-timeout fired early in virtual
terms, so timeout/retry behaviour sped up relative to the dilated world
(v1 known gap #1).
Every heap Entry now carries delay_stamp — the global delay ledger at
(re-)queue time, cfg-gated on smarm-causal. pop_due converts any debt
accrued since the stamp to wall time (tsc_hz) and shifts the effective
deadline; a not-yet-due entry is re-queued at the shifted deadline with a
fresh stamp, so it keeps chasing delay injected while it waits. seq is
preserved across re-queues, keeping send_after cancellation identity
intact (cancelled entries are discarded before any shift). Zero debt is
byte-identical to the old path; peek_deadline may under-report, costing
one spurious scheduler wake per injected chunk (documented).
This also makes the park-gated resume credit *correct* rather than
forgiving for sleepers: a sleeping actor now physically pays its debt by
sleeping longer, so the on_resume fast-forward reflects real payment
(sleeping_actor_pays_injected_delay pins this end-to-end through the
runtime).
New test hooks: inject_delay_cycles_for_test (deterministic ledger
driver, eagerly TSC-calibrating so conversion never stalls a scheduler
loop) and cycles_to_duration. The ledger is process-global, so the
delta-sensitive causal tests now serialize on a shared test mutex — they
were racy under the parallel test harness before this, in principle.
Samples were taken only when maybe_preempt's cold block happened to fire
in-site, so the interval between the last check and SiteGuard drop was
discarded on every site entry. Measured live on the 24-core box:
22-29us lost per entry, a constant attribution efficiency of ~0.93-0.94,
which under-reported every impact (+83.5% where theory says +100%; the
observed shortfall fits 1/(1-pct*eff)-1 at both 25% and 50%).
SiteGuard enter/drop now call site_transition(): leaving the target site
flushes the pending interval into the ledger (sample-only, never spins,
so safe under no-preempt regions); entering the target site re-arms the
sample clock so pre-site time is never attributed (the symmetric
over-attribution). Winner attribution is factored into attribute(),
shared by the cold check and the flush, with the same interval clamps.
Adds examples/causal_attrib_probe.rs (ground-truth in-site time vs
ledger attribution, the probe that confirmed the leak) and the
site_boundaries_flush_tail regression test (a site entry that never
hits a cold check must still be attributed). Also gates causal_probe
on smarm-causal in Cargo.toml - it never was, so featureless builds
of the examples were broken.
causal_site! scoped site guards per actor slot, progress! throughput
points, and a Coz-style virtual-speedup engine hooked into
maybe_preempt's amortized cold block: target-site samples grow a global
delay ledger; bystanders spin-absorb their debt at the next causal
check, with timeslice extension so injected delay is not charged
against the slice.
Resume credit (Coz's blocked-thread rule) is gated on a causal_parked
slot bit set only by a real park: crediting on every resume made any
yield-cadence actor delay-immune and every experiment inert (found
live on a 24-core run — dead-flat deltas across all sites).
Report normalization uses a measured TSC frequency (~50ms calibration
on first use) instead of the crate-wide 3 GHz assumption, which
uniformly inflated impact numbers on a 3.7 GHz box. impact_pct() is
the machine-readable form of the summary for programmatic checks.
examples/causal_pipeline.rs burns fixed *work* (calibrated LCG loop),
not fixed wall time — a timed busy-wait absorbs injected delay into
its own budget and reads as a no-op. Self-checking: exits nonzero if
causal separation fails; skips the verdict below 4 cores. Validated
on a 24-core box: reserve (true bottleneck) +29.3%@25/+83.5%@50;
serialize and background-compaction ~0%.
Known v1 gaps (jar): timer-heap deadlines unshifted, no-check!/no-alloc
actors undelayable, multi-scheduler coherence best-effort Relaxed,
off-CPU blame punted, Instant::now() uncorrected.
Zero-cost with the feature off; clippy -D warnings clean both ways;
full suite green with and without smarm-causal.
Root cause of soak20 signature 2 (refcount_test.exs 'watcher crash',
110235x fast {:error, :server_down} probes over the full await window):
by_name mapped name -> slot *index*, so a name whose holder died (no stop
path unregisters; prune is lazy) and whose slot was then re-tenanted read
as live-held: register failed NameTaken{holder: <unrelated tenant>} (which
the bridge macro's generated start() swallows -> start_server/1 reports :ok
for a server that never came up), while name resolution reached the
tenant's mailbox, missed on the message TypeId and failed fast WITHOUT
pruning — the wedge self-sustained for the tenant's lifetime. Name-
addressed send additionally judged liveness on the slot's *current*
mailbox pid, so a same-typed tenant would have received the message
(misdelivery) and a differently typed one a misleading NoChannel.
Fix: by_name: HashMap<&'static str, Pid> — every reader judges the
*stored* holder with the generation-checked live(), so a recycled slot's
tenant no longer impersonates a dead holder, and every touch (register /
whereis / resolve / send) prunes and heals a stale name. prune(index)
becomes prune_holder(pid): names bound to the holder go; the mailbox goes
only while still the holder's own (a tenant's replacement mailbox is left
untouched). Introspection matches names to mailboxes by full pid, so a
stale name never annotates a slot's new tenant.
Deterministic regression test added first and shown to fail pre-fix
(tests/stale_name_slot_reuse.rs: tiny slab forces re-tenanting; old slot
(1,0) died, tenant (1,1) took the index; register -> NameTaken pre-fix).
Post-fix it asserts the healed contract: whereis -> None (pruned), call ->
ServerDown, re-register -> Ok. Suite 33 ok-binaries, clippy gate clean.
In the wild the window opened at every splice_test teardown:
Splice.terminate -> exit_server('subtree') left the name bound; width 20
raised the re-tenant probability. Downstream (smarm_beam): install_child's
unregister-before-register workaround becomes dead code (removed there);
the #[smarm_server] macro's swallowed register error becomes truthful
idempotency (a NameTaken now really is a live holder).
A queued Envelope::Call was stranded until the last Sender dropped, so a
caller parked in gen_server::call was never released with ServerDown when a
*named* server was request_stop'd — the registry's inbox Sender clone (lazy
prune) kept the channel Arc, and the queued reply_tx, alive indefinitely.
Receiver::Drop now drains the queue (items dropped after releasing the lock,
since a reply_tx drop reaches a different channel's lock + the scheduler),
restoring the documented ServerDown guarantee on every teardown path.
Adds tests/stop_with_queued_call.rs: deterministic pure-smarm reproducer.
A `=> postpone` row (cast/call/info) defers the current event untouched,
to be replayed after the next real transition. `handle` is now two-phase:
a borrow-only postpone pre-pass that hands the event back as
`Step::Postponed(ev)`, then the existing consuming `match (state, event)`.
The loop owns a FIFO queue, drained in the new state ahead of further
intake; a replayed event may postpone again. A postponed `call` keeps its
Reply, so a later state answers it.
`handle` returns `Step` (Postponed / Transitioned / Stayed) so the loop
can see both deferral and transition without reading the state cell.
`Resolution::Postpone` is removed: postpone is a pre-dispatch routing
decision, not a consuming-dispatch outcome.
Add the two timeout flavours, both surfacing as ordinary events matched
in on-state arms:
- cx.state_timeout(d): fires a state_timeout event after d in the current
state, auto-reset by the loop on every real transition.
- cx.timeout(name, d): fires a timeout(name) event after d, surviving state
changes, keyed by name, with cx.cancel_timeout(name).
Both ride the existing timer min-heap via send_after_to onto a new per-loop
system channel, selected above the inbox so a fire can't be starved by inbox
traffic. A local-id stamp on each fire lets a reset/cancel that loses the race
discard a stale fire. The macro grows an info: clause and folds three internal
Ev variants (Info, StateTimeout, Timeout) alongside cast/call, with new row
keywords info / state_timeout / timeout. Unmatched info silently drops (the
gen_server default); state/named timeouts have no default, so a state that can
see one must handle it or the match is non-exhaustive.
Rename the hand-written expansion-target example fused -> expanded, retire the
deprecated Switch demo machine (its round-trip / enter / panic-down coverage
moves onto the timer machine), and refresh the macro docs to the door machine.
cargo build --all-targets warning-free; cargo test green.
- rename StatemRef/StatemCallError/StatemSendError -> GenStatem*
- move the inline unit test out of src; consolidate the Switch coverage
onto a single macro-driven harness in tests/gen_statem.rs
- drop the redundant hand-written Switch test machine and the two
untracked rejected-direction probes (succ_enums, typed_edges)
- rename examples statem_{fused,macro}.rs -> gen_statem_{fused,macro}.rs
- strip RFC/chunk/spike provenance and fix the mislabeled "throwaway"
example header and dead cross-references
Runtime support layer for gen_statem, built against existing public API
(channel + scheduler::spawn), sibling to gen_server. No macro: per review,
build the primitives first and hand-write the Switch example to evaluate
whether a statem! macro earns its place before committing to one.
- src/statem.rs: Machine trait (on_start/handle), Resolution<S> with
From<S>, Cx (on_unhandled), Reply<T> move-only reply handle, StatemRef
(send/call), spawn + inbox loop. Real time only; Postpone + Cx timeout
arming are in the type surface but not yet acted on (chunks 2-3).
- examples/statem_switch.rs: the RFC Switch machine hand-written against the
primitives, tagged USER vs MACRO to mark what a macro would generate.
Asserts the RFC end-state (flips=1, enters=3).
- tests/statem.rs: call/cast round-trip, enter-on-start/transition-not-stay,
panicking-handler -> Down.
Reply<T> included (the call helper needs a handle type; keeps the example
true to the RFC surface) but isolated and trivially removable if we drop it.
A thin GenServer consumer of the Chunk-1 read primitive — the live
observer process (D12). ObserverRequest/ObserverReply are the wire
contract (D11); the version rides along on the snapshot/tree payloads,
which already carry SNAPSHOT_FORMAT_VERSION (D1). Behind the new
`observer` Cargo feature, off by default (D10): the primitive stays
always-on, only the transport is gated. Cast is Infallible, so the
server takes no async traffic and handle_cast is statically unreachable.
Gated integration test proves each verb relays exactly what the
corresponding primitive returns (snapshot/tree/actor_info), incl. a
forged-pid None and a live Parked classification.
Accumulate on-CPU cycles per actor as ActorInfo.budget_cycles, behind
the off-by-default budget-accounting feature (D6) — a reductions-style
work metric for relative comparison across runs.
- Approximate by design (per Mark): charge now - slice-start once at the
yield point, reusing the timestamp reset_timeslice already sets, so
one RDTSC per resume not two. Wake-slot resumes inherit the slice and
so slightly over-attribute the chain's time to the woken actor — noise
that averages out; we trade exactness for half the hot-path cost.
- Field/ActorInfo member are unconditional (keeps the snapshot shape
stable across the feature flag, D1); only the accumulation is gated,
so default builds are byte-identical and pay nothing. Reads return 0
when off. Single-writer Relaxed like the other counters; reset in
reset_counters (D7).
Matrix: default + feature-on + rq-mpmc + rq-striped + release + trace all
green; loom unaffected (feature off under --cfg loom).
Tally each dequeued message against the receiving actor, surfaced as
ActorInfo.messages_received (the 'is this actor draining slower than its
mailbox fills' signal).
- messages-received, not sent (D4): the receiver counts on its own
thread, so it's a single-writer Relaxed load+store on a hot Slot
AtomicU64, no atomic RMW (D5); reuses the stashed *const Slot from 2a.
- Incremented at all six channel dequeue-success sites (recv,
recv_timeout x2, recv_match, try_recv_match, try_recv) via
preempt::note_message_received; no-op outside an actor (null slot).
- Resets with overruns in reset_counters across the three lifecycle
sites (D7).
Matrix: debug default + rq-mpmc + rq-striped + release green; loom
slot_state/run_queue models pass (the 3 send_after_to loom failures are
pre-existing at fc014c4 — plain #[test]s under --cfg loom, not Chunk 2).
Tally overruns at the slice-expiry site in preempt.rs (the RFC 006
signal), surfaced as ActorInfo.overruns.
- Counter is a hot-region AtomicU64 on Slot, single-writer Relaxed
load+store (no atomic RMW), read Relaxed by the snapshot (D5).
- Reached from the rare expiry branch via a stashed *const Slot in a
preempt thread-local, set/cleared on the same resume/return boundary
as CURRENT_STOP — one TLS load, no runtime lookup. Shared infra for
the messages-received counter next.
- Reset in all three slot-lifecycle sites: Slot::vacant, reclaim_slot,
install_actor (standing invariant, D7) — per-incarnation counts.
tree() / tree_from() fold a Chunk-1 snapshot into a forest by grouping
each actor under its parent pid — one O(n) pass, no new reads. tree_from
is public so a held (or synthetic) snapshot can be folded without a
second scan.
- D8: actors parented at ROOT_PID are genuine roots; an actor whose
recorded parent is absent from the snapshot is re-rooted under the
sentinel and flagged orphaned, so the forest stays total. take()-on-
place doubles as a guard against re-entering a node.
- D9: the edge is parent/spawned-by, documented as not necessarily
supervision.
snapshot() / actor_info() return owned ActorInfo over the slab: pid,
names, fine-grained scheduling state, parent edge, trap flag, mailbox
depth, and monitor/link/joiner counts. Pure reads, no hot-path change.
- ActorState maps the packed slot word (no new storage); introspect.rs.
- D1: RuntimeSnapshot carries SNAPSHOT_FORMAT_VERSION from day one.
- D2: per-slot tearing (ps semantics); actor_info coherent per actor.
- D3: mailbox depth included. Registry channels now stored behind an
ErasedSender trait (downcast for clone_sender + type-erased
queued_len); depth summed over published channels under the registry
Leaf (Leaf -> Channel), kept out of the cold-lock pass so no two
Leaves are ever held at once. Depth covers published channels only.
- Done slots surface as root-less tombstones.
Add a cancellable message-delivery timer on the existing timer.rs min-heap,
the substrate the gen_server time idioms (idle/receive timeout, periodic tick,
debounce/backoff) need.
- Reason::Send { fire }: a type-erased delivery thunk. send_after (Pid<A>, via
send_to) and send_after_named (Name<M>, via send) capture dest+msg and
resolve the address on fire, not at arm time, so a dead target / restarted
name is observed when it fires; a failed resolve or closed inbox is dropped
(Erlang erlang:send_after semantics).
- Cancellation via an "armed" set keyed on the entry seq, exposed as an opaque
TimerId. Only Send timers use it; Sleep/WaitTimeout keep their inert-stale
behaviour untouched. pop_due fires a Send only while still armed and removes
it, so cancel returns true iff it landed before the fire (the race signal).
cancel is unscoped: any holder of the id can cancel (e.g. racing two servers
and cancelling the slow path).
- peek_deadline contract documented as "<= true next deadline" so a future
hierarchical timing wheel can back Timers without touching send_after or the
scheduler idle path. call_timeout left on recv_timeout (blast radius).
Tests: Timers-level (fire/cancel/race/clear/ordering) plus scheduler-level
delivery for both Name<M> and Pid<A>, cancel-prevents-delivery, and silent
drop on unresolved name / dead pid. Green on rq-mutex/rq-mpmc/rq-striped.
The explicit fallible fallback for when only an untyped Pid is in hand (off a
Down, out of a future members()), so the typed send_to / send-by-name stay the
default and callers don't grow workarounds around a missing primitive.
- send_dyn::<M>(Pid, msg): one line over the shared send_to_pid core. Same
identity-bound, no-redirect liveness as send_to (Dead once the incarnation is
gone); the difference is the explicit M, so NoChannel is a real runtime
outcome here rather than the debug_assert it collapses to on the typed paths.
- Takes Pid<Erased> specifically: a typed Pid<A> must .erase() to reach it, so
opting into the fallible downcast stays visible at the call site.
tests/registry.rs: send_dyn_delivers_and_reports_wrong_type (right type
delivers; live actor with no channel for the asked type -> NoChannel) and
send_dyn_to_dead_pid_is_dead. Full suite + order-checker green, warning-free.
This completes the RFC 014 send surface: Name<M> (re-resolving), Pid<A>
(typed direct), send_dyn (bare-pid). send_after (§6) can now target any of
them — its Dest question is answered.
Make a Pid<A> messageable: install + send_to, the direct addressing mode that
dies with the actor and never redirects (the counterpart to the re-resolving
Name).
- install::<A: Addressable>(tx: Sender<A::Msg>) -> Pid<A>: opt-in, lazy,
nameless publish (RFC 014 §5). Files the actor's inbox into the by_index
mailbox table under TypeId::of::<A::Msg>() and re-types self's identity as
Pid<A>. Infallible: no name to collide on, self is always live in run().
- register and install now share publish_channel (the insert-or-extend-mailbox
step factored out); register additionally binds the name. Byte-identical
storage, so name- and pid-addressing populate the same table.
- send_to::<A>(Pid<A>, A::Msg): resolve by raw pid, deliver with NO redirect.
The stored mailbox must be this exact incarnation (generation included) and
live, else SendError::Dead — even when the slot now holds a different live
actor, which is left untouched. Resolution clones the Sender under the Leaf
lock, releases, then sends (Leaf -> Channel), as name send / pg / finalize.
- send_to_pid is the shared raw-pid core; send_dyn (§4.6) lands on it next.
- SendError gains Dead(M), the pid-path counterpart to name-path Unresolved;
into_inner / Display / Debug updated. Name send never yields Dead, pid send
never yields Unresolved — disjoint by construction, documented on the enum.
tests/registry.rs: install_then_send_to_pid_delivers; and the load-bearing
send_to_does_not_redirect_after_takeover (slot reused by a new incarnation, the
stale Pid<A> send returns Dead and the new occupant gets only its own message).
Full suite + order-checker green, warning-free across all targets.
Rip out the old name<->pid bimap; rebuild registry.rs as the live mailbox
directory. A name resolves to a SINGLE actor (many-per-label is pg's job); that
actor owns a SET of typed channels (channels are typed, so no single mailbox),
keyed by message TypeId. Resolution: name -> pid (one actor) -> Mailbox ->
channel for type M. Same name + different type parameter selects different
channels of the one actor, so capability separation (§4.7) needs no new type.
- register<M>(Name<M>, Sender<M>): captures the current actor's channel under a
name (one step, per decision). Adds channels cumulatively; NameTaken only vs a
different LIVE holder; stale/dangling bindings pruned on contact.
- send<M>(Name<M>, msg): the payoff — resolve, clone the Sender UNDER the Leaf
lock, release, then send (Leaf -> Channel; a send can unpark). Returns the msg
on Unresolved / NoChannel / Closed.
- whereis -> single live pid; unregister frees only the name (mailbox/other
names survive). name_of (reverse lookup) dropped — deferred to introspection.
- Contained erasure: each channel is Box<dyn Any+Send> filed under TypeId::of M
and downcast to its own keying type, so the downcast can't fail on good data
(debug-asserted via the stored type_name). Phantom M on Name re-imposes type.
- One Leaf RawMutex (the fold), so a name send stays on a single Leaf — two
Leaves at once is a hard panic. runtime.rs field/init unchanged (same Registry
name + new()).
tests/registry.rs rewritten for the new semantics: send-by-name delivery,
many typed channels on one actor, same-name/different-type routing, NameTaken,
takeover across slot reuse, Unresolved/NoChannel. Also fold in a Phase 1 fixup:
the phantom test key was an enum whose variants tripped dead-code under the test
build (only caught now that I run -D warnings on --tests). Full suite +
order-checker green, warning-free across all targets.
Wire eviction to actor death via the monitor subsystem, using drain-on-
contact (no scheduler hot-path hook).
- Each membership now carries its own Monitor alongside the group entry;
join() installs monitor(pid) (registration races finalize_actor under the
cold lock, as every monitor does) and a redundant join tears its extra
monitor back down.
- reap_group: every group op drains the touched group's monitors with a
non-blocking try_recv (a delivered Down or closed channel = dead) and, on
the first death, sweeps that pid out of *every* group via remove_where —
so a death vanishes from all its groups on next contact.
- Lock discipline: monitor()/demonitor() (cold = Leaf) run outside the group
lock; try_recv (Channel) runs under it, permitted by the Leaf->Channel
order; evicted/rejected monitors are dropped after the lock releases so no
receiver-drop wakeup runs under it. Validated by the debug-build lock-order
checker passing under the integration tests.
- remove_where now returns the evicted monitors (for deferred drop) and
preserves intra-group insertion order.
- Tests: unit (synthetic monitors) for reap live/dead/closed + sweep;
integration (real actors) for death-vanishes-from-every-group, pick on an
all-dead group, peer survival, leave isolation, dead-at-join eviction.
Outstanding: a miri pass (cargo +nightly miri test --lib pg::) is untried —
first build exceeds the sandbox window and the futex RawMutex may need a miri
shim; the mechanical Leaf->Channel checker is the primary soundness guard.
request_stop sets the flag and issues a wildcard unpark, but unpark
no-ops on a Queued actor (the pending run "is" the wake). If the
actor's first action on resume is a blocking park — no allocation, no
check!() on the way — the wake-side-only check in park_current is
unreachable: the actor parks with the stop flag already set and no
wake is ever coming. The runtime then idles forever.
Fix: check_cancelled at the ENTRY of park_current, before the switch.
The remaining window (flag lands after the entry check, before the
park) is covered by the existing protocol: the stop's unpark finds
Running / the prep-to-park window, sets Notified, and the park-return
re-queues into the wake-side check. Unwinding from the entry is the
same unwind path as the wake side; leftover wait registrations are
stale-epoch / dead-generation and self-clean at their wakers' failed
CAS.
This is bug #1 of the urus chunk-2 session (bug #2, the terminal
wake, is the preceding commit). Repro: tests/cancel.rs
stop_flagged_while_queued_lands_at_first_park — deadlocks without the
fix (watchdog-bounded), passes in <50ms with it.
An idle scheduler thread blocks in poll_wake on a snapshot of
peek_deadline / io_outstanding. Only the io threads ever write the wake
pipe (on completion push); enqueue does not. Mid-flight that is masked —
the thread that caused an enqueue is awake and processes it — but at
termination the snapshot can go terminally stale, two ways:
- Stale io_outstanding (infinite hang): an actor parked in wait_readable
is request_stop'ped. Cancellation deregisters the waiter and produces
NO completion, so the wake pipe is never written. A sibling blocked on
io_outstanding > 0 with no timers pending sits in poll(-1) forever.
- Orphaned timer deadline (finite stall): an actor cancelled out of a
long sleep leaves its timer entry behind. Clearing the wheel at
AllDone doesn't help a sibling that already blocked on that deadline;
it sleeps it out in full.
In both cases the remaining actors finish on the other scheduler thread,
which reaches the live==0 && io_out==0 verdict and returns without
waking the blocked one. Runtime::run then stalls or hangs in its worker
join. Found via urus's graceful-shutdown tests (~10% flake, one test per
variant); confirmed by gdb dumps of hung processes showing run() in
JoinHandle::join over a sibling in poll(wake_fd, 59750ms) resp. poll(-1).
Fix: Io::wake() writes the wake pipe directly; the AllDone arm calls it
after the timer clear. One byte wakes every poller; each re-runs the
verdict, independently reaches AllDone, and re-wakes — idempotent.
tests/terminal_wake.rs reproduces both variants deterministically: the
root busy-spins (no timer entries, occupies one scheduler thread) so the
sibling settles into the stale idle wait before the stop is issued. Both
hang without the fix and pass in <0.5s with it.
Known residual gap, deliberately unfixed: the timers-pending /
no-io-subsystem idle branch blocks in thread::sleep and has no wake
mechanism at all — the same stall exists for runtimes that never
initialize io. Roadmap candidate alongside cross-thread unpark and
entry-side check_cancelled in park_current (the lossy-QUEUED-stop bug).
A thread-local Cell<Option<Pid>> per scheduler, checked before the shared
run queue. Runtime-selected via Config { wake_slot: bool }, default OFF
until the slot shootout accepts it (one binary benches both arms).
Push policy: slot-eligible iff the wake originates from actor context
(current_pid().is_some()) — the slot push replaces run_queue.push at the
tail of the unpark protocol's Parked → Queued CAS, so at-most-once-enqueued
holds verbatim as (slot ⊕ shared queue). Scheduler-context wakes (timer/IO
drain) and spawns always go shared (spawns never reach unpark_inner at all).
Displacement is Go semantics: newest wake takes the slot, occupant pushed
shared — moved, never copied.
Pop order: slot, then shared. Slot-popped actors skip reset_timeslice() and
inherit the waker's remaining slice; a handoff chain is bounded by one
slice, after which the preempt-yield re-enqueue goes shared (a yield is not
a wake) — the one-slice starvation bound, zero new counters. Idle and
AllDone are only reachable with an empty local slot by pop order; an
occupied slot elsewhere holds a Queued (live) actor, so the counter-first
termination argument is untouched.
Observability: per-thread slot_hits / slot_displacements (reset at run()
start so post-run stats() reads are per-run), SlotPush/SlotPop trace events.
Bench plan (roadmap v0.9 item 2): rq_runtime gains the slot on/off
dimension (SMARM_BENCH_SLOT, default "0 1") — ping-pong-pairs is the
target metric, yield-storm the regression guard, spawn-storm the
neutrality check. RQCSV grows a slot column; RQSLOT lines carry the
counters; bench_rq.sh aggregates both. Tests pin the push policy through
the counters (actor-context hits, spawn/join bypass, displacement,
default-off, per-run reset).
FdArm composes fd readiness with channel arms on one wait epoch.
Selectable grows fallible sel_register and an eager-cleanup hook;
losing/stop-unwound/timed-out fd arms are unregistered (waiters entry
+ kernel ONESHOT) so the fd is never poisoned. try_select /
try_select_timeout surface registration errors (EBADF, EMFILE,
AlreadyExists) instead of RFC 008's permanently-ready lean, which
would busy-loop a healthy-but-unregistrable fd; select/select_timeout
stay infallible for channel-only arms. Adds wait_readable_timeout /
wait_writable_timeout as one-arm selects.
Known benign race (pre-existing, slightly widened): a queued FdReady
racing the cleanup DEL can spuriously wake a fresh waiter on that fd;
absorbed by select's defensive re-loop. Fixable by epoch-stamping
completions.
A stopped actor unwinding out of wait_fd's park leaked its waiters entry
and the kernel-side EPOLLONESHOT registration; the stale entry then failed
every future wait_*() on that fd with AlreadyExists (the defensive bare DEL
in epoll_register sits behind the contains_key check, so it never ran).
Fix where the invariant breaks: a drop guard in wait_fd, armed after a
successful register and forgotten on the normal wake (where FdReady already
removed + DEL'd). On unwind it cleans up iff the entry is still this wait's
(pid, epoch) — an entry consumed by a racing FdReady means the fd may carry
another actor's fresh registration, which must be left alone. Closes the
v0.2 fd-hygiene TODO.
Monitors are created at runtime (you watch a worker you just spawned in a
handler), so down arms can't ride the static info list. init grows a
&ServerCtx parameter (breaking; no-op default) whose clonable Watcher hands
Monitors to the loop over a control arm; the loop selects the live down
arms ahead of everything else. Arm priority: downs → control → infos →
inbox. A delivered Down retires its arm (monitors are one-shot); a state
that never clones the Watcher closes the control arm after init and the
loop falls back to the plain-inbox park.
type Info + handle_info (no-op default) on GenServer; ServerBuilder
(with_info/under/start) so start variants don't multiply — start/start_under
stay as wrappers. The loop selects [infos.., inbox] in priority order when
info arms exist, and keeps the plain recv() park when none do. Closed info
arms are silently removed (closed-arm-is-ready-forever); a closed inbox
still means graceful shutdown.
The deadline is one more stamped waker on the select's wait epoch: a unit
TimerTarget whose on_timeout is a bare unpark_at. Nothing registers in any
arm for it, so there is nothing to cancel or leak — an arm winning leaves
the timer entry to die at its epoch CAS; the timer winning leaves the arms'
registrations to self-clean like any select loser's. Classification is pure
channel state (wakes are precise): some arm ready -> Some(first, priority
order); none -> the timer was the only remaining stamped waker -> None.
Message-first on a raced deadline, same as recv_timeout.
Registration pass factored into register_arms, which owns the retire_wait
obligation on the ready-now exit for both entry points.
select(&[&dyn Selectable]) -> usize registers (pid, epoch) in every arm
under one wait epoch, parks once, and returns the first ready index in
documented priority order (BEAM-style; no fairness promise). A closed arm
counts as ready — and stays ready forever, so callers drop it from the set
once observed. Losing arms need no cancellation pass: the winning wake
consumed the epoch, so their registrations die at their wakers' failed CAS
or are overwritten by the receiver's next wait on that channel (the
single-receiver debug_asserts relax to 'none or own pid' accordingly).
The one genuinely new protocol piece is the no-park exit: returning with an
arm ready at registration time leaves earlier arms holding LIVE-epoch
registrations, whose wakes could fault a later one-shot park as a pending
notification. scheduler::retire_wait closes it — bump the epoch (in-flight
wakes die at their CAS), eat a notification that already landed
(StateWord::clear_notify), then re-observe the stop flag, in that order.
Proved by the new loom theorem retire_eats_late_arm_notification; the
integration probes in tests/select.rs fire stale loser-arm wakes and assert
a subsequent sleep's one-shot park holds its full duration.
The slot epoch is THE wait identity, so the hand-rolled per-primitive copies
go away: channel loses cur_wait/next_wait_seq/timed_out, mutex loses Wait.seq
and next_seq, TimerTarget::on_timeout takes the epoch. Registrations become
(pid, epoch) — channel parked_receiver, mutex waiters, io fd waiters,
Blocking io completions, sleep timers, joiner lists — and their wakers move
to unpark_at. begin_wait is lock-free, so each primitive opens the wait
inside the same critical section that publishes the registration.
recv_timeout's wake-classification loop collapses: wakes are precise, so
queue → Ok, senders==0 → Disconnected, else Timeout — the 'Defensive'
re-register branch is now unreachable by protocol, not by audit. Same for
Mutex::lock_timeout's one-shot park.
End-state invariant, auditable in one sentence: the only wildcard wake is
request_stop, which is terminal.
Deviates deliberately from the roadmap's monitor-based sketch
(monitor + wait reply-or-Down-or-deadline + demonitor): server death is
already observable on the reply channel itself - the reply sender drops
as the server unwinds, closing the channel and waking the parked caller.
So a bounded call is exactly recv_timeout on the reply channel, mapping
Disconnected -> ServerDown and Timeout -> Timeout. No registration
exists, so a timed-out call leaks nothing by construction - the
guarantee the monitor design had to engineer.
Semantics on timeout match Erlang: the request stays in the inbox and
is still handled; the late reply's send fails harmlessly against the
dropped reply receiver. CallTimeoutError keeps ServerDown and Timeout
distinguishable; the infallible call() is unchanged.
Tests: reply within deadline, timeout against a slow (parking) handler
with elapsed bounds, server survival across an abandoned call (late
reply discarded, bounded and unbounded calls keep working), and
ServerDown-not-Timeout for both death paths (mid-call panic and
already-gone inbox).
The per-recv deadline the roadmap deferred, built exactly the way
Mutex::lock_timeout already works: register the wait with a per-wait
seq, arm a timer::Reason::WaitTimeout with the channel inner (now a
TimerTarget) as the target, park. On expiry on_timeout cancels the wait
only if that same seq is still parked; satisfied or abandoned waits
leave a stale heap entry that no-ops on seq mismatch, per the
no-cancellation convention in timer.rs.
Race resolution is message-first: a send that lands by the time the
woken receiver runs is delivered even if the deadline had also passed.
Closure is reported as RecvTimeoutError::Disconnected, keeping timeout
and server/sender death distinguishable for callers (gen_server call
timeout builds on exactly this distinction next).
The timer is armed outside the channel critical section (the timers
lock must never nest under a Channel-class lock); the unpark race this
opens is absorbed by the RunningNotified protocol.
Tests: immediate delivery, actual timeout (with elapsed check), prompt
wake on send, Disconnected on close, zero-duration poll, post-timeout
seq isolation (stale entry must not cancel later waits), and a
4-scheduler mixed-outcome run (12 fed / 12 timed out).
register/whereis/unregister plus the inverse lookup name_of. Erlang
semantics: one name per pid, one pid per name, registering over a live
binding errors; same-binding re-register is an idempotent Ok.
Bimap = two HashMaps held in exact inverse under one Leaf RawMutex in
RuntimeInner; the invariant is debug-asserted on every mutation. The
inverse direction costs one extra String per binding and buys O(1)
pid->name for supervisors/tracing/diagnostics.
Cleanup is lazy: no finalize_actor hook, no Slot field (which would buy
into the reset-in-three-places invariant). Liveness rides the
generation-checked slot word - pids are never reused, so a stale binding
is detectable, never misdirected - and bindings to dead actors behave as
absent, pruned on contact by whereis/name_of/register/unregister.
Liveness checks under the registry lock read only the atomic slot word,
so the leaf rule holds trivially.
send_by_name is deliberately absent: smarm has no per-pid mailbox, so
there is nothing generic to send *to* - the registry yields a Pid,
usable with monitor/link/request_stop and as routing for user channels.
Tests: roundtrip both directions, idempotence, NameTaken on live holder,
one-name-per-pid, NoProc, death-evaporates-binding (via lookup pruning
and via register's own eviction path), unregister, cross-actor lookup
wired into request_stop, and a 4-scheduler name-contention churn test.
Phase 2 surfaced the hole this closes: channel guards were held with
preemption enabled, so a timeslice switch inside a channel critical
section could release the pthread mutex from a different OS thread (UB).
RawMutex disables preemption for the guard span and is
cross-thread-release sound by construction; poison goes away with it.
The strict single-class leaf rule cannot survive the migration: finalize
clones the supervisor/trap senders and monitor() clones the Down sender,
all under a cold lock, and those senders live in the slot - the nesting
is structural. The leaf check is therefore generalized to two classes:
Leaf (cold locks, free list, stack pool) and Channel. Order is
Leaf -> Channel, at most one of each; both directions of violation are
debug-asserted at the acquisition site. Channel critical sections call
only the lock-free unpark protocol, so the order is acyclic.
Tests: class-ordering unit tests in raw_mutex (allowed nesting + all
three rejected shapes), plus a multi-scheduler integration test driving
channels through monitor churn and actor death so any ordering
regression trips the debug assert instead of deadlocking.
benches/README.md: catalog of the two bench families (cross-runtime
comparisons + the v0.5 run-queue shootout), how to run the shootout
(scripts/bench_rq.sh, SMARM_BENCH_* knobs, the --no-default-features dance,
summary.csv/RQCSV format), honest-numbers caveats (core count is the
experiment; striped losing at low thread counts is expected), and the
conventions for adding a bench.
tests/README.md: catalog grouped by layer (low-level units / feature areas /
regression+stress), the run matrix (debug-first — that's where the invariant
asserts live — three queue variants, release, loom, trace), why loom tests
live in src/ rather than tests/, and the house conventions: one runtime per
test, oversubscription on purpose, regression tests validated against the
reintroduced bug, odds-stacking for stochastic tests, ordering-not-duration
time assertions, and new-invariant = assert + loom model.
Also fixes a stale header in tests/mutex.rs that claimed 'loom::Mutex' —
it tests smarm::Mutex, and the old name is actively confusing now that loom
is real in this repo.
The phase-1 split filled Idle::next_deadline from the timers lock after the
shared lock was released, which turned the stale-PID branch (formerly a 100µs
poll) into a potential sleep-until-timer-deadline while runnable actors sat in
the queue — a hard stall on exact(1). Stale pops are now PopResult::Retry and
loop without sleeping.
test(poison): strengthen regression coverage. The stop-storm test never fired
the sentinel under a lock (its actors only allocate at lock-free points). Add
self_stop_during_spawn_does_not_poison_shared_mutex: a stop-flagged actor whose
next allocation is the Box::new(closure) inside spawn's with_shared. Validated
both ways: passes with the gate, SIGABRTs (unwind-in-allocator) with the gate
removed. Also alloc_interval(1) so every allocation is an observation point.
- next_monitor_id -> AtomicU64 on RuntimeInner
- timers -> own Mutex<Timers>; io -> own Mutex<Option<IoThread>> (lock order: io-before-shared)
- pending_closures Vec folded into Slot::pending_closure
- termination check reads io liveness before shared; ordering argument documented
- poison fix: check_cancelled() no longer fires while preemption is disabled,
so a cancellation unwind can never poison a runtime/channel mutex
- regression test: tests/poison_stop.rs
- ROADMAP_v0.5.md added
Monitors could be installed but never taken down. That gap was about to
bite: the gen_server call timeout we want next is the Erlang dance —
monitor the server, wait for the reply or a Down or a deadline, then
demonitor — and without a way to remove a registration, every timed-out
call would leak a monitor on the server's slot and risk a stale Down
arriving later. So this lands the cleanup primitive before anything
depends on the old shape.
The decision flagged in the roadmap ("decide the monitor API NOW") is
resolved by giving each registration a process-unique MonitorId and
returning it to the caller. monitor() now hands back a
Monitor { id, target, rx } rather than a bare Receiver<Down>: read the
notice from rx as before, and pass &Monitor to demonitor to tear exactly
that registration down. The id comes from a monotonic counter in shared
state, bumped under the same lock that does the registration, so it's
deterministic and never reused — which is what lets demonitor name one of
several monitors on the same target unambiguously. target rode along on
the struct (over the sketched {id, rx}) purely so demonitor can go
straight to the slot instead of scanning every slot for the id.
demonitor returns Option<MonitorId> rather than a bool: Some(id) when a
live registration was found and removed, None when there was nothing left
to remove — it already fired (the registration is drained on finalize),
it was a NoProc, or the slot has been reclaimed. The generation half of
the pid quietly protects that last case: a recycled slot index fails
slot_mut's generation check, so a late demonitor is a clean no-op and can
never strip a different actor's monitor that happens to share the index.
The one piece of real care is reentrancy. Removing a registration drops
the slot's Sender, and Sender::drop can unpark a parked receiver, which
re-enters the shared mutex — which is not reentrant. So demonitor moves
the sender out of the Vec under the lock and lets it drop only after the
lock is released, the same discipline finalize_actor already follows for
its monitor and supervisor sends.
Flushing a Down the target already queued isn't a separate flag; it falls
out of dropping the Monitor. demonitor(&m); drop(m) stops future notices
and discards any queued one — the gen_server-call cleanup in one move.
Storage is now Vec<(MonitorId, Sender<Down>)>. The three slot-reset sites
were left alone on purpose: they clear/rebuild the Vec, which doesn't care
about the element type, so there's no fourth reset obligation. finalize
just destructures (_, m). Because chained_spawn and yield_many register no
monitors, that Vec stays empty on the hot path — taking an empty Vec costs
the same and the notify loop runs zero times — and a before/after general
probe confirmed both medians sit within noise.
Tests cover the three behaviours that matter: a demonitored watcher gets
no Down (its channel closes), demonitoring one of several leaves the
siblings firing normally, and demonitoring after the Down has already
fired reports None.
Thin request-reply layer on channels, no runtime change. A server is an
actor owning a GenServer state; clients hold a clonable ServerRef and issue
call (sync, parks for the reply) or cast (fire-and-forget).
- Single inbox carrying an internal Envelope { Call(req, reply_tx) | Cast },
forced by the no-select / no-unified-mailbox invariant; call makes a
one-shot reply channel and parks on it.
- Server-down detection is pure channel closure (no monitor): send fails if
the inbox is gone; the reply sender drops on the server's unwind so a
parked caller wakes to Err. Both collapse to CallError/CastError::ServerDown.
- init/terminate are optional trait hooks; terminate runs via a drop guard so
it fires on every exit path (clean close, panic, request_stop). Must stay
non-blocking — may run mid-unwind.
- ServerRef carries pid() for monitor/request_stop/link; start + start_under.
Tests: cast-then-call roundtrip, init/terminate ordering, both server-down
paths (reply-channel close on handler panic; inbox-send failure when gone).
Claude (sandbox)