Compare commits

..
29 Commits
Author SHA1 Message Date
smarm d4839f1d81 feat(runtime,io): driver-enqueues + park/wake idle path — retire the wake pipe
The swap (RFC 018). Schedulers no longer sleep on a shared level-triggered
wake pipe — the herd source that made the default 8-thread config 7x
slower than 2 threads (E1). They park on per-thread futex parkers via the
coordination layer; IO backends become producers behind a two-call
contract (make runnable, then the enqueue tail wakes exactly one parked
scheduler).

Deleted: the drain lock and the one-winner phase-1 drain; the shared
completions VecDeque; the wake pipe fds, poll_wake, drain_wake_pipe,
wake_scheduler, the FdReady/Blocking Completion enum; the 100us idle nap;
the per-pop io.lock liveness read; io.rs's as_millis timeout truncation.

Added:
- enqueue wake tail (fixes the silent enqueue): wake_one_if_idle, a fence
  + one Relaxed mask load when everyone is busy — the pure-compute hot
  path pays almost nothing.
- driver-enqueues: the pool thread stashes its result in the slot,
  decrements io_outstanding, unparks; the epoll thread removes+DELs the
  waiter under the waiters lock and unparks. Both reach the runtime via a
  Weak (no Arc cycle). The waiters map moves behind its own Arc<Mutex> so
  the epoll thread never takes the runtime io lock (teardown holds it
  while joining that thread).
- io_outstanding / io_fd_waiters atomics: the termination verdict reads
  two atomics instead of taking io.lock on every pop.
- timekeeper idle path: at most one parked scheduler holds the timer
  deadline (an expiry wakes one, not a herd); everyone else parks
  indefinitely and is woken by the enqueue tail.
- busy-path timer due-check (ratified design point (a)): under saturation
  nobody parks and no timekeeper exists, yet due timers must still fire —
  one Relaxed load of the earliest-deadline snapshot per loop, clock read
  only when a timer is armed. Maintained under the timers mutex.
- chain rule: a scheduler that pops with more work queued and a sibling
  parked wakes one, so surplus runs in parallel rather than behind it.

tests/park_wake.rs pins the two new observable properties: timers fire
under full scheduler saturation, and sub-ms sleeps are prompt (the
as_millis truncation regression). Full suite + all loom models green;
clippy --lib clean.
2026-07-24 09:12:12 +02:00
smarm 2854b560d6 feat(park): fenced producer fast path + earliest-deadline snapshot
Two integration-driven amendments ahead of the runtime swap:

wake_one_if_idle() realizes RFC 018's "empty-mask fast path is one
relaxed load" soundly: a bare relaxed load is a lost-wake in the Dekker
shape for the lock-free ring queues, so the producer publishes work,
fences (SeqCst), then reads the mask Relaxed — paired with a matching
fence between the consumer's bit-publish and its re-check in park().
The pure-compute hot path (mask 0) never takes the shared mask line
exclusive; the RMW read stays on the rare chain-rule path only. Loom
models 1/2 now drive the fenced pattern end to end.

next_deadline is the earliest KNOWN timer deadline, independent of
whether anyone is parked — which tk_armed cannot give: under saturation
nobody parks, nobody arms, yet due timers must still fire (ratified
design point (a): the busy-path due-check). Maintained under the timers
mutex (note_deadline on insert — which also carries the timekeeper
re-arm wake — refresh_deadline after pop/clear); read lock-free.
deadline_due() costs one Relaxed load and a branch when no timer exists;
the clock is read only when one does.
2026-07-24 09:12:12 +02:00
smarm 7b026cfe56 feat(park): scheduler coordination layer — parkers, idle mask, wake protocol (RFC 018)
Schedulers get an IO-agnostic sleep/wake primitive of their own: one
futex Parker per scheduler thread (permit semantics, std::thread::park
shaped — closes the check-then-park race), an AtomicU64 idle mask with a
set-bit → re-check → wait park protocol, wake_one (highest-bit LIFO,
CAS-clear before unpark: exactly one wakeup per call by construction),
wake_all for the terminal path, and the timekeeper role — at most one
parked scheduler holds the timer deadline, with an atomic armed-deadline
snapshot for the busy-path due-check and an insert-side re-arm wake.

Deadlines travel as nanosecond timespecs end to end; the wake pipe's
as_millis truncation is unrepresentable here. The Dekker publish/re-check
shape is resolved by the same-location-RMW handshake (AcqRel), not SeqCst
loads; loom verifies exactly this in four models (no-lost-wake, chain
propagation, timekeeper handoff, termination), run with
LOOM_MAX_PREEMPTIONS=3 — unbounded exploration is impractical for the
looped models. Loom/non-Linux builds park on a Mutex+Condvar via
sync_shim.

Standalone until the runtime swap (next commit): nothing outside tests
constructs a Coordinator yet, hence the temporary dead_code allow in
lib.rs.
2026-07-24 09:12:12 +02:00
smarm 006a3283e7 chore(hooks): clippy gate falls back to a nix-shell toolchain
Desktop migration: the home-manager rust here ships without the clippy
component. Prefer an installed cargo-clippy; otherwise run clippy from an
ephemeral nix-shell with a separate target dir (mixed-compiler artifacts
are an E0514 hard error). MSRV keeps the shell's older toolchain a
legitimate gate.
2026-07-24 09:12:12 +02:00
Markk116 8c764e9169 docs(monitor): user-facing rewrite of process monitors
Lead with the user's problem (learn when another actor dies without
it knowing you're watching), explain one-directional/one-shot
semantics and contrast briefly with link without assuming link.rs has
been read. Add a compiling doctest. Drop em-dashes. Correctness facts
about registration/death races and demonitor-after-fire safety kept,
reworded in plain terms and separated from the public item docs.
2026-07-24 08:44:56 +02:00
Markk116 41b9d6d056 docs(introspect): user-facing rewrite of runtime introspection
Was the worst offender for external-context references (RFC 016
Chunk 1/4, DECISION D1/D2, RFC 003/011), all removed. Lead with the
practical use cases (debugging, health checks, test assertions,
dashboards) for snapshot()/actor_info()/tree(), and explain the
per-actor-reads-not-a-world-freeze consistency model in plain terms
instead of citing a decision log. Add a compiling doctest.
2026-07-24 08:44:56 +02:00
Markk116 dd845f22fe docs(mutex): user-facing rewrite of the actor-blocking mutex
Every public item was previously undocumented. Lead with why Mutex<T>
exists (a channel/gen_server is overkill for plain shared state) and
how it differs from std::sync::Mutex (parks the actor not the OS
thread, every lock is timeout-bounded by default). Add a compiling
doctest. Document new/lock/lock_timeout/try_lock/set_default_timeout/
MutexGuard/LockTimeout/DEFAULT_TIMEOUT. Drop em-dashes; keep wake
protocol mechanics as contributor-facing comments on private internals.
2026-07-24 08:44:56 +02:00
Markk116 36a0a9832d docs(registry): user-facing rewrite of the name registry
Replace the 'what changed' diff-against-a-prior-design framing with a
plain explanation of what the registry is for (naming an actor so
others can find and message it by name) and a compiling doctest
(register/whereis/send/unregister). Cut all RFC/decision-number/bug-id
references and em-dashes; move type-erasure and locking-discipline
detail into an Implementation notes section for contributors.
2026-07-24 08:40:26 +02:00
Markk116 feda6517e5 docs(channel): user-facing rewrite of the MPSC channel primitive
Lead with what a channel is and how to use it (compiling doctest for
channel()/send/recv/close), before any internal rationale. Document
every previously-undocumented public item (channel(), Sender, Receiver,
SendError, RecvError). Move the RawMutex-vs-std::sync::Mutex rationale
and lock-class discipline into an Implementation notes section. Drop
em-dashes throughout.
2026-07-24 08:40:26 +02:00
Markk116 8625ae4c35 docs(scheduler): user-facing rewrite of the actor/spawn/run entry point
Lead with what an actor is and how to start one with run()/spawn(),
following gen_server.rs's example-first style. Add a compiling module
doctest. Drop RFC references and em-dashes; keep internal mechanics
(preemption gating, thread-local borrow rules) as plain contributor
comments rather than public-facing doc prose.
2026-07-24 08:40:26 +02:00
Claude (sandbox) d9addeba5e test(causal): controller test no longer races the sweep's site snapshot
run_experiments snapshots the site registry once at entry. stage-x is
registered lazily (first causal_site! execution in the worker), so on a
1-core box the snapshot deterministically wins whenever a sibling test
has already paid the tsc_hz calibration — stage-x missed the sweep and
the summary assert tripped (silently, pre-propagation; the earlier
cascade attribution was incomplete). The worker now signals after its
first site entry and the test waits on it before starting the sweep.
Jarred separately: the lazy-registration trap is a lib UX hazard worth a
doc note or warm-up guidance.
2026-07-18 21:59:21 +00:00
Claude (sandbox) d5a3ba1934 fix(causal): born current — slot reuse booked phantom park forgiveness
A fresh/reused slot started with causal_delay=0 and causal_parked=true:
the first resume then 'forgave' the entire monotone global backlog, once
per spawn, booked as park forgiveness. Under close-mode conn churn (~95k
spawns/s) that is millions of phantom forgiven ms per 700ms window, even
in 0% cells — the books could not balance under actor churn while
ka-mode stayed plausible (few spawns). Impacts were unaffected: the
first resume always precedes the first check, so nobody ever spun.

reset_counters now installs Coz's new-thread rule: causal_delay starts
at the current global ledger and causal_parked starts false — a newborn
neither owes nor is forgiven the process's history, and delay injected
while it sits spawn-queued (runnable, not blocked) is owed and paid at
its first check, the semantics audit_zero_pct_window_absorbs_leftover_
debt pins. That test (born failing, unmasked by the run() propagation
fix) and the new actor_churn_between_experiments_forgives_nothing
regression test both go green.
2026-07-18 21:55:56 +00:00
Claude (sandbox) 7eae56a296 fix(runtime): a root actor panic escapes run()
The trampoline caught the root's panic, recorded it as Outcome::Panic on
the slot, and run() dropped the initial handle without reading it — every
assert inside run(), the standard test-suite pattern, was silently
vacuous (found live: a failing-first test passed). run() now reads the
root outcome before the handle drop and resume_unwinds the payload after
full teardown, so a caller's catch_unwind leaves the Runtime reusable;
Exit and Stopped return normally. The payload message is printed before
re-raising (the throw-site hook output was suppressed in-actor).

Correct the two tests this unmasked, both born failing and never run:
the select loser-arm test kept a closed arm in the set (the documented
closed-arm rule: a closed arm reports ready forever — observe the
disconnect and drop it); the send_after-to-dead test expected Ok(None)
from a closed+empty channel (documented: Err(RecvError), which proves
nothing-delivered even more strongly).
2026-07-18 21:52:50 +00:00
Claude (sandbox) 527f045e17 feat(causal): offcpu column + closed-books eff in the attrib probe
The probe now prints eff+offcpu next to eff — attributed plus the
counted runnable off-CPU gaps over ground-truth in-site time — the
per-window check that the located mechanism accounts for the whole
residual (~1.00 = books closed, no remaining silent loss). Audit line
gains the offcpu column, same delta-terms convention as the lib
renderer. Header doc rewritten from hypothesis to resolution.
2026-07-13 12:46:58 +00:00
Claude (sandbox) 0ee3fe7330 feat(causal): offcpu audit bucket — the @50 deficit located (RFC 007)
GPU sweep decomposed the ~23ms/700ms @50 injection deficit: every
ledger bucket is ~zero (drop park 0, discards 0, drop yield ~0.3ms),
books balance at absorbed+forgiven = 4x injected in all 48 cells, and
the 0%-cell contamination signature is absent. The probe pins the
residual: eff 0.933-0.943, a constant 22-27µs missing per site entry
= ~4.9 slice-expiry yields/entry x ~5.6µs runqueue wait. The
"deficit" is runnable off-CPU time inside the site — wall time the
probe's ground truth counts but on-CPU attribution correctly skips
(Coz model: speeding the site's code does not shrink queue-wait).

Measure-only reclassification, no behaviour change: a yield in the
target site stashes (tsc, experiment epoch) on the slot; the next
on_resume counts the gap into OFFCPU_IN_SITE_{CYCLES,N} (would-be
delta terms, MAX_SAMPLE_CYCLES-capped) iff the epoch still matches
and the word is live — a gap straddling end()/a same-word begin()
(live in the probe's 50,50 schedule) is dropped, never a leaked
cooldown. Parks excluded: blocked time is forgiveness territory.
New offcpu column in render_ledger_audit; LedgerCounters and
ExperimentResult grow the two fields. Fidelity footer now states
the on-CPU basis (deliberate wording change to the pinned summary;
the substring pin test still holds). +2 tests (counted gap; epoch
straddle) + render assert.
2026-07-13 12:46:15 +00:00
Claude (sandbox) 9bfeb2c6a2 feat(causal): ledger-audit output in the pipeline demo and attrib probe
- causal_pipeline: SMARM_CAUSAL_AUDIT=1 appends render_ledger_audit()
  after the summary; pinned summary format untouched.
- causal_attrib_probe: per-pct audit line (absorbed/forgiven/drops/
  discards) under the existing eff line, so in-site-vs-attributed and
  the loss buckets land in one place for the sweep.

1-core smoke (work + wide): books balance — absorbed = 2x injected and
forgiven = 2x injected, i.e. owed = injected x (N-1) with N=5 actors,
zero outstanding. drop park = 0 even in wide mode (the bottleneck's
queue is never empty, so its in-guard recv never parks); drop yield is
~1500 events but ~0.1ms per window, confirming slice-expiry yields
sample at their own checkpoint. Deficit decomposition needs the
parallel box.
2026-07-13 12:11:24 +00:00
Claude (sandbox) a3be8f0977 feat(causal): ledger audit — decompose the @50 injection deficit (RFC 007)
Measure-only counters for the deficit hunt (~23ms short per 700ms window
at 50% on the bottleneck site; superlinear vs 25%). Nothing here changes
injection or absorption; the sweep decides the fix.

Buckets, windowed per cell into new ExperimentResult fields (audit
snapshot taken at end() — spin/attribution freeze there, forgiveness
does not):
- spin_absorbed / park_forgiven: where owed delay actually went. Spin
  during a 0% cell is the baseline-contamination signature — leftover
  debt from a prior window being paid in a later one (checks gate on
  the experiment word, so cooldowns pay nothing and debt carries over).
- drop_park / drop_yield (+counts): the deschedule path flushes no
  sample tail — an in-target-site park or yield silently loses
  [last sample -> now]; on_resume re-arms before the actor runs again.
  New on_deschedule hook in all three intent arms (real park; explicit/
  slice-expiry yield; requeued park counts as yield — it never blocked).
  Slice-expiry yields sample at the descheduling checkpoint, so a fat
  yield bucket points at explicit yield_now or requeued parks.
- discard_overmax (+count, in would-be delta terms so columns compare
  against injected_cycles) / discard_unarmed: the attribute() clamps,
  previously silent.

LedgerCounters + ledger_counters() expose cumulative totals (tests,
run-level prints); render_ledger_audit() is the per-cell companion to
render_summary, which stays byte-identical (pinned). ExperimentResult
now derives Default so literals survive future audit-field growth.

Tests: +7 (spin counted, forgiveness counted, in-site park drop, in-site
yield drop, overmax discard, 0%-window leftover absorption — synthesized
deterministically via inject_delay_cycles_for_test with no experiment
active — and the audit render). 22/22 causal.
2026-07-13 12:07:19 +00:00
Claude (sandbox) a2d0b7af18 feat(causal): fidelity footer in render_summary
One unconditional footer line whenever there are results:
"note: impacts are lower bounds — undershoot grows with speedup pct;
rankings unaffected" — surfacing the RFC 007 Validation fidelity statement
where users actually look, instead of only in the RFC. Wording pinned by
the summary test.
2026-07-13 11:11:20 +00:00
Claude (sandbox) a4647f368a feat(causal): wall-anchored send_after — user-facing timer opt-out (RFC 007)
send_after_wall / send_after_named_wall (+ Timers::insert_send_wall) arm a
message-delivery timer that opts out of the RFC 007 virtual-time shift and
fires at its raw deadline regardless of injected delay — the Send-reason
sibling of sleep_wall, closing the jar item whose substrate efbc254 landed.
For deadlines that reflect the outside world (protocol timeouts, wall-clock
schedules) rather than workload pacing. cancel_timer is anchor-agnostic and
unchanged; without the feature the API exists and is identical to send_after.

The gen_server timer layer (send_after_to, RFC 015 §5) deliberately stays
virtual-only — an opt-out there means new options on the gen_server/statem
timeout API, out of scope for now.

Tests: wall send fires at raw deadline while a virtual sibling shifts;
cancel on a wall send with debt outstanding; featureless delivery/cancel
smokes through the public named API. 15/15 causal, 34/34 binaries both
feature configs, lib clippy clean both.
2026-07-13 11:10:11 +00:00
Claude (sandbox) fec760a3c0 docs(causal): record the reserve-shortfall verdict in the demo header
Occupancy probe on 24 cores: δ = 0.3µs/item (0.1% of the serialized
path); wide guard leaves the @50% cell unchanged. The +84-vs-+100
shortfall is controller-side (injected 327ms of the ideal 350ms over
the 700ms window, plus ~3% real-rate dip during experiments), not
unguarded stage time. urus's ~70µs/request remainder remains the
guard-placement case; the occupancy probe discriminates the two.
2026-07-13 09:48:24 +00:00
Claude (sandbox) d5b6a8f66f feat(causal): pipeline demo modes for the reserve-shortfall experiment
SMARM_CAUSAL_MODE selects the reserve stage's guard placement:
work (default, unchanged) | wide (guard over recv+work+send, the whole
serialized per-item path) | occupancy (no experiments; per-segment
timing of reserve's loop at baseline, reporting the unguarded
remainder δ and the impact ceiling it implies).

Discriminates the two candidate explanations for the demo's +84-vs-+100
@50% shortfall: physical recv/send time outside the guard (occupancy
sees δ≈30µs, wide recovers ~2x) vs. injection-side credit loss
(occupancy sees δ≈0, wide caps at ~+84 too — guard cadence identical).
2026-07-13 09:40:27 +00:00
Claude (sandbox) efbc254634 feat(causal): wall-anchored timers — controller windows keep fixed wall length (RFC 007)
New timer anchor: insert_sleep_wall / scheduler::sleep_wall (exported) opt a
Sleep entry out of the RFC 007 virtual-time shift, so it fires at its raw
deadline regardless of injected delay. Featureless config is unchanged (the
API exists but is identical to sleep).

The causal controller's window/cooldown sleeps and the tsc_hz calibration
sleep use it on the actor path (the OS-thread path was already wall). This
fixes the controller's own sleeps dilating under its own injection —
experiment windows stretched ~2x at 50% speedup (337ms -> 646ms injected/
window). Deltas were rate-normalized so results were unbiased; this fixes
sweep cost, not bias. The general wall-anchored-timer-semantics jar item
(user-facing opt-out) remains open; this lands the substrate.

Test: wall_timer_ignores_injected_delay — wall entry fires at raw deadline
while a virtual sibling in the same heap shifts. 13/13 causal, 34/34
binaries both feature configs.
2026-07-13 07:44:51 +00:00
Claude (sandbox) 04dbac1f4b feat(causal): timer-heap virtual time — deadlines chase injected delay (RFC 007)
Injected delays dilate virtual time for the workload, but timer deadlines
stayed wall-anchored: a sleep or receive-timeout fired early in virtual
terms, so timeout/retry behaviour sped up relative to the dilated world
(v1 known gap #1).

Every heap Entry now carries delay_stamp — the global delay ledger at
(re-)queue time, cfg-gated on smarm-causal. pop_due converts any debt
accrued since the stamp to wall time (tsc_hz) and shifts the effective
deadline; a not-yet-due entry is re-queued at the shifted deadline with a
fresh stamp, so it keeps chasing delay injected while it waits. seq is
preserved across re-queues, keeping send_after cancellation identity
intact (cancelled entries are discarded before any shift). Zero debt is
byte-identical to the old path; peek_deadline may under-report, costing
one spurious scheduler wake per injected chunk (documented).

This also makes the park-gated resume credit *correct* rather than
forgiving for sleepers: a sleeping actor now physically pays its debt by
sleeping longer, so the on_resume fast-forward reflects real payment
(sleeping_actor_pays_injected_delay pins this end-to-end through the
runtime).

New test hooks: inject_delay_cycles_for_test (deterministic ledger
driver, eagerly TSC-calibrating so conversion never stalls a scheduler
loop) and cycles_to_duration. The ledger is process-global, so the
delta-sensitive causal tests now serialize on a shared test mutex — they
were racy under the parallel test harness before this, in principle.
2026-07-13 07:31:08 +00:00
Claude (sandbox) d496914d40 fix(causal): flush target-site samples at guard boundaries (RFC 007)
Samples were taken only when maybe_preempt's cold block happened to fire
in-site, so the interval between the last check and SiteGuard drop was
discarded on every site entry. Measured live on the 24-core box:
22-29us lost per entry, a constant attribution efficiency of ~0.93-0.94,
which under-reported every impact (+83.5% where theory says +100%; the
observed shortfall fits 1/(1-pct*eff)-1 at both 25% and 50%).

SiteGuard enter/drop now call site_transition(): leaving the target site
flushes the pending interval into the ledger (sample-only, never spins,
so safe under no-preempt regions); entering the target site re-arms the
sample clock so pre-site time is never attributed (the symmetric
over-attribution). Winner attribution is factored into attribute(),
shared by the cold check and the flush, with the same interval clamps.

Adds examples/causal_attrib_probe.rs (ground-truth in-site time vs
ledger attribution, the probe that confirmed the leak) and the
site_boundaries_flush_tail regression test (a site entry that never
hits a cold check must still be attributed). Also gates causal_probe
on smarm-causal in Cargo.toml - it never was, so featureless builds
of the examples were broken.
2026-07-12 19:35:06 +00:00
Claude (sandbox) 2668f4018f feat(causal): native causal profiling behind smarm-causal (RFC 007 v1)
causal_site! scoped site guards per actor slot, progress! throughput
points, and a Coz-style virtual-speedup engine hooked into
maybe_preempt's amortized cold block: target-site samples grow a global
delay ledger; bystanders spin-absorb their debt at the next causal
check, with timeslice extension so injected delay is not charged
against the slice.

Resume credit (Coz's blocked-thread rule) is gated on a causal_parked
slot bit set only by a real park: crediting on every resume made any
yield-cadence actor delay-immune and every experiment inert (found
live on a 24-core run — dead-flat deltas across all sites).

Report normalization uses a measured TSC frequency (~50ms calibration
on first use) instead of the crate-wide 3 GHz assumption, which
uniformly inflated impact numbers on a 3.7 GHz box. impact_pct() is
the machine-readable form of the summary for programmatic checks.

examples/causal_pipeline.rs burns fixed *work* (calibrated LCG loop),
not fixed wall time — a timed busy-wait absorbs injected delay into
its own budget and reads as a no-op. Self-checking: exits nonzero if
causal separation fails; skips the verdict below 4 cores. Validated
on a 24-core box: reserve (true bottleneck) +29.3%@25/+83.5%@50;
serialize and background-compaction ~0%.

Known v1 gaps (jar): timer-heap deadlines unshifted, no-check!/no-alloc
actors undelayable, multi-scheduler coherence best-effort Relaxed,
off-CPU blame punted, Instant::now() uncorrected.

Zero-cost with the feature off; clippy -D warnings clean both ways;
full suite green with and without smarm-causal.
2026-07-12 19:10:04 +00:00
smarm-agent 1c90a4ef5e fix(registry): by_name stores the full holder Pid — a dead name heals under slot reuse
Root cause of soak20 signature 2 (refcount_test.exs 'watcher crash',
110235x fast {:error, :server_down} probes over the full await window):
by_name mapped name -> slot *index*, so a name whose holder died (no stop
path unregisters; prune is lazy) and whose slot was then re-tenanted read
as live-held: register failed NameTaken{holder: <unrelated tenant>} (which
the bridge macro's generated start() swallows -> start_server/1 reports :ok
for a server that never came up), while name resolution reached the
tenant's mailbox, missed on the message TypeId and failed fast WITHOUT
pruning — the wedge self-sustained for the tenant's lifetime. Name-
addressed send additionally judged liveness on the slot's *current*
mailbox pid, so a same-typed tenant would have received the message
(misdelivery) and a differently typed one a misleading NoChannel.

Fix: by_name: HashMap<&'static str, Pid> — every reader judges the
*stored* holder with the generation-checked live(), so a recycled slot's
tenant no longer impersonates a dead holder, and every touch (register /
whereis / resolve / send) prunes and heals a stale name. prune(index)
becomes prune_holder(pid): names bound to the holder go; the mailbox goes
only while still the holder's own (a tenant's replacement mailbox is left
untouched). Introspection matches names to mailboxes by full pid, so a
stale name never annotates a slot's new tenant.

Deterministic regression test added first and shown to fail pre-fix
(tests/stale_name_slot_reuse.rs: tiny slab forces re-tenanting; old slot
(1,0) died, tenant (1,1) took the index; register -> NameTaken pre-fix).
Post-fix it asserts the healed contract: whereis -> None (pruned), call ->
ServerDown, re-register -> Ok. Suite 33 ok-binaries, clippy gate clean.

In the wild the window opened at every splice_test teardown:
Splice.terminate -> exit_server('subtree') left the name bound; width 20
raised the re-tenant probability. Downstream (smarm_beam): install_child's
unregister-before-register workaround becomes dead code (removed there);
the #[smarm_server] macro's swallowed register error becomes truthful
idempotency (a NameTaken now really is a live holder).
2026-07-12 07:23:30 +00:00
smarm-agent f6641cd266 runtime: name scheduler threads smarm-sched-{slot}
Extra scheduler threads (slots 1..N-1) are now spawned via thread::Builder
with the name smarm-sched-{slot}, so they are identifiable in
/proc/<pid>/task/*/comm, stack dumps and debuggers. Thread 0 keeps its
caller-given name (an embedder names the thread that calls run — smarm_beam
names it smarm-runtime). A refused spawn still panics, matching the previous
thread::spawn semantics.

Motivation: the §17 scheduler-width knob in smarm_beam asserts the *live*
width by counting these named threads, and a 20-scheduler soak needs the
threads tellable apart in wedge captures.
2026-07-11 19:17:52 +00:00
smarm-agent 0017c5b9a1 fix(runtime): consume wake-pipe bytes only under the drain lock
Lost-wakeup: schedule_loop's phase-1 drain uses drain_lock.try_lock(), and
try_lock losers skip the completion drain entirely. Both schedulers park on
one shared wake pipe and, until now, drained ALL its bytes right after their
idle poll_wake returned — outside the drain lock. A loser could therefore
eat the byte announcing a completion the winner had not seen (the winner was
already past drain_completions when the epoll thread pushed it), and both
threads would park with the completion stranded. Because the bridge eventfd
is registered EPOLLONESHOT, the kernel had already disarmed it at
epoll_wait, so no later write could re-fire it: the runtime slept until an
unrelated timer deadline forced another phase-1 pass.

Fix: drain_wake_pipe() moves inside the drain guard, immediately before
drain_completions(); the two post-poll drains in the Pop::Idle arms are
removed. Producers push their completion before writing the byte, so a byte
consumed under the guard always has its completion visible to the drain that
follows. An unconsumed byte keeps the (level-triggered) idle poll returning
instantly, so a try_lock loser spins briefly until the winner releases —
it can no longer sleep through stranded work.

Found via smarm_beam's ingress-cap drain barrier flaking under CPU load
(5/25 loaded suite runs wedged; mid-wedge stacks showed both schedulers in
poll_wake with an FdReady stranded and the eventfd disarmed). Post-fix:
60/60 loaded runs green, tight 5.8-6.8s timing band, no stall tail.
Root-cause notes: smarm_beam outputs/flake-rootcause-egress-overload.md.
2026-07-11 16:13:46 +00:00
smarm-agent 6c2b7e91cf channel: drop queued messages when the Receiver drops
A queued Envelope::Call was stranded until the last Sender dropped, so a
caller parked in gen_server::call was never released with ServerDown when a
*named* server was request_stop'd — the registry's inbox Sender clone (lazy
prune) kept the channel Arc, and the queued reply_tx, alive indefinitely.

Receiver::Drop now drains the queue (items dropped after releasing the lock,
since a reply_tx drop reaches a different channel's lock + the scheduler),
restoring the documented ServerDown guarantee on every teardown path.

Adds tests/stop_with_queued_call.rs: deterministic pure-smarm reproducer.
2026-06-24 20:53:27 +00:00
27 changed files with 6310 additions and 1180 deletions
+17 -1
View File
@@ -2,7 +2,23 @@
# smarm pre-commit gate: clippy the library (src/) with warnings as errors. # smarm pre-commit gate: clippy the library (src/) with warnings as errors.
# unwrap_used / expect_used are denied (Cargo.toml [lints.clippy]): library # unwrap_used / expect_used are denied (Cargo.toml [lints.clippy]): library
# code must not hide a panic behind unwrap/expect. Tests/examples are not gated. # code must not hide a panic behind unwrap/expect. Tests/examples are not gated.
#
# Toolchain resolution: prefer an installed cargo-clippy; on machines whose
# rust comes without the clippy component (e.g. NixOS home-manager), fall
# back to an ephemeral nix-shell toolchain. The fallback uses its own target
# dir (target/clippy) because the shell's rustc version may differ from the
# default toolchain's — mixed-compiler artifacts in one target dir are an
# E0514 hard error. MSRV (Cargo.toml rust-version) keeps the older shell
# toolchain a legitimate gate.
set -eu set -eu
[ -f "$HOME/.cargo/env" ] && . "$HOME/.cargo/env" [ -f "$HOME/.cargo/env" ] && . "$HOME/.cargo/env"
cd "$(git rev-parse --show-toplevel)" cd "$(git rev-parse --show-toplevel)"
cargo clippy --lib -- -D warnings if cargo clippy --version >/dev/null 2>&1; then
cargo clippy --lib -- -D warnings
elif command -v nix-shell >/dev/null 2>&1; then
nix-shell -p clippy -p cargo -p rustc \
--run 'CARGO_TARGET_DIR=target/clippy cargo clippy --lib -- -D warnings'
else
echo "pre-commit: cargo clippy unavailable and no nix-shell fallback" >&2
exit 1
fi
+1
View File
@@ -4,3 +4,4 @@ smarm_trace.json
/bench_results/ /bench_results/
__pycache__/ __pycache__/
*.pyc *.pyc
profile.coz
+16
View File
@@ -19,6 +19,10 @@ expect_used = "deny"
[features] [features]
default = ["rq-mutex"] default = ["rq-mutex"]
smarm-trace = [] smarm-trace = []
# RFC 007: native causal profiling. Zero cost when off (cf. smarm-trace): the
# hook in `maybe_preempt` and the resume-path fast-forward compile away; the
# two Slot ledger fields exist regardless and stay 0 (budget_cycles precedent).
smarm-causal = []
# RFC 016 Chunk 2: cycle-accurate per-actor time-budget accounting. Off by # RFC 016 Chunk 2: cycle-accurate per-actor time-budget accounting. Off by
# default — it costs two extra RDTSC reads per actor resume on the hot path # default — it costs two extra RDTSC reads per actor resume on the hot path
# (D6). The `ActorInfo.budget_cycles` field exists regardless; it just stays 0 # (D6). The `ActorInfo.budget_cycles` field exists regardless; it just stays 0
@@ -89,3 +93,15 @@ harness = false
[[example]] [[example]]
name = "observer" name = "observer"
required-features = ["observer"] required-features = ["observer"]
[[example]]
name = "causal_pipeline"
required-features = ["smarm-causal"]
[[example]]
name = "causal_attrib_probe"
required-features = ["smarm-causal"]
[[example]]
name = "causal_probe"
required-features = ["smarm-causal"]
+198
View File
@@ -0,0 +1,198 @@
//! Attribution-efficiency probe (RFC 007 follow-up).
//!
//! Original hypothesis: the ~4pt impact shortfall on the 24-core
//! validation (+29.3/+83.5 vs theoretical +33/+100) is a constant
//! attribution efficiency eff ≈ 0.91 from site-exit tail truncation.
//! The guard-drop flush closed that leak, yet eff held at ~0.93 —
//! RESOLVED (2026-07-13 sweep): the residual is runnable off-CPU time
//! inside the site (~4.9 slice-expiry yields/entry x ~5.6µs runqueue
//! wait), wall time the ground truth below counts but on-CPU
//! attribution correctly skips. The offcpu audit bucket now counts it;
//! `eff+offcpu` printed per window should sit at ~1.00 — the
//! closed-books check.
//!
//! Measurement: same pipeline as `causal_pipeline`, but the `reserve`
//! actor also measures its raw in-site time directly (rdtsc at guard
//! enter/exit) and counts site entries. For each experiment window at
//! pct%:
//!
//! eff = (Δglobal_delay / (pct/100)) / Δin_site_cycles
//!
//! and the missing time per site entry localizes the leak:
//!
//! tail_us/entry = (Δin_site Δglobal_delay/(pct/100)) / Δentries
//!
//! A constant eff across 25/50% with tail/entry in the tens of µs
//! localizes a per-entry mechanism; `eff+offcpu` ≈ 1.00 confirms the
//! runnable-gap account and rules out any remaining silent loss.
//!
//! Run: cargo run --release --example causal_attrib_probe --features smarm-causal
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
fn rdtsc() -> u64 {
// x86_64 only — same clock the ledger uses.
unsafe { core::arch::x86_64::_rdtsc() }
}
/// Same fixed-work loop as causal_pipeline (dependent LCG, preemptible).
fn work_iters(iters: u64) {
let mut acc = 0x2545_f491_4f6c_dd1du64;
let mut i = 0u64;
while i < iters {
let chunk_end = (i + 256).min(iters);
while i < chunk_end {
acc = acc.wrapping_mul(6364136223846793005).wrapping_add(i);
i += 1;
}
std::hint::black_box(acc);
smarm::check!();
}
}
fn calibrate_iters_per_us() -> u64 {
let n = 8_000_000u64;
let t = Instant::now();
work_iters(n);
(n / (t.elapsed().as_micros().max(1) as u64)).max(1)
}
static IN_SITE_CYCLES: AtomicU64 = AtomicU64::new(0);
static SITE_ENTRIES: AtomicU64 = AtomicU64::new(0);
fn main() {
let per_us = calibrate_iters_per_us();
println!("calibration: {per_us} work iters/µs");
let work_us = move |us: u64| work_iters(us * per_us);
let cores = std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1);
println!("cores: {cores}");
if cores < 4 {
println!("probe: SKIPPED (needs the stages in parallel)");
return;
}
smarm::init(smarm::Config::default()).run(move || {
let stop = Arc::new(AtomicBool::new(false));
let (tx_ab, rx_ab) = smarm::channel::<u64>();
let (tx_bc, rx_bc) = smarm::channel::<u64>();
let stop_p = stop.clone();
let producer = smarm::spawn(move || {
let mut i = 0u64;
while !stop_p.load(Ordering::Relaxed) {
{
let _g = smarm::causal_site!("serialize");
work_us(200);
}
if tx_ab.send(i).is_err() {
break;
}
i += 1;
}
});
// Reserve: the target — instrumented with ground-truth in-site time.
let reserve = smarm::spawn(move || {
while let Ok(item) = rx_ab.recv() {
{
let t0 = rdtsc();
let _g = smarm::causal_site!("reserve");
work_us(400);
// Measured before guard drop: exactly the span the
// ledger should be attributing.
IN_SITE_CYCLES.fetch_add(rdtsc().saturating_sub(t0), Ordering::Relaxed);
SITE_ENTRIES.fetch_add(1, Ordering::Relaxed);
}
if tx_bc.send(item).is_err() {
break;
}
}
});
let notify = smarm::spawn(move || {
while rx_bc.recv().is_ok() {
{
let _g = smarm::causal_site!("notify");
work_us(50);
}
smarm::progress!("orders-processed");
}
});
let stop_bg = stop.clone();
let background = smarm::spawn(move || {
while !stop_bg.load(Ordering::Relaxed) {
let _g = smarm::causal_site!("background-compaction");
work_us(500);
}
});
smarm::sleep(Duration::from_millis(300));
let hz = smarm::causal::tsc_hz();
println!("tsc_hz: {:.3} GHz", hz / 1e9);
// Manual windows so ledger/ground-truth snapshots align exactly.
for &pct in &[25u32, 50, 50, 25] {
let g0 = smarm::causal::global_delay_cycles();
let s0 = IN_SITE_CYCLES.load(Ordering::Relaxed);
let e0 = SITE_ENTRIES.load(Ordering::Relaxed);
let a0 = smarm::causal::ledger_counters();
smarm::causal::begin_experiment_for_test("reserve", pct);
smarm::sleep(Duration::from_millis(1000));
smarm::causal::end_experiment_for_test();
let audit = smarm::causal::ledger_counters().delta_since(&a0);
let injected = smarm::causal::global_delay_cycles() - g0;
let in_site = IN_SITE_CYCLES.load(Ordering::Relaxed) - s0;
let entries = SITE_ENTRIES.load(Ordering::Relaxed) - e0;
let attributed = injected as f64 / (pct as f64 / 100.0);
let eff = attributed / in_site as f64;
// Books-closure check: add back the runnable off-CPU gaps the
// audit counted (delta terms -> raw via /pct) — should be ~1.00.
let eff_closed = (injected as f64 + audit.offcpu_in_site_cycles as f64)
/ (pct as f64 / 100.0)
/ in_site as f64;
let missing = in_site as f64 - attributed;
let tail_us = if entries > 0 {
missing / entries as f64 / hz * 1e6
} else {
f64::NAN
};
println!(
"pct {pct:>2}% in_site {:>8.1}ms attributed {:>8.1}ms eff {eff:.3} eff+offcpu {eff_closed:.3} entries {entries} missing/entry {tail_us:.1}µs",
in_site as f64 / hz * 1e3,
attributed / hz * 1e3,
);
// RFC 007 deficit hunt: name the losses. Drop/discard columns are
// in would-be delta terms — divide by pct/100 to compare with the
// missing attribution above.
let ms = |c: u64| c as f64 / hz * 1e3;
println!(
" audit: absorbed {:>7.1}ms forgiven {:>6.1}ms drop park {:>5.2}ms/{:<5} yield {:>5.2}ms/{:<5} offcpu {:>6.2}ms/{:<5} discard >max {:>5.2}ms/{:<3} unarmed {}",
ms(audit.spin_absorbed_cycles),
ms(audit.park_forgiven_cycles),
ms(audit.drop_park_cycles),
audit.drop_park_n,
ms(audit.drop_yield_cycles),
audit.drop_yield_n,
ms(audit.offcpu_in_site_cycles),
audit.offcpu_in_site_n,
ms(audit.discard_overmax_cycles),
audit.discard_overmax_n,
audit.discard_unarmed_n
);
smarm::sleep(Duration::from_millis(150));
}
stop.store(true, Ordering::Relaxed);
producer.join().unwrap();
reserve.join().unwrap();
notify.join().unwrap();
background.join().unwrap();
println!("probe: DONE");
});
}
+296
View File
@@ -0,0 +1,296 @@
//! Causal-profiling demo (RFC 007): a pipeline where conventional profiling
//! lies and causal profiling doesn't.
//!
//! producer --(serialize ~200µs/item)--> reserve --(~400µs/item)--> notify
//! background: an actor burning CPU constantly, fully off the critical path
//!
//! `reserve` is the true bottleneck. `serialize` is hot but overlapped with
//! `reserve`'s backlog, and `background` is the hottest code in the process
//! while contributing nothing to throughput. A cycle profiler ranks them
//! background > reserve ≈ 2×serialize; the causal report instead shows
//! throughput responding to virtual speedups of `reserve` and (near-)ignoring
//! `serialize` and `background`.
//!
//! Stage cost is fixed *work* (a calibrated arithmetic loop), not fixed wall
//! time. This matters: a timed busy-wait absorbs injected causal delay into
//! its own budget and finishes on schedule regardless, making every
//! experiment read as a no-op (found live on a 24-core run: dead-flat
//! deltas). Real workloads are work-shaped, so the demo must be too.
//!
//! Run:
//! cargo run --release --example causal_pipeline --features smarm-causal
//!
//! Modes (`SMARM_CAUSAL_MODE`), for probing what the guard placement leaves
//! out of the measurement (a site speeds up only what it wraps; `recv`/`send`
//! on the serialized stage sit outside the canonical guard):
//! work (default) — guard wraps only the 400µs of work.
//! wide — guard widened over recv + work + send, the whole
//! serialized per-item path.
//! occupancy — no experiments; times each segment of reserve's loop
//! at baseline and reports the unguarded per-item
//! overhead δ plus the impact ceiling it implies.
//!
//! Result (24-core run, 2026-07-13, job f9305cbb): δ measured 0.3µs/item —
//! 0.1% of the serialized path — and `wide` does not move the @50% cell
//! (+83.5/+86.3 vs work's +81.5/+86.7). This demo's +84-vs-+100 @50%
//! shortfall is therefore NOT unguarded stage time; it is controller-side:
//! injected delay reaches ~327ms of the ideal 350ms over the 700ms window,
//! plus a ~3% real-throughput dip while experiments run. Contrast urus's
//! causal_bench, where the same arithmetic identified a real ~70µs/request
//! unguarded remainder (recv/reply outside the store guard). Sites measure
//! what they wrap — and the occupancy probe tells you which case you're in.
//!
//! Prints a summary, writes `profile.coz` (Coz plot-compatible), and — given
//! enough cores for the pipeline to actually run in parallel — checks the
//! expected separation and exits nonzero if it doesn't hold, so a CI box can
//! run this as a smoke test.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
/// LCG-mix `iters` times in dependent sequence (unvectorizable, un-elidable),
/// staying preemptible — and causal-sampleable/delayable — via `check!()`.
fn work_iters(iters: u64) {
let mut acc = 0x2545_f491_4f6c_dd1du64;
let mut i = 0u64;
while i < iters {
let chunk_end = (i + 256).min(iters);
while i < chunk_end {
acc = acc.wrapping_mul(6364136223846793005).wrapping_add(i);
i += 1;
}
std::hint::black_box(acc);
smarm::check!();
}
}
/// Measure how many `work_iters` iterations fit in a microsecond on this
/// machine, so stage costs below are meaningful in time while staying
/// work-shaped.
fn calibrate_iters_per_us() -> u64 {
let n = 8_000_000u64;
let t = Instant::now();
work_iters(n);
(n / (t.elapsed().as_micros().max(1) as u64)).max(1)
}
/// Guard placement for the `reserve` stage — see module doc.
#[derive(Clone, Copy, PartialEq)]
enum Mode {
Work,
Wide,
Occupancy,
}
fn main() {
let mode = match std::env::var("SMARM_CAUSAL_MODE").as_deref() {
Err(_) | Ok("") | Ok("work") => Mode::Work,
Ok("wide") => Mode::Wide,
Ok("occupancy") => Mode::Occupancy,
Ok(other) => {
eprintln!("unknown SMARM_CAUSAL_MODE {other:?} (work|wide|occupancy)");
std::process::exit(2);
}
};
println!(
"mode: {}",
match mode {
Mode::Work => "work",
Mode::Wide => "wide",
Mode::Occupancy => "occupancy",
}
);
let per_us = calibrate_iters_per_us();
println!("calibration: {per_us} work iters/µs");
let work_us = move |us: u64| work_iters(us * per_us);
let mut failures: Vec<String> = Vec::new();
smarm::init(smarm::Config::default()).run(move || {
let stop = Arc::new(AtomicBool::new(false));
let (tx_ab, rx_ab) = smarm::channel::<u64>();
let (tx_bc, rx_bc) = smarm::channel::<u64>();
// Producer: hot serialization, but upstream of the bottleneck.
let stop_p = stop.clone();
let producer = smarm::spawn(move || {
let mut i = 0u64;
while !stop_p.load(Ordering::Relaxed) {
{
let _g = smarm::causal_site!("serialize");
work_us(200);
}
if tx_ab.send(i).is_err() {
break;
}
i += 1;
}
// tx_ab drops here; downstream drains and exits.
});
// Reserve: the true bottleneck (~400µs of work per item).
let reserve = smarm::spawn(move || match mode {
Mode::Work => {
while let Ok(item) = rx_ab.recv() {
{
let _g = smarm::causal_site!("reserve");
work_us(400);
}
if tx_bc.send(item).is_err() {
break;
}
}
}
// Whole serialized per-item path under the guard: a virtual
// speedup now also compresses recv/send, so the @50% cell should
// recover the theoretical 2× that `work` mode's placement caps.
Mode::Wide => loop {
let _g = smarm::causal_site!("reserve");
let Ok(item) = rx_ab.recv() else { break };
work_us(400);
if tx_bc.send(item).is_err() {
break;
}
},
// Time each segment at baseline; the recv+send remainder δ is
// the serialized time a `work`-placed guard cannot speed up.
Mode::Occupancy => {
let (mut recv_ns, mut work_ns, mut send_ns, mut n) = (0u64, 0u64, 0u64, 0u64);
loop {
let t0 = Instant::now();
let Ok(item) = rx_ab.recv() else { break };
let t1 = Instant::now();
{
let _g = smarm::causal_site!("reserve");
work_us(400);
}
let t2 = Instant::now();
if tx_bc.send(item).is_err() {
break;
}
recv_ns += (t1 - t0).as_nanos() as u64;
work_ns += (t2 - t1).as_nanos() as u64;
send_ns += t2.elapsed().as_nanos() as u64;
n += 1;
}
let items = n.max(1) as f64;
let (r, w, s) = (
recv_ns as f64 / items / 1e3,
work_ns as f64 / items / 1e3,
send_ns as f64 / items / 1e3,
);
let delta = r + s;
let total = w + delta;
println!("occupancy: {n} items; per item recv {r:.1}µs + work(guarded) {w:.1}µs + send {s:.1}µs");
println!(
"occupancy: unguarded δ = {delta:.1}µs/item = {:.1}% of the serialized path",
100.0 * delta / total
);
for pct in [25u32, 50] {
let f = 1.0 - f64::from(pct) / 100.0;
println!(
"occupancy: predicted reserve impact @{pct}% -> {:+.1}% (ceiling if δ were guarded: {:+.1}%)",
100.0 * (total / (f * w + delta) - 1.0),
100.0 * (1.0 / f - 1.0)
);
}
}
});
// Notify: light tail stage; marks the unit of useful work.
let notify = smarm::spawn(move || {
while rx_bc.recv().is_ok() {
{
let _g = smarm::causal_site!("notify");
work_us(50);
}
smarm::progress!("orders-processed");
}
});
// Background: hottest code in the process, zero throughput relevance.
let stop_bg = stop.clone();
let background = smarm::spawn(move || {
while !stop_bg.load(Ordering::Relaxed) {
let _g = smarm::causal_site!("background-compaction");
work_us(500);
}
});
// Warm up so queues reach steady state before measuring.
smarm::sleep(Duration::from_millis(300));
if mode == Mode::Occupancy {
// No experiments: hold steady state for a window, then drain and
// let the reserve actor print its segment report.
smarm::sleep(Duration::from_millis(1500));
stop.store(true, Ordering::Relaxed);
producer.join().unwrap();
reserve.join().unwrap();
notify.join().unwrap();
background.join().unwrap();
return;
}
let results = smarm::causal::run_experiments(&smarm::causal::ExperimentPlan {
speedups_pct: vec![0, 25, 50],
experiment: Duration::from_millis(700),
cooldown: Duration::from_millis(150),
});
stop.store(true, Ordering::Relaxed);
producer.join().unwrap();
reserve.join().unwrap();
notify.join().unwrap();
background.join().unwrap();
print!("{}", smarm::causal::render_summary(&results));
// RFC 007 deficit hunt: SMARM_CAUSAL_AUDIT=1 appends the per-cell
// ledger audit (injected/absorbed/forgiven + drop and discard
// buckets) without touching the pinned summary format.
if std::env::var_os("SMARM_CAUSAL_AUDIT").is_some() {
print!("{}", smarm::causal::render_ledger_audit(&results));
}
let coz = smarm::causal::render_coz(&results);
match std::fs::write("profile.coz", coz) {
Ok(()) => println!("\nwrote profile.coz"),
Err(e) => eprintln!("\nfailed to write profile.coz: {e}"),
}
// Verdict. The separation only exists when the four pipeline actors
// actually run in parallel; on a small box, report and skip.
let cores = std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1);
if cores < 4 {
println!("verdict: SKIPPED ({cores} cores; separation needs the stages in parallel)");
return;
}
let impact = |site: &str| {
smarm::causal::impact_pct(&results, site, 25, "orders-processed")
};
let mut expect = |site: &str, ok: &dyn Fn(f64) -> bool, want: &str| match impact(site) {
Some(p) => {
let verdict = if ok(p) { "ok" } else { "FAIL" };
println!("verdict: {site} @25% -> {p:+.1}% (want {want}) {verdict}");
if !ok(p) {
failures.push(format!("{site}: {p:+.1}% (want {want})"));
}
}
None => {
println!("verdict: {site} @25% -> missing cell FAIL");
failures.push(format!("{site}: missing cell"));
}
};
expect("reserve", &|p| p > 15.0, "> +15%");
expect("serialize", &|p| p < 10.0, "< +10%");
expect("background-compaction", &|p| p < 10.0, "< +10%");
if failures.is_empty() {
println!("verdict: PASS — causal separation holds");
} else {
println!("verdict: FAIL — {}", failures.join("; "));
std::process::exit(1);
}
});
}
+145
View File
@@ -0,0 +1,145 @@
//! Diagnostic probe for RFC 007 on a target box. Measures, in order:
//! 1. TSC frequency against `Instant` (the crate assumes 3 GHz).
//! 2. TSC sanity under actor migration: distribution of wall time actually
//! spent in `burn_us(400)` across many runs — a bimodal/short tail means
//! cross-core TSC offsets are cutting burns short.
//! 3. Pipeline stage rates with no experiment running (who is the real
//! bottleneck?).
//! 4. The same rates during a 50% experiment on `background-compaction`
//! (a correct implementation must slow every stage; an off-critical-path
//! target must reduce end-to-end throughput proportionally).
//!
//! Run: cargo run --release --example causal_probe --features smarm-causal
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
fn burn_us(us: u64) {
let cycles = us * 3_000;
let start = smarm::preempt::rdtsc();
while smarm::preempt::rdtsc().saturating_sub(start) < cycles {
smarm::check!();
}
}
fn main() {
// 1. TSC calibration (plain OS thread, before the runtime starts).
let c0 = smarm::preempt::rdtsc();
let t0 = Instant::now();
std::thread::sleep(Duration::from_millis(200));
let hz = (smarm::preempt::rdtsc() - c0) as f64 / t0.elapsed().as_secs_f64();
println!("tsc_hz: {:.3e} (crate assumes 3.0e9)", hz);
smarm::init(smarm::Config::default()).run(move || {
// 2. burn_us(400) wall-time distribution inside a migrating actor.
let h = smarm::spawn(|| {
let mut samples: Vec<u64> = (0..500)
.map(|_| {
let t = Instant::now();
burn_us(400);
t.elapsed().as_micros() as u64
})
.collect();
samples.sort_unstable();
println!(
"burn_us(400) wall us: min {} p10 {} p50 {} p90 {} max {}",
samples[0], samples[50], samples[250], samples[450], samples[499]
);
});
h.join().unwrap();
// 3+4. Pipeline with per-stage counters.
let stop = Arc::new(AtomicBool::new(false));
let produced = Arc::new(AtomicU64::new(0));
let reserved = Arc::new(AtomicU64::new(0));
let notified = Arc::new(AtomicU64::new(0));
let (tx_ab, rx_ab) = smarm::channel::<u64>();
let (tx_bc, rx_bc) = smarm::channel::<u64>();
let stop_p = stop.clone();
let produced2 = produced.clone();
let producer = smarm::spawn(move || {
let mut i = 0u64;
while !stop_p.load(Ordering::Relaxed) {
{
let _g = smarm::causal_site!("serialize");
burn_us(200);
}
if tx_ab.send(i).is_err() {
break;
}
produced2.fetch_add(1, Ordering::Relaxed);
i += 1;
}
});
let reserved2 = reserved.clone();
let reserve = smarm::spawn(move || {
while let Ok(item) = rx_ab.recv() {
{
let _g = smarm::causal_site!("reserve");
burn_us(400);
}
reserved2.fetch_add(1, Ordering::Relaxed);
if tx_bc.send(item).is_err() {
break;
}
}
});
let notified2 = notified.clone();
let notify = smarm::spawn(move || {
while rx_bc.recv().is_ok() {
{
let _g = smarm::causal_site!("notify");
burn_us(50);
}
notified2.fetch_add(1, Ordering::Relaxed);
smarm::progress!("orders-processed");
}
});
let stop_bg = stop.clone();
let background = smarm::spawn(move || {
while !stop_bg.load(Ordering::Relaxed) {
let _g = smarm::causal_site!("background-compaction");
burn_us(500);
}
});
smarm::sleep(Duration::from_millis(300));
let window = |label: &str| {
let (p0, r0, n0) = (
produced.load(Ordering::Relaxed),
reserved.load(Ordering::Relaxed),
notified.load(Ordering::Relaxed),
);
let d0 = smarm::causal::global_delay_cycles();
let t = Instant::now();
smarm::sleep(Duration::from_millis(700));
let secs = t.elapsed().as_secs_f64();
println!(
"{label}: produced {:.0}/s reserved {:.0}/s notified {:.0}/s injected {:.0}ms(assumed-3GHz)",
(produced.load(Ordering::Relaxed) - p0) as f64 / secs,
(reserved.load(Ordering::Relaxed) - r0) as f64 / secs,
(notified.load(Ordering::Relaxed) - n0) as f64 / secs,
(smarm::causal::global_delay_cycles() - d0) as f64 / 3.0e9 * 1e3,
);
};
window("no-experiment ");
smarm::causal::begin_experiment_for_test("background-compaction", 50);
window("bg-comp @ 50% ");
smarm::causal::end_experiment_for_test();
window("post-experiment");
stop.store(true, Ordering::Relaxed);
producer.join().unwrap();
reserve.join().unwrap();
notify.join().unwrap();
background.join().unwrap();
});
}
+959
View File
@@ -0,0 +1,959 @@
//! Native causal profiling (RFC 007). Enabled by `--features smarm-causal`;
//! zero cost without it (same discipline as `smarm-trace`).
//!
//! The Coz algorithm, transposed onto actors: to estimate what speeding up
//! code site S by p% would do to throughput, we instead *slow everything
//! else down* by p% of the time spent in S, and watch the progress-point
//! rates respond. Where Coz must inject real `usleep`s into OS threads from
//! the outside, smarm owns every clock that matters:
//!
//! - Sampling and delay injection happen at `maybe_preempt`'s amortised
//! cadence — an existing, safe hook (never inside a prep-to-park region).
//! - Injected delay is subtracted from the actor's timeslice
//! (`preempt::extend_timeslice`), so experiments don't perturb scheduling.
//! - Delay bookkeeping is *actor*-granular: each `Slot` carries an absorbed-
//! delay ledger, compared against a global ledger. Parked actors absorb
//! accrued delay for free on resume (Coz's blocked-thread rule) — waiting
//! is never penalised.
//!
//! v1 scope (per RFC discussion): explicit scoped sites (`causal_site!`)
//! rather than PC sampling (jar Q1 stays open); throughput progress points
//! only; timer-heap deadlines are *not* shifted (documented gap — long
//! experiments can make real-time timeouts fire early in virtual terms);
//! multi-scheduler coherence is best-effort via global atomics.
//!
//! Usage:
//! ```ignore
//! let _g = smarm::causal_site!("inventory-reserve"); // in suspect code
//! smarm::progress!("orders-processed"); // per unit of work
//! let results = smarm::causal::run_experiments(&Default::default());
//! print!("{}", smarm::causal::render_summary(&results));
//! std::fs::write("profile.coz", smarm::causal::render_coz(&results))?;
//! ```
#[cfg(feature = "smarm-causal")]
mod inner {
use crate::preempt;
use std::cell::Cell;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Mutex, OnceLock};
use std::time::{Duration, Instant};
// -----------------------------------------------------------------------
// Global state
// -----------------------------------------------------------------------
/// Active experiment, packed `(site_id << 32) | speedup_pct`. 0 = idle.
/// A single word so the hot path reads one atomic; experiments are global
/// across scheduler threads (jar Q7, v1: plain Relaxed atomics).
static EXPERIMENT: AtomicU64 = AtomicU64::new(0);
/// Monotone experiment-window counter, bumped by every `begin()`. Lets
/// the offcpu-gap stash (RFC 007) tell apart two windows with an
/// identical site+pct word — live in the attrib probe's 50,50 schedule
/// — so a gap straddling `end()`/`begin()` never counts a cooldown.
static EXPERIMENT_EPOCH: AtomicU64 = AtomicU64::new(0);
/// Global virtual-delay ledger, in TSC cycles: the total delay every
/// actor *should* have experienced since startup. Grows while a sample
/// lands in the experiment's target site; each actor's `Slot` ledger
/// chases it by spin-absorbing at preemption checks.
static GLOBAL_DELAY: AtomicU64 = AtomicU64::new(0);
/// Registered site names; site id = index + 1 (0 = "no site").
static SITES: OnceLock<Mutex<Vec<&'static str>>> = OnceLock::new();
/// Registered progress points (leaked for `'static`, like trace's drain
/// state — the set is small and lives for the process).
static PROGRESS: OnceLock<Mutex<Vec<&'static ProgressPoint>>> = OnceLock::new();
thread_local! {
/// TSC at this thread's previous causal check, the sample "period"
/// denominator. Re-armed on every actor resume so scheduler time and
/// a previous actor's tail never count toward a sample. 0 = unarmed.
static LAST_SAMPLE_TSC: Cell<u64> = const { Cell::new(0) };
}
/// Guard against TSC weirdness (migration between unsynced sockets,
/// virtualisation steps): a single sample interval larger than this is
/// discarded rather than believed. ~33ms at 3 GHz — far beyond any real
/// gap between preemption checks inside a slice.
const MAX_SAMPLE_CYCLES: u64 = 100_000_000;
/// Cap on delay spun in one visit, so one check can never wedge an actor
/// for a human-visible pause; the remainder is absorbed on later visits.
/// ~3ms at 3 GHz.
const MAX_SPIN_PER_VISIT: u64 = 10_000_000;
// -----------------------------------------------------------------------
// Ledger audit (RFC 007 deficit hunt): where injected delay is born,
// paid, and forgiven — and where would-be attribution is silently lost
// (deschedule tails, clamp discards). Monotone Relaxed totals, read via
// `ledger_counters()`; `run_experiments` windows them into
// `ExperimentResult`. Measure-only: nothing here changes injection or
// absorption behaviour.
// -----------------------------------------------------------------------
/// Cycles bystanders actually spun to pay down the global ledger.
static SPIN_ABSORBED_CYCLES: AtomicU64 = AtomicU64::new(0);
/// Cycles waived at wake after a real park (the blocked-thread rule).
static PARK_FORGIVEN_CYCLES: AtomicU64 = AtomicU64::new(0);
/// Would-be attribution lost when the target-site actor parks mid-site.
static DROP_PARK_CYCLES: AtomicU64 = AtomicU64::new(0);
static DROP_PARK_N: AtomicU64 = AtomicU64::new(0);
/// Same loss at yields (explicit, slice-expiry, or a park that requeued).
static DROP_YIELD_CYCLES: AtomicU64 = AtomicU64::new(0);
static DROP_YIELD_N: AtomicU64 = AtomicU64::new(0);
/// Samples discarded by the TSC-weirdness clamp, in would-be delta terms.
static DISCARD_OVERMAX_CYCLES: AtomicU64 = AtomicU64::new(0);
static DISCARD_OVERMAX_N: AtomicU64 = AtomicU64::new(0);
/// In-site samples dropped because the thread's clock was unarmed.
static DISCARD_UNARMED_N: AtomicU64 = AtomicU64::new(0);
/// Would-be attribution over runnable off-CPU gaps inside the target
/// site (yield-descheduled -> resumed within the same window). Not a
/// loss: on-CPU-only attribution is the Coz model — queue-wait is not
/// shrunk by speeding the site's code — but counted so the audit books
/// close against wall in-site time (the located @50 "deficit").
static OFFCPU_IN_SITE_CYCLES: AtomicU64 = AtomicU64::new(0);
static OFFCPU_IN_SITE_N: AtomicU64 = AtomicU64::new(0);
fn sites() -> &'static Mutex<Vec<&'static str>> {
SITES.get_or_init(|| Mutex::new(Vec::new()))
}
fn progress_points() -> &'static Mutex<Vec<&'static ProgressPoint>> {
PROGRESS.get_or_init(|| Mutex::new(Vec::new()))
}
/// Recover from lock poisoning: all these registries hold plain data that
/// is valid at every instruction boundary, so a panicked registrant can't
/// leave them torn.
fn lock_unpoisoned<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match m.lock() {
Ok(g) => g,
Err(poisoned) => poisoned.into_inner(),
}
}
// -----------------------------------------------------------------------
// Sites
// -----------------------------------------------------------------------
/// Register (or look up) a causal site by name; returns its nonzero id.
/// Called once per `causal_site!` expansion via a `OnceLock`, so the
/// mutex is off every hot path.
pub fn site_id(name: &'static str) -> u32 {
let mut v = lock_unpoisoned(sites());
if let Some(pos) = v.iter().position(|n| *n == name) {
return (pos + 1) as u32;
}
v.push(name);
v.len() as u32
}
fn site_name(id: u32) -> Option<String> {
if id == 0 {
return None;
}
let v = lock_unpoisoned(sites());
v.get((id - 1) as usize).map(|s| (*s).to_string())
}
/// RAII marker: while alive, the current *actor* (not thread — the id
/// lives in its `Slot` and survives preemption/migration) is "inside"
/// the site. Nesting restores the outer site on drop. Inert outside an
/// actor (scheduler/OS-thread stacks).
pub struct SiteGuard {
/// Slot of the actor that entered, null if entered outside an actor.
/// Valid for the guard's whole life: the guard lives on the actor's
/// stack, and a slot is never reclaimed while its actor is alive —
/// the same argument as `preempt::check_cancelled`.
slot: *const crate::runtime::Slot,
prev: u32,
}
impl SiteGuard {
/// Enter `site` for the on-CPU actor.
pub fn enter(site: u32) -> Self {
let slot = preempt::current_slot_ptr();
if slot.is_null() {
return SiteGuard { slot, prev: 0 };
}
// SAFETY: non-null ⇒ points at the on-CPU actor's slot; see the
// field docs for the lifetime argument.
let prev = unsafe { (*slot).causal_site() };
unsafe { (*slot).set_causal_site(site) };
site_transition(slot, prev, site);
SiteGuard { slot, prev }
}
}
impl Drop for SiteGuard {
fn drop(&mut self) {
if !self.slot.is_null() {
// SAFETY (both): as in `enter` — the actor (and thus its
// slot) is alive for as long as this guard is on its stack.
let site = unsafe { (*self.slot).causal_site() };
unsafe { (*self.slot).set_causal_site(self.prev) };
site_transition(self.slot, site, self.prev);
}
}
}
/// Name of the site the on-CPU actor is currently inside, if any.
/// (Introspection/testing; not a hot path.)
pub fn current_site_name() -> Option<String> {
let slot = preempt::current_slot_ptr();
if slot.is_null() {
return None;
}
// SAFETY: on-CPU actor's slot, valid for the whole resume.
site_name(unsafe { (*slot).causal_site() })
}
// -----------------------------------------------------------------------
// Progress points
// -----------------------------------------------------------------------
/// A named throughput counter. One per distinct name; `progress!` call
/// sites sharing a name share the counter.
pub struct ProgressPoint {
name: &'static str,
count: AtomicU64,
}
impl ProgressPoint {
/// The hot path: one Relaxed RMW. (Contended across actors by design
/// — a progress point is a global rate meter.)
#[inline]
pub fn bump(&self) {
self.count.fetch_add(1, Ordering::Relaxed);
}
}
/// Register (or look up) a progress point. Called once per `progress!`
/// expansion via a `OnceLock`; the mutex is off the hot path.
pub fn register_progress(name: &'static str) -> &'static ProgressPoint {
let mut v = lock_unpoisoned(progress_points());
if let Some(p) = v.iter().find(|p| p.name == name) {
return p;
}
let p: &'static ProgressPoint = Box::leak(Box::new(ProgressPoint {
name,
count: AtomicU64::new(0),
}));
v.push(p);
p
}
/// Snapshot of all progress points as `(name, count)`.
pub fn progress_snapshot() -> Vec<(String, u64)> {
lock_unpoisoned(progress_points())
.iter()
.map(|p| (p.name.to_string(), p.count.load(Ordering::Relaxed)))
.collect()
}
// -----------------------------------------------------------------------
// The hot hook: sample + absorb
// -----------------------------------------------------------------------
/// Called from `maybe_preempt` at the amortised timeslice-check cadence,
/// under the `PREEMPTION_ENABLED` gate (so never in a prep-to-park or
/// no-preempt region — spinning here is as safe as yielding is).
///
/// One Relaxed load and out when no experiment is running.
#[inline]
pub(crate) fn check() {
let exp = EXPERIMENT.load(Ordering::Relaxed);
if exp == 0 {
return;
}
cold_check(exp);
}
/// The experiment-active path, kept out of the inlined fast path.
#[cold]
fn cold_check(exp: u64) {
let slot = preempt::current_slot_ptr();
if slot.is_null() {
return;
}
let now = preempt::rdtsc();
let last = LAST_SAMPLE_TSC.with(|c| c.replace(now));
let target_site = (exp >> 32) as u32;
let pct = exp & 0xffff_ffff;
// SAFETY (both derefs below): non-null ⇒ the on-CPU actor's slot,
// never reclaimed while the actor runs — see `check_cancelled`.
let my_site = unsafe { (*slot).causal_site() };
if my_site == target_site && pct > 0 {
// A sample landed in the target site: everyone else must fall
// behind by pct% of the sampled interval. Grow the global ledger
// and credit ourselves the same amount — the credited gap *is*
// the virtual speedup.
if last == 0 {
// Unarmed clock: no interval to attribute — count the loss.
DISCARD_UNARMED_N.fetch_add(1, Ordering::Relaxed);
return;
}
// SAFETY: `slot` is the on-CPU actor's slot (checked non-null
// above); see `check_cancelled` for the lifetime argument.
unsafe { attribute(slot, now.saturating_sub(last), pct) };
} else {
// Not the winner: chase the global ledger by spinning off the
// difference, then push the slice start forward so injected
// delay never counts as compute (the clock correction that Coz
// cannot do from outside).
let global = GLOBAL_DELAY.load(Ordering::Relaxed);
let mine = unsafe { (*slot).causal_delay() };
if mine >= global {
return;
}
let spin = (global - mine).min(MAX_SPIN_PER_VISIT);
let start = preempt::rdtsc();
while preempt::rdtsc().saturating_sub(start) < spin {
core::hint::spin_loop();
}
SPIN_ABSORBED_CYCLES.fetch_add(spin, Ordering::Relaxed);
unsafe { (*slot).set_causal_delay(mine.wrapping_add(spin)) };
preempt::extend_timeslice(spin);
// The spin is not part of the next sample interval either.
LAST_SAMPLE_TSC.with(|c| c.set(preempt::rdtsc()));
}
}
/// Attribute one target-site sample of `interval` cycles at `pct`%:
/// grow the global ledger and credit the sampling actor's own ledger by
/// the same amount — the credited gap *is* the virtual speedup. Shared
/// by the cold check and the guard-boundary flush. Applies the same
/// clamps as sampling always has: zero intervals and clock hiccups are
/// discarded, not the run.
///
/// SAFETY: `slot` must point at the on-CPU actor's slot (the
/// `check_cancelled` lifetime argument).
unsafe fn attribute(slot: *const crate::runtime::Slot, interval: u64, pct: u64) {
if interval == 0 {
return; // now == last: nothing to attribute, nothing lost
}
if interval > MAX_SAMPLE_CYCLES {
// TSC-weirdness clamp: the sample is discarded, not the run.
// Count the loss in would-be delta terms so the audit's columns
// compare directly against `injected_cycles`.
DISCARD_OVERMAX_N.fetch_add(1, Ordering::Relaxed);
DISCARD_OVERMAX_CYCLES
.fetch_add(interval.saturating_mul(pct) / 100, Ordering::Relaxed);
return;
}
let delta = interval.saturating_mul(pct) / 100;
GLOBAL_DELAY.fetch_add(delta, Ordering::Relaxed);
let mine = (*slot).causal_delay();
(*slot).set_causal_delay(mine.wrapping_add(delta));
}
/// Site-boundary hook, called by `SiteGuard` enter/drop when the
/// actor's current site changes from `old` to `new`. Sample-only —
/// never spins — so it is safe anywhere, including no-preempt regions
/// where `check()` cannot run.
///
/// - Leaving the experiment's target site: flush the pending interval.
/// Cold checks only sample when they happen to fire in-site, so the
/// tail between the last check and the guard drop was otherwise
/// discarded on every site entry — measured live at ~22-29µs/entry,
/// ~6-7% of all target time (eff 0.93), which under-reported every
/// impact (+83.5% where theory says +100%).
/// - Entering the target site: re-arm the sample clock, so time spent
/// *before* the site can never be attributed to it by the first
/// in-site check (the symmetric over-attribution).
#[inline]
fn site_transition(slot: *const crate::runtime::Slot, old: u32, new: u32) {
let exp = EXPERIMENT.load(Ordering::Relaxed);
if exp == 0 || old == new {
return;
}
let target = (exp >> 32) as u32;
let pct = exp & 0xffff_ffff;
if old == target && new != target {
let now = preempt::rdtsc();
let last = LAST_SAMPLE_TSC.with(|c| c.replace(now));
if pct > 0 {
if last != 0 {
// SAFETY: forwarded from the guard, which holds the on-CPU
// actor's slot for its whole life (see `SiteGuard::slot`).
unsafe { attribute(slot, now.saturating_sub(last), pct) };
} else {
DISCARD_UNARMED_N.fetch_add(1, Ordering::Relaxed);
}
}
} else if new == target && old != target {
LAST_SAMPLE_TSC.with(|c| c.set(preempt::rdtsc()));
}
}
/// Resume-path hook (scheduler thread, actor off-CPU). Two duties:
///
/// - If the last deschedule was a *real park*, time blocked absorbs any
/// delay accrued meanwhile for free — Coz's blocked-thread rule, which
/// keeps experiments from punishing actors for waiting. An actor that
/// merely yielded (slice expiry) was runnable the whole time and keeps
/// its debt: it must pay by spinning at its next check. Forgiving on
/// every resume would make any yield-cadence actor delay-immune and
/// experiments inert (found live on a 24-core run: nothing slowed).
/// - If the deschedule was a *yield* in the live experiment's target
/// site, count the off-CPU gap it opened into the offcpu audit bucket
/// (RFC 007: the located @50 deficit — runnable queue-wait is wall
/// time in-site that on-CPU attribution correctly skips). Same-window
/// only, enforced by the experiment epoch; measure-only.
/// - Arm this thread's sample clock so the first interval of the resume
/// excludes scheduler time.
#[inline]
pub(crate) fn on_resume(slot: &crate::runtime::Slot) {
let (desched_tsc, desched_epoch) = slot.take_causal_desched();
if desched_tsc != 0 && desched_epoch == EXPERIMENT_EPOCH.load(Ordering::Relaxed) {
// Same epoch ⇒ no `begin()` since the stash; a nonzero word ⇒
// no `end()` either — the gap closed inside its own window.
let exp = EXPERIMENT.load(Ordering::Relaxed);
if exp != 0 {
let pct = exp & 0xffff_ffff;
let gap = preempt::rdtsc()
.saturating_sub(desched_tsc)
.min(MAX_SAMPLE_CYCLES);
OFFCPU_IN_SITE_CYCLES
.fetch_add(gap.saturating_mul(pct) / 100, Ordering::Relaxed);
OFFCPU_IN_SITE_N.fetch_add(1, Ordering::Relaxed);
}
}
if slot.take_causal_parked() {
let global = GLOBAL_DELAY.load(Ordering::Relaxed);
let mine = slot.causal_delay();
if mine < global {
PARK_FORGIVEN_CYCLES.fetch_add(global - mine, Ordering::Relaxed);
slot.set_causal_delay(global);
}
}
LAST_SAMPLE_TSC.with(|c| c.set(preempt::rdtsc()));
}
/// Deschedule-path hook (scheduler side, same OS thread the actor just
/// ran on). If an experiment is live and the departing actor sits in the
/// target site, the sample tail `[last sample -> now]` is about to be
/// lost: nothing flushes it here, and `on_resume` re-arms the clock
/// before the actor runs again. Measure-only (RFC 007 deficit hunt) —
/// tally the would-be attribution into the park/yield drop buckets and
/// leave behaviour untouched. The interval is capped at
/// MAX_SAMPLE_CYCLES: past that the flush would have discarded it anyway
/// (counted separately). `now` includes the few hundred ns of scheduler
/// bookkeeping since the actor actually stopped — an acceptable
/// overcount for a diagnostic.
///
/// Slice-expiry yields sample at the same checkpoint that deschedules
/// them, so their tails are ~zero by construction; a fat yield bucket
/// therefore points at explicit `yield_now` calls or requeued parks.
///
/// Yields additionally stash the deschedule instant on the slot so
/// `on_resume` can count the runnable off-CPU gap (offcpu bucket).
pub(crate) fn on_deschedule(slot: &crate::runtime::Slot, real_park: bool) {
let exp = EXPERIMENT.load(Ordering::Relaxed);
if exp == 0 {
return;
}
let target = (exp >> 32) as u32;
let pct = exp & 0xffff_ffff;
if pct == 0 || slot.causal_site() != target {
return;
}
let now = preempt::rdtsc();
if !real_park {
// Runnable gap opens here; `on_resume` closes and counts it
// (offcpu bucket). Parks are excluded: blocked time is already
// represented by forgiveness, and blocked wall time is not
// queue-wait.
slot.set_causal_desched(now, EXPERIMENT_EPOCH.load(Ordering::Relaxed));
}
let last = LAST_SAMPLE_TSC.with(|c| c.get());
if last == 0 {
return;
}
let interval = now.saturating_sub(last).min(MAX_SAMPLE_CYCLES);
let would_be = interval.saturating_mul(pct) / 100;
if real_park {
DROP_PARK_N.fetch_add(1, Ordering::Relaxed);
DROP_PARK_CYCLES.fetch_add(would_be, Ordering::Relaxed);
} else {
DROP_YIELD_N.fetch_add(1, Ordering::Relaxed);
DROP_YIELD_CYCLES.fetch_add(would_be, Ordering::Relaxed);
}
}
// -----------------------------------------------------------------------
// Experiments
// -----------------------------------------------------------------------
fn begin(site: u32, pct: u32) {
EXPERIMENT_EPOCH.fetch_add(1, Ordering::Relaxed);
EXPERIMENT.store(((site as u64) << 32) | pct as u64, Ordering::Relaxed);
}
fn end() {
EXPERIMENT.store(0, Ordering::Relaxed);
}
/// Total virtual delay injected so far, in TSC cycles.
pub fn global_delay_cycles() -> u64 {
GLOBAL_DELAY.load(Ordering::Relaxed)
}
/// Cumulative ledger-audit totals since startup (RFC 007 deficit hunt).
/// All monotone; window a span by snapshotting before/after and taking
/// `delta_since`. Cycle fields are in would-be-injected delta terms so
/// they compare directly against `injected_cycles`.
#[derive(Clone, Copy, Debug, Default)]
pub struct LedgerCounters {
pub spin_absorbed_cycles: u64,
pub park_forgiven_cycles: u64,
pub drop_park_cycles: u64,
pub drop_park_n: u64,
pub drop_yield_cycles: u64,
pub drop_yield_n: u64,
pub discard_overmax_cycles: u64,
pub discard_overmax_n: u64,
pub discard_unarmed_n: u64,
pub offcpu_in_site_cycles: u64,
pub offcpu_in_site_n: u64,
}
impl LedgerCounters {
/// Field-wise difference against an earlier snapshot.
pub fn delta_since(&self, before: &LedgerCounters) -> LedgerCounters {
LedgerCounters {
spin_absorbed_cycles: self
.spin_absorbed_cycles
.saturating_sub(before.spin_absorbed_cycles),
park_forgiven_cycles: self
.park_forgiven_cycles
.saturating_sub(before.park_forgiven_cycles),
drop_park_cycles: self.drop_park_cycles.saturating_sub(before.drop_park_cycles),
drop_park_n: self.drop_park_n.saturating_sub(before.drop_park_n),
drop_yield_cycles: self
.drop_yield_cycles
.saturating_sub(before.drop_yield_cycles),
drop_yield_n: self.drop_yield_n.saturating_sub(before.drop_yield_n),
discard_overmax_cycles: self
.discard_overmax_cycles
.saturating_sub(before.discard_overmax_cycles),
discard_overmax_n: self.discard_overmax_n.saturating_sub(before.discard_overmax_n),
discard_unarmed_n: self.discard_unarmed_n.saturating_sub(before.discard_unarmed_n),
offcpu_in_site_cycles: self
.offcpu_in_site_cycles
.saturating_sub(before.offcpu_in_site_cycles),
offcpu_in_site_n: self.offcpu_in_site_n.saturating_sub(before.offcpu_in_site_n),
}
}
}
/// Snapshot the cumulative audit counters.
pub fn ledger_counters() -> LedgerCounters {
LedgerCounters {
spin_absorbed_cycles: SPIN_ABSORBED_CYCLES.load(Ordering::Relaxed),
park_forgiven_cycles: PARK_FORGIVEN_CYCLES.load(Ordering::Relaxed),
drop_park_cycles: DROP_PARK_CYCLES.load(Ordering::Relaxed),
drop_park_n: DROP_PARK_N.load(Ordering::Relaxed),
drop_yield_cycles: DROP_YIELD_CYCLES.load(Ordering::Relaxed),
drop_yield_n: DROP_YIELD_N.load(Ordering::Relaxed),
discard_overmax_cycles: DISCARD_OVERMAX_CYCLES.load(Ordering::Relaxed),
discard_overmax_n: DISCARD_OVERMAX_N.load(Ordering::Relaxed),
discard_unarmed_n: DISCARD_UNARMED_N.load(Ordering::Relaxed),
offcpu_in_site_cycles: OFFCPU_IN_SITE_CYCLES.load(Ordering::Relaxed),
offcpu_in_site_n: OFFCPU_IN_SITE_N.load(Ordering::Relaxed),
}
}
/// Absorbed-delay ledger of the on-CPU actor (testing/introspection).
pub fn my_absorbed_delay_cycles() -> u64 {
let slot = preempt::current_slot_ptr();
if slot.is_null() {
return 0;
}
// SAFETY: on-CPU actor's slot; see `check_cancelled`.
unsafe { (*slot).causal_delay() }
}
/// Test support: start an experiment targeting `site_name` at `pct`%
/// virtual speedup. Registers the site if needed.
pub fn begin_experiment_for_test(name: &'static str, pct: u32) {
begin(site_id(name), pct);
}
/// Test support: stop the running experiment.
pub fn end_experiment_for_test() {
end();
}
/// Test support: grow the global delay ledger directly, as if target-site
/// samples had attributed `cycles` — deterministic driver for the timer
/// virtual-time tests. Calibrates the TSC eagerly so conversion later
/// never stalls a scheduler loop.
pub fn inject_delay_cycles_for_test(cycles: u64) {
let _ = tsc_hz();
GLOBAL_DELAY.fetch_add(cycles, Ordering::Relaxed);
}
/// Convert ledger cycles to wall time at the measured TSC rate.
pub fn cycles_to_duration(cycles: u64) -> Duration {
Duration::from_secs_f64(cycles as f64 / tsc_hz())
}
/// Controller parameters: which speedups to try per site, and the
/// experiment/cooldown windows.
pub struct ExperimentPlan {
pub speedups_pct: Vec<u32>,
pub experiment: Duration,
pub cooldown: Duration,
}
impl Default for ExperimentPlan {
fn default() -> Self {
ExperimentPlan {
speedups_pct: vec![0, 25, 50],
experiment: Duration::from_millis(500),
cooldown: Duration::from_millis(100),
}
}
}
/// One completed experiment cell.
#[derive(Default)]
pub struct ExperimentResult {
pub site: String,
pub speedup_pct: u32,
pub duration: Duration,
/// Progress-point deltas over the window, `(name, count)`.
pub deltas: Vec<(String, u64)>,
/// Virtual delay injected during the window (cycles).
pub injected_cycles: u64,
// Ledger-audit deltas over the window (RFC 007 deficit hunt); see
// `LedgerCounters` for field semantics. `spin_absorbed_cycles > 0`
// in a 0% cell means the window paid debt left over from an earlier
// one — the baseline-contamination signature.
pub spin_absorbed_cycles: u64,
pub park_forgiven_cycles: u64,
pub drop_park_cycles: u64,
pub drop_park_n: u64,
pub drop_yield_cycles: u64,
pub drop_yield_n: u64,
pub discard_overmax_cycles: u64,
pub discard_overmax_n: u64,
pub discard_unarmed_n: u64,
pub offcpu_in_site_cycles: u64,
pub offcpu_in_site_n: u64,
}
/// Run the plan synchronously on the calling (OS) thread: for every
/// registered site × speedup, run one experiment window and record
/// progress-point deltas, with a cooldown between cells. Sites and
/// progress points must already be registered (the workload has to be
/// running); the caller owns workload start/stop.
///
/// v1 controller: exhaustive sweep, fixed windows, no adaptive site
/// selection or confidence stopping (jar Q5).
///
/// Callable from a plain OS thread *or* from inside an actor: sleeping
/// parks the green thread when we're on one (so no scheduler thread is
/// blocked), and falls back to `thread::sleep` otherwise.
pub fn run_experiments(plan: &ExperimentPlan) -> Vec<ExperimentResult> {
fn controller_sleep(d: Duration) {
if preempt::current_slot_ptr().is_null() {
std::thread::sleep(d);
} else {
// Wall-anchored: the controller's window/cooldown sleeps
// *define* the experiment's wall length; letting them chase
// the delay it is itself injecting would stretch every window
// (observed ~2x at 50% speedup). Deltas are rate-normalized
// either way — this fixes cost, not bias.
crate::scheduler::sleep_wall(d);
}
}
// Calibrate before any window so report rendering never has to sleep.
let _ = tsc_hz();
let site_list: Vec<(u32, String)> = {
let v = lock_unpoisoned(sites());
v.iter()
.enumerate()
.map(|(i, n)| ((i + 1) as u32, (*n).to_string()))
.collect()
};
let mut out = Vec::new();
for (sid, sname) in &site_list {
for &pct in &plan.speedups_pct {
let before = progress_snapshot();
let injected_before = global_delay_cycles();
let audit_before = ledger_counters();
let t0 = Instant::now();
begin(*sid, pct);
controller_sleep(plan.experiment);
end();
// Snapshot immediately: injection and spin freeze at `end()`
// (checks gate on the experiment word), but forgiveness does
// not — a later snapshot would leak cooldown wakes into the
// window.
let audit = ledger_counters().delta_since(&audit_before);
let elapsed = t0.elapsed();
let after = progress_snapshot();
let deltas = after
.iter()
.map(|(n, c)| {
let b = before
.iter()
.find(|(bn, _)| bn == n)
.map(|(_, bc)| *bc)
.unwrap_or(0);
(n.clone(), c.saturating_sub(b))
})
.collect();
out.push(ExperimentResult {
site: sname.clone(),
speedup_pct: pct,
duration: elapsed,
deltas,
injected_cycles: global_delay_cycles() - injected_before,
spin_absorbed_cycles: audit.spin_absorbed_cycles,
park_forgiven_cycles: audit.park_forgiven_cycles,
drop_park_cycles: audit.drop_park_cycles,
drop_park_n: audit.drop_park_n,
drop_yield_cycles: audit.drop_yield_cycles,
drop_yield_n: audit.drop_yield_n,
discard_overmax_cycles: audit.discard_overmax_cycles,
discard_overmax_n: audit.discard_overmax_n,
discard_unarmed_n: audit.discard_unarmed_n,
offcpu_in_site_cycles: audit.offcpu_in_site_cycles,
offcpu_in_site_n: audit.offcpu_in_site_n,
});
controller_sleep(plan.cooldown);
}
}
out
}
// -----------------------------------------------------------------------
// Reports
// -----------------------------------------------------------------------
/// Measured TSC frequency (Hz), calibrated once. The crate-wide 3 GHz
/// constant is fine for the *relative* timeslice check, but report
/// normalisation divides wall time by injected time, so a 20% Hz error
/// skews every impact number — measured live: a 3.7 GHz box inflated all
/// baselines uniformly. Calibrated against `Instant` over ~50ms on first
/// use; `run_experiments` triggers it before its first window (using the
/// park-aware sleep, so no scheduler thread is blocked when called from
/// an actor).
static TSC_HZ_MEASURED: OnceLock<f64> = OnceLock::new();
/// Measured TSC frequency in Hz. Calibrates on first call (~50ms).
pub fn tsc_hz() -> f64 {
*TSC_HZ_MEASURED.get_or_init(|| {
let c0 = preempt::rdtsc();
let t0 = Instant::now();
let d = Duration::from_millis(50);
if preempt::current_slot_ptr().is_null() {
std::thread::sleep(d);
} else {
// Wall-anchored: calibration divides TSC delta by *wall*
// elapsed; a virtual sleep dilated by concurrent injection
// would still measure correctly (elapsed() is wall) but
// waste window time — and must never depend on the ledger
// it exists to convert.
crate::scheduler::sleep_wall(d);
}
(preempt::rdtsc().wrapping_sub(c0)) as f64 / t0.elapsed().as_secs_f64()
})
}
/// Normalized rate for one cell: count over the *virtual* window
/// (wall injected) — Coz's normalization: injected delay does not
/// exist in the virtual timeline. A bottleneck site keeps its raw count
/// while shrinking the divisor → positive impact; a fully overlapped
/// site loses count proportionally → ~zero.
fn normalized_rate(r: &ExperimentResult, point: &str) -> Option<f64> {
let count = r.deltas.iter().find(|(n, _)| n == point).map(|(_, c)| *c)?;
let injected_secs = r.injected_cycles as f64 / tsc_hz();
let virtual_secs = (r.duration.as_secs_f64() - injected_secs).max(1e-9);
Some(count as f64 / virtual_secs)
}
/// Impact of virtually speeding up `site` by `speedup_pct` on progress
/// point `point`, in percent relative to that site's own 0% baseline
/// cell. `None` if either cell or the point is missing, or the baseline
/// rate is zero. This is the machine-readable form of the summary's
/// "vs baseline" column, for programmatic checks (CI, examples).
pub fn impact_pct(
results: &[ExperimentResult],
site: &str,
speedup_pct: u32,
point: &str,
) -> Option<f64> {
let cell = results
.iter()
.find(|r| r.site == site && r.speedup_pct == speedup_pct)?;
let base = results.iter().find(|r| r.site == site && r.speedup_pct == 0)?;
let rate = normalized_rate(cell, point)?;
let b = normalized_rate(base, point)?;
if b <= 0.0 {
return None;
}
Some((rate / b - 1.0) * 100.0)
}
/// Human-readable summary: per (site, progress point), the throughput at
/// each virtual speedup and the change relative to that site's own 0%
/// baseline. A near-zero column across speedups means: optimising this
/// site buys you nothing — the RFC's headline answer.
///
/// Ends with a one-line fidelity note (RFC 007 Validation): reported
/// impacts are conservative — attribution counts on-CPU site time only,
/// so runnable queue-wait inside the site (the located @50 "deficit",
/// eff ≈ 0.93 live) is never injected and gains are lower bounds; site
/// *rankings* are unaffected.
pub fn render_summary(results: &[ExperimentResult]) -> String {
use std::fmt::Write;
let mut s = String::new();
let _ = writeln!(s, "== smarm causal profile ==");
let mut sites_seen: Vec<&str> = Vec::new();
for r in results {
if !sites_seen.contains(&r.site.as_str()) {
sites_seen.push(&r.site);
}
}
for site in sites_seen {
let _ = writeln!(s, "site {site}");
for r in results.iter().filter(|r| r.site == site) {
for (name, _) in &r.deltas {
let rate = match normalized_rate(r, name) {
Some(x) => x,
None => continue,
};
let rel = impact_pct(results, site, r.speedup_pct, name)
.map(|p| format!("{p:+.1}%"))
.unwrap_or_else(|| "n/a".to_string());
let _ = writeln!(
s,
" speedup {:>3}% {name:<24} {rate:>12.1}/s vs baseline {rel} (injected {:.1}ms)",
r.speedup_pct,
r.injected_cycles as f64 / tsc_hz() * 1e3
);
}
}
}
if !results.is_empty() {
let _ = writeln!(
s,
"note: impacts are lower bounds — site time counts on-CPU only (runnable queue-wait is not attributed); rankings unaffected"
);
}
s
}
/// Ledger-audit companion to `render_summary` (RFC 007 deficit hunt):
/// per cell, where the window's virtual delay went — born (injected),
/// paid (absorbed), waived (forgiven at wake) — and the attribution the
/// sampler lost: tails dropped at parks/yields inside the target site,
/// plus clamp discards. Cycle columns in ms at the calibrated TSC rate.
/// `absorbed` above `injected` in a cell (0% especially) means it paid
/// debt left over from earlier windows.
pub fn render_ledger_audit(results: &[ExperimentResult]) -> String {
use std::fmt::Write;
let hz = tsc_hz();
let ms = |c: u64| c as f64 / hz * 1e3;
let mut s = String::new();
let _ = writeln!(s, "== smarm causal ledger audit ==");
for r in results {
let _ = writeln!(
s,
"site {:<22} @{:>2}% injected {:>7.1}ms absorbed {:>7.1}ms forgiven {:>7.1}ms \
drop park {:>6.2}ms/{:<4} yield {:>6.2}ms/{:<4} offcpu {:>6.2}ms/{:<5} \
discard >max {:>6.2}ms/{:<3} unarmed {}",
r.site,
r.speedup_pct,
ms(r.injected_cycles),
ms(r.spin_absorbed_cycles),
ms(r.park_forgiven_cycles),
ms(r.drop_park_cycles),
r.drop_park_n,
ms(r.drop_yield_cycles),
r.drop_yield_n,
ms(r.offcpu_in_site_cycles),
r.offcpu_in_site_n,
ms(r.discard_overmax_cycles),
r.discard_overmax_n,
r.discard_unarmed_n
);
}
s
}
/// Coz-compatible profile text (`profile.coz`), so Coz's existing plot
/// tooling renders our experiments — the RFC's "don't build a UI" call.
pub fn render_coz(results: &[ExperimentResult]) -> String {
use std::fmt::Write;
let mut s = String::new();
let _ = writeln!(s, "startup\ttime=0");
for r in results {
let _ = writeln!(
s,
"experiment\tselected={}\tspeedup={:.2}\tduration={}\tselected-samples=1",
r.site,
r.speedup_pct as f64 / 100.0,
r.duration.as_nanos()
);
for (name, count) in &r.deltas {
let _ = writeln!(s, "throughput-point\tname={name}\tdelta={count}");
}
}
s
}
}
#[cfg(feature = "smarm-causal")]
pub use inner::*;
/// Mark one unit of useful work complete at a named throughput progress
/// point (RFC 007). One Relaxed increment when `smarm-causal` is on; nothing
/// at all when it's off.
#[cfg(feature = "smarm-causal")]
#[macro_export]
macro_rules! progress {
($name:literal) => {{
static __SMARM_PP: ::std::sync::OnceLock<&'static $crate::causal::ProgressPoint> =
::std::sync::OnceLock::new();
__SMARM_PP
.get_or_init(|| $crate::causal::register_progress($name))
.bump();
}};
}
#[cfg(not(feature = "smarm-causal"))]
#[macro_export]
macro_rules! progress {
($name:literal) => {{}};
}
/// Enter a named causal-profiling site for the current actor; the returned
/// guard exits it (restoring any enclosing site) on drop. Site identity is
/// stored in the actor's slot, so it survives preemption and migration.
/// Expands to a unit no-op without `smarm-causal`.
#[cfg(feature = "smarm-causal")]
#[macro_export]
macro_rules! causal_site {
($name:literal) => {{
static __SMARM_SITE: ::std::sync::OnceLock<u32> = ::std::sync::OnceLock::new();
$crate::causal::SiteGuard::enter(*__SMARM_SITE.get_or_init(|| $crate::causal::site_id($name)))
}};
}
#[cfg(not(feature = "smarm-causal"))]
#[macro_export]
macro_rules! causal_site {
($name:literal) => {
()
};
}
+282 -198
View File
@@ -1,38 +1,102 @@
//! Unbounded MPSC channels. //! Unbounded multi-producer, single-consumer channels: how actors talk to
//! each other.
//! //!
//! Inner state is `Arc<RawMutex<Inner<T>>>` so channels can be sent across OS //! A channel is a queue with a typed [`Sender`] on one end and a typed
//! threads (required for the multi-scheduler runtime where a sender and //! [`Receiver`] on the other. Any number of actors can hold a clone of the
//! receiver may run on different scheduler threads simultaneously). //! `Sender` and push messages onto the same queue; exactly one [`Receiver`]
//! reads them back out, in the order they arrived. This is the basic wiring
//! smarm's other actor primitives (`gen_server`, `pg`, the registry) are all
//! built out of, and it is directly usable on its own for a worker that just
//! needs an inbox.
//! //!
//! ## Why `RawMutex` (Channel class), not `std::sync::Mutex` //! ## A first channel
//! //!
//! An actor holding a guard with preemption *enabled* can be timesliced //! ```
//! inside the critical section and resume on a different OS thread — the //! use smarm::{channel, run, spawn};
//! pthread mutex would then be released from a thread that didn't lock it,
//! which is UB (Linux futexes happen to tolerate it, but it's not
//! guaranteed). `RawMutex` disables preemption for the guard's span and is
//! cross-thread-release sound by construction, closing the hole. It also
//! cannot poison. Channel locks form their own [`LockClass::Channel`]
//! (raw_mutex.rs): they may be taken under a cold (Leaf) lock — finalize and
//! `monitor()` clone senders that live in slots — but nothing may be locked
//! under them, which the debug build enforces. `recv_match` runs its user
//! predicate under this lock: keep it cheap, pure, and channel-free.
//! //!
//! Semantics: //! run(|| {
//! - Senders are clonable; the last sender drop closes the channel. //! let (tx, rx) = channel::<u64>();
//! - `Receiver::recv` on an empty open channel parks the receiver. //!
//! - `Receiver::recv` on an empty closed channel returns `Err(RecvError)`. //! let worker = spawn(move || {
//! - `Sender::send` on an open channel always succeeds. //! // Blocks until a message arrives.
//! - `Sender::send` on a closed channel (receiver dropped) returns //! let n = rx.recv().unwrap();
//! `Err(SendError(value))`. //! assert_eq!(n, 42);
//! - When a send pushes to a previously empty queue and a receiver is //!
//! parked, the receiver is unparked. //! // Once every Sender is dropped, recv() reports the channel closed
//! // instead of blocking forever.
//! assert!(rx.recv().is_err());
//! });
//!
//! tx.send(42).unwrap();
//! drop(tx); // last sender gone: the channel is now closed
//! worker.join().unwrap();
//! });
//! ```
//!
//! ## Sending
//!
//! [`Sender`] is cheaply clonable: hand a clone to every actor that needs to
//! push messages into this queue. The channel stays open as long as at least
//! one clone exists; [`Sender::send`] never blocks and always succeeds while
//! the channel is open, since the queue is unbounded. Once the [`Receiver`]
//! has been dropped, `send` returns the message back to you in
//! [`SendError`] instead of delivering it.
//!
//! ## Receiving
//!
//! There is exactly one [`Receiver`] per channel (it is not clonable).
//! [`Receiver::recv`] returns the next message in arrival order, parking the
//! calling actor if the queue is currently empty. Once every `Sender` has
//! been dropped and the queue has been drained, `recv` stops parking and
//! returns [`RecvError`] instead, so a receiver never blocks forever waiting
//! on senders that are never coming back.
//!
//! Beyond plain `recv`, three variants cover the common needs:
//!
//! - [`Receiver::try_recv`]: never parks: reports an empty-but-open channel
//! as `Ok(None)` instead of waiting.
//! - [`Receiver::recv_timeout`]: parks, but gives up and returns
//! [`RecvTimeoutError::Timeout`] if no message arrives before a deadline.
//! - [`Receiver::recv_match`] / [`Receiver::try_recv_match`]: selective
//! receive. Instead of taking whatever is at the front of the queue, pick
//! out the first message matching a predicate, leaving the rest queued in
//! order. Handy for an actor that wants to prioritise one kind of message
//! over others already waiting.
//!
//! ## Waiting on several channels: `select`
//!
//! [`select`] parks an actor across several receivers at once and reports
//! the index of the first one that is ready (has a message queued, or has
//! been closed). [`select_timeout`] adds a deadline, the way `recv_timeout`
//! does for a single channel. See their docs for the full contract,
//! including the priority-order and no-fairness guarantee.
//!
//! ## Implementation notes
//!
//! The queue and its bookkeeping live behind `Arc<RawMutex<Inner<T>>>`
//! rather than a `std::sync::Mutex`, so that a channel can be freely shared
//! and sent across the OS threads backing the multi-scheduler runtime.
//! `RawMutex` matters here for a subtler reason too: an ordinary pthread
//! mutex can be released from a different OS thread than the one that took
//! it (smarm's preemption can migrate a timesliced actor between scheduler
//! threads mid-critical-section), and doing that to a `std::sync::Mutex` is
//! undefined behavior. `RawMutex` disables preemption for the guard's short
//! lifetime instead, so the release always happens on the thread that
//! acquired it, and it has no poisoning to worry about besides. Channel
//! locks are cheap and are never held across another lock acquisition or a
//! blocking call; the predicate passed to `recv_match` runs under this lock,
//! which is why it needs to stay cheap, pure, and must not call back into
//! the same channel.
use crate::pid::Pid; use crate::pid::Pid;
use crate::raw_mutex::RawMutex; use crate::raw_mutex::RawMutex;
use std::collections::VecDeque; use std::collections::VecDeque;
use std::sync::Arc; use std::sync::Arc;
/// Create a new channel and return its `(Sender, Receiver)` halves.
///
/// The channel is unbounded (no capacity limit) and open until every
/// `Sender` has been dropped.
pub fn channel<T>() -> (Sender<T>, Receiver<T>) { pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
let inner = Arc::new(RawMutex::new_channel(Inner { let inner = Arc::new(RawMutex::new_channel(Inner {
queue: VecDeque::new(), queue: VecDeque::new(),
@@ -45,29 +109,40 @@ pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
struct Inner<T> { struct Inner<T> {
queue: VecDeque<T>, queue: VecDeque<T>,
/// The parked receiver's `(pid, park-epoch)`. The epoch is the slot /// The parked receiver's `(pid, park-epoch)`, if one is currently
/// word's runtime-wide wait identity (see slot_state.rs): wakers call /// waiting. The epoch identifies exactly which wait this is, so a waker
/// `unpark_at(pid, epoch)`, so an entry left over from an already-woken /// left over from a wait that already ended (a losing `select` arm, a
/// wait — a `select` loser arm, a satisfied `recv_timeout`'s timer — is /// `recv_timeout` whose timer fired after it was already satisfied) is
/// inert: the wake fails the word's epoch CAS and no-ops. This replaces /// inert and does nothing when it fires.
/// the old per-channel `cur_wait`/`next_wait_seq`/`timed_out` trio: wait
/// identity now exists exactly once, in the slot word.
parked_receiver: Option<(Pid, u32)>, parked_receiver: Option<(Pid, u32)>,
senders: usize, senders: usize,
receiver_alive: bool, receiver_alive: bool,
} }
/// The sending half of a channel, created by [`channel`]. Clonable: every
/// clone pushes onto the same queue, and the channel stays open as long as
/// any clone is alive. Dropping the last `Sender` closes the channel, which
/// wakes a parked [`Receiver`] so it can observe the closure.
pub struct Sender<T> { pub struct Sender<T> {
inner: Arc<RawMutex<Inner<T>>>, inner: Arc<RawMutex<Inner<T>>>,
} }
/// The receiving half of a channel, created by [`channel`]. Not clonable:
/// a channel has exactly one receiver. Reads messages in the order they
/// were sent, via [`recv`](Receiver::recv) and its variants.
pub struct Receiver<T> { pub struct Receiver<T> {
inner: Arc<RawMutex<Inner<T>>>, inner: Arc<RawMutex<Inner<T>>>,
} }
/// Returned by [`Sender::send`] when the channel's [`Receiver`] has already
/// been dropped. Carries the message back so it is never silently lost;
/// recover it with `.0` or by matching.
#[derive(Debug, PartialEq, Eq)] #[derive(Debug, PartialEq, Eq)]
pub struct SendError<T>(pub T); pub struct SendError<T>(pub T);
/// Returned by [`Receiver::recv`] (and the other receive methods, in their
/// own error types) when the channel is closed: every `Sender` has been
/// dropped and no message is left queued.
#[derive(Debug, PartialEq, Eq, Clone, Copy)] #[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub struct RecvError; pub struct RecvError;
@@ -84,8 +159,8 @@ impl std::error::Error for RecvError {}
pub enum RecvTimeoutError { pub enum RecvTimeoutError {
/// The deadline passed with no message available. /// The deadline passed with no message available.
Timeout, Timeout,
/// All senders dropped with no message available — the bounded analogue /// Every sender was dropped with no message available. The
/// of [`RecvError`]. /// timeout-aware counterpart of plain [`RecvError`].
Disconnected, Disconnected,
} }
@@ -115,8 +190,8 @@ impl<T> Drop for Sender<T> {
// Wake the parked receiver on the last sender drop regardless of // Wake the parked receiver on the last sender drop regardless of
// whether the queue is empty. A plain `recv` only ever parks on an // whether the queue is empty. A plain `recv` only ever parks on an
// empty queue (so this is unchanged for it), but a selective // empty queue (so this is unchanged for it), but a selective
// `recv_match` may be parked on a *non-empty* queue holding only // `recv_match` may be parked on a non-empty queue holding only
// non-matching messages — it must wake to observe closure and // non-matching messages. It must wake to observe closure and
// return Err rather than sleep forever. // return Err rather than sleep forever.
if g.senders == 0 { if g.senders == 0 {
g.parked_receiver.take() g.parked_receiver.take()
@@ -132,19 +207,37 @@ impl<T> Drop for Sender<T> {
impl<T> Drop for Receiver<T> { impl<T> Drop for Receiver<T> {
fn drop(&mut self) { fn drop(&mut self) {
self.inner.lock().receiver_alive = false; // The only consumer is gone: queued messages can never be delivered.
// Drop them now instead of leaving them queued until the last Sender
// happens to go away, which can be long after this receiver's owner
// has exited if some other part of the runtime is still holding a
// clone of the Sender. Draining runs each queued message's own drop
// glue, which matters for a gen_server call: dropping a queued call
// envelope drops its reply channel too, which wakes the caller with
// an error instead of leaving it parked forever. Drain under the
// lock, then run the drops after releasing it, since a message's
// drop glue may itself touch a different channel or the scheduler.
let drained = {
let mut g = self.inner.lock();
g.receiver_alive = false;
std::mem::take(&mut g.queue)
};
drop(drained);
} }
} }
impl<T> Sender<T> { impl<T> Sender<T> {
/// Number of messages currently queued behind this channel. Introspection /// Number of messages currently queued and not yet received. For
/// only (RFC 016 mailbox depth); takes the channel lock, so callers reach /// introspection and monitoring; takes the channel's internal lock, so
/// it under the registry Leaf (Leaf → Channel) via the erased probe in /// avoid calling it from a hot path.
/// `registry.rs`, never on a hot path.
pub(crate) fn queued_len(&self) -> usize { pub(crate) fn queued_len(&self) -> usize {
self.inner.lock().queue.len() self.inner.lock().queue.len()
} }
/// Push `value` onto the channel. Succeeds unconditionally as long as
/// the [`Receiver`] is still alive: the queue has no capacity limit, so
/// this never blocks and never fails except when the channel is closed,
/// in which case `value` comes back in [`SendError`].
pub fn send(&self, value: T) -> Result<(), SendError<T>> { pub fn send(&self, value: T) -> Result<(), SendError<T>> {
let unpark = { let unpark = {
let mut g = self.inner.lock(); let mut g = self.inner.lock();
@@ -165,6 +258,10 @@ impl<T> Sender<T> {
} }
impl<T> Receiver<T> { impl<T> Receiver<T> {
/// Block until a message is available and return it. Messages come back
/// in the order they were sent. If the queue is empty and every
/// [`Sender`] has already been dropped, returns [`RecvError`] instead of
/// blocking forever.
pub fn recv(&self) -> Result<T, RecvError> { pub fn recv(&self) -> Result<T, RecvError> {
loop { loop {
{ {
@@ -184,15 +281,15 @@ impl<T> Receiver<T> {
g.parked_receiver.is_none_or(|(p, _)| p == me), g.parked_receiver.is_none_or(|(p, _)| p == me),
"channel has more than one receiver" "channel has more than one receiver"
); );
// begin_wait is lock-free legal under the Channel lock; // begin_wait is lock-free, so it's legal under the Channel lock;
// registering in the same critical section makes the epoch // registering in the same critical section makes the epoch
// atomic with the senders' view of the registration. // atomic with the senders' view of the registration.
g.parked_receiver = Some((me, crate::scheduler::begin_wait())); g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
crate::te!(crate::trace::Event::RecvPark(me)); crate::te!(crate::trace::Event::RecvPark(me));
} }
// Release the lock before parking the unparker will need it. // Release the lock before parking: the unparker will need it.
crate::scheduler::park_current(); crate::scheduler::park_current();
// Woken up — record it before looping to check the queue. // Woken up. Record it before looping to check the queue.
crate::te!(crate::trace::Event::RecvWake(match crate::actor::current_pid() { crate::te!(crate::trace::Event::RecvWake(match crate::actor::current_pid() {
Some(p) => p, Some(p) => p,
None => panic!("smarm: RecvWake outside an actor (core corrupt)"), None => panic!("smarm: RecvWake outside an actor (core corrupt)"),
@@ -200,29 +297,18 @@ impl<T> Receiver<T> {
} }
} }
/// Bounded receive: like [`recv`](Self::recv), but gives up once /// Like [`recv`](Self::recv), but gives up and returns
/// `timeout` has elapsed, returning [`RecvTimeoutError::Timeout`]. /// [`RecvTimeoutError::Timeout`] if no message has arrived by the time
/// `timeout` elapses.
/// ///
/// Built on the same timer machinery as `Mutex::lock_timeout`: the wait /// If a message arrives at essentially the same moment the deadline
/// registers a `WaitTimeout` entry stamped with the wait's park-epoch; /// passes, the message wins: you get `Ok` rather than `Timeout`. If
/// on expiry the channel (as the /// every sender is dropped before a message arrives or the deadline
/// [`TimerTarget`](crate::timer::TimerTarget)) checks whether *this* /// passes, you get [`RecvTimeoutError::Disconnected`].
/// wait is still parked and, only then, cancels it. A wake that races
/// the deadline resolves message-first: if a message is available when
/// the receiver runs, it is delivered even if the timer had already
/// fired. A satisfied or abandoned wait leaves its timer entry to expire
/// as a no-op (registration gone; epoch consumed), per the
/// no-cancellation convention in `timer.rs`.
/// ///
/// The wake is classified from state alone — wakes are precise (the only /// `Duration::ZERO` is a valid timeout: it still gives any
/// stamped wakers of this wait are a send, the last-sender drop, and the /// already-queued message a chance to be returned, and only then
/// timer; a stop wake unwinds out of `park_current` and never reaches /// reports `Timeout`.
/// the classification), so: message queued → `Ok`; `senders == 0` →
/// `Disconnected`; neither → it was the timer → `Timeout`.
///
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-
/// due timer is drained, then reports `Timeout` unless a message was
/// already queued.
pub fn recv_timeout(&self, timeout: std::time::Duration) -> Result<T, RecvTimeoutError> pub fn recv_timeout(&self, timeout: std::time::Duration) -> Result<T, RecvTimeoutError>
where where
T: Send + 'static, T: Send + 'static,
@@ -254,7 +340,7 @@ impl<T> Receiver<T> {
// Arm the timer after releasing the channel lock (insert takes the // Arm the timer after releasing the channel lock (insert takes the
// timers lock; never nest under a Channel lock). A send or even the // timers lock; never nest under a Channel lock). A send or even the
// timer itself may unpark us before we park the RunningNotified // timer itself may unpark us before we park; the runtime's wake
// protocol makes the park below return immediately in that case. // protocol makes the park below return immediately in that case.
let deadline = crate::timer::deadline_from_now(timeout); let deadline = crate::timer::deadline_from_now(timeout);
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = self.inner.clone(); let target: std::sync::Arc<dyn crate::timer::TimerTarget> = self.inner.clone();
@@ -276,16 +362,23 @@ impl<T> Receiver<T> {
Err(RecvTimeoutError::Timeout) Err(RecvTimeoutError::Timeout)
} }
/// Selective receive: remove and return the first queued message for which /// Selective receive: find and return the first queued message for
/// `pred` holds, leaving the rest in arrival order. If no queued message /// which `pred` returns `true`, leaving every other message in the
/// matches, parks and re-scans on every send (a selective receiver may park /// queue untouched and in order. Useful when an actor's inbox mixes
/// on a *non-empty* queue). Returns `Err(RecvError)` only once the channel /// message kinds and it wants to handle one kind out of turn, without
/// is closed and no queued message matches. /// discarding the rest.
/// ///
/// `pred` is run while the channel lock is held: keep it cheap and pure, /// If nothing queued matches, this blocks and re-checks every time a new
/// and do not call back into this channel from inside it. It is modelled as /// message arrives, the same way [`recv`](Self::recv) blocks on an empty
/// `Fn` (not `FnMut`) deliberately — it is re-run from scratch on every /// queue: a selective receiver can be waiting even while the queue holds
/// scan, so a stateful predicate would observe surprising re-counting. /// messages, just none that match yet. Returns [`RecvError`] only once
/// the channel is closed and still nothing matches.
///
/// `pred` runs while the channel is locked, so keep it cheap, side
/// effect free, and make sure it never calls back into this same
/// channel. It takes `&T` and is called fresh on every scan (not `FnMut`
/// with running state), so it should judge each message purely on its
/// own content.
pub fn recv_match<F>(&self, pred: F) -> Result<T, RecvError> pub fn recv_match<F>(&self, pred: F) -> Result<T, RecvError>
where where
F: Fn(&T) -> bool, F: Fn(&T) -> bool,
@@ -317,7 +410,7 @@ impl<T> Receiver<T> {
g.parked_receiver = Some((me, crate::scheduler::begin_wait())); g.parked_receiver = Some((me, crate::scheduler::begin_wait()));
crate::te!(crate::trace::Event::RecvPark(me)); crate::te!(crate::trace::Event::RecvPark(me));
} }
// Release the lock before parking the unparker will need it. // Release the lock before parking: the unparker will need it.
crate::scheduler::park_current(); crate::scheduler::park_current();
crate::te!(crate::trace::Event::RecvWake(match crate::actor::current_pid() { crate::te!(crate::trace::Event::RecvWake(match crate::actor::current_pid() {
Some(p) => p, Some(p) => p,
@@ -326,10 +419,12 @@ impl<T> Receiver<T> {
} }
} }
/// Non-blocking selective receive. `Ok(Some(v))` if a queued message /// The non-blocking counterpart of [`recv_match`](Self::recv_match):
/// matched `pred` (removed, rest left in order), `Ok(None)` if the channel /// returns immediately either way. `Ok(Some(v))` if a queued message
/// is open but nothing matched, `Err(RecvError)` if closed and nothing /// matched `pred` (removed; the rest stay queued in order), `Ok(None)`
/// matched. Same predicate contract as [`recv_match`](Self::recv_match). /// if the channel is open but nothing currently matches, `Err(RecvError)`
/// if the channel is closed and nothing matches. Same predicate contract
/// as `recv_match`.
pub fn try_recv_match<F>(&self, pred: F) -> Result<Option<T>, RecvError> pub fn try_recv_match<F>(&self, pred: F) -> Result<Option<T>, RecvError>
where where
F: Fn(&T) -> bool, F: Fn(&T) -> bool,
@@ -349,8 +444,10 @@ impl<T> Receiver<T> {
Ok(None) Ok(None)
} }
/// Non-blocking. `Ok(Some(v))` if a message was available, `Ok(None)` if /// The non-blocking counterpart of [`recv`](Self::recv): returns
/// the channel is empty but open, `Err(RecvError)` if closed and drained. /// immediately either way. `Ok(Some(v))` if a message was queued,
/// `Ok(None)` if the channel is open but currently empty, `Err(RecvError)`
/// if the channel is closed and the queue is drained.
pub fn try_recv(&self) -> Result<Option<T>, RecvError> { pub fn try_recv(&self) -> Result<Option<T>, RecvError> {
let mut g = self.inner.lock(); let mut g = self.inner.lock();
if let Some(v) = g.queue.pop_front() { if let Some(v) = g.queue.pop_front() {
@@ -365,18 +462,18 @@ impl<T> Receiver<T> {
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// TimerTarget the expiry half of recv_timeout // TimerTarget: the expiry half of recv_timeout
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
impl<T: Send + 'static> crate::timer::TimerTarget for RawMutex<Inner<T>> { impl<T: Send + 'static> crate::timer::TimerTarget for RawMutex<Inner<T>> {
fn on_timeout(&self, pid: Pid, epoch: u32) { fn on_timeout(&self, pid: Pid, epoch: u32) {
// Cancel the wait only if THIS wait (epoch match) is still // Cancel the wait only if THIS wait (epoch match) is still
// registered. If a sender already took `parked_receiver`, the // registered. If a sender already took `parked_receiver`, the
// receiver is waking with a message message wins, the timer // receiver is waking with a message: message wins, the timer
// no-ops. If a later wait by the same receiver is registered, the // no-ops. If a later wait by the same receiver is registered, the
// epoch mismatches stale entry, no-op. (The unpark_at would fail // epoch mismatches: stale entry, no-op. (unpark_at would fail its
// its word CAS in either case anyway; checking under the lock keeps // internal check in either case anyway; checking under the lock
// the registration bookkeeping exact.) // keeps the registration bookkeeping exact.)
let unpark = { let unpark = {
let mut g = self.lock(); let mut g = self.lock();
if g.parked_receiver == Some((pid, epoch)) { if g.parked_receiver == Some((pid, epoch)) {
@@ -386,7 +483,7 @@ impl<T: Send + 'static> crate::timer::TimerTarget for RawMutex<Inner<T>> {
false false
} }
}; };
// Unpark outside the channel lock it may take the run-queue lock; // Unpark outside the channel lock: it may take the run-queue lock;
// legal under a Channel lock, but pointless to nest. // legal under a Channel lock, but pointless to nest.
if unpark { if unpark {
crate::scheduler::unpark_at(pid, epoch); crate::scheduler::unpark_at(pid, epoch);
@@ -395,7 +492,7 @@ impl<T: Send + 'static> crate::timer::TimerTarget for RawMutex<Inner<T>> {
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// select ready-index wait over multiple receivers // select: ready-index wait over multiple receivers
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
pub(crate) mod sealed { pub(crate) mod sealed {
@@ -403,33 +500,34 @@ pub(crate) mod sealed {
} }
impl<T> sealed::Sealed for Receiver<T> {} impl<T> sealed::Sealed for Receiver<T> {}
/// An arm of a [`select`]. Implemented by [`Receiver`]; sealed, because the /// An arm of a [`select`]: something you can wait on alongside other arms
/// registration contract below is part of the runtime's wake protocol. /// and be told when it becomes ready. Implemented by [`Receiver`]; sealed
/// (cannot be implemented outside this crate), since the registration
/// contract below is part of the runtime's internal wake protocol.
/// ///
/// Contract (all under the arm's own lock): `sel_register` checks-or- /// Contract (all under the arm's own lock): `sel_register` checks-or-
/// registers atomically — if the arm is ready it does NOT register and /// registers atomically. If the arm is ready it does not register and
/// returns `Ok(false)`; otherwise it publishes `(pid, epoch)` where its /// returns `Ok(false)`; otherwise it publishes `(pid, epoch)` where its
/// wakers will find it and returns `Ok(true)`. "Ready" means a receive /// wakers will find it and returns `Ok(true)`. "Ready" means a receive
/// would not park: a message is queued, or the arm is closed. `Err` means /// would not block: a message is queued, or the arm is closed. `Err` means
/// the arm could not register at all (only fd arms can fail; channel /// the arm could not register at all (only fd arms can fail; channel
/// registration is infallible) — the wait must be retired and earlier /// registration always succeeds), and the wait must be retired and earlier
/// eager-cleanup arms unregistered. /// eager-cleanup arms unregistered.
pub trait Selectable: sealed::Sealed { pub trait Selectable: sealed::Sealed {
#[doc(hidden)] #[doc(hidden)]
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool>; fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool>;
#[doc(hidden)] #[doc(hidden)]
fn sel_ready(&self) -> bool; fn sel_ready(&self) -> bool;
/// Remove this arm's `(pid, epoch)` registration if and only if it /// Remove this arm's `(pid, epoch)` registration if, and only if, it is
/// is still in place. Default no-op: a losing channel arm's stale /// still in place. Default no-op: a losing channel arm's stale
/// registration is inert (its wakers die at the epoch CAS; the next /// registration is harmless and self-cleans. Fd arms override this:
/// wait overwrites the slot). Fd arms override this: their staleness /// their staleness would otherwise leave the fd unusable for future
/// poisons the fd (waiters entry + kernel-side ONESHOT registration) /// selects, so they need an eager cleanup pass.
/// and needs an eager cleanup pass.
#[doc(hidden)] #[doc(hidden)]
fn sel_unregister(&self, _pid: Pid, _epoch: u32) {} fn sel_unregister(&self, _pid: Pid, _epoch: u32) {}
/// Whether this arm requires the eager cleanup pass at all. Gates the /// Whether this arm requires the eager cleanup pass at all. Gates the
/// post-wake `sel_unregister` sweep so channel-only selects keep /// post-wake `sel_unregister` sweep so channel-only selects keep their
/// today's zero-cancellation hot path. /// cheap, cleanup-free path.
#[doc(hidden)] #[doc(hidden)]
fn sel_eager_cleanup(&self) -> bool { fn sel_eager_cleanup(&self) -> bool {
false false
@@ -456,38 +554,35 @@ impl<T> Selectable for Receiver<T> {
} }
} }
/// Park on every arm at once; return the index of the first ready one. /// Wait on several channels at once and return the index of the first one
/// that is ready, instead of blocking on just one with [`Receiver::recv`].
/// ///
/// "Ready" means a receive on that arm would not park: a message is queued, /// "Ready" means a receive on that arm would not block: a message is
/// or the arm is **closed** (so the caller's `try_recv` observes the /// queued, or the arm is closed (so the caller's own `try_recv` observes
/// disconnect a dead arm is an event, not a hang). The caller consumes the /// the disconnect: a dead arm is something to react to, not something to
/// arm itself, typically via [`Receiver::try_recv`]; single-receiver /// hang on). `select` only tells you which arm is ready; read the actual
/// channels guarantee nothing can steal the message in between. /// message yourself, typically with [`Receiver::try_recv`] on that arm.
/// ///
/// A closed arm stays ready *forever*: once its disconnect has been /// A closed arm stays ready forever. Once you have observed its disconnect,
/// observed, drop it from the arm set — under priority order it would /// drop it from the arm set you pass in next time: otherwise, under the
/// otherwise win every subsequent call and starve every higher-indexed arm. /// priority order below, it would win every subsequent call and starve
/// every arm listed after it.
/// ///
/// Arms are scanned **in order**: index 0 is the highest priority, both on /// Arms are checked **in order**: index 0 is the highest priority, both
/// the immediate-ready path and after a wake. This is a documented /// when checking immediately and after being woken. This is a deliberate,
/// guarantee (compose like BEAM receive clauses: put control channels /// documented guarantee, not an accident of implementation: put a control
/// first), not an accident — and therefore there is NO fairness promise; a /// or shutdown channel first so it is always noticed promptly. The
/// saturated arm 0 starves arm 1 by design. /// flip side is that there is **no fairness guarantee**: a busy arm 0 can
/// starve arm 1 indefinitely by design.
/// ///
/// One actor may select on a channel and later `recv` on it (or select on /// One actor can `select` on a channel and later plain `recv` on it (or
/// overlapping sets) freely. What stays illegal is what was always illegal: /// `select` again on an overlapping set of arms) with no restriction. What
/// two *different* actors receiving on one channel. /// stays illegal is what was always illegal for a channel: two *different*
/// actors receiving on the same one.
/// ///
/// Built on the consuming-wake protocol (see slot_state.rs): all arms are /// Panics if `arms` is empty, if called outside an actor, or if an fd arm
/// registered under one wait epoch; the winning wake consumes it, so losing /// fails to register (see [`try_select`] for the fallible form; a
/// arms' registrations are inert and need no cancellation pass — they /// channel-only `select` can never fail).
/// self-clean at their wakers' failed CAS, or get overwritten by this
/// receiver's next wait on that channel.
///
/// Panics if `arms` is empty, when called outside an actor, or if an fd
/// arm fails to register (EBADF, EMFILE, a second waiter on one fd —
/// see [`try_select`] for the fallible form; channel-only selects cannot
/// fail).
pub fn select(arms: &[&dyn Selectable]) -> usize { pub fn select(arms: &[&dyn Selectable]) -> usize {
match try_select(arms) { match try_select(arms) {
Ok(i) => i, Ok(i) => i,
@@ -495,9 +590,10 @@ pub fn select(arms: &[&dyn Selectable]) -> usize {
} }
} }
/// [`select`], fallible: `Err` when an arm fails to register (only fd /// The fallible form of [`select`]: `Err` when an arm fails to register.
/// arms can — EBADF, EMFILE on the epoll set, or a second waiter on an /// Only fd arms can fail this way (for example, the file descriptor is
/// fd that already has one). On `Err` the wait is fully retired and no /// invalid, or something else is already waiting on it); a channel-only
/// select can never fail. On `Err` the wait is fully retired and no
/// registration is left behind: every arm registered before the failing /// registration is left behind: every arm registered before the failing
/// one has been unregistered. /// one has been unregistered.
pub fn try_select(arms: &[&dyn Selectable]) -> std::io::Result<usize> { pub fn try_select(arms: &[&dyn Selectable]) -> std::io::Result<usize> {
@@ -513,13 +609,12 @@ pub fn try_select(arms: &[&dyn Selectable]) -> std::io::Result<usize> {
} }
// Stale fd registrations are not harmless (a losing fd arm's // Stale fd registrations are not harmless (a losing fd arm's
// waiters entry poisons the fd with AlreadyExists and its // leftover registration can make the fd unusable for the next
// kernel-side ONESHOT registration can fire arbitrarily late), so // select until a kernel event happens to clear it), so selects
// selects containing fd arms run an eager cleanup pass after the // containing fd arms run an eager cleanup pass after the park,
// park — including when a terminal stop unwinds out of it, via // including when a terminal stop unwinds out of it, via the guard.
// the guard. Channel-only selects skip all of it: `eager` is // Channel-only selects skip all of it: `eager` is false, the guard
// false, the guard is disarmed, and the loser-arm self-cleaning // is disarmed, and the loser-arm self-cleaning story is unchanged.
// story is unchanged.
let eager = arms.iter().any(|a| a.sel_eager_cleanup()); let eager = arms.iter().any(|a| a.sel_eager_cleanup());
let mut guard = UnregisterGuard { arms, me, epoch, armed: eager }; let mut guard = UnregisterGuard { arms, me, epoch, armed: eager };
@@ -532,22 +627,22 @@ pub fn try_select(arms: &[&dyn Selectable]) -> std::io::Result<usize> {
drop(guard); drop(guard);
// Woken precisely: an arm's send (message) or last-sender drop // Woken precisely: an arm's send (message) or last-sender drop
// (closure) consumed our epoch, and both leave their arm ready // (closure) is what woke us, and both leave their arm ready.
// return the first one, in priority order (which may be a // Return the first ready one, in priority order (which may be a
// different, higher-priority arm than the one that woke us; its // different, higher-priority arm than the one that woke us; its
// message stays queued and re-reports ready on the next call). // message stays queued and re-reports ready on the next call).
// Fd arms classify by a fresh zero-timeout poll, so they too are // Fd arms classify by a fresh zero-timeout poll, so they too are
// a pure function of state independent of the registration the // a pure function of current state, independent of the
// cleanup pass just removed. // registration the cleanup pass just removed.
for (i, arm) in arms.iter().enumerate() { for (i, arm) in arms.iter().enumerate() {
if arm.sel_ready() { if arm.sel_ready() {
return Ok(i); return Ok(i);
} }
} }
// Unreachable by protocol (a stop wake unwinds out of // Unreachable in practice (a stop wake unwinds out of
// park_current). Defensive: re-open the wait and re-register — // park_current before we get here). Defensive: re-open the wait
// stale own-registrations are overwritten (channels) or were // and re-register; stale own-registrations are overwritten
// removed by the cleanup pass above (fds). // (channels) or were removed by the cleanup pass above (fds).
} }
} }
@@ -562,11 +657,11 @@ fn unregister_arms(arms: &[&dyn Selectable], me: Pid, epoch: u32) {
} }
} }
/// Stop-unwind twin of the explicit cleanup pass: a terminal stop unwinds // Stop-unwind twin of the explicit cleanup pass: a terminal stop unwinds
/// out of `park_current`, and a registered fd arm must not outlive its // out of `park_current`, and a registered fd arm must not outlive its
/// actor (the generalization of `wait_fd`'s `Dereg`). Disarmed on the // actor. Disarmed on the normal path after the explicit pass runs; never
/// normal path after the explicit pass runs; never armed when no fd arm // armed when no fd arm is registered, keeping the channel-only path
/// registered, keeping the channel-only path guard-free in effect. // guard-free in effect.
struct UnregisterGuard<'a> { struct UnregisterGuard<'a> {
arms: &'a [&'a dyn Selectable], arms: &'a [&'a dyn Selectable],
me: Pid, me: Pid,
@@ -582,20 +677,16 @@ impl Drop for UnregisterGuard<'_> {
} }
} }
/// The registration pass shared by [`select`] and [`select_timeout`]: // The registration pass shared by `select` and `select_timeout`: check-or-
/// check-or-register each arm, in priority order, each atomically under its // register each arm, in priority order, each atomically under its own lock.
/// own lock. Cross-arm atomicity is unnecessary: an arm becoming ready // Cross-arm atomicity is unnecessary: an arm becoming ready right after its
/// right after its registration wakes the caller through the protocol (the // registration still wakes the caller through the normal wake path.
/// prep-to-park window is closed by RunningNotified). //
/// // `Ok(Some(i))` = arm `i` was already ready, the pass stopped, and the wait
/// `Ok(Some(i))` = arm `i` was ready, the pass stopped, and the wait has // has been fully retired (no park may follow): earlier fd arms are
/// been RETIRED (no park may follow): earlier arms hold live-epoch // unregistered eagerly so none are left dangling. `Err` = an arm failed to
/// registrations, so earlier *fd* arms are unregistered eagerly, then the // register; same unwind (earlier fd arms unregistered, wait retired).
/// epoch is bumped, a landed notification eaten, and a pending stop // `Ok(None)` = every arm registered successfully; the caller parks.
/// re-observed — without which a stale arm wake could fault a later
/// one-shot park. `Err` = an arm failed to register; identical unwind
/// (earlier fd arms unregistered, wait retired). `Ok(None)` = every arm
/// registered; the caller parks.
fn register_arms( fn register_arms(
me: Pid, me: Pid,
epoch: u32, epoch: u32,
@@ -619,10 +710,10 @@ fn register_arms(
Ok(None) Ok(None)
} }
/// The [`select_timeout`] timer target: stateless, because precise wakes // The `select_timeout` timer target: stateless, because a wake's cause can
/// make classification a pure function of channel state. The entry is // always be read back off plain channel state (an arm ready, or not). If
/// stamped with the select's epoch; if an arm already won, this unpark dies // an arm already won before the deadline, this timer's fire is simply
/// at the word's epoch CAS (the no-cancellation convention in `timer.rs`). // ignored, the way any other stale wakeup is.
struct SelectTimeout; struct SelectTimeout;
impl crate::timer::TimerTarget for SelectTimeout { impl crate::timer::TimerTarget for SelectTimeout {
fn on_timeout(&self, pid: Pid, epoch: u32) { fn on_timeout(&self, pid: Pid, epoch: u32) {
@@ -630,27 +721,21 @@ impl crate::timer::TimerTarget for SelectTimeout {
} }
} }
/// [`select`] with a deadline: returns `Some(index)` like `select`, or /// Like [`select`], but gives up and returns `None` if no arm becomes
/// `None` once `timeout` elapses with no arm ready. /// ready before `timeout` elapses.
/// ///
/// All of `select`'s semantics carry over (priority order, closed arms /// All of `select`'s semantics carry over: arms are still checked in
/// permanently ready, no fairness promise). The timeout is one more stamped /// priority order, a closed arm is still permanently ready, and there is
/// waker on the same wait epoch — nothing is registered in any arm for it, /// still no fairness guarantee across arms. A message that arrives at
/// so there is nothing to cancel or leak: an arm winning leaves the timer /// essentially the same moment the deadline passes still wins, the same
/// entry to expire as a stale-epoch no-op; the timer winning leaves the /// way [`Receiver::recv_timeout`] resolves that race.
/// arms' registrations to self-clean exactly as a `select` loser's would.
/// ///
/// The wake is classified from state alone (wakes are precise): some arm /// `Duration::ZERO` is a valid timeout: it still gives an already-ready arm
/// ready → `Some` of the first, in priority order; none ready → the timer /// a chance to be reported before falling through to `None`.
/// was the only remaining stamped waker → `None`. A message that races the
/// deadline resolves message-first, as `recv_timeout` does.
/// ///
/// `Duration::ZERO` is a valid timeout: it parks until the immediately-due /// Panics if `arms` is empty, if called outside an actor, or if an fd arm
/// timer is drained, then reports `None` unless an arm was already ready. /// fails to register (see [`try_select_timeout`] for the fallible form; a
/// /// channel-only select can never fail).
/// Panics if `arms` is empty, when called outside an actor, or if an fd
/// arm fails to register (see [`try_select_timeout`] for the fallible
/// form; channel-only selects cannot fail).
pub fn select_timeout( pub fn select_timeout(
arms: &[&dyn Selectable], arms: &[&dyn Selectable],
timeout: std::time::Duration, timeout: std::time::Duration,
@@ -663,9 +748,9 @@ pub fn select_timeout(
} }
} }
/// [`select_timeout`], fallible: `Err` when an arm fails to register /// The fallible form of [`select_timeout`]: `Err` when an arm fails to
/// (only fd arms can). On `Err` the wait is fully retired and no /// register (only fd arms can). On `Err` the wait is fully retired and no
/// registration — arm-side or kernel-side — is left behind. /// registration is left behind on any arm.
pub fn try_select_timeout( pub fn try_select_timeout(
arms: &[&dyn Selectable], arms: &[&dyn Selectable],
timeout: std::time::Duration, timeout: std::time::Duration,
@@ -680,17 +765,16 @@ pub fn try_select_timeout(
return Ok(Some(i)); // ready now: the timer was never armed return Ok(Some(i)); // ready now: the timer was never armed
} }
// Arm the timer after the registration pass, outside every Channel // Arm the timer after the registration pass, outside every channel
// lock (insert takes the timers lock). // lock (inserting a timer takes the timers lock).
let deadline = crate::timer::deadline_from_now(timeout); let deadline = crate::timer::deadline_from_now(timeout);
let target: std::sync::Arc<dyn crate::timer::TimerTarget> = std::sync::Arc::new(SelectTimeout); let target: std::sync::Arc<dyn crate::timer::TimerTarget> = std::sync::Arc::new(SelectTimeout);
crate::scheduler::insert_wait_timer(deadline, me, target, epoch); crate::scheduler::insert_wait_timer(deadline, me, target, epoch);
// Same eager-cleanup story as `try_select`: the timer arm needs none // Same eager-cleanup story as `try_select`: a timer win in particular
// (stateless, stale entries die at the epoch CAS), channel arms need // leaves every fd arm's registration behind, which without this pass
// none, fd arms do — and a timer win in particular leaves every fd // would leave those fds unusable until a kernel event happened to
// arm's registration behind, which without this pass would poison // clear them.
// those fds until a kernel event happened to fire.
let eager = arms.iter().any(|a| a.sel_eager_cleanup()); let eager = arms.iter().any(|a| a.sel_eager_cleanup());
let mut guard = UnregisterGuard { arms, me, epoch, armed: eager }; let mut guard = UnregisterGuard { arms, me, epoch, armed: eager };
+181 -83
View File
@@ -1,26 +1,85 @@
//! RFC 016 — runtime introspection (Chunk 1: the read primitive). //! Inspect what is running right now: which actors exist, what state each one
//! is in, and how they are related.
//! //!
//! A synchronous, internal read of the slab that returns *owned* data. This is //! This is the tool for questions like "is my server still alive", "how many
//! the mechanism the whole RFC hangs off: tests, the future observer //! actors are currently parked waiting on something", or "what does the spawn
//! gen_server (Chunk 4), and a later control plane (RFC 003) are all consumers //! tree look like". It is meant for debugging, test assertions, a health check
//! of [`snapshot`] / [`actor_info`], never of the runtime internals directly. //! endpoint, or a monitoring dashboard: anywhere you want to look at the
//! runtime from the outside without stopping it or coupling your code to its
//! internals.
//! //!
//! ## Consistency (DECISION D2 — per-slot tearing, `ps` semantics) //! Three entry points, in order of scope:
//! //!
//! [`snapshot`] is point-in-time and mildly racy *across* actors: each slot's //! - [`snapshot`] returns every actor that currently exists, as a plain
//! scheduling state is a lock-free word load, so an actor reported `Running` //! owned `Vec`, so you can filter, count, or search it however you like.
//! may already be `Parked`, and an actor can die mid-scan. This is the cheap, //! - [`actor_info`] returns a coherent view of exactly one actor, by pid.
//! useful model (a coherent stop-the-world cut is expensive and rarely wanted). //! Cheaper than filtering a whole snapshot down to one entry, and more
//! [`actor_info`] is coherent for the single actor it names. //! precise (see "Consistency" below).
//! - [`tree`] returns the same actors as [`snapshot`], folded into a
//! parent/child forest that mirrors who spawned whom.
//! //!
//! ## Locking //! ```
//! use smarm::{actor_info, channel, run, snapshot, spawn, ActorState};
//! //!
//! The lock order is **Leaf → Channel, at most one of each** (`raw_mutex.rs`); //! run(|| {
//! cold locks, the registry, and the free list are all Leaves, so we may never //! let (ready_tx, ready_rx) = channel::<()>();
//! hold two at once. The read is therefore phased: first a single registry-leaf //! let (gate_tx, gate_rx) = channel::<()>();
//! pass for names and mailbox depth (the per-channel length read is a Channel //!
//! lock taken under that Leaf — legal), released before the slab scan takes any //! let worker = spawn(move || {
//! per-slot cold Leaf. //! ready_tx.send(()).unwrap();
//! gate_rx.recv().unwrap(); // blocks here until released
//! });
//! ready_rx.recv().unwrap();
//!
//! // `snapshot` sees every actor, including this one and the worker.
//! let snap = snapshot();
//! assert!(snap.actors.len() >= 2);
//!
//! // `actor_info` gives a coherent view of just the worker. It is
//! // blocked on the gate channel, so it must be Parked.
//! let pid = worker.pid();
//! let info = actor_info(pid).expect("worker is still alive");
//! assert_eq!(info.state, ActorState::Parked);
//!
//! gate_tx.send(()).unwrap();
//! worker.join().unwrap();
//!
//! // Once joined, the pid no longer names a live actor.
//! assert!(actor_info(pid).is_none());
//! });
//! ```
//!
//! ## Consistency
//!
//! [`snapshot`] is not a single atomic pause-the-world freeze: it walks every
//! actor's state one after another, so it is a series of independent,
//! cheap, lock-free reads rather than one coherent moment in time. Between
//! reading actor A and actor B, either one can change state, and an actor can
//! even finish and disappear mid-scan. In practice this is exactly what you
//! want: a coherent stop-the-world snapshot would mean pausing every actor in
//! the runtime just to look at it, which is expensive and rarely necessary
//! for a dashboard, a test assertion, or a debugging session.
//!
//! [`actor_info`], in contrast, is coherent for the one actor it names: all of
//! its fields describe the same instant for that actor, because a single
//! actor's data cannot tear the way a scan across many actors can.
//!
//! ## Implementation notes
//!
//! These details matter if you are working on smarm itself; they are not part
//! of the public contract.
//!
//! The read never stops the scheduler and never holds a lock across the whole
//! scan. Each actor's scheduling state is a single lock-free word load
//! (hence the possible tearing described above). Reading the rest of an
//! actor's cold data (its supervisor, monitors, links, and so on) takes a
//! brief per-actor lock, just long enough to copy those fields out; nothing
//! is held across actors. Locking follows the crate-wide rule that at most
//! one "leaf" lock (a per-actor lock, the registry lock, or the free list
//! lock) is held at a time, with no leaf lock held while acquiring another.
//! The read is phased accordingly: first one pass over the registry to
//! collect every actor's registered names and mailbox depth, released before
//! the per-actor scan begins.
use crate::pid::Pid; use crate::pid::Pid;
use crate::registry::MailboxInfo; use crate::registry::MailboxInfo;
@@ -31,15 +90,28 @@ use crate::slot_state::{
}; };
use std::collections::HashMap; use std::collections::HashMap;
/// Snapshot wire-format version (DECISION D1). [`RuntimeSnapshot`] is treated as /// The format version carried by every [`RuntimeSnapshot`] and
/// a stable type from day one: it becomes the observer protocol (Chunk 4) and /// [`RuntimeTree`], as [`RuntimeSnapshot::format_version`] /
/// crosses a version boundary the moment a remote observer attaches (RFC 011), /// [`RuntimeTree::format_version`]. If you serialize a snapshot (for example
/// so the version travels with the data from the start. /// to send it somewhere else, or to compare snapshots taken with different
/// versions of smarm) check this field: a change in its value means the shape
/// of [`ActorInfo`] or its neighbors has changed and old and new snapshots
/// should not be assumed compatible. If you only ever read a snapshot
/// in-process in the same version of smarm that produced it, you can ignore
/// this field.
pub const SNAPSHOT_FORMAT_VERSION: u16 = 1; pub const SNAPSHOT_FORMAT_VERSION: u16 = 1;
/// Fine-grained scheduling state, mapped from the packed slot word with no new /// What an actor is doing right now, from the scheduler's point of view.
/// storage. `RunningNotified` collapses into `Notified` — a wake landed while ///
/// the actor was on-CPU and it will re-queue when it yields. /// - `Queued`: runnable, waiting for a scheduler thread to pick it up.
/// - `Running`: currently executing on a scheduler thread.
/// - `Notified`: was running and got woken up (for example, a message
/// arrived) before it had a chance to yield or park; it will be re-queued
/// as soon as it does.
/// - `Parked`: blocked, waiting on something such as a channel receive, a
/// mutex, a timer, or an IO event.
/// - `Done`: has finished (returned or panicked) but its slot has not been
/// reclaimed for reuse yet, so it is still visible to introspection.
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ActorState { pub enum ActorState {
Queued, Queued,
@@ -49,8 +121,8 @@ pub enum ActorState {
Done, Done,
} }
/// Classify a packed state word. `None` for a Vacant slot (skipped by the scan) /// Classify a packed state word. `None` for a Vacant slot (skipped by the
/// — the only state that is not an actor. /// scan): a vacant slot holds no actor at all, live or done.
fn classify(w: u64) -> Option<ActorState> { fn classify(w: u64) -> Option<ActorState> {
Some(match word_state(w) { Some(match word_state(w) {
ST_QUEUED => ActorState::Queued, ST_QUEUED => ActorState::Queued,
@@ -62,61 +134,76 @@ fn classify(w: u64) -> Option<ActorState> {
}) })
} }
/// Owned, point-in-time view of one actor — no borrows of runtime internals, so /// An owned, self-contained view of one actor at (approximately) one moment.
/// it is safe to hand to any consumer. /// It borrows nothing from the runtime, so you can keep it, send it
/// elsewhere, or print it long after the actor it describes has changed
/// state or even exited.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct ActorInfo { pub struct ActorInfo {
pub pid: Pid, pub pid: Pid,
/// Registered names, inverted from the registry (usually 0 or 1). /// Names this actor is currently registered under (see the
/// [`registry`](crate::registry) module). Usually empty or one name;
/// an actor can have more if it registered several.
pub names: Vec<&'static str>, pub names: Vec<&'static str>,
pub state: ActorState, pub state: ActorState,
/// Spawn-time parent edge (DECISION D9): `spawn_under` sets it to the /// The actor that spawned this one: whoever called `spawn` or
/// supervisor, plain `spawn` to the spawning actor — so it is parentage, /// `spawn_under` to create it. This is a parentage record, not
/// not necessarily a supervision relationship. `ROOT_PID` for the run's /// necessarily a supervision relationship: `spawn_under` records the
/// root actor and for `Done` tombstones (whose `Actor` is already gone). /// supervisor you asked for, while plain `spawn` records the spawning
/// actor itself, whether or not it supervises anything. It is the
/// runtime's root pid for the run's own root actor, and for a `Done`
/// actor whose bookkeeping has already been cleared.
pub supervisor: Pid, pub supervisor: Pid,
pub trap_exit: bool, pub trap_exit: bool,
pub monitors: u32, pub monitors: u32,
pub links: u32, pub links: u32,
pub joiners: u32, pub joiners: u32,
/// Queued messages summed over the actor's *published* channels (register / /// Messages currently queued and not yet delivered, summed across every
/// install / spawn_addr / gen_server). 0 for an actor that holds only a /// channel this actor has published (via `register`, `install`,
/// private `channel()` receiver — those are invisible to the registry. /// `spawn_addr`, or starting a gen_server). This is 0 for an actor that
/// only holds a private, unpublished `channel()` receiver, since nothing
/// outside the actor can see that channel exists.
pub mailbox_depth: u32, pub mailbox_depth: u32,
/// Timeslice overruns tallied for this incarnation (RFC 016 Chunk 2): how /// How many times this actor has been preempted for running past its
/// many times the actor was preempted for exceeding its slice. Resets on /// scheduling timeslice. Counts only since the actor's current start (a
/// restart (per-incarnation, D7). /// supervisor restart begins a fresh count).
pub overruns: u64, pub overruns: u64,
/// Messages this actor has received (dequeued) this incarnation (RFC 016 /// How many messages this actor has received (taken off its inbox), since
/// Chunk 2) — answers "is this actor a hotspot / draining slower than its /// its current start. Useful for spotting an actor whose mailbox is
/// mailbox fills." Counts received, not sent (D4). Per-incarnation (D7). /// filling up faster than it can drain it: compare this against
/// `mailbox_depth` over time.
pub messages_received: u64, pub messages_received: u64,
/// Approximate on-CPU cycles this incarnation has consumed (RFC 016 Chunk 2) /// Approximate CPU cycles this actor has spent running, since its current
/// — a reductions-like work metric for relative comparison. Always 0 unless /// start. A relative measure for comparing actors against each other, not
/// the `budget-accounting` feature is enabled (it costs an RDTSC per resume, /// an absolute or wall-clock figure. Always 0 unless the crate's
/// D6). Per-incarnation (D7). /// `budget-accounting` feature is enabled, since measuring it costs a
/// timestamp read on every resume.
pub budget_cycles: u64, pub budget_cycles: u64,
} }
/// A whole-runtime snapshot. See the module docs for the D2 tearing model. /// A snapshot of every actor in the runtime at (approximately) one moment.
/// See the module docs' "Consistency" section for what "approximately" means
/// here.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct RuntimeSnapshot { pub struct RuntimeSnapshot {
pub format_version: u16, pub format_version: u16,
pub actors: Vec<ActorInfo>, pub actors: Vec<ActorInfo>,
} }
/// Snapshot every live (and `Done`-but-not-yet-reclaimed) actor on the slab. /// Every actor that currently exists: running, queued, parked, or finished
/// O(n) over the slot table, running with preemption disabled (like every /// but not yet cleaned up. Cheap and lock-free per actor; see the module
/// runtime primitive) but holding no lock across the scan. Panics outside /// docs for what "approximately one moment" means for the result as a whole.
/// `Runtime::run()`; callable from actor code and the run thread. /// Panics if called outside [`run`](crate::run).
pub fn snapshot() -> RuntimeSnapshot { pub fn snapshot() -> RuntimeSnapshot {
with_runtime(|inner| { with_runtime(|inner| {
// Phase A: one registry-leaf pass for names + mailbox depth, released // First pass: one registry lock to collect names + mailbox depth for
// before any cold leaf (no two Leaves at once). // every actor, released before touching any per-actor lock below.
let mail = inner.registry.lock().introspect_map(); let mail = inner.registry.lock().introspect_map();
// Phase B: lock-free slab scan; per-slot cold leaf only to copy cold // Second pass: walk the actor table. Each actor's scheduling state is
// fields. Tearing across slots is intentional (D2). // a lock-free word load; only copying its other fields takes a brief
// per-actor lock. Tearing across actors is expected here (see the
// module docs' "Consistency" section).
let mut actors = Vec::new(); let mut actors = Vec::new();
for (idx, slot) in inner.slots.iter().enumerate() { for (idx, slot) in inner.slots.iter().enumerate() {
let idx = idx as u32; let idx = idx as u32;
@@ -128,8 +215,11 @@ pub fn snapshot() -> RuntimeSnapshot {
}) })
} }
/// Coherent view of a single actor, or `None` if the pid is stale, out of /// A coherent view of exactly one actor, or `None` if `pid` does not name a
/// range, or names a Vacant slot. /// currently-live entry: it is stale (that actor has already exited and its
/// slot was reused by another), out of range, or was never a real pid at
/// all. Unlike [`snapshot`], every field of the result describes the same
/// instant, since there is only one actor to read.
pub fn actor_info(pid: Pid) -> Option<ActorInfo> { pub fn actor_info(pid: Pid) -> Option<ActorInfo> {
with_runtime(|inner| { with_runtime(|inner| {
let slot = inner.slot_at(pid)?; let slot = inner.slot_at(pid)?;
@@ -141,11 +231,12 @@ pub fn actor_info(pid: Pid) -> Option<ActorInfo> {
}) })
} }
/// Build one `ActorInfo` for slot `idx`, or `None` if Vacant or /// Build one `ActorInfo` for slot `idx`, or `None` if the slot is empty or
/// racing-reclaimed. State is classified from a lock-free word load (the torn /// was reclaimed while this read was in progress. The scheduling state comes
/// read); the cold lock then pins the generation (reclaim bumps it under that /// from a lock-free word load (the source of the tearing described in the
/// same lock) so the cold fields are coherent for this incarnation. `mail` is /// module docs); the per-actor lock then confirms the actor has not since
/// this slot's registry entry, if any. /// exited and been replaced, so the rest of the fields are coherent for this
/// exact actor. `mail` is this slot's registry entry, if any.
fn read_slot(slot: &Slot, idx: u32, mail: Option<&MailboxInfo>) -> Option<ActorInfo> { fn read_slot(slot: &Slot, idx: u32, mail: Option<&MailboxInfo>) -> Option<ActorInfo> {
let w = slot.state_word(); let w = slot.state_word();
let state = classify(w)?; let state = classify(w)?;
@@ -153,10 +244,11 @@ fn read_slot(slot: &Slot, idx: u32, mail: Option<&MailboxInfo>) -> Option<ActorI
let pid = Pid::new(idx, gen); let pid = Pid::new(idx, gen);
let cold = slot.cold.lock(); let cold = slot.cold.lock();
// If the generation moved between the lock-free load and acquiring the cold // If the generation moved between the lock-free load and acquiring the
// lock, the slot was reclaimed (and maybe reused) — drop it rather than mix // per-actor lock, this actor exited (and the slot may already hold a new
// one incarnation's state with another's cold data. (ps semantics: a racing // one). Drop it rather than mix one actor's state with another's data; a
// actor may simply be missed mid-scan.) // racing actor may simply be missed by this scan, which is expected (see
// the module docs' "Consistency" section).
if word_gen(slot.state_word()) != gen { if word_gen(slot.state_word()) != gen {
return None; return None;
} }
@@ -172,7 +264,7 @@ fn read_slot(slot: &Slot, idx: u32, mail: Option<&MailboxInfo>) -> Option<ActorI
let joiners = cold.waiters.len() as u32; let joiners = cold.waiters.len() as u32;
drop(cold); drop(cold);
// Counters are hot-region atomics, read lock-free (RFC 016 Chunk 2). // Counters are plain atomics, read lock-free.
let overruns = slot.overruns(); let overruns = slot.overruns();
let messages_received = slot.messages_received(); let messages_received = slot.messages_received();
let budget_cycles = slot.budget_cycles(); let budget_cycles = slot.budget_cycles();
@@ -202,39 +294,45 @@ fn read_slot(slot: &Slot, idx: u32, mail: Option<&MailboxInfo>) -> Option<ActorI
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Chunk 3 — tree view (pure derivation over a Chunk-1 snapshot) // Tree view: a pure derivation over a snapshot
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// One node in the parentage forest. `children` are the actors whose recorded /// One node in the parentage forest returned by [`tree`]. `children` are the
/// parent edge points at this node's pid. /// actors whose recorded parent (see [`ActorInfo::supervisor`]) points at
/// this node's actor.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct TreeNode { pub struct TreeNode {
pub info: ActorInfo, pub info: ActorInfo,
/// The actor's recorded parent was absent from the snapshot (already /// True if this actor's recorded parent was not found in the snapshot
/// Done/Vacant, or itself a tombstone), so it was re-rooted under the forest /// (it had already exited, or was itself missing), so this node was
/// sentinel rather than dropped — the tree stays total (DECISION D8). /// placed at the top of the forest instead of being dropped. This keeps
/// every actor in the snapshot visible somewhere in the tree, even one
/// whose parent is gone.
pub orphaned: bool, pub orphaned: bool,
pub children: Vec<TreeNode>, pub children: Vec<TreeNode>,
} }
/// The parentage forest. Roots are actors parented at `ROOT_PID` (genuine /// The parentage forest: every actor from a snapshot, arranged by who spawned
/// roots) plus re-rooted orphans. The edge is *spawned-by / parent*, not /// whom. Roots are actors with no parent in the snapshot (including the
/// necessarily supervision (DECISION D9) — see [`ActorInfo::supervisor`]. /// run's own root actor) plus any orphaned actors (see [`TreeNode::orphaned`]).
/// This mirrors spawn parentage, not necessarily a supervision tree; see
/// [`ActorInfo::supervisor`].
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct RuntimeTree { pub struct RuntimeTree {
pub format_version: u16, pub format_version: u16,
pub roots: Vec<TreeNode>, pub roots: Vec<TreeNode>,
} }
/// Take a live [`snapshot`] and fold it into the parentage forest. /// Take a fresh [`snapshot`] and fold it into the parentage forest.
pub fn tree() -> RuntimeTree { pub fn tree() -> RuntimeTree {
tree_from(snapshot()) tree_from(snapshot())
} }
/// Fold an existing snapshot into a forest by grouping each actor under its /// Fold an existing snapshot into a parentage forest by grouping each actor
/// parent pid — a single O(n) pass, no new reads. Exposed separately so a /// under its parent, without taking a new snapshot. Useful if you already
/// consumer that already holds a snapshot (or a synthetic one, in tests) can /// have one (for example, one built in a test, or one you took earlier and
/// derive the tree without a second scan. /// want to inspect again) and want the tree view of it without re-reading
/// the runtime.
pub fn tree_from(snap: RuntimeSnapshot) -> RuntimeTree { pub fn tree_from(snap: RuntimeSnapshot) -> RuntimeTree {
let RuntimeSnapshot { format_version, actors } = snap; let RuntimeSnapshot { format_version, actors } = snap;
@@ -254,7 +352,7 @@ pub fn tree_from(snap: RuntimeSnapshot) -> RuntimeTree {
children_of.entry(parent).or_default().push(i); children_of.entry(parent).or_default().push(i);
} else { } else {
// Parent is the forest sentinel (genuine root) or absent from the // Parent is the forest sentinel (genuine root) or absent from the
// snapshot (orphan, D8) — either way a root of the forest. // snapshot (orphan): either way, a root of the forest.
orphaned[i] = parent != ROOT_PID; orphaned[i] = parent != ROOT_PID;
roots.push(i); roots.push(i);
} }
+156 -224
View File
@@ -13,44 +13,68 @@
//! leaves the actor, no copying through an intermediary thread. Built on //! leaves the actor, no copying through an intermediary thread. Built on
//! these are the conveniences `read(fd, &mut buf)` and `write(fd, &buf)`. //! these are the conveniences `read(fd, &mut buf)` and `write(fd, &buf)`.
//! //!
//! Architecture //! Architecture (RFC 018: driver-enqueues)
//! ============ //! =======================================
//! Per `run()`, two OS threads: //! Per `run()`, two OS threads, each a *producer* behind the runtime's
//! - **epoll thread**: owns the epollfd. Loops in `epoll_wait`. On a //! two-call contract — make the actor runnable (`unpark_at`, whose enqueue
//! ready fd, pushes `Completion::FdReady { pid, fd, events }` to the //! tail wakes a parked scheduler), nothing else:
//! shared completion queue and writes the scheduler-wake pipe. On the
//! shutdown pipe (also registered in epollfd), exits.
//! - **pool thread**: blocks on the request mpsc. Runs the closure
//! inside `catch_unwind`, pushes `Completion::Blocking { pid, result }`,
//! writes the scheduler-wake pipe.
//! //!
//! Both threads share a single `completions: Arc<Mutex<VecDeque<Completion>>>` //! - **epoll thread**: owns `epoll_wait` on the epollfd. On a ready fd it
//! and the same scheduler-wake pipe. //! removes the parked waiter from the shared `waiters` map and DELs the
//! fd (both under the waiters lock — see below), then unparks the
//! actor directly. On the shutdown pipe (also registered in the
//! epollfd), exits.
//! - **pool thread**: blocks on the request mpsc. Runs the closure inside
//! `catch_unwind`, stashes the result in the actor's slot
//! (`pending_io_result`, under the cold lock, generation-checked),
//! decrements the runtime's `io_outstanding`, and unparks the actor.
//! //!
//! `epoll_ctl` (register/unregister fd interest) is called by the //! There is no shared completion queue and no wake pipe: each producer
//! scheduler thread *directly* on the epollfd. That's well-defined per //! routes its own completion, so the whole byte-vs-completion visibility
//! `epoll_ctl(2)`: a thread may be calling `epoll_wait` on the epollfd //! discipline of the drain era — and the stranded-completion hazards it
//! while another thread calls `epoll_ctl`. Avoids needing a second mpsc //! defended against — is unrepresentable. Producers reach the runtime
//! and a second wake mechanism. //! through a `Weak<RuntimeInner>`: upgraded per completion (the path is
//! syscall-bound; the refcount op is noise) and avoiding an Arc cycle
//! through `RuntimeInner::io`.
//!
//! `epoll_ctl` (register fd interest) is called by the scheduler thread
//! directly on the epollfd. That's well-defined per `epoll_ctl(2)`: a
//! thread may be calling `epoll_wait` on the epollfd while another thread
//! calls `epoll_ctl`.
//! //!
//! Epoll mode //! Epoll mode
//! ========== //! ==========
//! Level-triggered with EPOLLONESHOT. After a wakeup the kernel //! Level-triggered with EPOLLONESHOT. After a wakeup the kernel
//! auto-disarms the fd, so we never get two wakeups for one //! auto-disarms the fd, so we never get two wakeups for one
//! `wait_readable` call. The scheduler explicitly `EPOLL_CTL_DEL`s the fd //! `wait_readable` call. The epoll thread explicitly `EPOLL_CTL_DEL`s the
//! on completion to free the slot for re-registration. Net effect: each //! fd on readiness to free the slot for re-registration. Net effect: each
//! `wait_readable(fd)` is one ADD, one wakeup, one DEL — symmetric and //! `wait_readable(fd)` is one ADD, one wakeup, one DEL — symmetric and
//! stateless between calls. //! stateless between calls.
//! //!
//! ## The waiters lock is the ADD/DEL serialization
//!
//! Registration (scheduler thread: check-vacant, defensive DEL, ADD,
//! insert) and readiness consumption (epoll thread: remove, DEL) each run
//! entirely under the `waiters` mutex. This is what makes the
//! oneshot-rearm race unrepresentable: a woken actor re-registering the
//! same fd cannot interleave with the epoll thread's DEL for the *previous*
//! registration — whichever takes the lock second sees a consistent
//! kernel-side state. Lock order: `io` (the runtime's outer mutex, held by
//! scheduler-side callers) → `waiters` → slot/queue leaves via `unpark_at`.
//! The epoll thread takes `waiters` without `io` — it must never take
//! `io`, both for lock-order hygiene and because teardown holds `io` while
//! joining it.
//!
//! Fd hygiene //! Fd hygiene
//! ========== //! ==========
//! An actor stopped while waiting on an fd unwinds out of `wait_fd`'s park; //! An actor stopped while waiting on an fd unwinds out of `wait_fd`'s park;
//! a drop guard there (armed after a successful register, forgotten on a //! a drop guard there (armed after a successful register, forgotten on a
//! normal wake) removes the `waiters` entry iff it is still that wait's //! normal wake) calls [`IoThread::cancel_waiter`], which removes the
//! `(pid, epoch)` and only then `EPOLL_CTL_DEL`s the fd — an entry already //! `waiters` entry iff it is still that wait's `(pid, epoch)` and only then
//! consumed by a racing `FdReady` means the fd may carry someone else's //! `EPOLL_CTL_DEL`s the fd — an entry already consumed by the epoll thread
//! fresh registration, which must be left alone. `epoll_register` keeps a //! means the fd may carry someone else's fresh registration, which must be
//! defensive bare DEL before ADD as belt-and-braces. //! left alone. `epoll_register` keeps a defensive bare DEL before ADD as
//! belt-and-braces.
//! //!
//! Buffers used with `read`/`write` should be on fds opened with //! Buffers used with `read`/`write` should be on fds opened with
//! `O_NONBLOCK`. If they aren't, the syscall may block the scheduler //! `O_NONBLOCK`. If they aren't, the syscall may block the scheduler
@@ -68,13 +92,14 @@
//! they have no equivalent panic-propagation path. //! they have no equivalent panic-propagation path.
use crate::pid::Pid; use crate::pid::Pid;
use crate::runtime::RuntimeInner;
use std::any::Any; use std::any::Any;
use std::collections::{HashMap, VecDeque}; use std::collections::HashMap;
use std::io; use std::io;
use std::os::fd::RawFd; use std::os::fd::RawFd;
use std::panic; use std::panic;
use std::sync::mpsc; use std::sync::atomic::Ordering;
use std::sync::{Arc, Mutex}; use std::sync::{mpsc, Arc, Mutex, Weak};
use std::thread::JoinHandle as OsJoinHandle; use std::thread::JoinHandle as OsJoinHandle;
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -86,42 +111,29 @@ use std::thread::JoinHandle as OsJoinHandle;
pub type IoResult = Result<Box<dyn Any + Send>, Box<dyn Any + Send>>; pub type IoResult = Result<Box<dyn Any + Send>, Box<dyn Any + Send>>;
struct Request { struct Request {
/// The submitter's park-epoch — carried through to the `Blocking` /// The submitter's park-epoch — the eventual wake is epoch-matched.
/// completion so the wake is epoch-matched.
epoch: u32, epoch: u32,
pid: Pid, pid: Pid,
/// The work to perform. Returns the wire-form result directly. /// The work to perform. Returns the wire-form result directly.
work: Box<dyn FnOnce() -> IoResult + Send>, work: Box<dyn FnOnce() -> IoResult + Send>,
} }
/// Completion message from either IO thread back to the scheduler. /// The parked-waiter map, shared between scheduler-side registration and
pub enum Completion { /// the epoll thread's readiness consumption. See the module docs on why
/// A `block_on_io` closure has finished (Ok = return value, Err = panic /// this single lock is the ADD/DEL serialization.
/// payload). type Waiters = Arc<Mutex<HashMap<RawFd, (Pid, u32)>>>;
Blocking { pid: Pid, epoch: u32, result: IoResult },
/// An fd registered via `wait_readable`/`wait_writable` is ready. The
/// scheduler looks up the parked pid in `waiters`, unparks it, and
/// removes the entry. `pid` isn't in this variant because the epoll
/// thread doesn't have access to the `waiters` map; the scheduler
/// thread owns that.
FdReady { fd: RawFd, events: u32 },
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// IoThread — created per `run()`, owned by `SchedulerState`. // IoThread — created per `run()`, owned by `RuntimeInner::io`.
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
pub struct IoThread { pub struct IoThread {
// ----- Channels & queues -----
/// Submission queue into the blocking-work pool. /// Submission queue into the blocking-work pool.
tx: mpsc::Sender<Request>, tx: mpsc::Sender<Request>,
/// Shared completion queue, fed by both the pool and the epoll thread. /// One parked actor per registered fd. Populated by `epoll_register`,
completions: Arc<Mutex<VecDeque<Completion>>>, /// consumed by the epoll thread on readiness or `cancel_waiter` on an
/// Pipe the scheduler polls in its idle path. Both IO threads write to /// unwound wait.
/// `wake_write` after pushing a completion. waiters: Waiters,
wake_read: RawFd,
wake_write: RawFd,
// ----- Epoll machinery ----- // ----- Epoll machinery -----
@@ -133,39 +145,25 @@ pub struct IoThread {
/// shutdown. /// shutdown.
shutdown_read: RawFd, shutdown_read: RawFd,
shutdown_write: RawFd, shutdown_write: RawFd,
/// One parked actor per registered fd. Populated by `wait_readable` /
/// `wait_writable` and drained by the scheduler when a `FdReady`
/// completion is processed.
pub waiters: HashMap<RawFd, (Pid, u32)>,
// ----- Threads ----- // ----- Threads -----
pool_thread: Option<OsJoinHandle<()>>, pool_thread: Option<OsJoinHandle<()>>,
epoll_thread: Option<OsJoinHandle<()>>, epoll_thread: Option<OsJoinHandle<()>>,
/// Number of `block_on_io` requests in-flight. Used by the scheduler's
/// idle path to decide whether to wait on the pipe or exit. Fd waits
/// are not counted here; they're counted by `waiters.len()`.
pub outstanding: u32,
} }
impl IoThread { impl IoThread {
pub fn start() -> io::Result<Self> { /// Start the pool and epoll threads. `rt` is the producers' route back
// Scheduler-facing wake pipe. /// into the runtime (slot table + unpark protocol); a `Weak` so the
let (wake_read, wake_write) = make_pipe()?; /// `RuntimeInner → IoThread → RuntimeInner` cycle never forms.
// Pool submission channel + shared completion queue. pub(crate) fn start(rt: Weak<RuntimeInner>) -> io::Result<Self> {
// Pool submission channel.
let (tx, rx) = mpsc::channel::<Request>(); let (tx, rx) = mpsc::channel::<Request>();
let completions: Arc<Mutex<VecDeque<Completion>>> = let waiters: Waiters = Arc::new(Mutex::new(HashMap::new()));
Arc::new(Mutex::new(VecDeque::new()));
// Epoll machinery. // Epoll machinery.
let epollfd = unsafe { libc::epoll_create1(libc::EPOLL_CLOEXEC) }; let epollfd = unsafe { libc::epoll_create1(libc::EPOLL_CLOEXEC) };
if epollfd < 0 { if epollfd < 0 {
// Best-effort fd cleanup before bailing.
unsafe {
libc::close(wake_read);
libc::close(wake_write);
}
return Err(io::Error::last_os_error()); return Err(io::Error::last_os_error());
} }
@@ -174,8 +172,6 @@ impl IoThread {
Err(e) => { Err(e) => {
unsafe { unsafe {
libc::close(epollfd); libc::close(epollfd);
libc::close(wake_read);
libc::close(wake_write);
} }
return Err(e); return Err(e);
} }
@@ -202,42 +198,37 @@ impl IoThread {
libc::close(epollfd); libc::close(epollfd);
libc::close(shutdown_read); libc::close(shutdown_read);
libc::close(shutdown_write); libc::close(shutdown_write);
libc::close(wake_read);
libc::close(wake_write);
} }
return Err(e); return Err(e);
} }
// Spawn pool thread. // Spawn pool thread.
let pool_comps = completions.clone(); let pool_rt = rt.clone();
let pool_thread = std::thread::Builder::new() let pool_thread = std::thread::Builder::new()
.name("smarm-io-pool".into()) .name("smarm-io-pool".into())
.spawn(move || pool_loop(rx, pool_comps, wake_write))?; .spawn(move || pool_loop(rx, pool_rt))?;
// Spawn epoll thread. // Spawn epoll thread.
let epoll_comps = completions.clone(); let epoll_waiters = waiters.clone();
let epoll_thread = std::thread::Builder::new() let epoll_thread = std::thread::Builder::new()
.name("smarm-io-epoll".into()) .name("smarm-io-epoll".into())
.spawn(move || epoll_loop(epollfd, epoll_comps, wake_write))?; .spawn(move || epoll_loop(epollfd, epoll_waiters, rt))?;
Ok(Self { Ok(Self {
tx, tx,
completions, waiters,
wake_read,
wake_write,
epollfd, epollfd,
shutdown_read, shutdown_read,
shutdown_write, shutdown_write,
waiters: HashMap::new(),
pool_thread: Some(pool_thread), pool_thread: Some(pool_thread),
epoll_thread: Some(epoll_thread), epoll_thread: Some(epoll_thread),
outstanding: 0,
}) })
} }
/// Hand a request to the pool. Increments `outstanding`. /// Hand a request to the pool. The caller (scheduler.rs) increments
/// `io_outstanding` BEFORE calling — the pool decrements on completion,
/// and an increment that trailed the completion would underflow.
pub fn submit(&mut self, pid: Pid, epoch: u32, work: Box<dyn FnOnce() -> IoResult + Send>) { pub fn submit(&mut self, pid: Pid, epoch: u32, work: Box<dyn FnOnce() -> IoResult + Send>) {
self.outstanding += 1;
// Send can only fail if the pool has hung up, which only happens // Send can only fail if the pool has hung up, which only happens
// on shutdown. submit during shutdown is a bug. // on shutdown. submit during shutdown is a bug.
if self.tx.send(Request { pid, epoch, work }).is_err() { if self.tx.send(Request { pid, epoch, work }).is_err() {
@@ -245,39 +236,13 @@ impl IoThread {
} }
} }
/// Drain every available completion. Caller (the scheduler) routes the
/// results and updates `outstanding` / `waiters` accordingly.
pub fn drain_completions(&mut self) -> Vec<Completion> {
let mut q = match self.completions.lock() {
Ok(g) => g,
Err(e) => panic!("smarm: io completions lock poisoned (core corrupt): {e}"),
};
let mut out = Vec::with_capacity(q.len());
while let Some(c) = q.pop_front() {
out.push(c);
}
out
}
pub fn wake_fd(&self) -> RawFd {
self.wake_read
}
/// Write the wake pipe directly: rouse every scheduler thread blocked in
/// its idle `poll_wake`. Used by the terminal (AllDone) path — an idle
/// sibling may be blocked on a snapshot that nothing will ever refresh
/// (an orphaned timer deadline, or `io_outstanding` from a waiter that
/// was stop-cancelled and so never produces a completion).
pub fn wake(&self) {
wake_scheduler(self.wake_write);
}
/// Register interest in `fd` becoming readable/writable; record `pid` /// Register interest in `fd` becoming readable/writable; record `pid`
/// as the parked waiter. The epoll thread will push a `FdReady` /// as the parked waiter. The epoll thread unparks it on readiness.
/// completion when the kernel signals. /// The caller increments `io_fd_waiters` BEFORE calling (mirror of
/// `submit`'s contract) and decrements it again if this errors.
/// ///
/// EPOLLONESHOT: one wakeup per registration. The scheduler must /// EPOLLONESHOT: one wakeup per registration; the epoll thread DELs on
/// `epoll_del` on completion to free the slot for re-registration. /// readiness, `cancel_waiter` DELs on an unwound wait.
pub fn epoll_register( pub fn epoll_register(
&mut self, &mut self,
fd: RawFd, fd: RawFd,
@@ -286,20 +251,24 @@ impl IoThread {
readable: bool, readable: bool,
writable: bool, writable: bool,
) -> io::Result<()> { ) -> io::Result<()> {
let mut waiters = match self.waiters.lock() {
Ok(g) => g,
Err(e) => panic!("smarm: io waiters lock poisoned (core corrupt): {e}"),
};
// Two actors waiting on the same fd would be a misuse: the kernel // Two actors waiting on the same fd would be a misuse: the kernel
// delivers exactly one EPOLLONESHOT wakeup, so the second waiter // delivers exactly one EPOLLONESHOT wakeup, so the second waiter
// would hang. Reject up front. // would hang. Reject up front.
if self.waiters.contains_key(&fd) { if waiters.contains_key(&fd) {
return Err(io::Error::new( return Err(io::Error::new(
io::ErrorKind::AlreadyExists, io::ErrorKind::AlreadyExists,
"fd already has a parked waiter", "fd already has a parked waiter",
)); ));
} }
// Belt-and-braces: the unwind guard in `wait_fd` is responsible for // Belt-and-braces: `cancel_waiter` is responsible for cleaning up a
// cleaning up a stopped waiter's registration, but a bare DEL is // stopped waiter's registration, but a bare DEL is harmless if the
// harmless if the fd isn't registered (ENOENT) and removes any leak // fd isn't registered (ENOENT) and removes any leak a path we
// a path we haven't thought of might leave behind. // haven't thought of might leave behind.
unsafe { unsafe {
libc::epoll_ctl(self.epollfd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut()); libc::epoll_ctl(self.epollfd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut());
} }
@@ -321,20 +290,30 @@ impl IoThread {
if r < 0 { if r < 0 {
return Err(io::Error::last_os_error()); return Err(io::Error::last_os_error());
} }
self.waiters.insert(fd, (pid, epoch)); waiters.insert(fd, (pid, epoch));
Ok(()) Ok(())
} }
/// Remove `fd` from the epollfd. Called by the scheduler after a /// Remove `fd`'s waiter iff it is still `(pid, epoch)`, DELing the fd
/// `FdReady` completion, so the next `wait_readable(fd)` can ADD again. /// from the epollfd in the same critical section. Returns whether the
/// /// entry was removed (the caller then decrements `io_fd_waiters`).
/// Does NOT touch `waiters` — that's the scheduler's bookkeeping; this /// `false` means the epoll thread consumed the registration first —
/// is purely the kernel-side cleanup. /// the fd may already carry someone else's fresh ADD; hands off.
pub fn epoll_deregister(&mut self, fd: RawFd) { pub fn cancel_waiter(&mut self, fd: RawFd, pid: Pid, epoch: u32) -> bool {
let mut waiters = match self.waiters.lock() {
Ok(g) => g,
Err(e) => panic!("smarm: io waiters lock poisoned (core corrupt): {e}"),
};
if waiters.get(&fd) == Some(&(pid, epoch)) {
waiters.remove(&fd);
// EPOLL_CTL_DEL of an already-removed fd returns ENOENT; ignore. // EPOLL_CTL_DEL of an already-removed fd returns ENOENT; ignore.
unsafe { unsafe {
libc::epoll_ctl(self.epollfd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut()); libc::epoll_ctl(self.epollfd, libc::EPOLL_CTL_DEL, fd, std::ptr::null_mut());
} }
true
} else {
false
}
} }
} }
@@ -354,7 +333,10 @@ impl Drop for IoThread {
let real_tx = std::mem::replace(&mut self.tx, dead_tx); let real_tx = std::mem::replace(&mut self.tx, dead_tx);
drop(real_tx); drop(real_tx);
// 3. Join both threads. // 3. Join both threads. Safe even while the caller holds the
// runtime's `io` mutex: neither thread ever takes it (they reach
// the runtime through a Weak they upgrade per completion, and
// the epoll thread's only lock is `waiters`).
if let Some(h) = self.epoll_thread.take() { if let Some(h) = self.epoll_thread.take() {
let _ = h.join(); let _ = h.join();
} }
@@ -367,8 +349,6 @@ impl Drop for IoThread {
libc::close(self.epollfd); libc::close(self.epollfd);
libc::close(self.shutdown_read); libc::close(self.shutdown_read);
libc::close(self.shutdown_write); libc::close(self.shutdown_write);
libc::close(self.wake_read);
libc::close(self.wake_write);
} }
} }
} }
@@ -379,36 +359,38 @@ impl Drop for IoThread {
const SHUTDOWN_EPOLL_TOKEN: u64 = u64::MAX; const SHUTDOWN_EPOLL_TOKEN: u64 = u64::MAX;
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Pool loop // Pool loop (producer: Blocking completions)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
fn pool_loop( fn pool_loop(rx: mpsc::Receiver<Request>, rt: Weak<RuntimeInner>) {
rx: mpsc::Receiver<Request>,
completions: Arc<Mutex<VecDeque<Completion>>>,
wake_write: RawFd,
) {
while let Ok(Request { pid, epoch, work }) = rx.recv() { while let Ok(Request { pid, epoch, work }) = rx.recv() {
let result: IoResult = match panic::catch_unwind(panic::AssertUnwindSafe(work)) { let result: IoResult = match panic::catch_unwind(panic::AssertUnwindSafe(work)) {
Ok(r) => r, Ok(r) => r,
Err(payload) => Err(payload), Err(payload) => Err(payload),
}; };
match completions.lock() { let Some(inner) = rt.upgrade() else { return };
Ok(mut g) => g.push_back(Completion::Blocking { pid, epoch, result }), // Stash the result under the cold lock (generation-checked: an
Err(e) => panic!("smarm: io completions lock poisoned (core corrupt): {e}"), // actor stopped with the op in flight discards it), decrement the
// in-flight count, then wake through the epoch-matched unpark. The
// unpark's enqueue tail wakes a parked scheduler; the actor stays
// `live` until it resumes and finalizes, so the decrement's
// ordering against the termination verdict is not load-bearing.
if let Some(slot) = inner.slot_at(pid) {
let mut cold = slot.cold.lock();
if slot.generation() == pid.generation() {
cold.pending_io_result = Some(result);
} }
wake_scheduler(wake_write); }
inner.io_outstanding.fetch_sub(1, Ordering::AcqRel);
inner.unpark_at(pid, epoch);
} }
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Epoll loop // Epoll loop (producer: FdReady completions)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
fn epoll_loop( fn epoll_loop(epollfd: RawFd, waiters: Waiters, rt: Weak<RuntimeInner>) {
epollfd: RawFd,
completions: Arc<Mutex<VecDeque<Completion>>>,
wake_write: RawFd,
) {
// Buffer for epoll_wait. 64 is plenty for our scale; if a real load // Buffer for epoll_wait. 64 is plenty for our scale; if a real load
// appears that needs more, this is a one-line change. // appears that needs more, this is a one-line change.
const MAX_EVENTS: usize = 64; const MAX_EVENTS: usize = 64;
@@ -436,29 +418,41 @@ fn epoll_loop(
} }
let mut shutdown_requested = false; let mut shutdown_requested = false;
let mut pushed_any = false;
{
let mut q = match completions.lock() {
Ok(g) => g,
Err(e) => panic!("smarm: io completions lock poisoned (core corrupt): {e}"),
};
for ev in events.iter().take(n as usize) { for ev in events.iter().take(n as usize) {
if ev.u64 == SHUTDOWN_EPOLL_TOKEN { if ev.u64 == SHUTDOWN_EPOLL_TOKEN {
shutdown_requested = true; shutdown_requested = true;
continue; continue;
} }
let fd = ev.u64 as RawFd; let fd = ev.u64 as RawFd;
let evs = ev.events; // Consume the registration: remove + DEL under the waiters
q.push_back(Completion::FdReady { // lock (the ADD/DEL serialization — see module docs). A
// vanished entry means `cancel_waiter` beat us: the wake is
// already moot.
let entry = {
let mut w = match waiters.lock() {
Ok(g) => g,
Err(e) => {
panic!("smarm: io waiters lock poisoned (core corrupt): {e}")
}
};
let entry = w.remove(&fd);
if entry.is_some() {
unsafe {
libc::epoll_ctl(
epollfd,
libc::EPOLL_CTL_DEL,
fd, fd,
events: evs, std::ptr::null_mut(),
}); );
pushed_any = true;
} }
} }
entry
if pushed_any { };
wake_scheduler(wake_write); if let Some((pid, epoch)) = entry {
let Some(inner) = rt.upgrade() else { return };
inner.io_fd_waiters.fetch_sub(1, Ordering::AcqRel);
inner.unpark_at(pid, epoch);
}
} }
if shutdown_requested { if shutdown_requested {
return; return;
@@ -466,27 +460,8 @@ fn epoll_loop(
} }
} }
/// Write one byte to the scheduler's wake pipe. Retries on EINTR; ignores
/// EAGAIN (pipe full means there's already an outstanding wake we haven't
/// consumed yet, which is sufficient).
fn wake_scheduler(wake_write: RawFd) {
let buf: [u8; 1] = [0];
unsafe {
loop {
let n = libc::write(wake_write, buf.as_ptr() as *const _, 1);
if n < 0 {
let e = *libc::__errno_location();
if e == libc::EINTR {
continue;
}
}
break;
}
}
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Pipe helpers (unchanged from v0.2) // Pipe helper
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
fn make_pipe() -> io::Result<(RawFd, RawFd)> { fn make_pipe() -> io::Result<(RawFd, RawFd)> {
@@ -497,46 +472,3 @@ fn make_pipe() -> io::Result<(RawFd, RawFd)> {
} }
Ok((fds[0], fds[1])) Ok((fds[0], fds[1]))
} }
/// Drain pending bytes from the wake pipe. The scheduler calls this after
/// a `poll` wakeup so the next idle call sees an empty pipe.
pub fn drain_wake_pipe(fd: RawFd) {
let mut buf = [0u8; 64];
loop {
let n = unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) };
if n <= 0 {
break;
}
}
}
/// Block on `fd` for up to `timeout`, returning when either there's data
/// to read or the timeout elapses. `None` for `timeout` means wait forever.
pub fn poll_wake(fd: RawFd, timeout: Option<std::time::Duration>) {
let timeout_ms: libc::c_int = match timeout {
None => -1,
Some(d) => {
let ms = d.as_millis();
if ms > i32::MAX as u128 {
i32::MAX
} else {
ms as i32
}
}
};
let mut pfd = libc::pollfd {
fd,
events: libc::POLLIN,
revents: 0,
};
loop {
let r = unsafe { libc::poll(&mut pfd as *mut _, 1, timeout_ms) };
if r < 0 {
let e = unsafe { *libc::__errno_location() };
if e == libc::EINTR {
continue;
}
}
break;
}
}
+4 -1
View File
@@ -32,12 +32,14 @@ pub mod introspect;
#[cfg(feature = "observer")] #[cfg(feature = "observer")]
pub mod observer; pub mod observer;
pub mod runtime; pub mod runtime;
pub(crate) mod park;
pub(crate) mod raw_mutex; pub(crate) mod raw_mutex;
pub(crate) mod slot_state; pub(crate) mod slot_state;
pub(crate) mod sync_shim; pub(crate) mod sync_shim;
#[doc(hidden)] // pub only so benches/rq_micro.rs can drive the raw structures #[doc(hidden)] // pub only so benches/rq_micro.rs can drive the raw structures
pub mod run_queue; pub mod run_queue;
pub mod trace; pub mod trace;
pub mod causal;
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Global allocator // Global allocator
@@ -79,7 +81,8 @@ pub use registry::{
}; };
pub use runtime::{init, Config, Runtime}; pub use runtime::{init, Config, Runtime};
pub use scheduler::{ pub use scheduler::{
block_on_io, cancel_timer, request_stop, run, self_pid, send_after, send_after_named, sleep, block_on_io, cancel_timer, request_stop, run, self_pid, send_after, send_after_named,
send_after_named_wall, send_after_wall, sleep, sleep_wall,
spawn, spawn_addr, spawn_under, wait_readable, wait_readable_timeout, wait_writable, spawn, spawn_addr, spawn_under, wait_readable, wait_readable_timeout, wait_writable,
wait_writable_timeout, yield_now, FdArm, JoinError, JoinHandle, wait_writable_timeout, yield_now, FdArm, JoinError, JoinHandle,
}; };
+102 -62
View File
@@ -1,49 +1,85 @@
//! Process monitors. //! Find out when another actor dies, without it knowing or caring that you're
//! watching.
//! //!
//! `monitor(target)` asks the runtime to deliver a single [`Down`] when //! Say one actor manages a pool of workers and needs to know when a worker
//! `target` terminates, and hands back a [`Monitor`] — the [`Receiver`] to read //! exits, so it can replace it. The worker does not need to know it is being
//! it from, plus the identity (`id`, `target`) needed to take the registration //! watched, and nothing about the worker's own behavior should change because
//! back down with [`demonitor`]. A monitor is: //! someone is watching it. That is what [`monitor`] is for: call
//! `monitor(target)` to get a [`Monitor`], and read exactly one [`Down`]
//! message off `monitor.rx` whenever `target` terminates, however it
//! terminates.
//! //!
//! - **unidirectional** — the watcher learns of the target's death, but the //! ```
//! target learns nothing of the watcher, and the watcher is unaffected by //! use smarm::{monitor, run, spawn, DownReason};
//! the death beyond the notification (contrast a *link*, which propagates
//! failure);
//! - **one-shot** — exactly one `Down` is ever sent for a given monitor.
//! The returned channel closes afterwards, so a second `recv()` yields
//! `Err(RecvError)`.
//! //!
//! This generalizes the older single-`supervisor_channel` mechanism: a //! run(|| {
//! supervisor is just a hard-wired monitor that the parent installs at spawn //! let worker = spawn(|| {
//! time. Here any actor may monitor any pid, any number of times. //! // does some work, then returns
//! });
//! let pid = worker.pid();
//! //!
//! ## Reasons //! let m = monitor(pid);
//! let _ = worker.join();
//! //!
//! [`DownReason`] is deliberately payload-free. A panicking actor's payload //! let down = m.rx.recv().expect("monitor channel closed before Down");
//! has a single owner and is delivered to whoever `join()`s the actor (as //! assert_eq!(down.pid, pid);
//! `JoinError`); a monitor only learns *that* it panicked, not the value. //! assert_eq!(down.reason, DownReason::Exit);
//! Monitoring a pid that is already gone (reclaimed, or never alive) yields //! });
//! [`DownReason::NoProc`] immediately, mirroring Erlang's `noproc`. //! ```
//! //!
//! ## Demonitoring //! A monitor is one-directional and one-shot:
//! //!
//! Each `monitor()` registration is tagged with a process-unique [`MonitorId`]. //! - **One-directional**: the watcher learns that the target died, but the
//! [`demonitor`] removes the registration named by a [`Monitor`] from its //! target is completely unaffected. It never learns it was being watched,
//! target's slot, returning `Some(id)` if a live registration was found or //! and its own behavior and lifetime do not change because of the monitor.
//! `None` if it had already fired (or the target is gone). Dropping the //! This is the opposite of a [`link`](mod@crate::link), which is bidirectional:
//! [`Monitor`] afterwards discards any `Down` that the target had *already* //! linking two actors means an abnormal death on either side can bring the
//! queued — the equivalent of Erlang's `demonitor(Ref, [flush])`. //! other down too. Reach for a monitor when you just want to *know*; reach
//! for a link when a peer's crash should actually stop you.
//! - **One-shot**: you get exactly one [`Down`] per `monitor()` call, then the
//! channel closes. Calling `monitor` again on the same target (or a
//! different one) gives you an independent registration with its own
//! [`Monitor`] and its own one-shot channel; nothing stops you from
//! monitoring the same actor many times over; each call is watched and
//! fires on its own.
//! //!
//! ## Races //! ## Why a monitor never hands you the panic value
//! //!
//! Registration (below) and `finalize_actor` (in `runtime`) both run under the //! If the target panicked, [`Down`] tells you *that* it panicked
//! shared-state mutex, so a target that is still alive when its monitor is //! ([`DownReason::Panic`]), but not the panic's payload. The payload has a
//! registered is guaranteed to deliver a real `Down`; there is no window in //! single owner: it is handed to whichever caller `join()`s the actor's
//! which the death slips between the liveness check and the registration. //! [`JoinHandle`](crate::JoinHandle), as a `JoinError`. A monitor only needs
//! `demonitor` is protected by the generation half of the pid: if the target //! to know that something went wrong, not reproduce the exact value that
//! has died and its slot index been recycled, `slot_mut(target)` fails the //! caused it, so it gets the reason and nothing else.
//! generation check and `demonitor` is a clean no-op — it can never strip a //!
//! *different* actor's monitor that happens to share the slot index. //! Monitoring a target that is already gone (it finished and was cleaned up,
//! or the pid never pointed at a real actor) is not an error: you get a
//! [`Down`] with [`DownReason::NoProc`] right away, instead of waiting
//! forever for something that already happened.
//!
//! ## Stopping a monitor early
//!
//! [`demonitor`] cancels a monitor before it fires. If the registration was
//! still live, it removes it and returns `Some` of the monitor's id: no
//! `Down` will arrive on that channel from here on. If the target had already
//! died and its `Down` already sent, there is nothing left to cancel and
//! `demonitor` returns `None`; the `Down` you already have (or that is
//! already sitting in the channel) is unaffected.
//!
//! If you want to cancel *and* make sure a `Down` that already arrived is
//! discarded without reading it, just drop the [`Monitor`]: dropping it closes
//! its receiver, and any queued `Down` is dropped along with it.
//!
//! ## Correctness notes for implementers
//!
//! A target that is still alive at the moment `monitor()` registers is
//! guaranteed to eventually produce a real `Down`: registration and the
//! target's own termination bookkeeping run under the same lock, so there is
//! no window in which the target could die without the just-added
//! registration seeing it. `demonitor` is similarly race-free against a target
//! that has since died and had its slot reused by a new, unrelated actor: it
//! is checked against the exact monitored incarnation, so it can never remove
//! a different actor's registration by accident, it simply reports `None`.
use crate::channel::{channel, Receiver, Sender}; use crate::channel::{channel, Receiver, Sender};
use crate::pid::Pid; use crate::pid::Pid;
@@ -51,8 +87,8 @@ use crate::scheduler::with_runtime;
/// Why a monitored actor went down. /// Why a monitored actor went down.
/// ///
/// `Copy` because it carries no payload see the module docs for why the /// Carries no payload: see the module docs for why a monitor never receives
/// panic payload is *not* included here. /// the panic value itself.
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DownReason { pub enum DownReason {
/// The target returned normally. /// The target returned normally.
@@ -76,21 +112,22 @@ pub struct Down {
pub reason: DownReason, pub reason: DownReason,
} }
/// A process-unique identifier for one `monitor()` registration. /// A unique identifier for one [`monitor`] registration.
/// ///
/// Opaque and `Copy`. Allocated from a monotonic counter in shared state, so /// Opaque and `Copy`. Never reused for the life of the runtime, so if you
/// it is never reused for the lifetime of the runtime — distinct `monitor()` /// monitor the same target more than once, each call's id is distinct. This
/// calls on the same target get distinct ids, which is what lets [`demonitor`] /// is what lets [`demonitor`] tear down exactly one of several monitors on
/// tear down exactly one of several monitors on a target. /// the same target without disturbing the others.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct MonitorId(pub(crate) u64); pub struct MonitorId(pub(crate) u64);
/// A live monitor: the receiving end of the one-shot [`Down`] channel, plus the /// A live monitor: the receiving end of the one-shot [`Down`] channel, plus the
/// identity needed to [`demonitor`] it. /// identity needed to [`demonitor`] it.
/// ///
/// Read the notification from [`Monitor::rx`]. Not `Clone` (the receiver is a /// Read the notification from [`Monitor::rx`]. Not `Clone`, since only one
/// single consumer). Dropping it closes the receiving end; if a `Down` was /// side is meant to consume it. Dropping a `Monitor` closes the receiving
/// already queued it is discarded with the channel. /// end; if a `Down` had already arrived but was never read, it is discarded
/// along with it.
pub struct Monitor { pub struct Monitor {
/// This registration's process-unique id. /// This registration's process-unique id.
pub id: MonitorId, pub id: MonitorId,
@@ -110,10 +147,11 @@ pub fn monitor<A>(target: Pid<A>) -> Monitor {
let target = target.erase(); let target = target.erase();
let (tx, rx) = channel::<Down>(); let (tx, rx) = channel::<Down>();
// Register under the target's cold lock. `tx.clone()` takes the channel's // Implementation note: registration happens under the target's cold
// own lock — a Channel-class RawMutex, explicitly permitted *under* a Leaf // lock. `tx.clone()` takes the channel's own lock, a Channel-class
// (cold) lock by the lock order (see raw_mutex.rs). We must still not // RawMutex, which is explicitly permitted under a Leaf (cold) lock by
// *send* under the lock, as `Sender::send` can unpark a parked receiver, // the lock order documented in raw_mutex.rs. We must still not *send*
// under the lock, since `Sender::send` can unpark a parked receiver,
// and there's no reason to nest that. // and there's no reason to nest that.
let (id, registered) = with_runtime(|inner| { let (id, registered) = with_runtime(|inner| {
let id = inner.alloc_monitor_id(); let id = inner.alloc_monitor_id();
@@ -140,19 +178,21 @@ pub fn monitor<A>(target: Pid<A>) -> Monitor {
} }
/// Cancel the monitor `m`. Returns `Some(id)` if a live registration was found /// Cancel the monitor `m`. Returns `Some(id)` if a live registration was found
/// on the target's slot and removed, or `None` if there was nothing to remove /// and removed, so no `Down` will arrive on `m.rx` from here on. Returns
/// — the target already fired its `Down` (the registration is drained on /// `None` if there was nothing left to remove: the target had already gone
/// finalize), was never alive (`NoProc`), or has been reclaimed. /// down and its `Down` was already sent (or is already sitting in the
/// channel, unread).
/// ///
/// This stops any *future* `Down`. To also discard a `Down` the target may have /// This only stops a *future* `Down`. If you also want to discard a `Down`
/// *already* queued (the finalize-races-demonitor case), drop `m` afterwards; /// that already arrived (or is about to, in a race with this call), drop `m`
/// dropping the [`Monitor`] closes its receiver and the queued notice goes with /// instead of, or in addition to, calling this: dropping the [`Monitor`]
/// it — the analogue of Erlang's `demonitor(Ref, [flush])`. /// closes its receiver and any queued notice is discarded with it.
pub fn demonitor(m: &Monitor) -> Option<MonitorId> { pub fn demonitor(m: &Monitor) -> Option<MonitorId> {
// Remove the registration under the target's cold lock, but move the // Implementation note: the registration is removed under the target's
// `Sender` *out* and let it drop only after the lock is released: // cold lock, but the `Sender` is moved *out* and dropped only after the
// dropping the last sender runs `Sender::drop`, which may unpark a parked // lock is released. Dropping the last sender runs `Sender::drop`, which
// receiver legal under a cold lock, but pointless to nest. // may unpark a parked receiver; legal under a cold lock, but pointless
// to nest.
let removed: Option<(MonitorId, Sender<Down>)> = with_runtime(|inner| { let removed: Option<(MonitorId, Sender<Down>)> = with_runtime(|inner| {
let slot = inner.slot_at(m.target)?; let slot = inner.slot_at(m.target)?;
let mut cold = slot.cold.lock(); let mut cold = slot.cold.lock();
+120 -9
View File
@@ -1,12 +1,89 @@
//! Actor-aware mutex with mandatory timeout. //! Shared mutable state across actors, when a channel is overkill.
//! //!
//! `Mutex<T>` parks the calling *green* thread on contention rather than //! smarm actors normally coordinate by sending messages, and for a piece of
//! blocking the OS thread. Every lock attempt is bounded by a timeout. //! owned state the right tool is usually a `gen_server`: one actor holds the
//! data and everyone else talks to it. Sometimes that is more machinery than
//! you need, and plain shared, lockable state is simpler: [`Mutex<T>`] is
//! that escape hatch. It behaves like `std::sync::Mutex<T>`, guarding a value
//! of type `T` behind a guard that gives you `&mut T` while held, but it is
//! built for smarm's actors rather than OS threads.
//! //!
//! Internals use `Arc<std::sync::Mutex<...>>` so the type is genuinely //! The key difference from `std::sync::Mutex` is what happens on contention.
//! `Send + Sync` and can be shared across scheduler threads. //! [`Mutex::lock`] parks the calling actor (a cooperatively scheduled green
//! thread) rather than blocking the underlying OS thread, so other actors on
//! the same OS thread keep running while it waits. And every lock attempt is
//! bounded by a timeout: an actor that hangs on to the lock forever (stuck in
//! a bug, or just slow) would otherwise wedge every other actor waiting on
//! it, so smarm makes the wait bounded by default instead of leaving it up
//! to you to remember.
//! //!
//! Fairness: FIFO. Poisoning: none. Reentrance: deadlock (caller bug). //! ## A first lock
//!
//! ```
//! use smarm::{run, spawn, Mutex};
//!
//! run(|| {
//! let counter = Mutex::new(0u32);
//!
//! // Mutex::clone() is cheap and hands out another handle to the SAME
//! // underlying value, much like Arc::clone: every clone shares one lock
//! // and one value, so mutations through one are visible through all.
//! let a = counter.clone();
//! let b = counter.clone();
//!
//! let h1 = spawn(move || {
//! let mut guard = a.lock().unwrap();
//! *guard += 1;
//! });
//! let h2 = spawn(move || {
//! let mut guard = b.lock().unwrap();
//! *guard += 1;
//! });
//! h1.join().unwrap();
//! h2.join().unwrap();
//!
//! assert_eq!(*counter.lock().unwrap(), 2);
//! });
//! ```
//!
//! ## Choosing a timeout
//!
//! [`Mutex::lock`] waits up to [`DEFAULT_TIMEOUT`] (30 seconds) before giving
//! up with [`LockTimeout`]. To use a different bound for one call, use
//! [`Mutex::lock_timeout`] instead; to change the default for every future
//! `lock()` call on this mutex (including through its clones), use
//! [`Mutex::set_default_timeout`]. If you never want to wait at all, use
//! [`Mutex::try_lock`], which returns immediately whether or not the lock was
//! free.
//!
//! ## Fairness and panics
//!
//! Waiters are granted the lock in the order they started waiting (FIFO), so
//! no actor can be starved by later arrivals repeatedly cutting in line.
//!
//! This mutex never poisons. `std::sync::Mutex` marks itself poisoned if a
//! thread panics while holding the lock, because a partly mutated value might
//! be left behind for the next lock holder to see. smarm's actors already
//! rely on `Drop` running during unwinding to release the lock, so if a
//! holder panics, [`MutexGuard::drop`] still runs and the next waiter is
//! granted the lock normally. It is the same tradeoff `std::sync::Mutex`
//! offers you if you choose to ignore poisoning: you may see a value left
//! mid-update by the panicking actor, so a panic inside a critical section is
//! still a bug worth fixing, just not one that also wedges every future lock
//! attempt.
//!
//! Locking a mutex you already hold (on the same actor) does not queue
//! behind yourself: it deadlocks, the same way relocking a non-reentrant
//! `std::sync::Mutex` does. Don't call `lock` while already holding a guard
//! from the same `Mutex`.
//!
//! ## Outside the runtime
//!
//! `Mutex<T>` also works when called from plain code that is not running as
//! a smarm actor (for example, in a test's setup code before calling
//! [`run`](crate::run)). There, an actor's cooperative park has no meaning,
//! so a lock attempt instead blocks the calling OS thread directly until the
//! mutex is free; there is no timeout on this path.
use crate::pid::Pid; use crate::pid::Pid;
use crate::scheduler; use crate::scheduler;
@@ -15,8 +92,14 @@ use std::collections::VecDeque;
use std::sync::{Arc, Mutex as StdMutex}; use std::sync::{Arc, Mutex as StdMutex};
use std::time::Duration; use std::time::Duration;
/// How long [`Mutex::lock`] waits for the lock before giving up, unless
/// overridden per-mutex with [`Mutex::set_default_timeout`] or per-call with
/// [`Mutex::lock_timeout`].
pub const DEFAULT_TIMEOUT: Duration = Duration::from_secs(30); pub const DEFAULT_TIMEOUT: Duration = Duration::from_secs(30);
/// Returned by [`Mutex::lock`] / [`Mutex::lock_timeout`] when the timeout
/// elapses before the lock became available. The lock attempt is abandoned;
/// nothing was acquired, and the mutex's value is unaffected.
#[derive(Debug, PartialEq, Eq, Clone, Copy)] #[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub struct LockTimeout; pub struct LockTimeout;
@@ -70,7 +153,7 @@ impl TimerTarget for MutexCore {
}; };
// Remove from waiters only if still there with matching epoch. // Remove from waiters only if still there with matching epoch.
// If the lock was already granted (holder == Some(pid)), the // If the lock was already granted (holder == Some(pid)), the
// timer fired after the grant treat as no-op; the actor // timer fired after the grant: treat as no-op; the actor
// will see `is_holder == true` and return Ok. // will see `is_holder == true` and return Ok.
if st.holder == Some(pid) { if st.holder == Some(pid) {
return; return;
@@ -100,6 +183,8 @@ pub struct Mutex<T> {
} }
impl<T> Mutex<T> { impl<T> Mutex<T> {
/// Wrap `value` in a new mutex, initially unlocked, with the default
/// lock timeout ([`DEFAULT_TIMEOUT`]).
pub fn new(value: T) -> Self { pub fn new(value: T) -> Self {
Self { Self {
core: Arc::new(MutexCore::new(DEFAULT_TIMEOUT)), core: Arc::new(MutexCore::new(DEFAULT_TIMEOUT)),
@@ -107,6 +192,11 @@ impl<T> Mutex<T> {
} }
} }
/// Change how long future [`lock`](Self::lock) calls on this mutex wait
/// before giving up. Applies to every clone of this `Mutex` (they share
/// one underlying lock), and to `lock` calls already in progress that
/// have not yet started waiting. Does not affect [`lock_timeout`](Self::lock_timeout)
/// calls, which always use the timeout passed in.
pub fn set_default_timeout(&self, timeout: Duration) { pub fn set_default_timeout(&self, timeout: Duration) {
match self.core.state.lock() { match self.core.state.lock() {
Ok(mut st) => st.default_timeout = timeout, Ok(mut st) => st.default_timeout = timeout,
@@ -114,6 +204,12 @@ impl<T> Mutex<T> {
} }
} }
/// Acquire the lock, waiting up to this mutex's default timeout
/// ([`DEFAULT_TIMEOUT`], or whatever [`set_default_timeout`](Self::set_default_timeout)
/// last set) if it is currently held elsewhere. Returns a [`MutexGuard`]
/// that releases the lock when dropped, or [`LockTimeout`] if the
/// deadline passes first. To use a one-off timeout instead of the
/// mutex's default, call [`lock_timeout`](Self::lock_timeout) directly.
pub fn lock(&self) -> Result<MutexGuard<'_, T>, LockTimeout> { pub fn lock(&self) -> Result<MutexGuard<'_, T>, LockTimeout> {
let timeout = match self.core.state.lock() { let timeout = match self.core.state.lock() {
Ok(st) => st.default_timeout, Ok(st) => st.default_timeout,
@@ -122,6 +218,10 @@ impl<T> Mutex<T> {
self.lock_timeout(timeout) self.lock_timeout(timeout)
} }
/// Acquire the lock, waiting up to `timeout` (ignoring this mutex's
/// default) if it is currently held elsewhere. Returns a [`MutexGuard`]
/// that releases the lock when dropped, or [`LockTimeout`] if `timeout`
/// elapses first with the lock still unavailable.
pub fn lock_timeout(&self, timeout: Duration) -> Result<MutexGuard<'_, T>, LockTimeout> { pub fn lock_timeout(&self, timeout: Duration) -> Result<MutexGuard<'_, T>, LockTimeout> {
// Outside the runtime (e.g. in tests, after run() returns) there is no // Outside the runtime (e.g. in tests, after run() returns) there is no
// current actor PID. Fall back to a blocking std::sync::Mutex acquire. // current actor PID. Fall back to a blocking std::sync::Mutex acquire.
@@ -157,7 +257,7 @@ impl<T> Mutex<T> {
Ok(g) => g, Ok(g) => g,
Err(e) => panic!("smarm: mutex state lock poisoned (core corrupt): {e}"), Err(e) => panic!("smarm: mutex state lock poisoned (core corrupt): {e}"),
}; };
// begin_wait is lock-free legal under the state lock; this // begin_wait is lock-free (legal under the state lock); this
// makes the epoch atomic with the registration's visibility to // makes the epoch atomic with the registration's visibility to
// grants and timeouts. // grants and timeouts.
let epoch = scheduler::begin_wait(); let epoch = scheduler::begin_wait();
@@ -170,7 +270,7 @@ impl<T> Mutex<T> {
scheduler::insert_wait_timer(deadline, me, target, epoch); scheduler::insert_wait_timer(deadline, me, target, epoch);
scheduler::park_current(); scheduler::park_current();
// Resumed precisely: only our grant or our timer can wake this // Resumed, precisely: only our grant or our timer can wake this
// wait (both epoch-stamped; a stop wake unwinds out of // wait (both epoch-stamped; a stop wake unwinds out of
// park_current). The one-shot interpretation below is therefore // park_current). The one-shot interpretation below is therefore
// exhaustive. Are we the holder? // exhaustive. Are we the holder?
@@ -193,6 +293,9 @@ impl<T> Mutex<T> {
} }
} }
/// Acquire the lock only if it is immediately available: never parks and
/// never waits. Returns `Some` with a [`MutexGuard`] if the lock was
/// free, `None` if it is currently held elsewhere.
pub fn try_lock(&self) -> Option<MutexGuard<'_, T>> { pub fn try_lock(&self) -> Option<MutexGuard<'_, T>> {
let me = crate::actor::current_pid()?; let me = crate::actor::current_pid()?;
let mut st = match self.core.state.lock() { let mut st = match self.core.state.lock() {
@@ -234,6 +337,10 @@ impl<T> Mutex<T> {
} }
impl<T> Clone for Mutex<T> { impl<T> Clone for Mutex<T> {
/// Cheap: hands back another handle to the same underlying lock and
/// value, the way `Arc::clone` does. All clones of a `Mutex` share one
/// lock and one protected value; locking through any clone excludes
/// every other clone.
fn clone(&self) -> Self { fn clone(&self) -> Self {
Self { core: self.core.clone(), value: self.value.clone() } Self { core: self.core.clone(), value: self.value.clone() }
} }
@@ -247,6 +354,10 @@ unsafe impl<T: Send> Sync for Mutex<T> {}
// Guard // Guard
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Grants access to the value inside a [`Mutex`] while the lock is held.
/// Dereferences to `&T` and `&mut T`. Dropping the guard releases the lock
/// and, if another actor is waiting, wakes the next one in arrival order.
/// Returned by [`Mutex::lock`], [`Mutex::lock_timeout`], and [`Mutex::try_lock`].
pub struct MutexGuard<'a, T> { pub struct MutexGuard<'a, T> {
mutex: &'a Mutex<T>, mutex: &'a Mutex<T>,
value: Option<T>, value: Option<T>,
+994
View File
@@ -0,0 +1,994 @@
//! Scheduler park/wake coordination layer (RFC 018).
//!
//! Schedulers never touch an fd to sleep: they park on a per-thread
//! [`Parker`] and are woken through an idle-mask protocol the runtime owns
//! outright. IO backends (epoll today, io_uring later) are *producers*
//! behind a two-call contract — make actors runnable, then wake — which is
//! what makes backend selection tractable (RFC 018 §step-back).
//!
//! Three pieces, all in [`Coordinator`]:
//!
//! - **Parker** (one per scheduler): permit semantics, `std::thread::park`
//! shaped — an unpark delivered before the park sets a permit; the next
//! park consumes it and returns immediately. This single property closes
//! the check-then-park race. Linux: `futex(2)` `FUTEX_WAIT`/`FUTEX_WAKE`
//! (private) with a nanosecond-precision relative `timespec` — the
//! `as_millis` truncation defect of the retired wake pipe is
//! unrepresentable here. Loom / non-Linux: `Mutex<bool>` + `Condvar`
//! (the loom models run against this build).
//! - **Idle mask**: an `AtomicU64` bitmask of parked scheduler ids
//! (construction asserts ≤ 64 schedulers). Park protocol: set own bit,
//! run the caller's mandatory post-publish re-check, then wait. A
//! producer that published work before observing our bit has left us
//! work the re-check finds; one that observes the bit wakes us.
//!
//! The publish/re-check pair is a store-buffer (Dekker) shape. Two sound
//! resolutions coexist here, chosen per call-site cost profile: the
//! **fence handshake** on the hot producer path
//! ([`Coordinator::wake_one_if_idle`]: publish work; `fence(SeqCst)`;
//! one *Relaxed* mask load — pairing with the consumer's `fetch_or(bit)`;
//! `fence(SeqCst)`; re-check inside [`Coordinator::park`]), so the
//! pure-compute hot path (everyone busy, mask 0) never takes the shared
//! mask line exclusive — the RFC's "one relaxed load" fast path, made
//! sound; and the **same-location-RMW read** (`fetch_or(0)`, in
//! [`Coordinator::wake_one`] / [`Coordinator::idle_mask`]) for the rare
//! paths (chain rule — at most one per wake) where reading the latest
//! mask by modification-order coherence is worth an RMW. The loom models
//! drive the fence pattern end to end (a fence-less plain-load draft
//! would — and did — fail model 1/2 with a lost wake, as it must).
//! - **Timekeeper**: at most one parked scheduler holds the timer
//! deadline (RFC 018 §timers) so a timer expiry wakes one scheduler,
//! not a herd. The role is an atomic `(holder id, armed deadline)`
//! pair; a timer insertion with an earlier deadline wakes the holder to
//! re-peek. Arm / insert-check MUST be serialized by the caller (the
//! timers mutex in the runtime) — the atomics exist so the *busy-path
//! due-check* (one Relaxed load, [`Coordinator::armed_deadline_nanos`])
//! and the wake stay lock-free. All races are biased over-wake: a
//! spurious permit costs one failed pop; a missed wake would cost a
//! stranded actor, and is unrepresentable under the serialization rule.
//!
//! Every wake here is *at most one* futex round-trip and wakes *exactly
//! one* scheduler by construction (`wake_one` CASes a bit clear before
//! unparking its owner) — there is no shared level-triggered anything
//! left to herd on.
//!
//! Standalone until the runtime swap (RFC 018 commit 2): nothing outside
//! tests constructs a [`Coordinator`] yet.
use crate::sync_shim::{fence, AtomicU64, Ordering};
use std::time::Instant;
/// Sentinel for "no timekeeper" in the holder word.
const NO_TIMEKEEPER: u64 = u64::MAX;
/// Sentinel for "no armed deadline" in the deadline word. Also what the
/// busy-path due-check compares against: `now_nanos < armed` is one branch.
pub(crate) const NO_DEADLINE: u64 = u64::MAX;
/// Outcome of a [`Coordinator::park`] call.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum ParkResult {
/// A permit was consumed (wake delivered before or during the park).
Woken,
/// The deadline passed with no wake. Only possible when a deadline was
/// supplied (and never under loom, which has no time — see `park`).
TimedOut,
/// The post-publish re-check found work; the thread never blocked.
/// A permit may still be pending (a racing `wake_one` picked us after
/// the bit was set) — it will surface as one spurious `Woken` on a
/// later park. Benign: over-wake by design.
WorkFound,
}
// ---------------------------------------------------------------------------
// Parker — permit-semantics thread parking
// ---------------------------------------------------------------------------
/// Linux, non-loom: futex on a state word.
///
/// States: EMPTY (no permit, nobody waiting), PARKED (a thread is, or is
/// about to be, in `futex_wait`), NOTIFIED (permit pending). The classic
/// std-parker protocol: `unpark` swaps to NOTIFIED and futex-wakes iff it
/// displaced PARKED; `park` CASes EMPTY→PARKED, waits, and consumes
/// NOTIFIED on every exit path.
#[cfg(all(target_os = "linux", not(loom)))]
mod parker {
use std::sync::atomic::{AtomicU32, Ordering};
use std::time::Instant;
const EMPTY: u32 = 0;
const PARKED: u32 = 1;
const NOTIFIED: u32 = 2;
pub(super) struct Parker {
state: AtomicU32,
}
impl Parker {
pub(super) fn new() -> Self {
Self { state: AtomicU32::new(EMPTY) }
}
/// Returns `true` = woken (permit consumed), `false` = timed out.
pub(super) fn park(&self, deadline: Option<Instant>) -> bool {
// Fast path: consume a pending permit without blocking.
if self
.state
.compare_exchange(EMPTY, PARKED, Ordering::AcqRel, Ordering::Acquire)
.is_err()
{
// Only NOTIFIED can be here (one thread parks at a time).
self.state.store(EMPTY, Ordering::Release);
return true;
}
loop {
let timeout = match deadline {
None => None,
Some(d) => {
let now = Instant::now();
if d <= now {
// Deadline passed: cancel the park. The swap
// races a concurrent unpark — if it delivered
// NOTIFIED first, report Woken (never lose a
// permit).
return self.state.swap(EMPTY, Ordering::AcqRel) == NOTIFIED;
}
Some(d - now)
}
};
futex_wait(&self.state, PARKED, timeout);
if self
.state
.compare_exchange(NOTIFIED, EMPTY, Ordering::AcqRel, Ordering::Acquire)
.is_ok()
{
return true;
}
// Spurious wake or timeout with state still PARKED: loop —
// the deadline check at the top decides.
}
}
pub(super) fn unpark(&self) {
if self.state.swap(NOTIFIED, Ordering::AcqRel) == PARKED {
futex_wake(&self.state, 1);
}
}
}
/// `FUTEX_WAIT` with a *relative* nanosecond timeout (`CLOCK_MONOTONIC`
/// per futex(2) for relative waits). No millisecond conversion anywhere:
/// the timespec carries the full sub-ms remainder (RFC 018 kills the
/// `as_millis` truncation structurally).
fn futex_wait(word: &AtomicU32, expected: u32, timeout: Option<std::time::Duration>) {
let ts;
let ts_ptr: *const libc::timespec = match timeout {
Some(d) => {
ts = libc::timespec {
tv_sec: d.as_secs() as libc::time_t,
tv_nsec: d.subsec_nanos() as libc::c_long,
};
&ts
}
None => std::ptr::null(),
};
// Errors (EAGAIN: word changed; ETIMEDOUT; EINTR) all mean "return
// and let the caller's state machine decide" — deliberately ignored.
unsafe {
libc::syscall(
libc::SYS_futex,
word.as_ptr(),
libc::FUTEX_WAIT | libc::FUTEX_PRIVATE_FLAG,
expected,
ts_ptr,
);
}
}
fn futex_wake(word: &AtomicU32, n: u32) {
unsafe {
libc::syscall(
libc::SYS_futex,
word.as_ptr(),
libc::FUTEX_WAKE | libc::FUTEX_PRIVATE_FLAG,
n,
);
}
}
}
/// Loom / non-Linux: `Mutex<bool>` permit + `Condvar` — loom's own model
/// of a parker, and the portable fallback. Under loom the deadline is
/// ignored (loom has no time); the models exercise wake paths only.
#[cfg(any(loom, not(target_os = "linux")))]
mod parker {
use crate::sync_shim::{Condvar, Mutex};
use std::time::Instant;
pub(super) struct Parker {
permit: Mutex<bool>,
cv: Condvar,
}
impl Parker {
pub(super) fn new() -> Self {
Self { permit: Mutex::new(false), cv: Condvar::new() }
}
/// Returns `true` = woken (permit consumed), `false` = timed out.
pub(super) fn park(&self, deadline: Option<Instant>) -> bool {
let mut permit = match self.permit.lock() {
Ok(g) => g,
Err(_) => panic!("smarm: parker permit lock poisoned (core corrupt)"),
};
loop {
if *permit {
*permit = false;
return true;
}
#[cfg(loom)]
{
// Loom has no clock: block until a wake. Models must
// deliver one (a park nobody wakes is a real deadlock
// and loom reports it as such).
let _ = deadline;
permit = match self.cv.wait(permit) {
Ok(g) => g,
Err(_) => panic!("smarm: parker cv poisoned (core corrupt)"),
};
}
#[cfg(not(loom))]
{
match deadline {
None => {
permit = match self.cv.wait(permit) {
Ok(g) => g,
Err(_) => panic!("smarm: parker cv poisoned (core corrupt)"),
};
}
Some(d) => {
let now = Instant::now();
if d <= now {
return false;
}
permit = match self.cv.wait_timeout(permit, d - now) {
Ok((g, _)) => g,
Err(_) => panic!("smarm: parker cv poisoned (core corrupt)"),
};
}
}
}
}
}
pub(super) fn unpark(&self) {
let mut permit = match self.permit.lock() {
Ok(g) => g,
Err(_) => panic!("smarm: parker permit lock poisoned (core corrupt)"),
};
*permit = true;
self.cv.notify_one();
}
}
}
use parker::Parker;
// ---------------------------------------------------------------------------
// Coordinator — idle mask + wake protocol + timekeeper
// ---------------------------------------------------------------------------
pub(crate) struct Coordinator {
parkers: Box<[Parker]>,
/// Bit `i` set = scheduler `i` is parked or committed to parking (set
/// before the re-check; cleared by `wake_one`'s CAS or by the parker
/// itself on return). AcqRel same-location-RMW handshake — see module
/// docs (no SeqCst needed: every producer-side read is an RMW).
idle: AtomicU64,
/// Timekeeper holder id, or `NO_TIMEKEEPER`. Written under the
/// caller's timer serialization (arm/disarm/insert-check); read
/// lock-free by the insert wake path.
tk_holder: AtomicU64,
/// Armed deadline as nanos since `origin`, or `NO_DEADLINE`. Written
/// only by the timekeeper arm/disarm protocol.
tk_armed: AtomicU64,
/// Earliest KNOWN timer deadline (nanos since `origin`), or
/// `NO_DEADLINE` — independent of whether any scheduler is parked,
/// which is what the timekeeper's `tk_armed` cannot give: under
/// saturation nobody parks and nobody arms, yet due timers must still
/// fire (ratified design point (a)). Maintained under the caller's
/// timers mutex (`note_deadline` on insert, `refresh_deadline` after a
/// pop/peek); read lock-free by the busy-path due-check.
next_deadline: AtomicU64,
/// Time origin for the nanos encoding.
origin: Instant,
}
impl Coordinator {
pub(crate) fn new(schedulers: usize) -> Self {
assert!(
(1..=64).contains(&schedulers),
"smarm: scheduler count must be 1..=64 (idle mask is one u64); got {schedulers}"
);
Self {
parkers: (0..schedulers).map(|_| Parker::new()).collect(),
idle: AtomicU64::new(0),
tk_holder: AtomicU64::new(NO_TIMEKEEPER),
tk_armed: AtomicU64::new(NO_DEADLINE),
next_deadline: AtomicU64::new(NO_DEADLINE),
origin: Instant::now(),
}
}
/// Encode a deadline for the armed snapshot / busy-path compare.
/// Saturating: a deadline at-or-before `origin` encodes as 0 (always
/// due), one beyond ~584 years as `NO_DEADLINE - 1`.
pub(crate) fn deadline_nanos(&self, deadline: Instant) -> u64 {
let nanos = deadline
.checked_duration_since(self.origin)
.map(|d| d.as_nanos())
.unwrap_or(0);
if nanos >= NO_DEADLINE as u128 {
NO_DEADLINE - 1
} else {
nanos as u64
}
}
/// Park scheduler `id` until a wake, the deadline, or a positive
/// re-check. Protocol: (1) publish own idle bit (SeqCst), (2) run
/// `recheck` — it MUST re-read the work source with an ordering that
/// pairs with the producer's publish (SeqCst load, or take the mutex
/// the producer publishes under); a `true` aborts the park, (3) block.
pub(crate) fn park(
&self,
id: usize,
deadline: Option<Instant>,
recheck: impl FnOnce() -> bool,
) -> ParkResult {
debug_assert!(id < self.parkers.len(), "park: scheduler id out of range");
let bit = 1u64 << id;
// (1) publish. AcqRel RMW: the acquire half is the handshake — if
// this lands after a producer's mask RMW in modification order, we
// read-from it and the producer's earlier work publication is
// visible to the re-check below (see module docs).
let prev = self.idle.fetch_or(bit, Ordering::AcqRel);
debug_assert_eq!(prev & bit, 0, "park: idle bit already set for this id");
// Fence half of the producer handshake (see `wake_one_if_idle`):
// orders our bit-publish before the re-check's loads, so it pairs
// with the producer's publish→fence→mask-load — at least one side
// must see the other's store, whichever queue backend is in play.
fence(Ordering::SeqCst);
// (2) the mandatory post-publish re-check.
if recheck() {
self.idle.fetch_and(!bit, Ordering::AcqRel);
return ParkResult::WorkFound;
}
// (3) block. The permit closes the window between the re-check and
// the futex wait: a wake_one that picked us in that window has
// already CASed our bit clear and set the permit.
let woken = self.parkers[id].park(deadline);
// Clear own bit — a no-op when a waker already CASed it clear.
self.idle.fetch_and(!bit, Ordering::AcqRel);
if woken {
ParkResult::Woken
} else {
ParkResult::TimedOut
}
}
/// Wake exactly one parked scheduler, if any: pick the highest set idle
/// bit (LIFO-ish — warmest cache), CAS it clear, deliver a permit.
/// Empty mask = no-op (everyone is awake and will find work by
/// popping). Returns whether a scheduler was woken.
pub(crate) fn wake_one(&self) -> bool {
// RMW read, not a load: reads the latest mask by modification-order
// coherence, closing the Dekker race with a parking consumer (see
// module docs). The release side of the RMW is what a later-parking
// consumer's fetch_or acquires to make its re-check sound.
let mut mask = self.idle.fetch_or(0, Ordering::AcqRel);
loop {
if mask == 0 {
return false;
}
let id = 63 - mask.leading_zeros() as usize; // highest set bit
let bit = 1u64 << id;
// The CAS is the exactly-one guarantee: whoever clears the bit
// owns the wake; a racing wake_one retries on the observed value
// (coherence: a failed CAS can never read older than `mask`).
match self.idle.compare_exchange(
mask,
mask & !bit,
Ordering::AcqRel,
Ordering::Acquire,
) {
Ok(_) => {
self.parkers[id].unpark();
return true;
}
Err(m) => mask = m,
}
}
}
/// The producer-side wake tail (`enqueue`'s fast path, RFC 018 "enqueue
/// wakes"). The caller has just published work (queue push); we fence,
/// then read the mask with ONE Relaxed load — 0 means every scheduler
/// is awake and the pure-compute hot path pays no RMW on the shared
/// mask line. Soundness is the fence handshake (module docs): our
/// fence orders the caller's push before the mask load; the consumer's
/// fence (in `park`) orders its bit-publish before its re-check — at
/// least one side must observe the other's store, so a consumer we
/// miss here is a consumer whose re-check finds the caller's work.
pub(crate) fn wake_one_if_idle(&self) -> bool {
fence(Ordering::SeqCst);
if self.idle.load(Ordering::Relaxed) == 0 {
return false;
}
self.wake_one()
}
/// Terminal wake (replaces the AllDone wake-pipe byte): clear the mask
/// and deliver a permit to *every* parker, parked or not. A permit set
/// on a busy scheduler costs one spurious park return — nothing at the
/// terminal boundary. Idempotent.
pub(crate) fn wake_all(&self) {
self.idle.store(0, Ordering::Release);
for p in self.parkers.iter() {
p.unpark();
}
}
/// Latest idle mask (RMW read — same handshake as `wake_one`). A
/// test-only observer: production expresses the chain rule through
/// `wake_one_if_idle` (fence + Relaxed load), not a mask read.
#[cfg(test)]
pub(crate) fn idle_mask(&self) -> u64 {
self.idle.fetch_or(0, Ordering::AcqRel)
}
// ----- timekeeper -----
/// Try to take the timekeeper role for scheduler `id` with `deadline`.
/// MUST be called under the caller's timer serialization (the timers
/// mutex), with `deadline` the heap minimum peeked under that same
/// hold — this is what makes the insert-check race-free. Returns
/// whether the role was taken (false = someone else holds it; park
/// with no deadline).
pub(crate) fn try_arm_timer(&self, id: usize, deadline: Instant) -> bool {
debug_assert!(id < self.parkers.len(), "try_arm_timer: id out of range");
if self
.tk_holder
.compare_exchange(NO_TIMEKEEPER, id as u64, Ordering::SeqCst, Ordering::SeqCst)
.is_err()
{
return false;
}
// Holder-then-deadline order: an insert-check that sees the holder
// with the deadline still NO_DEADLINE compares `new < MAX` = true
// and over-wakes — the benign direction. (Under the mandated timer
// serialization this interleaving cannot occur anyway.)
self.tk_armed.store(self.deadline_nanos(deadline), Ordering::SeqCst);
true
}
/// Release the timekeeper role (the holder, on wake, before it
/// re-peeks/fires). Callable without the timer serialization: a
/// racing insert may wake a no-longer-holder — over-wake, benign.
pub(crate) fn disarm_timer(&self, id: usize) {
debug_assert_eq!(
self.tk_holder.load(Ordering::SeqCst),
id as u64,
"disarm_timer by a non-holder"
);
// Deadline first: a concurrent insert-check then sees NO_DEADLINE
// and skips the (now pointless) wake instead of waking a stale
// holder id. Either order is correct; this one wastes less.
self.tk_armed.store(NO_DEADLINE, Ordering::SeqCst);
self.tk_holder.store(NO_TIMEKEEPER, Ordering::SeqCst);
}
/// Insert-side re-arm check: if `deadline` is earlier than the armed
/// snapshot, wake the timekeeper to re-peek. MUST be called under the
/// same timer serialization as `try_arm_timer` (see there).
pub(crate) fn timer_inserted(&self, deadline: Instant) {
if self.deadline_nanos(deadline) < self.tk_armed.load(Ordering::SeqCst) {
let holder = self.tk_holder.load(Ordering::SeqCst);
if holder != NO_TIMEKEEPER {
// Direct unpark, not wake_one: the wake targets the
// timekeeper specifically (it must re-peek the heap). Its
// idle bit stays set until it returns from park — a
// concurrent wake_one may pick it too; over-wake, benign.
self.parkers[holder as usize].unpark();
}
}
}
/// The armed-deadline snapshot (nanos since origin; `NO_DEADLINE` =
/// none). Test-only introspection on the timekeeper's armed value; the
/// busy-path due-check reads `next_deadline`, not this.
#[cfg(test)]
pub(crate) fn armed_deadline_nanos(&self) -> u64 {
self.tk_armed.load(Ordering::Relaxed)
}
// ----- earliest-deadline snapshot (busy-path due-check) -----
/// Record a newly inserted timer deadline. MUST be called under the
/// timers mutex (same serialization rule as `try_arm_timer`), which is
/// why plain compare+store suffices for the min-maintenance. Also runs
/// the timekeeper re-arm check (`timer_inserted`) — one call site for
/// both consequences of an insert.
pub(crate) fn note_deadline(&self, deadline: Instant) {
let n = self.deadline_nanos(deadline);
if n < self.next_deadline.load(Ordering::Relaxed) {
self.next_deadline.store(n, Ordering::Release);
}
self.timer_inserted(deadline);
}
/// Re-anchor the snapshot to the heap minimum (`None` = heap empty)
/// after a `pop_due` / `clear`. MUST be called under the timers mutex.
pub(crate) fn refresh_deadline(&self, next: Option<Instant>) {
let n = next.map_or(NO_DEADLINE, |d| self.deadline_nanos(d));
self.next_deadline.store(n, Ordering::Release);
}
/// Busy-path due-check: is the earliest known deadline at or past
/// `now`? One Relaxed load when no deadline is armed — the clock is
/// read only when a timer actually exists (matching the old drain
/// phase's is_empty guard), so the pure-compute hot path pays a load
/// and a branch.
pub(crate) fn deadline_due(&self) -> bool {
let n = self.next_deadline.load(Ordering::Relaxed);
n != NO_DEADLINE && self.deadline_nanos(Instant::now()) >= n
}
/// The earliest-deadline snapshot as an `Instant` (`None` = no timer
/// pending). Test-only: the idle path arms the timekeeper from the
/// timer heap's own `peek_deadline` under the timers mutex.
#[cfg(test)]
pub(crate) fn next_deadline_instant(&self) -> Option<Instant> {
let n = self.next_deadline.load(Ordering::Acquire);
if n == NO_DEADLINE {
None
} else {
self.origin.checked_add(std::time::Duration::from_nanos(n))
}
}
}
// ---------------------------------------------------------------------------
// Unit tests (std build)
// ---------------------------------------------------------------------------
#[cfg(all(test, not(loom)))]
mod tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering as O};
use std::sync::Arc;
use std::time::{Duration, Instant};
#[test]
fn permit_before_park_returns_immediately() {
let c = Coordinator::new(1);
// Deliver the wake first (nobody parked: wake_one no-ops on the
// mask, so use the timekeeper-direct path? No — permit semantics
// are the parker's own; exercise via wake_all which permits all).
c.wake_all();
let t0 = Instant::now();
let r = c.park(0, None, || false);
assert_eq!(r, ParkResult::Woken);
assert!(t0.elapsed() < Duration::from_millis(100), "park blocked despite permit");
}
#[test]
fn submillisecond_deadline_is_honored() {
// Regression for the retired as_millis truncation: a 500µs deadline
// must neither busy-return instantly forever nor round to 0/∞.
let c = Coordinator::new(1);
let t0 = Instant::now();
let r = c.park(0, Some(t0 + Duration::from_micros(500)), || false);
let dt = t0.elapsed();
assert_eq!(r, ParkResult::TimedOut);
assert!(dt >= Duration::from_micros(400), "woke too early: {dt:?}");
assert!(dt < Duration::from_millis(50), "overslept: {dt:?}");
}
#[test]
fn recheck_true_aborts_park_and_clears_bit() {
let c = Coordinator::new(2);
let r = c.park(1, None, || true);
assert_eq!(r, ParkResult::WorkFound);
assert_eq!(c.idle_mask(), 0, "bit not cleared after WorkFound");
}
#[test]
fn recheck_observes_own_bit_published() {
let c = Coordinator::new(2);
let seen = std::cell::Cell::new(0u64);
let r = c.park(1, None, || {
seen.set(c.idle_mask());
true
});
assert_eq!(r, ParkResult::WorkFound);
assert_eq!(seen.get() & 0b10, 0b10, "bit not published before re-check");
}
#[test]
fn wake_one_wakes_exactly_one_of_n() {
const N: usize = 4;
let c = Arc::new(Coordinator::new(N));
let woken = Arc::new(AtomicUsize::new(0));
let mut ts = Vec::new();
for id in 0..N {
let c = c.clone();
let woken = woken.clone();
ts.push(std::thread::spawn(move || {
let r = c.park(id, None, || false);
assert_eq!(r, ParkResult::Woken);
woken.fetch_add(1, O::SeqCst);
}));
}
// Wait until all four are published idle.
let t0 = Instant::now();
while c.idle_mask().count_ones() != N as u32 {
assert!(t0.elapsed() < Duration::from_secs(5), "threads never parked");
std::thread::yield_now();
}
assert!(c.wake_one());
// Exactly one wakes; give the others a beat to (incorrectly) wake.
let t0 = Instant::now();
while woken.load(O::SeqCst) == 0 {
assert!(t0.elapsed() < Duration::from_secs(5), "wake_one woke nobody");
std::thread::yield_now();
}
std::thread::sleep(Duration::from_millis(100));
assert_eq!(woken.load(O::SeqCst), 1, "wake_one woke more than one");
assert_eq!(c.idle_mask().count_ones(), (N - 1) as u32);
c.wake_all();
for t in ts {
match t.join() {
Ok(()) => {}
Err(p) => std::panic::resume_unwind(p),
}
}
assert_eq!(woken.load(O::SeqCst), N);
assert_eq!(c.idle_mask(), 0);
}
#[test]
fn wake_one_prefers_highest_bit() {
let c = Arc::new(Coordinator::new(3));
let woken_id = Arc::new(AtomicUsize::new(usize::MAX));
let mut ts = Vec::new();
for id in 0..3 {
let c = c.clone();
let woken_id = woken_id.clone();
ts.push(std::thread::spawn(move || {
if c.park(id, None, || false) == ParkResult::Woken {
let _ = woken_id.compare_exchange(usize::MAX, id, O::SeqCst, O::SeqCst);
}
}));
}
let t0 = Instant::now();
while c.idle_mask() != 0b111 {
assert!(t0.elapsed() < Duration::from_secs(5));
std::thread::yield_now();
}
assert!(c.wake_one());
let t0 = Instant::now();
while woken_id.load(O::SeqCst) == usize::MAX {
assert!(t0.elapsed() < Duration::from_secs(5));
std::thread::yield_now();
}
assert_eq!(woken_id.load(O::SeqCst), 2, "LIFO-ish: highest bit first");
c.wake_all();
for t in ts {
let _ = t.join();
}
}
#[test]
fn wake_one_on_empty_mask_is_noop() {
let c = Coordinator::new(2);
assert!(!c.wake_one());
assert_eq!(c.idle_mask(), 0);
}
#[test]
fn timekeeper_arm_is_exclusive_and_snapshot_readable() {
let c = Coordinator::new(2);
let d2 = Instant::now() + Duration::from_secs(10);
let d1 = Instant::now() + Duration::from_secs(1);
assert_eq!(c.armed_deadline_nanos(), NO_DEADLINE);
assert!(c.try_arm_timer(0, d2));
assert!(!c.try_arm_timer(1, d1), "second arm must fail while held");
assert_eq!(c.armed_deadline_nanos(), c.deadline_nanos(d2));
c.disarm_timer(0);
assert_eq!(c.armed_deadline_nanos(), NO_DEADLINE);
assert!(c.try_arm_timer(1, d1), "role must be re-takeable after disarm");
c.disarm_timer(1);
}
#[test]
fn earlier_insert_wakes_timekeeper() {
let c = Coordinator::new(2);
let far = Instant::now() + Duration::from_secs(60);
let near = Instant::now() + Duration::from_millis(1);
assert!(c.try_arm_timer(0, far));
// Holder not yet parked: the wake must land as a permit.
c.timer_inserted(near);
let t0 = Instant::now();
let r = c.park(0, Some(far), || false);
assert_eq!(r, ParkResult::Woken, "re-arm wake lost");
assert!(t0.elapsed() < Duration::from_secs(5), "slept toward the stale deadline");
c.disarm_timer(0);
}
#[test]
fn later_insert_does_not_wake_timekeeper() {
let c = Coordinator::new(2);
let near = Instant::now() + Duration::from_millis(20);
let far = Instant::now() + Duration::from_secs(60);
assert!(c.try_arm_timer(0, near));
c.timer_inserted(far); // later than armed: no wake
let r = c.park(0, Some(near), || false);
assert_eq!(r, ParkResult::TimedOut, "spurious wake for a later insert");
c.disarm_timer(0);
}
#[test]
#[should_panic(expected = "1..=64")]
fn more_than_64_schedulers_asserts() {
let _ = Coordinator::new(65);
}
#[test]
fn wake_one_if_idle_noop_on_empty_and_wakes_on_parked() {
let c = Arc::new(Coordinator::new(1));
assert!(!c.wake_one_if_idle(), "empty mask must be a no-op");
let c2 = c.clone();
let t = std::thread::spawn(move || {
assert_eq!(c2.park(0, None, || false), ParkResult::Woken);
});
let t0 = Instant::now();
while c.idle_mask() == 0 {
assert!(t0.elapsed() < Duration::from_secs(5), "never parked");
std::thread::yield_now();
}
assert!(c.wake_one_if_idle());
match t.join() {
Ok(()) => {}
Err(p) => std::panic::resume_unwind(p),
}
}
#[test]
fn deadline_snapshot_min_maintenance_and_due_check() {
let c = Coordinator::new(1);
assert!(!c.deadline_due(), "no deadline: never due");
assert_eq!(c.next_deadline_instant(), None);
let far = Instant::now() + Duration::from_secs(60);
let near = Instant::now() + Duration::from_millis(1);
c.note_deadline(far);
assert!(!c.deadline_due());
c.note_deadline(near); // min wins
assert!(c.next_deadline_instant().is_some_and(|d| d <= near));
c.note_deadline(far); // later insert must NOT raise the snapshot
assert!(c.next_deadline_instant().is_some_and(|d| d <= near));
std::thread::sleep(Duration::from_millis(2));
assert!(c.deadline_due(), "past deadline not reported due");
c.refresh_deadline(Some(far));
assert!(!c.deadline_due(), "refresh did not re-anchor");
c.refresh_deadline(None);
assert_eq!(c.next_deadline_instant(), None);
// A deadline at/before origin encodes as 0: always due.
c.note_deadline(Instant::now() - Duration::from_secs(1));
assert!(c.deadline_due());
}
#[test]
fn note_deadline_earlier_wakes_timekeeper_via_snapshot_path() {
// note_deadline must carry the timer_inserted re-arm wake too.
let c = Coordinator::new(2);
let far = Instant::now() + Duration::from_secs(60);
let near = Instant::now() + Duration::from_millis(1);
assert!(c.try_arm_timer(0, far));
c.note_deadline(near);
let r = c.park(0, Some(far), || false);
assert_eq!(r, ParkResult::Woken, "re-arm wake lost through note_deadline");
c.disarm_timer(0);
}
}
// ---------------------------------------------------------------------------
// loom models — RUSTFLAGS="--cfg loom" cargo test --lib --release park
// ---------------------------------------------------------------------------
#[cfg(all(test, loom))]
mod loom_tests {
use super::*;
use loom::sync::atomic::{AtomicU64 as LAtomicU64, Ordering as O};
use loom::sync::{Arc, Mutex as LMutex};
use loom::thread;
use std::time::{Duration, Instant};
/// RFC 018 loom model 1 — no lost wake.
/// producer{publish item; wake_one} ∥ consumer{set bit; re-check; park}:
/// the consumer always observes the item or a permit; it can never
/// sleep past a published item (loom's deadlock detector is the
/// assertion — a consumer parked forever fails the model).
#[test]
fn no_lost_wake() {
loom::model(|| {
let c = Arc::new(Coordinator::new(1));
let item = Arc::new(LAtomicU64::new(0));
let prod = {
let c = c.clone();
let item = item.clone();
thread::spawn(move || {
// The enqueue shape: publish, then the fenced fast-path
// wake tail (this is what the runtime's enqueue calls).
item.store(1, O::SeqCst);
c.wake_one_if_idle();
})
};
// Consumer: loop until the item is popped. Guarded load+CAS,
// not a blind swap — a swap writes 0 even when empty, and
// coherence allows that write to land after the producer's
// store in modification order, destroying the item (a model
// bug loom caught in an earlier draft of this test).
loop {
if item.load(O::SeqCst) == 1
&& item.compare_exchange(1, 0, O::SeqCst, O::SeqCst).is_ok()
{
break;
}
let _ = c.park(0, None, || item.load(O::SeqCst) == 1);
}
prod.join().unwrap();
});
}
/// RFC 018 loom model 2 — chain propagation.
/// Two items, two sleepers, ONE producer wake: the chain rule (a woken
/// consumer that sees surplus work and a non-empty mask wakes again)
/// must get both items consumed with no further producer action.
#[test]
fn chain_propagation() {
loom::model(|| {
let c = Arc::new(Coordinator::new(2));
let items = Arc::new(LAtomicU64::new(0));
let consumed = Arc::new(LAtomicU64::new(0));
let mut hs = Vec::new();
for id in 0..2usize {
let c = c.clone();
let items = items.clone();
let consumed = consumed.clone();
hs.push(thread::spawn(move || loop {
if consumed.load(O::SeqCst) == 2 {
c.wake_all(); // release a sibling still parked
break;
}
let cur = items.load(O::SeqCst);
if cur > 0
&& items
.compare_exchange(cur, cur - 1, O::SeqCst, O::SeqCst)
.is_ok()
{
consumed.fetch_add(1, O::SeqCst);
// THE CHAIN RULE, exactly as production expresses it
// (runtime.rs schedule_loop): surplus ⇒ the fenced
// fast-path wake. Models the Relaxed-load chain path,
// not just the RMW one.
if items.load(O::SeqCst) > 0 {
c.wake_one_if_idle();
}
continue;
}
let _ = c.park(id, None, || {
items.load(O::SeqCst) > 0 || consumed.load(O::SeqCst) == 2
});
}));
}
// Producer (main): two items, ONE wake, via the enqueue-shaped
// fenced fast path.
items.store(2, O::SeqCst);
c.wake_one_if_idle();
for h in hs {
h.join().unwrap();
}
assert_eq!(consumed.load(O::SeqCst), 2);
assert_eq!(items.load(O::SeqCst), 0);
});
}
/// RFC 018 loom model 3 — timekeeper handoff.
/// An earlier-deadline insert racing the parking timekeeper: the
/// earlier deadline is always honored — either the timekeeper armed it
/// directly (insert landed first under the timers lock) or the insert
/// wakes the timekeeper to re-peek. A timekeeper sleeping toward the
/// stale later deadline would deadlock the model (loom has no time).
#[test]
fn timekeeper_handoff() {
loom::model(|| {
let origin = Instant::now();
let d_far = origin + Duration::from_secs(60);
let d_near = origin + Duration::from_secs(1);
let c = Arc::new(Coordinator::new(1));
// The timers-mutex stand-in: heap min under a lock.
let heap_min = Arc::new(LMutex::new(d_far));
let tk = {
let c = c.clone();
let heap_min = heap_min.clone();
thread::spawn(move || {
// Peek + arm under the lock (the serialization rule).
let armed = {
let g = heap_min.lock().unwrap();
let min = *g;
assert!(c.try_arm_timer(0, min));
min
};
if armed == d_far {
// Insert hadn't landed: it MUST wake us. Parking
// toward d_far with no wake = model deadlock.
let r = c.park(0, Some(armed), || false);
assert_eq!(r, ParkResult::Woken, "re-arm wake lost");
}
// Woken (or armed the near deadline directly): re-peek.
c.disarm_timer(0);
let g = heap_min.lock().unwrap();
assert_eq!(*g, d_near, "earlier deadline not visible on re-peek");
})
};
// Inserter (main): publish the earlier deadline under the lock,
// then the insert-check.
{
let mut g = heap_min.lock().unwrap();
*g = d_near;
c.timer_inserted(d_near);
}
tk.join().unwrap();
});
}
/// RFC 018 loom model 4 — termination.
/// The AllDone verdict: producer flips done and wake_all()s; consumers
/// must never park forever past done (park's re-check + wake_all's
/// permits close every interleaving; a stuck consumer = loom deadlock).
#[test]
fn termination_no_park_past_done() {
loom::model(|| {
let c = Arc::new(Coordinator::new(2));
let done = Arc::new(LAtomicU64::new(0));
let mut hs = Vec::new();
for id in 0..2usize {
let c = c.clone();
let done = done.clone();
hs.push(thread::spawn(move || loop {
if done.load(O::SeqCst) == 1 {
break;
}
let _ = c.park(id, None, || done.load(O::SeqCst) == 1);
}));
}
done.store(1, O::SeqCst);
c.wake_all();
for h in hs {
h.join().unwrap();
}
});
}
}
+24
View File
@@ -98,6 +98,25 @@ pub(crate) fn clear_current_slot() {
CURRENT_SLOT.with(|c| c.set(std::ptr::null())); CURRENT_SLOT.with(|c| c.set(std::ptr::null()));
} }
/// RFC 007 (`smarm-causal`) — raw pointer to the on-CPU actor's slot, null on
/// the scheduler's own stack. Same lifetime argument as `note_overrun`: the
/// slot is never reclaimed while its actor is on-CPU.
#[cfg(feature = "smarm-causal")]
#[inline]
pub(crate) fn current_slot_ptr() -> *const crate::runtime::Slot {
CURRENT_SLOT.with(|c| c.get())
}
/// RFC 007 (`smarm-causal`) — push the slice start forward by `cycles`, so
/// virtually-injected delay spun inside `maybe_preempt` does not count against
/// the actor's timeslice (the clock-correction half of the RFC: the runtime
/// owns this clock, so it can subtract its own perturbation).
#[cfg(feature = "smarm-causal")]
#[inline]
pub(crate) fn extend_timeslice(cycles: u64) {
TIMESLICE_START.with(|c| c.set(c.get().wrapping_add(cycles)));
}
/// Tally a timeslice overrun against the on-CPU actor (RFC 016 Chunk 2). A /// Tally a timeslice overrun against the on-CPU actor (RFC 016 Chunk 2). A
/// no-op if no actor is bound (the scheduler's own stack). Reached only from /// no-op if no actor is bound (the scheduler's own stack). Reached only from
/// the slice-expiry branch, which is already the yield path, so its cost is /// the slice-expiry branch, which is already the yield path, so its cost is
@@ -247,6 +266,11 @@ pub fn maybe_preempt() {
// Observe a pending stop first: if we are being cancelled // Observe a pending stop first: if we are being cancelled
// there is no point yielding, we unwind instead. // there is no point yielding, we unwind instead.
check_cancelled(); check_cancelled();
// RFC 007: causal-profiling sample/absorb point. Shares the
// amortised cadence, and the PREEMPTION_ENABLED gate — so it
// can never spin inside a prep-to-park or no-preempt region.
#[cfg(feature = "smarm-causal")]
crate::causal::check();
let start = TIMESLICE_START.with(|s| s.get()); let start = TIMESLICE_START.with(|s| s.get());
if rdtsc().saturating_sub(start) > CONFIGURED_TIMESLICE_CYCLES.with(|t| t.get()) { if rdtsc().saturating_sub(start) > CONFIGURED_TIMESLICE_CYCLES.with(|t| t.get()) {
// Tally the overrun (RFC 016 Chunk 2) before handing back — // Tally the overrun (RFC 016 Chunk 2) before handing back —
+288 -207
View File
@@ -1,55 +1,108 @@
//! Named mailbox registry — resolve a name (or pid) to a *messageable* actor.
//! Give an actor a name so other actors can find it and message it.
//! //!
//! ## What changed (RFC 014) //! Without the registry, the only way to reach an actor is to already be
//! holding its [`Pid`], usually because you spawned it yourself or someone
//! passed it to you. That is fine for a worker you just created, but it does
//! not work for a well-known service that arbitrary parts of your program
//! need to find independently, like a logger, a config store, or a
//! connection pool. The registry solves this: an actor claims a name once,
//! and from then on any other actor can look that name up, or send to it
//! directly, without ever having been handed a `Pid`.
//! //!
//! The old registry was a `name <-> pid` bimap: `whereis` handed back a `Pid` //! ```
//! you could not send to, because a pid is just `(index, generation)` with no //! use smarm::{channel, register, run, send, spawn, unregister, whereis, Name};
//! delivery endpoint. This rework makes resolution yield something messageable.
//! //!
//! Two facts shape the structure: //! const COUNTER: Name<u64> = Name::new("counter");
//! //!
//! 1. **A name resolves to a single actor.** Many actors under one label is //! run(|| {
//! what *process groups* (`pg`) are for; the registry is one-name-one-actor //! let (ready_tx, ready_rx) = channel::<()>();
//! (several names *may* point at the same actor). //! let (tx, rx) = channel::<u64>();
//! 2. **Channels are typed**, so an actor has no single untyped mailbox. An
//! actor instead owns a *set* of typed channels — one [`Sender`] per message
//! type it accepts. So the registry maps name/pid to a [`Mailbox`]: a small
//! structure holding that actor's pid plus all of its typed channels, keyed
//! by message [`TypeId`].
//! //!
//! Resolution is therefore: `name -> pid` (single actor) `-> Mailbox -> the //! let worker = spawn(move || {
//! channel for message type M`. A `Name<Cmd>` and a `Name<Admin>` on the *same* //! // Claim the name for this actor's inbox. Any actor holding
//! actor select *different* channels purely by their type parameter, so //! // `COUNTER` can now reach this one by name.
//! capability separation (RFC 014 §4.7) needs no extra machinery. //! register(COUNTER, tx).unwrap();
//! ready_tx.send(()).unwrap();
//! assert_eq!(rx.recv().unwrap(), 42);
//! });
//! //!
//! ## Type erasure is contained //! ready_rx.recv().unwrap(); // wait for the worker to register
//! //!
//! Each stored channel is a `Box<dyn Any + Send>` that is concretely a //! // Look the name up, or just send to it directly.
//! `Sender<M>`, filed under `TypeId::of::<M>()`. A resolve for `M` looks up //! assert_eq!(whereis("counter"), Some(worker.pid()));
//! that exact `TypeId` and downcasts to `Sender<M>` — keyed by the very type we //! send(COUNTER, 42).unwrap();
//! downcast to, so the downcast cannot fail on correct data; a failure is a
//! smarm bug, asserted in debug. The phantom `M` on [`Name`] re-imposes the
//! type at the call site, so callers never touch the erasure.
//! //!
//! ## Cleanup is lazy (prune-on-contact) //! worker.join().unwrap();
//! //!
//! As before, there is no `finalize` hook and no name field on the slot. Every //! // The name dies with the actor: nobody holds it anymore.
//! operation that touches a binding checks the target pid's liveness via the //! assert_eq!(whereis("counter"), None);
//! generation-checked slot word; a binding to a dead actor behaves as absent //! });
//! and is pruned on contact (its [`Mailbox`] and every name pointing at it are //! ```
//! dropped). The cost is a dead binding lingering until something looks at it;
//! the payoff is zero coupling to the actor lifecycle.
//! //!
//! ## Locking //! ## Names carry a message type
//! //!
//! One `RawMutex` (Leaf class) in `RuntimeInner`, exactly like the old //! A [`Name<M>`] is a plain string plus a type parameter `M`: the message
//! registry. The fold (name index *and* handles under the one lock) is what //! type that name expects to receive. [`Name::new`] is `const`, so the usual
//! keeps a name-addressed `send` on a single Leaf — `raw_mutex` panics on a //! pattern is a module-level constant like `COUNTER` above, shared by every
//! second Leaf acquired while one is held. The send path clones the `Sender` //! caller. The type parameter means a name is only ever sent the kind of
//! **under** the Leaf lock (a `Sender::clone` takes a Channel lock, permitted //! message it was declared for. If two different constants share the same
//! under a Leaf), then **releases** the Leaf and only *then* sends — a send can //! string but have different message types, they still address two
//! unpark a receiver, and wakeup-bearing work runs outside the Leaf. Order is //! independent channels on the same actor: registering both just gives that
//! **Leaf -> Channel**, as `pg`/`finalize`. //! actor two ways to be reached, one per message type. This is how you give
//! one actor a "public" channel and a separate, differently-typed "admin"
//! channel under related names, without inventing an enum to merge them.
//!
//! ## One actor per name, looked up fresh every time
//!
//! A name always points at exactly one actor at a time (contrast a *process
//! group*, from the [`pg`](crate::pg) module, which is one name mapping to
//! many actors). Unlike a plain [`Pid`], which names one specific actor
//! forever and stops working the moment that actor dies, a name is
//! re-resolved on every [`send`]: if the actor holding it dies and a new one
//! registers under the same name, the next `send` reaches the new holder
//! automatically. Use a name for a long-lived service whose exact identity
//! you do not want to track by hand; use a `Pid` when you already have one
//! and want to talk to that exact actor.
//!
//! ## Registration ends when the actor does
//!
//! There is no separate step to clean up a name when its actor exits: dying
//! is enough. The next operation that touches a dead binding (a [`whereis`],
//! a [`send`], or another actor's [`register`] of the same name) notices the
//! actor is gone and clears the stale entry as a side effect, so the name
//! becomes free again. [`unregister`] is only for a live actor voluntarily
//! giving up a name it no longer wants; nothing has to call it on the way
//! out.
//!
//! ## Implementation notes
//!
//! These details matter if you are working on smarm itself; they are not
//! part of the public contract.
//!
//! Internally, each live actor that has published at least one channel owns
//! a `Mailbox`: its pid plus a set of typed channels, keyed by the message
//! type's `TypeId`. A stored channel is a `Box<dyn Any + Send>` that
//! is concretely a `Sender<M>`; resolving for `M` looks up that exact
//! `TypeId` and downcasts, so the downcast cannot fail on correct data (a
//! failure would be a bug in the registry itself, checked in debug builds).
//! Registering a name therefore means: find or create the actor's mailbox,
//! insert the channel under its type, and point the name at the actor's pid.
//!
//! There is no callback when an actor exits. Every operation that touches a
//! binding checks the target pid's liveness directly against the scheduler's
//! slot table (which also tracks a generation counter, so a dead actor's
//! reused slot index is never mistaken for the same actor). A binding to a
//! dead actor is treated as absent and dropped right there. This keeps the
//! registry decoupled from actor teardown, at the cost of a dead binding
//! lingering until something happens to look at it.
//!
//! The whole registry (both the name index and the per-actor mailboxes) sits
//! behind one lock, which is what lets a name-addressed [`send`] resolve and
//! clone the target's sender in a single critical section. The sender is
//! cloned while that lock is held, then the lock is released before the
//! actual send, since delivering a message can wake a parked receiver and
//! that wakeup work should not run while the registry is locked.
use crate::channel::Sender; use crate::channel::Sender;
use crate::pid::{Addressable, Name, Pid}; use crate::pid::{Addressable, Name, Pid};
@@ -80,28 +133,33 @@ impl std::fmt::Display for RegisterError {
impl std::error::Error for RegisterError {} impl std::error::Error for RegisterError {}
/// Why a name-addressed [`send`] did not deliver. Carries the message back so /// Why a send did not deliver. Every variant carries the undelivered message
/// the caller never loses it (mirrors [`crate::channel::SendError`]). /// back, mirroring [`crate::channel::SendError`], so a failed send never
/// silently drops what you tried to send.
/// ///
/// `Debug`/`Display` are hand-written so neither demands `M: Debug` — the /// `Debug` and `Display` are hand-written so neither requires `M: Debug`,
/// payload is returned, not printed. /// since the payload is handed back to you, not printed.
pub enum SendError<M> { pub enum SendError<M> {
/// No live actor is currently registered under this name. Name-addressed /// No live actor is currently registered under this name. Returned only
/// [`send`] only; the pid-addressed counterpart is [`SendError::Dead`]. /// by name-addressed [`send`]; the pid-addressed counterpart of "nothing
/// there" is [`SendError::Dead`].
Unresolved(M), Unresolved(M),
/// The pid-addressed actor is no longer the live incarnation this pid names /// The actor this pid identifies has died, even if its slot has since
/// — it has died, even if its slot now holds a *different* actor (a direct /// been taken over by a different, live actor. A direct `Pid<A>` send
/// `Pid<A>` send never redirects; contrast name-addressed [`send`]). Pid /// never redirects to that new occupant; contrast name-addressed
/// paths ([`send_to`] / [`send_dyn`]) only. /// [`send`], which would reach it. Returned by the pid-addressed sends,
/// [`send_to`] and [`send_dyn`].
Dead(M), Dead(M),
/// The actor is live but exposes no channel for this message type. /// The actor is live but has not published a channel for this message
/// type.
NoChannel(M), NoChannel(M),
/// The actor's channel for this message type is closed (its receiver is gone). /// The actor's channel for this message type is closed (its receiver has
/// been dropped).
Closed(M), Closed(M),
/// No live member to deliver to — a [`dispatch`](crate::dispatch) over an /// No live member was available to deliver to: returned by
/// empty (or all-dead) process group. Group-addressed dispatch only; the /// [`dispatch`](crate::dispatch) when the target process group is empty
/// name-addressed counterpart is [`SendError::Unresolved`]. The message is /// or every member in it has died. The name-addressed counterpart of
/// handed back undelivered. /// this case is [`SendError::Unresolved`].
NoMember(M), NoMember(M),
} }
@@ -150,9 +208,9 @@ impl<M> std::error::Error for SendError<M> {}
/// A registry-stored channel, type-erased over its message type. The stored /// A registry-stored channel, type-erased over its message type. The stored
/// object must serve two readers: `clone_sender` (downcast back to the concrete /// object must serve two readers: `clone_sender` (downcast back to the concrete
/// `Sender<M>`) and the RFC 016 snapshot (queued length without knowing `M`). /// `Sender<M>`) and the runtime introspection snapshot (queued length without
/// A bare `Box<dyn Any>` gives the first but not the second, so we erase behind /// knowing `M`). A bare `Box<dyn Any>` gives the first but not the second, so
/// this small trait instead. /// we erase behind this small trait instead.
trait ErasedSender: Send { trait ErasedSender: Send {
fn as_any(&self) -> &dyn Any; fn as_any(&self) -> &dyn Any;
fn queued_len(&self) -> usize; fn queued_len(&self) -> usize;
@@ -169,7 +227,7 @@ impl<M: Send + 'static> ErasedSender for Sender<M> {
/// One typed channel of an actor, type-erased. Concretely a `Sender<M>` filed /// One typed channel of an actor, type-erased. Concretely a `Sender<M>` filed
/// under `TypeId::of::<M>()`; `msg_type` is `type_name::<M>()`, kept for /// under `TypeId::of::<M>()`; `msg_type` is `type_name::<M>()`, kept for
/// observers (RFC 014 §4.5) and as the debug cross-check on the downcast. /// observability tooling and as the debug cross-check on the downcast.
struct Channel { struct Channel {
sender: Box<dyn ErasedSender>, sender: Box<dyn ErasedSender>,
msg_type: &'static str, msg_type: &'static str,
@@ -204,7 +262,7 @@ impl Mailbox {
} }
} }
/// Per-actor registry view handed to RFC 016 introspection: registered names /// Per-actor registry view handed to runtime introspection: registered names
/// and summed mailbox depth, tagged with the mailbox's `pid` so a stale /// and summed mailbox depth, tagged with the mailbox's `pid` so a stale
/// incarnation can be filtered against the slab. Covers only *published* /// incarnation can be filtered against the slab. Covers only *published*
/// channels (`register` / `install` / `spawn_addr` / gen_server start); an /// channels (`register` / `install` / `spawn_addr` / gen_server start); an
@@ -217,14 +275,21 @@ pub(crate) struct MailboxInfo {
} }
/// The directory. Invariant (held under the registry lock): every value in /// The directory. Invariant (held under the registry lock): every value in
/// `by_name` is the index of a [`Mailbox`] present in `by_index`, and that /// `by_name` is the full [`Pid`] (index *and* generation) of an actor that
/// mailbox's `pid.index()` equals the key. Stale entries (dead actors) violate /// published a [`Mailbox`] into `by_index` at registration time. Stale entries
/// nothing — they are simply pruned on contact. /// (dead holders, including holders whose slot has since been re-tenanted by
/// a different actor) violate nothing: they are pruned on contact, and the
/// generation makes "dead" decidable even after slot reuse.
pub(crate) struct Registry { pub(crate) struct Registry {
/// `pid.index() -> the actor's mailbox`. The handle store. /// `pid.index() -> the actor's mailbox`. The handle store.
by_index: HashMap<u32, Mailbox>, by_index: HashMap<u32, Mailbox>,
/// `name -> pid.index()`. Several names may map to one actor. /// `name -> holder pid`. Several names may map to one actor. The full pid
by_name: HashMap<&'static str, u32>, /// (not just the index) is load-bearing: an index alone cannot tell a dead
/// holder from the live actor now tenanting its recycled slot. Comparing
/// only the index would make such a name read as live-held (unresolvable
/// and unregisterable at once) and could misdeliver to whatever new,
/// same-typed actor now sits in that slot.
by_name: HashMap<&'static str, Pid>,
} }
impl Registry { impl Registry {
@@ -232,24 +297,31 @@ impl Registry {
Self { by_index: HashMap::new(), by_name: HashMap::new() } Self { by_index: HashMap::new(), by_name: HashMap::new() }
} }
/// Drop a dead actor's mailbox and every name that pointed at it. /// Drop a dead holder's artifacts: every name bound to it, and its
fn prune(&mut self, index: u32) { /// mailbox, but only while the mailbox is still *its own*. A recycled
self.by_index.remove(&index); /// slot's mailbox belongs to the live tenant (publish replaces it
self.by_name.retain(|_, idx| *idx != index); /// wholesale on pid mismatch) and is left untouched.
fn prune_holder(&mut self, holder: Pid) {
self.by_name.retain(|_, p| *p != holder);
if self.by_index.get(&holder.index()).is_some_and(|mb| mb.pid == holder) {
self.by_index.remove(&holder.index());
}
} }
/// RFC 016 snapshot input: per-slot-index registry view — the actor's /// Runtime introspection input: per-slot-index registry view, giving the
/// registered names (inverted from `by_name`) and its mailbox depth (queued /// actor's registered names (inverted from `by_name`) and its mailbox
/// messages summed across every published typed channel). Built in one pass /// depth (queued messages summed across every published typed channel).
/// under the registry Leaf; the per-channel `queued_len` takes a Channel /// Carries each mailbox's full `pid` so the caller can discard a stale
/// lock, legal under the Leaf (Leaf → Channel). Carries each mailbox's full /// incarnation's entry against the slab's live generation. Names are
/// `pid` so the caller can discard a stale incarnation's entry against the /// matched to mailboxes by *full pid*, so a stale name (dead holder)
/// slab's live generation. Names that dangle (point at no mailbox) are /// still annotates the corpse's own mailbox if that survives, but never a
/// dropped — they violate no invariant and get pruned on next contact. /// recycled slot's new tenant; names that attach to no mailbox are
/// dropped, since that violates no invariant and they get pruned on next
/// contact.
pub(crate) fn introspect_map(&self) -> HashMap<u32, MailboxInfo> { pub(crate) fn introspect_map(&self) -> HashMap<u32, MailboxInfo> {
let mut names: HashMap<u32, Vec<&'static str>> = HashMap::new(); let mut names: HashMap<Pid, Vec<&'static str>> = HashMap::new();
for (&name, &idx) in &self.by_name { for (&name, &pid) in &self.by_name {
names.entry(idx).or_default().push(name); names.entry(pid).or_default().push(name);
} }
let mut out: HashMap<u32, MailboxInfo> = HashMap::with_capacity(self.by_index.len()); let mut out: HashMap<u32, MailboxInfo> = HashMap::with_capacity(self.by_index.len());
for (&idx, mb) in &self.by_index { for (&idx, mb) in &self.by_index {
@@ -258,7 +330,7 @@ impl Registry {
idx, idx,
MailboxInfo { MailboxInfo {
pid: mb.pid, pid: mb.pid,
names: names.remove(&idx).unwrap_or_default(), names: names.remove(&mb.pid).unwrap_or_default(),
depth: depth.min(u32::MAX as usize) as u32, depth: depth.min(u32::MAX as usize) as u32,
}, },
); );
@@ -268,15 +340,16 @@ impl Registry {
/// Single-actor form of [`introspect_map`](Self::introspect_map): the /// Single-actor form of [`introspect_map`](Self::introspect_map): the
/// registry view for one slot index, or `None` if no mailbox is published /// registry view for one slot index, or `None` if no mailbox is published
/// there. Used by `actor_info` so its cost stays proportional to the one /// there. Used by the runtime's per-actor introspection so its cost stays
/// actor rather than locking every channel in the runtime. /// proportional to the one actor rather than locking every channel in the
/// runtime.
pub(crate) fn introspect_one(&self, idx: u32) -> Option<MailboxInfo> { pub(crate) fn introspect_one(&self, idx: u32) -> Option<MailboxInfo> {
let mb = self.by_index.get(&idx)?; let mb = self.by_index.get(&idx)?;
let depth: usize = mb.channels.values().map(|c| c.sender.queued_len()).sum(); let depth: usize = mb.channels.values().map(|c| c.sender.queued_len()).sum();
let names = self let names = self
.by_name .by_name
.iter() .iter()
.filter_map(|(&n, &i)| (i == idx).then_some(n)) .filter_map(|(&n, &p)| (p == mb.pid).then_some(n))
.collect(); .collect();
Some(MailboxInfo { pid: mb.pid, names, depth: depth.min(u32::MAX as usize) as u32 }) Some(MailboxInfo { pid: mb.pid, names, depth: depth.min(u32::MAX as usize) as u32 })
} }
@@ -287,14 +360,18 @@ fn live(inner: &crate::runtime::RuntimeInner, pid: Pid) -> bool {
inner.slot_at(pid).is_some_and(|s| s.is_live_for(pid)) inner.slot_at(pid).is_some_and(|s| s.is_live_for(pid))
} }
/// Publish the current actor's `Sender<M>` under `name`, capturing the channel /// Give the current actor's channel a name, so other actors can find and
/// so the name becomes messageable. Idempotent for the same `(name, type)`; /// message it by that name instead of needing its [`Pid`].
/// registering a *second* type under the same (or another) name on the same
/// actor just adds another channel to the actor's mailbox.
/// ///
/// Fails with [`RegisterError::NameTaken`] if the name is held by a *different* /// Calling this again with the same `(name, type)` from the same actor is
/// live actor (a binding to a dead actor is pruned and the name treated as /// harmless. Registering a *second* message type under the same (or a
/// free). Panics if called outside `Runtime::run()`. /// different) name from the same actor just adds another typed channel to
/// that actor's mailbox; it does not replace the first.
///
/// Fails with [`RegisterError::NameTaken`] if the name is currently held by a
/// *different* live actor. A name held by an actor that has since died is not
/// considered taken: it is quietly reclaimed and handed to you. Panics if
/// called outside [`run`](crate::run).
pub fn register<M: Send + 'static>(name: Name<M>, tx: Sender<M>) -> Result<(), RegisterError> { pub fn register<M: Send + 'static>(name: Name<M>, tx: Sender<M>) -> Result<(), RegisterError> {
register_with(self_pid(), name.as_str(), tx) register_with(self_pid(), name.as_str(), tx)
} }
@@ -302,8 +379,8 @@ pub fn register<M: Send + 'static>(name: Name<M>, tx: Sender<M>) -> Result<(), R
/// Bind `name` to `pid`'s mailbox and publish `tx` under `M`'s [`TypeId`], for /// Bind `name` to `pid`'s mailbox and publish `tx` under `M`'s [`TypeId`], for
/// an explicit (already-live) actor rather than `self`. The shared core of /// an explicit (already-live) actor rather than `self`. The shared core of
/// [`register`] (which passes `self_pid()`) and the parent-side server-name /// [`register`] (which passes `self_pid()`) and the parent-side server-name
/// bind in `gen_server` (which names a freshly spawned server before its body /// bind in `gen_server`, which names a freshly spawned server before its body
/// has run, so the name resolves the instant `start()` returns). Same collision /// has run, so the name resolves the instant `start()` returns. Same collision
/// rules and lock discipline as `register`. /// rules and lock discipline as `register`.
pub(crate) fn register_with<M: Send + 'static>( pub(crate) fn register_with<M: Send + 'static>(
me: Pid, me: Pid,
@@ -315,21 +392,22 @@ pub(crate) fn register_with<M: Send + 'static>(
if !live(inner, me) { if !live(inner, me) {
return Err(RegisterError::NoProc); return Err(RegisterError::NoProc);
} }
if let Some(&holder_idx) = reg.by_name.get(key) { if let Some(&holder) = reg.by_name.get(key) {
match reg.by_index.get(&holder_idx).map(|m| m.pid) { if holder == me {
Some(holder) if holder == me => {} // same actor: just add the channel below // Same actor: just add the channel below.
Some(holder) if live(inner, holder) => { } else if live(inner, holder) {
return Err(RegisterError::NameTaken { holder }); return Err(RegisterError::NameTaken { holder });
} } else {
Some(_) => reg.prune(holder_idx), // dead holder: free the name // Dead holder: free the name (and its other stale artifacts).
None => { // Liveness is judged against the *stored* pid, generation
reg.by_name.remove(key); // dangling name: free it // included, so a recycled slot's live tenant no longer makes a
} // dead name read as taken.
reg.prune_holder(holder);
} }
} }
// Publish (or extend) the mailbox with this channel, then bind the name. // Publish (or extend) the mailbox with this channel, then bind the name.
publish_channel::<M>(&mut reg, me, tx); publish_channel::<M>(&mut reg, me, tx);
reg.by_name.insert(key, me.index()); reg.by_name.insert(key, me);
Ok(()) Ok(())
}) })
} }
@@ -352,14 +430,14 @@ fn publish_channel<M: Send + 'static>(reg: &mut Registry, me: Pid, tx: Sender<M>
/// Publish the current actor's `Sender<A::Msg>` into its mailbox **without** /// Publish the current actor's `Sender<A::Msg>` into its mailbox **without**
/// binding a name, and hand back the typed [`Pid<A>`] that addresses this /// binding a name, and hand back the typed [`Pid<A>`] that addresses this
/// actor directly. This is the opt-in, lazy install of RFC 014 §5: an actor /// actor directly.
/// that wants to be reachable by a direct, identity-bound [`Pid<A>`] (rather
/// than only via a re-resolving [`Name`]) calls this once with its inbox
/// sender, then hands the returned pid out.
/// ///
/// Unlike [`register`] there is no name to collide on, and `self` is always a /// This is for an actor that wants to be reachable directly by its pid,
/// live actor inside `run()`, so this is infallible. Panics if called outside /// rather than only through a re-resolving [`Name`]: call this once with your
/// `Runtime::run()`. /// inbox sender, then hand the returned `Pid<A>` to whoever should be able to
/// message you. Unlike [`register`] there is no name to collide on, and the
/// current actor is always live while inside `run()`, so this cannot fail.
/// Panics if called outside [`run`](crate::run).
pub fn install<A: Addressable>(tx: Sender<A::Msg>) -> Pid<A> { pub fn install<A: Addressable>(tx: Sender<A::Msg>) -> Pid<A> {
let me = self_pid(); let me = self_pid();
with_runtime(|inner| { with_runtime(|inner| {
@@ -375,11 +453,12 @@ pub fn install<A: Addressable>(tx: Sender<A::Msg>) -> Pid<A> {
/// Publish `tx` into `pid`'s mailbox under `M`'s [`TypeId`], for an explicit /// Publish `tx` into `pid`'s mailbox under `M`'s [`TypeId`], for an explicit
/// (freshly minted, already-live) actor rather than `self`. The parent-side /// (freshly minted, already-live) actor rather than `self`. The parent-side
/// half of [`spawn_addr`](crate::spawn_addr): the spawner makes the inbox and /// half of [`spawn_addr`](crate::spawn_addr): the spawner makes the inbox and
/// publishes the sender here *before* handing back the `Pid<A>`, so an immediate /// publishes the sender here *before* handing back the `Pid<A>`, so an
/// `send_to` on the returned pid always resolves — the address is live the /// immediate `send_to` on the returned pid always resolves. The address is
/// instant the caller holds it, with no dependence on the body having run yet. /// live the instant the caller holds it, with no dependence on the spawned
/// actor's body having run yet.
/// ///
/// Caller guarantees `pid` is the just-installed actor (Queued, this exact /// Caller guarantees `pid` is the just-installed actor (queued, this exact
/// incarnation); `publish_channel` replaces any stale leftover at the slot. /// incarnation); `publish_channel` replaces any stale leftover at the slot.
pub(crate) fn install_for<M: Send + 'static>(pid: Pid, tx: Sender<M>) { pub(crate) fn install_for<M: Send + 'static>(pid: Pid, tx: Sender<M>) {
with_runtime(|inner| { with_runtime(|inner| {
@@ -389,109 +468,106 @@ pub(crate) fn install_for<M: Send + 'static>(pid: Pid, tx: Sender<M>) {
}); });
} }
/// The single actor currently registered under `name`, or `None` if unbound or /// Look up which actor currently holds `name`, if any. Returns `None` if the
/// no longer live (the stale binding is pruned on the way out). /// name is unbound, or if it was bound to an actor that has since died (the
/// stale binding is cleared as a side effect of this call).
pub fn whereis(name: &str) -> Option<Pid> { pub fn whereis(name: &str) -> Option<Pid> {
with_runtime(|inner| { with_runtime(|inner| {
let mut reg = inner.registry.lock(); let mut reg = inner.registry.lock();
let idx = *reg.by_name.get(name)?; let pid = *reg.by_name.get(name)?;
match reg.by_index.get(&idx).map(|m| m.pid) { if live(inner, pid) {
Some(pid) if live(inner, pid) => Some(pid), Some(pid)
Some(_) => { } else {
reg.prune(idx); // Generation-checked against the stored holder: a recycled slot's
// live tenant reads dead here, and the stale name heals.
reg.prune_holder(pid);
None None
} }
None => {
reg.by_name.remove(name);
None
}
}
}) })
} }
/// Resolve `name` to a *typed* [`Pid<A>`] — the identity-bound counterpart of /// Like [`whereis`], but returns a *typed* [`Pid<A>`] instead of a bare
/// [`whereis`] (RFC 014 §4.4). Recovers the compile-checked /// [`Pid`], so a follow-up [`send_to`] is compile-checked instead of needing
/// [`send_to`] path from a durable name: looks the name up, /// the untyped [`send_dyn`] escape hatch. `None` if the name is unbound or its
/// then re-types the erased pid as `Pid<A>` via the unchecked /// holder has died.
/// [`assert_type`](crate::pid::assert_type) primitive. A wrong `A` is not
/// unsound — it degrades to [`SendError::NoChannel`] on the next send (routing
/// is by message `TypeId`), never a misdelivery. `None` if unbound or dead.
/// ///
/// Panics if called outside `Runtime::run()`. /// The type `A` is not checked against what the name's holder actually
/// published: if you pick the wrong `A`, this still succeeds, but the next
/// send against the returned pid degrades to [`SendError::NoChannel`] rather
/// than reaching the wrong actor or the wrong channel.
///
/// Panics if called outside [`run`](crate::run).
pub fn lookup_as<A: Addressable>(name: &str) -> Option<Pid<A>> { pub fn lookup_as<A: Addressable>(name: &str) -> Option<Pid<A>> {
whereis(name).map(crate::pid::assert_type::<A>) whereis(name).map(crate::pid::assert_type::<A>)
} }
/// Resolve `name` to its actor's pid and a cloned `Sender<M>`, under the Leaf /// Resolve `name` to its actor's pid and a cloned `Sender<M>`, all under one
/// lock (clone-under-lock, then release). The crate-internal building block for /// lock acquisition. The crate-internal building block for `gen_server`'s
/// `gen_server`'s by-name addressing: a named server publishes its inbox as a /// by-name addressing: a named server publishes its inbox as a
/// `Sender<Envelope<G>>` (via [`register_with`]), and `whereis_server` / `call` /// `Sender<Envelope<G>>` (via [`register_with`]), and the server's `call` /
/// / `cast` recover that exact typed sender here to rebuild a `GenServerRef<G>`. /// `cast` / `whereis_server` recover that exact typed sender here to rebuild a
/// `None` if unbound, dead (pruned on the way out), or holding no `M` channel. /// `GenServerRef<G>`. `None` if unbound, dead (pruned on the way out), or
/// holding no `M` channel.
pub(crate) fn resolve_named_sender<M: Send + 'static>(name: &str) -> Option<(Pid, Sender<M>)> { pub(crate) fn resolve_named_sender<M: Send + 'static>(name: &str) -> Option<(Pid, Sender<M>)> {
with_runtime(|inner| { with_runtime(|inner| {
let mut reg = inner.registry.lock(); let mut reg = inner.registry.lock();
let idx = *reg.by_name.get(name)?; let pid = *reg.by_name.get(name)?;
let pid = match reg.by_index.get(&idx).map(|m| m.pid) { if !live(inner, pid) {
Some(pid) if live(inner, pid) => pid, // Stored-pid liveness, generation included: a name whose holder
Some(_) => { // died is pruned (heals) even if the slot has a new tenant.
reg.prune(idx); // Otherwise the tenant's mailbox would make the name unresolvable
// without pruning, wedging it for the tenant's lifetime.
reg.prune_holder(pid);
return None; return None;
} }
None => { // A live holder's mailbox is its own (publish replaces wholesale on
reg.by_name.remove(name); // pid mismatch, and one live actor per slot), so index lookup is safe.
return None; let tx = reg.by_index.get(&pid.index()).and_then(Mailbox::clone_sender::<M>)?;
}
};
let tx = reg.by_index.get(&idx).and_then(Mailbox::clone_sender::<M>)?;
Some((pid, tx)) Some((pid, tx))
}) })
} }
/// Remove the binding for `name`, returning the actor it pointed at if still /// Give up a name. Returns the actor it pointed at, if that actor was still
/// live. Only the *name* is freed; the actor's mailbox (and any other names for /// live. Only the *name* is freed; the actor's mailbox (and any other names
/// it) remain. A binding to a dead actor is reported as `None`. /// bound to it) are unaffected. A binding to an already-dead actor reports
/// `None`, since there was nothing live to release.
pub fn unregister(name: &str) -> Option<Pid> { pub fn unregister(name: &str) -> Option<Pid> {
with_runtime(|inner| { with_runtime(|inner| {
let mut reg = inner.registry.lock(); let mut reg = inner.registry.lock();
let idx = reg.by_name.remove(name)?; let pid = reg.by_name.remove(name)?;
match reg.by_index.get(&idx).map(|m| m.pid) { if live(inner, pid) { Some(pid) } else { None }
Some(pid) if live(inner, pid) => Some(pid),
_ => None,
}
}) })
} }
/// Resolve `name` to its actor's `Sender<M>` and deliver `msg`. The whole point /// Look `name` up and deliver `msg` to whichever actor currently holds it.
/// of the rework: a name you can *send* to. /// This is the point of naming an actor: a name you can send a message to
/// directly, without a separate lookup step.
/// ///
/// Errors (message returned in every case): [`SendError::Unresolved`] if no /// On failure the message comes back to you, wrapped in the [`SendError`]
/// live actor holds the name, [`SendError::NoChannel`] if that actor has no /// variant that explains why: [`SendError::Unresolved`] if no live actor
/// channel for `M`, [`SendError::Closed`] if its `M` channel's receiver is /// currently holds the name, [`SendError::NoChannel`] if the actor that holds
/// gone. Panics if called outside `Runtime::run()`. /// it never published a channel for `M`, or [`SendError::Closed`] if it
/// published one but has since dropped the receiving end. Panics if called
/// outside [`run`](crate::run).
pub fn send<M: Send + 'static>(name: Name<M>, msg: M) -> Result<(), SendError<M>> { pub fn send<M: Send + 'static>(name: Name<M>, msg: M) -> Result<(), SendError<M>> {
let key = name.as_str(); let key = name.as_str();
with_runtime(|inner| { with_runtime(|inner| {
// Resolve + clone the sender under the Leaf lock, then drop the lock // Resolve + clone the sender under the registry lock, then drop the
// before sending (a send can unpark a receiver). // lock before sending (a send can unpark a receiver).
let tx = { let tx = {
let mut reg = inner.registry.lock(); let mut reg = inner.registry.lock();
let idx = match reg.by_name.get(key) { let pid = match reg.by_name.get(key) {
Some(&i) => i, Some(&p) => p,
None => return Err(SendError::Unresolved(msg)), None => return Err(SendError::Unresolved(msg)),
}; };
let pid = match reg.by_index.get(&idx).map(|m| m.pid) {
Some(pid) => pid,
None => {
reg.by_name.remove(key);
return Err(SendError::Unresolved(msg));
}
};
if !live(inner, pid) { if !live(inner, pid) {
reg.prune(idx); // Stored-pid liveness (generation included), so a recycled
// slot's new live tenant is never mistaken for the name's
// original (now-dead) holder.
reg.prune_holder(pid);
return Err(SendError::Unresolved(msg)); return Err(SendError::Unresolved(msg));
} }
match reg.by_index.get(&idx).and_then(Mailbox::clone_sender::<M>) { match reg.by_index.get(&pid.index()).and_then(Mailbox::clone_sender::<M>) {
Some(tx) => tx, Some(tx) => tx,
None => return Err(SendError::NoChannel(msg)), None => return Err(SendError::NoChannel(msg)),
} }
@@ -502,18 +578,19 @@ pub fn send<M: Send + 'static>(name: Name<M>, msg: M) -> Result<(), SendError<M>
/// Resolve a *raw* pid to its mailbox and deliver `msg` on the channel for `M`, /// Resolve a *raw* pid to its mailbox and deliver `msg` on the channel for `M`,
/// with **no redirect**. The stored mailbox must be this exact incarnation /// with **no redirect**. The stored mailbox must be this exact incarnation
/// (generation included) and still live; otherwise the actor this pid named is /// (generation included) and still live; otherwise the actor this pid named
/// gone and the result is [`SendError::Dead`] even when the slot now holds a /// is gone and the result is [`SendError::Dead`], even when the slot now
/// different, live actor (which we leave untouched). Shared by [`send_to`] /// holds a different, live actor (which is left untouched). Shared by
/// (typed, `M = A::Msg`, channel guaranteed on an installed actor) and /// [`send_to`] (typed, `M = A::Msg`, channel guaranteed on an installed
/// [`send_dyn`] (explicit `M`, where `NoChannel` is a real outcome). /// actor) and [`send_dyn`] (explicit `M`, where `NoChannel` is a real
/// outcome).
fn send_to_pid<M: Send + 'static>( fn send_to_pid<M: Send + 'static>(
inner: &crate::runtime::RuntimeInner, inner: &crate::runtime::RuntimeInner,
pid: Pid, pid: Pid,
msg: M, msg: M,
) -> Result<(), SendError<M>> { ) -> Result<(), SendError<M>> {
// Resolve + clone the sender under the Leaf lock, then drop the lock before // Resolve + clone the sender under the registry lock, then drop the lock
// sending (a send can unpark a receiver) — Leaf -> Channel, as name `send`. // before sending (a send can unpark a receiver), same order as `send`.
let tx = { let tx = {
let mut reg = inner.registry.lock(); let mut reg = inner.registry.lock();
match reg.by_index.get(&pid.index()).map(|m| m.pid) { match reg.by_index.get(&pid.index()).map(|m| m.pid) {
@@ -526,7 +603,7 @@ fn send_to_pid<M: Send + 'static>(
} }
// Our incarnation's mailbox, but the actor has died: prune + Dead. // Our incarnation's mailbox, but the actor has died: prune + Dead.
Some(stored) if stored == pid => { Some(stored) if stored == pid => {
reg.prune(pid.index()); reg.prune_holder(pid);
return Err(SendError::Dead(msg)); return Err(SendError::Dead(msg));
} }
// A different incarnation (or nothing) occupies the slot: the actor // A different incarnation (or nothing) occupies the slot: the actor
@@ -537,32 +614,36 @@ fn send_to_pid<M: Send + 'static>(
tx.send(msg).map_err(|crate::channel::SendError(m)| SendError::Closed(m)) tx.send(msg).map_err(|crate::channel::SendError(m)| SendError::Closed(m))
} }
/// Deliver `msg` to the exact actor named by `pid` — RFC 014 §4.2's direct, /// Deliver `msg` directly to the exact actor identified by `pid`. Unlike
/// identity-bound addressing mode. Unlike name-addressed [`send`] there is **no /// name-addressed [`send`], there is **no redirect**: if that specific actor
/// redirect**: if that incarnation has died the message comes back as /// has died, the message comes back as [`SendError::Dead`], even if its slot
/// [`SendError::Dead`], even if its slot now holds a different actor. /// has since been taken over by a different, live actor. Use this when you
/// already hold a `Pid<A>` and want to talk to that one actor specifically;
/// use [`send`] with a [`Name`] when you want whichever actor currently holds
/// a name.
/// ///
/// The message type is the actor's `A::Msg`, so on a live actor that has /// The message type is the actor's `A::Msg`, so on a live actor that has
/// installed its inbox (via [`install`] or [`register`]) the channel is always /// installed its inbox (via [`install`] or [`register`]) the channel is
/// present; [`SendError::NoChannel`] therefore means the actor is live but /// always present; [`SendError::NoChannel`] therefore means the actor is live
/// never published a `Pid<A>`-reachable inbox. Panics if called outside /// but never published a `Pid<A>`-reachable inbox. Panics if called outside
/// `Runtime::run()`. /// [`run`](crate::run).
pub fn send_to<A: Addressable>(pid: Pid<A>, msg: A::Msg) -> Result<(), SendError<A::Msg>> { pub fn send_to<A: Addressable>(pid: Pid<A>, msg: A::Msg) -> Result<(), SendError<A::Msg>> {
with_runtime(|inner| send_to_pid::<A::Msg>(inner, pid.erase(), msg)) with_runtime(|inner| send_to_pid::<A::Msg>(inner, pid.erase(), msg))
} }
/// The explicit bare-pid escape hatch (RFC 014 §4.6): deliver `msg` of type `M` /// The escape hatch for sending to a bare, untyped [`Pid`] when the typed
/// to `pid` when all you hold is an untyped [`Pid`] — a pid off a [`Down`], or /// [`send_to`] is unavailable, for example a pid recovered from a [`Down`]
/// out of a future `members()` — so the typed [`send_to`] is unavailable. /// notification or a group's `members()` list, where you no longer know the
/// actor's message type at compile time.
/// ///
/// This is the one send whose message type can genuinely be wrong: the actor /// Because the message type is not checked at compile time here, this is the
/// may be live yet expose no channel for `M`, returning [`SendError::NoChannel`] /// one send that can genuinely be live-but-wrong: the actor may be alive yet
/// (on the typed paths that downcast collapses to a `debug_assert`). It is /// expose no channel for `M`, in which case you get [`SendError::NoChannel`]
/// named and documented as the fallible fallback so the typed `Pid<A>` / /// back instead of a misdelivery. Liveness and redirect behavior are
/// `Name<M>` paths stay the obvious default and an agentic caller reaches for a /// otherwise identical to [`send_to`]: identity-bound, no redirect,
/// present primitive instead of inventing a workaround. Liveness is identical /// [`SendError::Dead`] once the addressed incarnation is gone. Prefer
/// to [`send_to`]: identity-bound, no redirect, [`SendError::Dead`] once the /// `send_to` with a typed `Pid<A>` whenever you have one; reach for this only
/// addressed incarnation is gone. Panics if called outside `Runtime::run()`. /// when you don't. Panics if called outside [`run`](crate::run).
/// ///
/// [`Down`]: crate::Down /// [`Down`]: crate::Down
pub fn send_dyn<M: Send + 'static>(pid: Pid, msg: M) -> Result<(), SendError<M>> { pub fn send_dyn<M: Send + 'static>(pid: Pid, msg: M) -> Result<(), SendError<M>> {
+388 -161
View File
@@ -86,21 +86,28 @@
//! # Termination (counter-based) //! # Termination (counter-based)
//! //!
//! The old all-clear scanned the slot table under the big lock. Now: //! The old all-clear scanned the slot table under the big lock. Now:
//! exit when `io_out == 0` (read *before* the queue lock, phase-1 ordering) //! exit when `io_outstanding + io_fd_waiters == 0` (two Relaxed/Acquire
//! and, under the queue lock, the queue is empty and `live_actors == 0`. //! atomic loads, read *before* the queue pop) and, under the queue lock,
//! `live_actors` is incremented in `spawn` before the enqueue and decremented //! the queue is empty and `live_actors == 0`. `live_actors` is incremented
//! at the very END of `finalize_actor`, strictly after every wakeup that //! in `spawn` before the enqueue and decremented at the very END of
//! finalize produces has been enqueued. The soundness crux: any enqueue //! `finalize_actor`, strictly after every wakeup that finalize produces has
//! targets a live (not-yet-finalized) actor, so `live == 0` implies no wakeup //! been enqueued. The soundness crux: any enqueue targets a live
//! can still be in flight; combined with "spawner is itself live", observing //! (not-yet-finalized) actor, so `live == 0` implies no wakeup can still be
//! `(queue empty, live == 0)` under the queue lock means no work can ever //! in flight; combined with "spawner is itself live", observing
//! appear again. //! `(queue empty, live == 0)` means no work can ever appear again.
//! //!
//! # Timer / IO drain (try-lock, one-winner) //! # Scheduler park/wake (RFC 018)
//! //!
//! Unchanged from phase 1: one winner per round drains due timers and IO //! Schedulers sleep on per-thread futex parkers via the coordination layer
//! completions from their own mutexes; wakeups go through the unpark //! (`park.rs`), NOT on a shared wake pipe. IO backends are producers behind
//! protocol like everyone else's. //! a two-call contract — make the actor runnable (`unpark_at`), whose
//! `enqueue` tail wakes exactly one parked scheduler. The blocking pool and
//! epoll thread each route their own completions (driver-enqueues); there
//! is no shared completion queue, no drain lock, no one-winner drain phase.
//! Timers fire two ways: a busy-path due-check every loop iteration (one
//! Relaxed load of the earliest-deadline snapshot when no timer is armed),
//! and the timekeeper — at most one parked scheduler holds the timer
//! deadline, so an expiry wakes one scheduler, not a herd.
use crate::actor::{ use crate::actor::{
clear_current_pid, is_actor_done, reset_actor_done, set_current_actor_box, clear_current_pid, is_actor_done, reset_actor_done, set_current_actor_box,
@@ -445,6 +452,42 @@ pub(crate) struct Slot {
/// and only when the `budget-accounting` feature is on (it costs two RDTSC /// and only when the `budget-accounting` feature is on (it costs two RDTSC
/// per resume); stays 0 otherwise. Same single-writer Relaxed discipline. /// per resume); stays 0 otherwise. Same single-writer Relaxed discipline.
budget_cycles: AtomicU64, budget_cycles: AtomicU64,
/// RFC 007 (`smarm-causal`) — id of the causal-profiling site this actor
/// is currently inside (0 = none). Lives in the slot, not a thread-local,
/// so it survives preemption and cross-scheduler migration. Written only
/// by the actor itself (guard enter/exit on its own thread), read by that
/// same thread in `maybe_preempt` — single-writer Relaxed, like `overruns`.
/// Exists regardless of the feature; stays 0 without it.
causal_site: AtomicU32,
/// RFC 007 (`smarm-causal`) — virtual-speedup delay cycles this actor has
/// absorbed (or been credited). Compared against the global ledger in
/// `causal::check`; fast-forwarded on resume-from-park so blocked time
/// absorbs delay for free (Coz's blocked-thread rule). Same single-writer
/// discipline.
causal_delay: AtomicU64,
/// RFC 007 (`smarm-causal`) — true iff this actor's last deschedule was a
/// real park (a successful `park_return`, not a slice yield and not the
/// instant-wake re-queue). Consumed by `causal::on_resume`: only a wake
/// from genuine blocking forgives outstanding virtual delay; a merely
/// preempted (runnable) actor stays in debt and must pay by spinning.
/// Starts false: a fresh spawn is born current (`reset_counters` sets
/// `causal_delay` to the global ledger), and anything injected while it
/// sits spawn-queued is owed, not waived. Written by the owning
/// scheduler thread at the deschedule /
/// resume boundary only — same single-writer discipline as `causal_delay`.
causal_parked: AtomicBool,
/// RFC 007 — TSC at this actor's last *runnable* (yield) deschedule
/// while it sat in the live experiment's target site; 0 = no gap
/// pending. Paired with `causal_desched_epoch`; written on the
/// deschedule path, consumed at the next resume — same single-writer
/// discipline as `causal_delay`.
causal_desched_tsc: AtomicU64,
/// RFC 007 — experiment epoch live at that deschedule (see
/// `causal::EXPERIMENT_EPOCH`): the resume counts the gap only into
/// the same window, so one straddling `end()` — or a later `begin()`
/// with an identical site+pct word — is dropped instead of leaking a
/// cooldown across windows.
causal_desched_epoch: AtomicU64,
/// Cold lifecycle data. See [`SlotCold`]. /// Cold lifecycle data. See [`SlotCold`].
pub(crate) cold: RawMutex<SlotCold>, pub(crate) cold: RawMutex<SlotCold>,
} }
@@ -459,6 +502,14 @@ impl Slot {
overruns: AtomicU64::new(0), overruns: AtomicU64::new(0),
messages_received: AtomicU64::new(0), messages_received: AtomicU64::new(0),
budget_cycles: AtomicU64::new(0), budget_cycles: AtomicU64::new(0),
causal_site: AtomicU32::new(0),
causal_delay: AtomicU64::new(0),
// Overwritten at every occupancy by `reset_counters` (born
// current); false so a hypothetical reset-skipping path cannot
// waive the whole global backlog at first resume.
causal_parked: AtomicBool::new(false),
causal_desched_tsc: AtomicU64::new(0),
causal_desched_epoch: AtomicU64::new(0),
cold: RawMutex::new(SlotCold { cold: RawMutex::new(SlotCold {
actor: None, actor: None,
waiters: Vec::new(), waiters: Vec::new(),
@@ -547,6 +598,92 @@ impl Slot {
self.overruns.store(0, Ordering::Relaxed); self.overruns.store(0, Ordering::Relaxed);
self.messages_received.store(0, Ordering::Relaxed); self.messages_received.store(0, Ordering::Relaxed);
self.budget_cycles.store(0, Ordering::Relaxed); self.budget_cycles.store(0, Ordering::Relaxed);
self.causal_site.store(0, Ordering::Relaxed);
// Born current (Coz's new-thread rule): a fresh incarnation neither
// owes the process's accumulated delay history nor books it as park
// forgiveness. The old init (delay 0, parked true) waived the whole
// monotone backlog once per spawn via the first resume — found live
// under close-mode conn churn (~95k spawns/s): millions of phantom
// forgiven ms per 700ms window, even in 0% cells. Parked starts
// false: delay injected while spawn-queued is *owed* (the newborn is
// runnable, not blocked) and paid at its first check — the semantics
// `audit_zero_pct_window_absorbs_leftover_debt` pins.
#[cfg(feature = "smarm-causal")]
self.causal_delay
.store(crate::causal::global_delay_cycles(), Ordering::Relaxed);
#[cfg(not(feature = "smarm-causal"))]
self.causal_delay.store(0, Ordering::Relaxed);
self.causal_parked.store(false, Ordering::Relaxed);
self.causal_desched_tsc.store(0, Ordering::Relaxed);
self.causal_desched_epoch.store(0, Ordering::Relaxed);
}
/// RFC 007 — mark that this actor's deschedule was a genuine park.
/// Called from the scheduler's `YieldIntent::Park` branch on a successful
/// `park_return` only.
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn set_causal_parked(&self) {
self.causal_parked.store(true, Ordering::Relaxed);
}
/// RFC 007 — consume the parked marker at resume: returns whether the
/// last deschedule was a real park, and clears it so the next resume
/// defaults to "was runnable" unless the park branch says otherwise.
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn take_causal_parked(&self) -> bool {
self.causal_parked.swap(false, Ordering::Relaxed)
}
/// RFC 007 — stash the runnable-deschedule instant and the experiment
/// epoch it happened under (offcpu-gap audit; see `causal::on_resume`).
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn set_causal_desched(&self, tsc: u64, epoch: u64) {
self.causal_desched_tsc.store(tsc, Ordering::Relaxed);
self.causal_desched_epoch.store(epoch, Ordering::Relaxed);
}
/// RFC 007 — consume the stash: `(tsc, epoch)`; `(0, _)` = none pending.
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn take_causal_desched(&self) -> (u64, u64) {
let tsc = self.causal_desched_tsc.swap(0, Ordering::Relaxed);
let epoch = self.causal_desched_epoch.load(Ordering::Relaxed);
(tsc, epoch)
}
/// RFC 007 — current causal site id (0 = none). Single-writer: only the
/// on-CPU actor's thread writes, via the site-guard enter/exit.
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn causal_site(&self) -> u32 {
self.causal_site.load(Ordering::Relaxed)
}
/// RFC 007 — write the current causal site id (guard enter/exit).
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn set_causal_site(&self, id: u32) {
self.causal_site.store(id, Ordering::Relaxed);
}
/// RFC 007 — absorbed/credited virtual-delay cycles.
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn causal_delay(&self) -> u64 {
self.causal_delay.load(Ordering::Relaxed)
}
/// RFC 007 — set the absorbed-delay ledger (spin-absorb, credit, or the
/// resume-path fast-forward). Single-writer per the resume protocol: the
/// actor's own thread while on-CPU, the resuming scheduler thread at the
/// resume boundary — never both at once.
#[cfg_attr(not(feature = "smarm-causal"), allow(dead_code))]
#[inline]
pub(crate) fn set_causal_delay(&self, v: u64) {
self.causal_delay.store(v, Ordering::Relaxed);
} }
/// A pid's-eye snapshot of the slot. Cold paths re-read this under the /// A pid's-eye snapshot of the slot. Cold paths re-read this under the
@@ -620,8 +757,21 @@ pub(crate) struct RuntimeInner {
pub(crate) io: Mutex<Option<IoThread>>, pub(crate) io: Mutex<Option<IoThread>>,
/// Monotonic `MonitorId` source. Never reused. /// Monotonic `MonitorId` source. Never reused.
pub(crate) next_monitor_id: AtomicU64, pub(crate) next_monitor_id: AtomicU64,
/// Try-lock: exactly one scheduler thread drains timers/IO per iteration. /// RFC 018: the scheduler coordination layer — per-scheduler parkers,
drain_lock: Mutex<()>, /// idle mask, wake protocol, timekeeper role, earliest-deadline
/// snapshot. Arc'd because `Timers` shares it (insert-side deadline
/// notes run under the timers mutex).
pub(crate) coord: Arc<crate::park::Coordinator>,
/// `block_on_io` requests in flight. Incremented by the submitter
/// BEFORE submit (underflow-proof), decremented by the pool thread on
/// completion. Read lock-free by the idle path's termination verdict —
/// the per-pop `io.lock` of the drain era is gone.
pub(crate) io_outstanding: AtomicU32,
/// Parked fd waiters. Incremented by the registrar BEFORE
/// `epoll_register` (rolled back on error), decremented by whoever
/// consumes the registration (epoll thread on readiness, canceller on
/// an unwound wait). Same lock-free verdict read as `io_outstanding`.
pub(crate) io_fd_waiters: AtomicU32,
/// Per-thread stats, indexed by scheduler thread slot (0..N). /// Per-thread stats, indexed by scheduler thread slot (0..N).
pub(crate) stats: Vec<SchedulerStats>, pub(crate) stats: Vec<SchedulerStats>,
/// Global counters for RFC 000 primitives. /// Global counters for RFC 000 primitives.
@@ -672,6 +822,12 @@ impl RuntimeInner {
let slots: Box<[Slot]> = (0..max_actors).map(|_| Slot::vacant()).collect(); let slots: Box<[Slot]> = (0..max_actors).map(|_| Slot::vacant()).collect();
// Low indices on top of the stack so early spawns get low pids. // Low indices on top of the stack so early spawns get low pids.
let free: Vec<u32> = (0..max_actors as u32).rev().collect(); let free: Vec<u32> = (0..max_actors as u32).rev().collect();
// RFC 018: the coordination layer (asserts thread_count <= 64), and
// the timers' hook into it — every insert under the timers mutex
// notes its deadline (busy-path snapshot + timekeeper re-arm).
let coord = Arc::new(crate::park::Coordinator::new(thread_count));
let mut timers = Timers::new();
timers.attach_coordinator(coord.clone());
Arc::new(Self { Arc::new(Self {
run_queue: crate::run_queue::RunQueue::new(thread_count, max_actors), run_queue: crate::run_queue::RunQueue::new(thread_count, max_actors),
slots, slots,
@@ -680,10 +836,12 @@ impl RuntimeInner {
root_bits: AtomicU64::new(u64::MAX), root_bits: AtomicU64::new(u64::MAX),
root_exited: AtomicBool::new(false), root_exited: AtomicBool::new(false),
root_swept: AtomicBool::new(false), root_swept: AtomicBool::new(false),
timers: Mutex::new(Timers::new()), timers: Mutex::new(timers),
io: Mutex::new(None), io: Mutex::new(None),
next_monitor_id: AtomicU64::new(0), next_monitor_id: AtomicU64::new(0),
drain_lock: Mutex::new(()), coord,
io_outstanding: AtomicU32::new(0),
io_fd_waiters: AtomicU32::new(0),
stats, stats,
io_parked: AtomicU32::new(0), io_parked: AtomicU32::new(0),
sleeping: AtomicU32::new(0), sleeping: AtomicU32::new(0),
@@ -744,6 +902,13 @@ impl RuntimeInner {
); );
self.run_queue.push(pid); self.run_queue.push(pid);
crate::te!(crate::trace::Event::Enqueue(pid)); crate::te!(crate::trace::Event::Enqueue(pid));
// RFC 018 enqueue wake (fixes the silent enqueue): if a scheduler
// is parked, wake exactly one. The fast path when everyone is busy
// is a fence + one Relaxed load of an unmodified line — the
// pure-compute hot path pays (almost) nothing. Bias is over-wake:
// a spurious wake costs one futex round-trip and a failed pop; a
// missed wake would cost a stranded actor.
self.coord.wake_one_if_idle();
} }
/// Make `pid` runnable if it is parked; coalesce or defer otherwise. /// Make `pid` runnable if it is parked; coalesce or defer otherwise.
@@ -899,6 +1064,11 @@ pub fn init(config: Config) -> Runtime {
impl Runtime { impl Runtime {
/// Run `f` as the initial actor, block until all actors finish. /// Run `f` as the initial actor, block until all actors finish.
/// Can be called multiple times sequentially on the same `Runtime`. /// Can be called multiple times sequentially on the same `Runtime`.
///
/// A panic in the initial actor propagates out of `run` (re-raised
/// after teardown, so the `Runtime` stays reusable even under a
/// caller's `catch_unwind`). Panics in other actors surface through
/// joins, links, monitors, and supervisors as usual.
pub fn run(&self, f: impl FnOnce() + Send + 'static) { pub fn run(&self, f: impl FnOnce() + Send + 'static) {
// Install smarm's panic hook on first call. The default Rust hook is // Install smarm's panic hook on first call. The default Rust hook is
// not reentrant — concurrent actor panics can trigger a double-panic // not reentrant — concurrent actor panics can trigger a double-panic
@@ -941,7 +1111,16 @@ impl Runtime {
self.inner.live_actors.load(Ordering::Acquire), 0, self.inner.live_actors.load(Ordering::Acquire), 0,
"run() called while previous run still active" "run() called while previous run still active"
); );
let io_thread = match IoThread::start() { // RFC 018: the IO producers reach the runtime (slot table + unpark)
// through a Weak, so no RuntimeInner → IoThread → RuntimeInner cycle
// forms. Reset the in-flight counters BEFORE the threads can touch
// them (a prior run left them at 0 on a clean exit; the asserts pin
// that).
debug_assert_eq!(self.inner.io_outstanding.load(Ordering::Acquire), 0);
debug_assert_eq!(self.inner.io_fd_waiters.load(Ordering::Acquire), 0);
self.inner.io_outstanding.store(0, Ordering::Release);
self.inner.io_fd_waiters.store(0, Ordering::Release);
let io_thread = match IoThread::start(Arc::downgrade(&self.inner)) {
Ok(io) => io, Ok(io) => io,
Err(e) => panic!("failed to start IO thread: {e}"), Err(e) => panic!("failed to start IO thread: {e}"),
}; };
@@ -967,16 +1146,27 @@ impl Runtime {
self.inner.root_swept.store(false, Ordering::Relaxed); self.inner.root_swept.store(false, Ordering::Relaxed);
self.inner.set_root(initial_handle.pid()); self.inner.set_root(initial_handle.pid());
// Launch N-1 extra scheduler threads. The calling thread is thread 0. // Launch N-1 extra scheduler threads, named `smarm-sched-{slot}` so
// they are identifiable in `/proc/<pid>/task/*/comm`, stack dumps and
// debuggers. The calling thread is thread 0 and keeps its caller-given
// name (an embedder typically names it when spawning `run`).
let mut os_threads = Vec::new(); let mut os_threads = Vec::new();
for slot in 1..self.thread_count { for slot in 1..self.thread_count {
let inner = self.inner.clone(); let inner = self.inner.clone();
let t = thread::spawn(move || { let t = thread::Builder::new()
.name(format!("smarm-sched-{slot}"))
.spawn(move || {
RUNTIME.with(|r| *r.borrow_mut() = Some(inner.clone())); RUNTIME.with(|r| *r.borrow_mut() = Some(inner.clone()));
SCHED_SLOT.with(|s| s.set(slot)); SCHED_SLOT.with(|s| s.set(slot));
schedule_loop(&inner, slot); schedule_loop(&inner, slot);
RUNTIME.with(|r| *r.borrow_mut() = None); RUNTIME.with(|r| *r.borrow_mut() = None);
}); });
// `thread::spawn` (the previous form) also panics when the OS
// refuses a thread, so this keeps the failure semantics.
let t = match t {
Ok(t) => t,
Err(e) => panic!("failed to spawn smarm scheduler thread: {e}"),
};
os_threads.push(t); os_threads.push(t);
} }
@@ -989,6 +1179,23 @@ impl Runtime {
let _ = t.join(); let _ = t.join();
} }
// The root's outcome, read before its handle drops (the outstanding
// handle pins the slot: no reclaim, no generation change under us).
// A root panic must escape `run()` — propagated at the tail below,
// after teardown. Dropping it unread here made every assert inside
// `run` silently vacuous (found live: a failing-first test passed).
let root_outcome = {
let slot = match self.inner.slot_at(initial_handle.pid()) {
Some(slot) => slot,
None => panic!("run(): root pid out of range"),
};
debug_assert!(
slot.status_for(initial_handle.pid()) == Status::Done,
"root not Done at run() teardown"
);
slot.cold.lock().outcome.take()
};
// Drop initial handle (decrements outstanding_handles count). // Drop initial handle (decrements outstanding_handles count).
drop(initial_handle); drop(initial_handle);
@@ -1016,12 +1223,34 @@ impl Runtime {
} }
self.inner.io_parked.store(0, Ordering::Relaxed); self.inner.io_parked.store(0, Ordering::Relaxed);
self.inner.sleeping.store(0, Ordering::Relaxed); self.inner.sleeping.store(0, Ordering::Relaxed);
self.inner.io_outstanding.store(0, Ordering::Relaxed);
self.inner.io_fd_waiters.store(0, Ordering::Relaxed);
RUNTIME.with(|r| *r.borrow_mut() = None); RUNTIME.with(|r| *r.borrow_mut() = None);
// Flush trace to disk (no-op without smarm-trace). // Flush trace to disk (no-op without smarm-trace).
#[cfg(feature = "smarm-trace")] #[cfg(feature = "smarm-trace")]
crate::trace::flush(); crate::trace::flush();
// Propagate a root panic — last, after the full teardown above, so
// a `catch_unwind` around `run()` leaves the `Runtime` reusable.
// `Exit` and `Stopped` return normally: a cooperatively stopped
// root is not an error. `resume_unwind` re-raises the original
// payload; the throw-site hook output was suppressed in-actor, so
// the caller sees the payload message without the origin file:line.
if let Some(Outcome::Panic(payload)) = root_outcome {
// Surface the message: the throw-site hook output was suppressed
// in-actor, so without this a harness shows a bare FAILED.
let msg: Option<&str> = payload
.downcast_ref::<&'static str>()
.copied()
.or_else(|| payload.downcast_ref::<String>().map(String::as_str));
eprintln!(
"smarm: root actor panicked: {}",
msg.unwrap_or("<non-string panic payload>")
);
std::panic::resume_unwind(payload);
}
} }
/// Snapshot of runtime statistics for introspection / tests. /// Snapshot of runtime statistics for introspection / tests.
@@ -1311,40 +1540,30 @@ fn stop_live_actors(inner: &Arc<RuntimeInner>) {
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// schedule_loop — runs on each scheduler OS thread // Timer firing — shared by the busy-path due-check and the timekeeper
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) { /// Pop and dispatch every due timer. `pop_due` re-anchors the
crate::preempt::configure_preempt(inner.alloc_interval, inner.timeslice_cycles); /// earliest-deadline snapshot under the timers mutex before returning, so
let stats = &inner.stats[slot_idx]; /// a caller that raced a concurrent insert simply comes back on the next
/// due-check. Dispatch runs with the timers lock released.
loop { fn fire_due_timers(inner: &Arc<RuntimeInner>, try_only: bool) {
// ---------------------------------------------------------------- let due = if try_only {
// 1. Try to win the drain lock (timers + IO). One winner per round; // Busy path: if another scheduler is already in the timers mutex
// losers skip immediately and proceed to step 2. // (firing, inserting, or peeking) skip — the snapshot stays due
// ---------------------------------------------------------------- // until someone actually pops, so the check re-fires next loop.
if let Ok(_drain_guard) = inner.drain_lock.try_lock() { match inner.timers.try_lock() {
// Timers and IO live behind their own mutexes (phase 1), so the Ok(mut t) => t.pop_due(std::time::Instant::now()),
// pure-yield / pure-compute hot path never contends a global lock Err(std::sync::TryLockError::WouldBlock) => return,
// just to discover there is nothing to drain. The clock is read Err(std::sync::TryLockError::Poisoned(e)) => {
// only when the timer heap is non-empty. panic!("smarm: timers lock poisoned (core corrupt): {e}")
let due = { }
let mut t = match inner.timers.lock() { }
Ok(t) => t, } else {
match inner.timers.lock() {
Ok(mut t) => t.pop_due(std::time::Instant::now()),
Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"), Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"),
};
if t.is_empty() {
Vec::new()
} else {
t.pop_due(std::time::Instant::now())
} }
};
let completions = match inner.io.lock() {
Ok(mut io) => io
.as_mut()
.map(|io| io.drain_completions())
.unwrap_or_default(),
Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"),
}; };
for entry in due { for entry in due {
match entry.reason { match entry.reason {
@@ -1353,9 +1572,7 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
// actor is between `timers.insert_sleep` and // actor is between `timers.insert_sleep` and
// `park_current`; RunningNotified makes the upcoming park // `park_current`; RunningNotified makes the upcoming park
// re-queue), or gone (no-op). // re-queue), or gone (no-op).
crate::timer::Reason::Sleep { epoch } => { crate::timer::Reason::Sleep { epoch } => inner.unpark_at(entry.pid, epoch),
inner.unpark_at(entry.pid, epoch)
}
crate::timer::Reason::WaitTimeout { target, epoch } => { crate::timer::Reason::WaitTimeout { target, epoch } => {
// The callback may call unpark_at itself. // The callback may call unpark_at itself.
target.on_timeout(entry.pid, epoch); target.on_timeout(entry.pid, epoch);
@@ -1370,57 +1587,26 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
crate::timer::Reason::Send { fire } => fire(), crate::timer::Reason::Send { fire } => fire(),
} }
} }
}
for completion in completions { // ---------------------------------------------------------------------------
match completion { // schedule_loop — runs on each scheduler OS thread
crate::io::Completion::Blocking { pid, epoch, result } => { // ---------------------------------------------------------------------------
match inner.io.lock() {
Ok(mut io) => { fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
if let Some(io) = io.as_mut() { crate::preempt::configure_preempt(inner.alloc_interval, inner.timeslice_cycles);
io.outstanding = io.outstanding.saturating_sub(1); let stats = &inner.stats[slot_idx];
loop {
// ----------------------------------------------------------------
// 1. Busy-path timer due-check (RFC 018 design point (a)): under
// saturation nobody parks, so no timekeeper exists — due timers
// must still fire. One Relaxed load + branch when no timer is
// armed; the clock is read only when one is.
// ----------------------------------------------------------------
if inner.coord.deadline_due() {
fire_due_timers(inner, true);
} }
}
Err(e) => {
panic!("smarm: io lock poisoned (core corrupt): {e}")
}
}
// Stash the result under the cold lock, then unpark.
// The protocol also covers the submit→park window
// (RunningNotified), which the old code missed for
// Blocking completions — a latent lost wakeup.
if let Some(slot) = inner.slot_at(pid) {
{
let mut cold = slot.cold.lock();
if slot.generation() == pid.generation() {
cold.pending_io_result = Some(result);
} else {
// Actor died (stopped) with the op in
// flight; discard the result.
}
}
inner.unpark_at(pid, epoch);
}
}
crate::io::Completion::FdReady { fd, events: _ } => {
// Resolve the parked pid under the io lock, then wake
// through the protocol. Lock order: io before all.
let parked = match inner.io.lock() {
Ok(mut io) => io.as_mut().and_then(|io| {
let entry = io.waiters.remove(&fd);
io.epoll_deregister(fd);
entry
}),
Err(e) => {
panic!("smarm: io lock poisoned (core corrupt): {e}")
}
};
if let Some((pid, epoch)) = parked {
inner.unpark_at(pid, epoch);
}
}
}
}
} // drain_guard drops here
// ---------------------------------------------------------------- // ----------------------------------------------------------------
// 2. Pop a runnable pid. Pop order (RFC 005): wake slot first, then // 2. Pop a runnable pid. Pop order (RFC 005): wake slot first, then
@@ -1429,7 +1615,7 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
// ---------------------------------------------------------------- // ----------------------------------------------------------------
enum Pop { enum Pop {
Got(Pid), Got(Pid),
Idle { io_outstanding: u32, wake_fd: Option<std::os::fd::RawFd> }, Idle,
AllDone, AllDone,
/// Root has exited and nothing is runnable: stop the parked-forever /// Root has exited and nothing is runnable: stop the parked-forever
/// remainder, then re-pop. Fires at most once per run. /// remainder, then re-pop. Fires at most once per run.
@@ -1454,19 +1640,12 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
crate::te!(crate::trace::Event::SlotPop(pid)); crate::te!(crate::trace::Event::SlotPop(pid));
pid pid
} else { } else {
// Read IO liveness BEFORE the queue lock (phase-1 ordering: a // Read IO liveness BEFORE the queue pop — two atomic loads now
// completion resurrects an actor only via the drain path, whose // (RFC 018), not a per-pop `io.lock`: a completion resurrects
// enqueue would be visible under the queue lock we take next). // an actor via the producer's own unpark→enqueue, whose entry
let (io_out, io_fd) = { // would be visible to the pop below.
let io = match inner.io.lock() { let io_out = inner.io_outstanding.load(Ordering::Acquire)
Ok(io) => io, + inner.io_fd_waiters.load(Ordering::Acquire);
Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"),
};
match io.as_ref() {
Some(io) => (io.outstanding + io.waiters.len() as u32, Some(io.wake_fd())),
None => (0, None),
}
};
stats.run_queue_len.store(inner.run_queue.len(), Ordering::Relaxed); stats.run_queue_len.store(inner.run_queue.len(), Ordering::Relaxed);
let pop = match inner.run_queue.pop() { let pop = match inner.run_queue.pop() {
@@ -1498,7 +1677,7 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
// the idle wait below on the next pass. // the idle wait below on the next pass.
Pop::RootDrain Pop::RootDrain
} else { } else {
Pop::Idle { io_outstanding: io_out, wake_fd: io_fd } Pop::Idle
} }
} }
}; };
@@ -1513,22 +1692,15 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
Ok(mut timers) => timers.clear(), Ok(mut timers) => timers.clear(),
Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"), Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"),
} }
// Terminal wake: a sibling scheduler may be blocked in its // Terminal wake (replaces the wake-pipe byte): a sibling
// idle wait on a snapshot that is now terminally stale — an // may be parked on a snapshot that is now terminally
// orphaned long deadline (it would sleep it out in full) or // stale — an orphaned long deadline, or a stale
// a stale `io_outstanding > 0` from a stop-cancelled waiter // `io_fd_waiters > 0` from a stop-cancelled waiter
// (it would block in poll(-1) forever; cancellation produces // (cancellation produces no completion, so nothing else
// no completion, so nothing else writes the wake pipe). // will ever wake it). `wake_all` permits every parker;
// One byte wakes every poller; each re-runs the verdict, // each sibling re-runs the verdict, reaches AllDone
// reaches AllDone itself, and re-wakes — idempotent. // itself, and re-wakes — idempotent.
match inner.io.lock() { inner.coord.wake_all();
Ok(io) => {
if let Some(io) = io.as_ref() {
io.wake();
}
}
Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"),
}
return; return;
} }
Pop::RootDrain => { Pop::RootDrain => {
@@ -1538,34 +1710,51 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
stop_live_actors(inner); stop_live_actors(inner);
continue; continue;
} }
Pop::Idle { io_outstanding, wake_fd } => { Pop::Idle => {
// Something is still in flight. Sleep on the appropriate // Something is still in flight. Park on our own futex
// source to avoid hammering the queue mutex; retry on wake. // until a producer wakes us (enqueue tail), a deadline
let next_deadline = match inner.timers.lock() { // passes, or the re-check finds the world changed.
Ok(timers) => timers.peek_deadline(), //
Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"), // Timekeeper (RFC 018): at most one parked scheduler
// holds the timer deadline — the first idler to arm it
// parks with a timeout, the rest park indefinitely, so a
// timer expiry wakes one scheduler, not a herd. Peek and
// arm under the timers mutex (the serialization that
// makes the insert-side re-arm race-free).
let tk_deadline = {
let timers = match inner.timers.lock() {
Ok(t) => t,
Err(e) => {
panic!("smarm: timers lock poisoned (core corrupt): {e}")
}
}; };
match (next_deadline, wake_fd) { timers
(Some(deadline), fd_opt) => { .peek_deadline()
let now = std::time::Instant::now(); .filter(|d| inner.coord.try_arm_timer(slot_idx, *d))
if deadline > now { };
let timeout = deadline - now; // The mandatory post-publish re-check: a producer that
match fd_opt { // enqueued (or a verdict input that flipped) before it
Some(fd) => { // could see our idle bit has left us the evidence.
crate::io::poll_wake(fd, Some(timeout)); let _ = inner.coord.park(slot_idx, tk_deadline, || {
crate::io::drain_wake_pipe(fd); !inner.run_queue.is_empty()
} || (inner.live_actors.load(Ordering::Acquire) == 0
None => thread::sleep(timeout), && inner.io_outstanding.load(Ordering::Acquire) == 0
} && inner.io_fd_waiters.load(Ordering::Acquire) == 0)
} || (inner.root_exited.load(Ordering::Acquire)
} && !inner.root_swept.load(Ordering::Acquire))
(None, Some(fd)) if io_outstanding > 0 => { || inner.coord.deadline_due()
crate::io::poll_wake(fd, None); });
crate::io::drain_wake_pipe(fd); if tk_deadline.is_some() {
} // Hand the role back BEFORE firing: pop_due can run
_ => { // `Send` thunks that insert new timers, and the
thread::sleep(std::time::Duration::from_micros(100)); // insert-side re-arm check must see either no
} // timekeeper (skip) or a real parked one — never us,
// awake and about to re-peek anyway.
inner.coord.disarm_timer(slot_idx);
// Woken for the deadline, for work, or to re-peek
// after an earlier insert — fire whatever is due;
// the next idle pass re-arms with the new minimum.
fire_due_timers(inner, false);
} }
continue; continue;
} }
@@ -1578,6 +1767,21 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
// by the at-most-once-enqueued invariant nothing else can have // by the at-most-once-enqueued invariant nothing else can have
// changed the state of a queued actor. // changed the state of a queued actor.
// ---------------------------------------------------------------- // ----------------------------------------------------------------
// RFC 018 chain rule: we just took one runnable; if more remain and
// a sibling is parked, wake exactly one so the surplus runs in
// PARALLEL rather than serially behind us (without this the surplus
// is not stranded — we re-pop it after resuming — but it waits out
// our whole timeslice while an idle core sits available). Cheap: the
// queue-length check is queue-local, and `wake_one_if_idle` is a
// fence + one Relaxed mask load when nobody is parked. A Relaxed
// miss here is safe — the enqueue that created the surplus already
// issued its own wake (RFC 018 no-lost-wake); this only sharpens
// parallelism latency.
if !inner.run_queue.is_empty() {
inner.coord.wake_one_if_idle();
}
let slot = match inner.slot_at(pid) { let slot = match inner.slot_at(pid) {
Some(s) => s, Some(s) => s,
None => continue, // can't happen for real pids; defensive None => continue, // can't happen for real pids; defensive
@@ -1616,6 +1820,11 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
crate::preempt::reset_timeslice(); crate::preempt::reset_timeslice();
} }
PREEMPTION_ENABLED.with(|c| c.set(true)); PREEMPTION_ENABLED.with(|c| c.set(true));
// RFC 007: delay accrued while this actor was off-CPU is absorbed for
// free (Coz's blocked-thread rule) — fast-forward its ledger and arm
// the per-thread sample clock before it runs.
#[cfg(feature = "smarm-causal")]
crate::causal::on_resume(slot);
crate::te!(crate::trace::Event::Resume(pid)); crate::te!(crate::trace::Event::Resume(pid));
unsafe { switch_to_actor() }; unsafe { switch_to_actor() };
@@ -1642,6 +1851,11 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
let gen = pid.generation(); let gen = pid.generation();
match intent { match intent {
YieldIntent::Yield => { YieldIntent::Yield => {
// RFC 007 audit: measure the sample tail this yield drops
// (near-zero for slice-expiry yields, which sample at the
// same checkpoint; fat for explicit yield_now in-site).
#[cfg(feature = "smarm-causal")]
crate::causal::on_deschedule(slot, false);
// Running OR RunningNotified → Queued; a notification // Running OR RunningNotified → Queued; a notification
// arriving mid-run coalesces into the re-queue. // arriving mid-run coalesces into the re-queue.
crate::te!(crate::trace::Event::Yield(pid)); crate::te!(crate::trace::Event::Yield(pid));
@@ -1650,11 +1864,24 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
} }
YieldIntent::Park => { YieldIntent::Park => {
if slot.word.park_return(gen) { if slot.word.park_return(gen) {
// RFC 007 audit: an in-site park drops its sample
// tail (nothing flushes it; on_resume re-arms).
#[cfg(feature = "smarm-causal")]
crate::causal::on_deschedule(slot, true);
// RFC 007: a real park — the eventual wake forgives
// delay accrued while blocked (the instant-wake
// re-queue below does NOT: that actor never blocked).
#[cfg(feature = "smarm-causal")]
slot.set_causal_parked();
crate::te!(crate::trace::Event::Park(pid)); crate::te!(crate::trace::Event::Park(pid));
} else { } else {
// An unpark landed in the prep-to-park window; the // An unpark landed in the prep-to-park window; the
// word is back to Queued — re-queue instead of // word is back to Queued — re-queue instead of
// parking. The lost-wakeup window, closed. // parking. The lost-wakeup window, closed: this actor
// never blocked, so its dropped tail counts as a
// yield in the RFC 007 audit.
#[cfg(feature = "smarm-causal")]
crate::causal::on_deschedule(slot, false);
crate::te!(crate::trace::Event::UnparkFlagConsumed(pid)); crate::te!(crate::trace::Event::UnparkFlagConsumed(pid));
inner.enqueue(pid); inner.enqueue(pid);
} }
+441 -173
View File
@@ -1,11 +1,69 @@
//! Scheduler public API — thin façade over the multi-scheduler runtime. //! Start actors and control them: `run`, `spawn`, `sleep`, timers, and IO waits.
//! //!
//! All heavy lifting lives in `runtime.rs`. This module exposes the same //! ## What is an actor?
//! surface that the rest of the codebase (channel, mutex, io, timer, actor)
//! calls into, plus the public API re-exported from `lib.rs`.
//! //!
//! The single-threaded `run()` entry point is kept as a convenience wrapper //! An actor in smarm is a *green thread*: a lightweight, cooperatively
//! around `runtime::init(Config::exact(1)).run(f)`. //! scheduled unit of execution with its own stack, running alongside
//! thousands of others on a handful of OS threads. You give it a closure;
//! smarm gives it a [`Pid`] and a [`JoinHandle`]. Actors talk to each other by
//! sending messages over [`channel`](mod@crate::channel)s, not by sharing memory,
//! so most of the concurrency bugs that come from shared mutable state simply
//! don't arise.
//!
//! Everything in smarm (channels, [`Mutex`](crate::Mutex), timers, IO waits,
//! `gen_server`) needs to run on top of a scheduler. [`run`] starts one and
//! blocks the calling OS thread until the actor you give it finishes; from
//! inside that actor (or any actor it spawns), call [`spawn`] to start more.
//!
//! ```
//! use smarm::{run, spawn};
//!
//! run(|| {
//! let handle = spawn(|| {
//! println!("hello from another actor");
//! });
//! handle.join().unwrap();
//! });
//! ```
//!
//! ## Waiting for an actor: `JoinHandle`
//!
//! [`spawn`] returns a [`JoinHandle`], your only handle on the actor's
//! outcome. Call [`JoinHandle::join`] to block the calling actor until the
//! spawned one finishes:
//!
//! - if it returned normally, `join` returns `Ok(())`;
//! - if it panicked, `join` returns `Err(`[`JoinError`]`)` carrying the panic
//! payload, so a crash in one actor never silently vanishes and never
//! crashes the process; the caller decides what to do with it (log it,
//! propagate it, ignore it).
//!
//! If you don't need the result, drop the `JoinHandle` (or never bind it):
//! the actor keeps running independently. To watch an actor without
//! blocking, or to react to failures across a whole tree of actors, see
//! [`monitor`](mod@crate::monitor), [`link`](mod@crate::link), and
//! [`supervisor`](crate::supervisor) instead.
//!
//! ## Sleeping, timeouts, and IO
//!
//! [`sleep`] parks only the calling actor, not the OS thread underneath it,
//! so thousands of sleeping actors cost nothing but the timer entry.
//! [`send_after`] and [`send_after_named`] schedule a message to be delivered
//! later (Erlang-style `send_after`), and [`cancel_timer`] can call one off
//! before it fires.
//!
//! For blocking or file-descriptor-based IO, see [`block_on_io`],
//! [`wait_readable`], and [`wait_writable`]: they park the calling actor and
//! resume it when the work completes or the fd is ready, again without
//! blocking an OS thread.
//!
//! ## Stopping an actor from the outside
//!
//! [`request_stop`] asks an actor to cooperatively unwind: it's the
//! mechanism `gen_server` shutdown, timeouts, and supervisor restarts are
//! built on. It's best-effort: an actor that never yields, allocates, or
//! blocks (a tight loop with nothing else in it) has no opportunity to
//! notice the request.
use crate::actor::current_pid; use crate::actor::current_pid;
use crate::channel::Sender; use crate::channel::Sender;
@@ -14,27 +72,21 @@ use crate::runtime::{
self, RuntimeInner, YieldIntent, RUNTIME, self, RuntimeInner, YieldIntent, RUNTIME,
}; };
use crate::supervisor::Signal; use crate::supervisor::Signal;
use std::sync::atomic::Ordering;
use std::sync::Arc; use std::sync::Arc;
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// with_runtime / try_with_runtime // with_runtime / try_with_runtime
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Borrow the current runtime. Panics if called outside `Runtime::run()`. // Borrow the current runtime. Panics if called outside `Runtime::run()`.
/// //
/// The whole span runs with preemption disabled. Two reasons, both load- // Preemption is disabled for the whole span. `f` holds a thread-local borrow
/// bearing: // of `RUNTIME`; if a preemption-driven context switch moved the actor to a
/// // different OS thread in the middle of `f`, the borrow guard would be
/// - The `RUNTIME` thread-local borrow is live across `f`. A preemption- // released on the wrong thread's copy of the thread-local, corrupting its
/// driven context switch inside `f` can resume the actor on a DIFFERENT OS // borrow count. `f` is also always runtime bookkeeping that should run to
/// thread; the borrow guard would then increment this thread's RefCell but // completion without the actor being suspended or unwound partway through.
/// decrement the other's — a count underflow that leaves that thread's
/// RefCell permanently "mutably borrowed". Holding a thread-local guard
/// across a potential switch point is the one unforgivable sin of green
/// threads; disabling preemption makes "no switch inside `f`" structural
/// instead of an accident of which bodies happen to allocate.
/// - `f` is runtime bookkeeping. Suspending an actor halfway through it (or
/// unwinding via the stop sentinel, which shares the gate) is never wanted.
pub(crate) fn with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> R { pub(crate) fn with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> R {
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false)); let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| { let result = RUNTIME.with(|r| {
@@ -49,9 +101,9 @@ pub(crate) fn with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> R {
result result
} }
/// Borrow the runtime if present; returns `None` otherwise. // Borrow the runtime if present, otherwise `None`. Used on cleanup paths
/// Used on cleanup paths (channel Drop during teardown). // (e.g. a channel's Drop impl during teardown) that may run after the
/// Same preemption gate as [`with_runtime`]; same reasons. // runtime has already gone away. Same preemption gate as `with_runtime`.
pub(crate) fn try_with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> Option<R> { pub(crate) fn try_with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> Option<R> {
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false)); let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| r.borrow().as_ref().map(f)); let result = RUNTIME.with(|r| r.borrow().as_ref().map(f));
@@ -63,19 +115,49 @@ pub(crate) fn try_with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> Op
// JoinHandle / JoinError // JoinHandle / JoinError
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// The spawned actor panicked. Returned by [`JoinHandle::join`]; `payload`
/// is exactly what the panic carried (the value passed to `panic!`, or
/// whatever a library panicked with), the same payload you'd get from
/// [`std::thread::JoinHandle::join`]. Downcast it if you need to inspect it:
///
/// ```
/// use smarm::{run, spawn};
///
/// run(|| {
/// let h = spawn(|| panic!("boom"));
/// let err = h.join().unwrap_err();
/// let msg = err.payload.downcast_ref::<&str>().copied().unwrap_or("?");
/// assert_eq!(msg, "boom");
/// });
/// ```
#[derive(Debug)] #[derive(Debug)]
pub struct JoinError { pub struct JoinError {
pub payload: Box<dyn std::any::Any + Send>, pub payload: Box<dyn std::any::Any + Send>,
} }
/// A handle to a spawned actor, returned by [`spawn`], [`spawn_under`], and
/// friends. Use [`join`](Self::join) to wait for the actor to finish and
/// collect its outcome, or [`pid`](Self::pid) to get its identity for use
/// with [`request_stop`], [`monitor`](crate::monitor::monitor), or
/// [`link`](crate::link::link).
///
/// If you never call `join` (or drop the handle instead), the actor is not
/// affected: it keeps running and its resources are still reclaimed when it
/// finishes. `join` is how you find out *what happened*, not a requirement
/// for the actor to make progress or clean up.
pub struct JoinHandle { pub struct JoinHandle {
pid: Pid, pid: Pid,
consumed: bool, consumed: bool,
} }
impl JoinHandle { impl JoinHandle {
/// The identity of the actor this handle refers to.
pub fn pid(&self) -> Pid { self.pid } pub fn pid(&self) -> Pid { self.pid }
/// Block the calling actor until the spawned actor finishes, then
/// report how it finished: `Ok(())` if it returned normally or stopped
/// cooperatively via [`request_stop`], `Err(`[`JoinError`]`)` if it
/// panicked.
pub fn join(mut self) -> Result<(), JoinError> { pub fn join(mut self) -> Result<(), JoinError> {
use crate::actor::Outcome; use crate::actor::Outcome;
@@ -177,6 +259,19 @@ impl Drop for JoinHandle {
// spawn / spawn_under / self_pid // spawn / spawn_under / self_pid
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Start a new actor running `f`, and return a [`JoinHandle`] for it.
///
/// The new actor runs concurrently with its caller and with every other
/// actor in the runtime; smarm schedules it cooperatively across the
/// available OS threads. `spawn` can be called from inside `run`'s closure,
/// or from inside any actor (spawning a child from a child works the same
/// way); it cannot be called before `run` has started or after it returns.
///
/// The returned [`JoinHandle`] is how you learn how the actor finished. If
/// you don't need that, it's fine to drop it: the actor still runs to
/// completion either way.
///
/// Panics if called outside `Runtime::run()`.
pub fn spawn(f: impl FnOnce() + Send + 'static) -> JoinHandle { pub fn spawn(f: impl FnOnce() + Send + 'static) -> JoinHandle {
let parent = current_pid().unwrap_or_else(|| { let parent = current_pid().unwrap_or_else(|| {
// Outside an actor but inside run(): the initial spawn. with_runtime // Outside an actor but inside run(): the initial spawn. with_runtime
@@ -186,6 +281,11 @@ pub fn spawn(f: impl FnOnce() + Send + 'static) -> JoinHandle {
spawn_under(parent, f) spawn_under(parent, f)
} }
/// Like [`spawn`], but explicitly attaches the new actor to `supervisor`
/// instead of the calling actor. Ordinary code should reach for [`spawn`];
/// this exists for supervision trees (see [`supervisor`](crate::supervisor))
/// and other cases that need to place a child under a specific ancestor
/// rather than its true caller.
pub fn spawn_under<A>(supervisor: Pid<A>, f: impl FnOnce() + Send + 'static) -> JoinHandle { pub fn spawn_under<A>(supervisor: Pid<A>, f: impl FnOnce() + Send + 'static) -> JoinHandle {
let supervisor = supervisor.erase(); let supervisor = supervisor.erase();
// Stack + closure boxing happen before ANY runtime lock is taken: no // Stack + closure boxing happen before ANY runtime lock is taken: no
@@ -208,19 +308,20 @@ pub fn spawn_under<A>(supervisor: Pid<A>, f: impl FnOnce() + Send + 'static) ->
JoinHandle { pid, consumed: false } JoinHandle { pid, consumed: false }
} }
/// Spawn a typed, single-message actor and hand back its identity-bound /// Spawn an actor that other actors can message directly by its [`Pid<A>`],
/// [`Pid<A>`] (RFC 014's typed-path producer). The runtime makes the actor's /// rather than only by holding on to a channel `Sender` you passed it
/// inbox, hands the body its [`Receiver<A::Msg>`], installs the sender, and /// yourself.
/// returns the parent a `Pid<A>`.
/// ///
/// The inbox is published from the parent side **before** the pid is returned /// `body` receives the [`Receiver<A::Msg>`](crate::channel::Receiver) smarm
/// (see [`registry::install_for`](crate::registry::install_for)), so the address /// creates for it; `spawn_addr` publishes the matching `Sender` and hands
/// is live the instant the caller holds it: an immediate /// back the actor's typed address. That address is usable the instant you
/// [`send_to`](crate::send_to) always resolves, never racing the body's first /// hold it: an immediate [`send_to`](crate::send_to) on the returned pid
/// instruction. The actor is detached — its lifetime is governed by its own /// always finds the inbox, even if `body` hasn't started running yet.
/// logic (an explicit stop message, or returning), like ///
/// [`GenServerBuilder::start`](crate::GenServerBuilder::start) — so the backing join /// The spawned actor is detached (there is no [`JoinHandle`] to join): its
/// handle is dropped. Spawns under the current actor (via [`spawn`]). /// lifetime is up to its own logic, for example running until it receives a
/// stop message, or until the actor decides to return. This mirrors how
/// [`GenServerBuilder::start`](crate::GenServerBuilder::start) works.
/// ///
/// Panics if called outside `Runtime::run()`. /// Panics if called outside `Runtime::run()`.
pub fn spawn_addr<A: crate::pid::Addressable>( pub fn spawn_addr<A: crate::pid::Addressable>(
@@ -237,6 +338,11 @@ pub fn spawn_addr<A: crate::pid::Addressable>(
use crate::context::init_actor_stack; use crate::context::init_actor_stack;
/// The identity of the actor currently running. Use it to hand your own
/// address to another actor (for a reply, a monitor, or a link).
///
/// Panics if called outside an actor (for example, from the closure passed
/// to [`run`] itself, before any [`spawn`]).
pub fn self_pid() -> Pid { pub fn self_pid() -> Pid {
match current_pid() { match current_pid() {
Some(pid) => pid, Some(pid) => pid,
@@ -248,6 +354,12 @@ pub fn self_pid() -> Pid {
// yield_now / park_current / unpark // yield_now / park_current / unpark
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Voluntarily give up the CPU so another runnable actor gets a turn, then
/// resume as soon as the scheduler gets back around to you. Use this in a
/// long-running, allocation-free loop that you want to stay cooperative with
/// the rest of the runtime (see also the [`check!`](crate::check) macro,
/// which does the same thing conditionally, only when your timeslice has
/// actually run out).
pub fn yield_now() { pub fn yield_now() {
runtime::set_yield_intent(YieldIntent::Yield); runtime::set_yield_intent(YieldIntent::Yield);
unsafe { crate::context::switch_to_scheduler() }; unsafe { crate::context::switch_to_scheduler() };
@@ -255,48 +367,44 @@ pub fn yield_now() {
crate::preempt::check_cancelled(); crate::preempt::check_cancelled();
} }
// Suspend the current actor until something wakes it (a message arrives, a
// timer fires, a lock is granted, and so on). This is the low-level parking
// primitive that every blocking smarm operation (channel recv, sleep,
// Mutex::lock, IO waits, JoinHandle::join) is built on; application code
// should reach for one of those rather than calling this directly.
//
// Checks for a pending cooperative-stop request both before parking (a stop
// requested while merely queued to run would otherwise have no future wake
// to catch it) and after resuming (so a stop that arrived while parked is
// noticed as soon as we wake, and unwinds from here exactly like any other
// blocking call would).
pub fn park_current() { pub fn park_current() {
// Entry-side observation point: a stop flagged while we were QUEUED is
// otherwise lost — the stop's wildcard unpark no-ops on a Queued actor
// (the pending run "is" the wake), so if our first action on resume is
// this park, no wake is ever coming and the wake-side check below is
// unreachable. Checking here closes that hole; a flag that lands after
// this check is covered by the existing protocol (the stop's unpark
// finds Running / the prep-to-park window, sets Notified, and the
// park-return re-queues us into the wake-side check). Unwinding from
// here is the same unwind path as the wake side: leftover wait
// registrations are stale-epoch / dead-generation and self-clean at
// their wakers' failed CAS.
crate::preempt::check_cancelled(); crate::preempt::check_cancelled();
runtime::set_yield_intent(YieldIntent::Park); runtime::set_yield_intent(YieldIntent::Park);
unsafe { crate::context::switch_to_scheduler() }; unsafe { crate::context::switch_to_scheduler() };
// Observation point on the wakeup side of every blocking primitive
// (recv/sleep/mutex/io/join). Past the prep-to-park window, so this never
// races a wakeup: a stop unparks us, we resume here, and unwind out of
// whatever blocking call parked us — running Drop along the way.
crate::preempt::check_cancelled(); crate::preempt::check_cancelled();
} }
// Wake `pid` unconditionally, regardless of what it's currently waiting for.
// Reserved for terminal wakes (`request_stop`); anything waking an actor from
// a specific registered wait (a channel send, a mutex grant, a timer, an IO
// completion) must use `unpark_at` instead, so a stale wakeup can never be
// mistaken for the one the actor is actually waiting on.
pub fn unpark(pid: Pid) { pub fn unpark(pid: Pid) {
// The whole protocol lives on the slot's packed word (gen + epoch +
// state in one CAS) — see slot_state.rs. No runtime lock unless we
// enqueue. WILDCARD form: reserved for terminal wakes (request_stop);
// every registration-based waker must use `unpark_at`.
let _ = try_with_runtime(|inner| inner.unpark(pid)); let _ = try_with_runtime(|inner| inner.unpark(pid));
} }
/// Epoch-matched unpark: wake `pid` only if its current wait is still the // Wake `pid` only if its current wait is still the one this waker
/// one this waker registered for. The form every registration-based waker // registered for (an "epoch-matched" unpark). Every registration-based
/// (channel senders, mutex grants, wait-timers, io completions, joiner // waker (channel senders, mutex grants, wait-timers, IO completions, joiner
/// wakes) must use — see slot_state.rs for the consuming-wake rules. // wakes) must use this rather than the unconditional `unpark`.
pub(crate) fn unpark_at(pid: Pid, epoch: u32) { pub(crate) fn unpark_at(pid: Pid, epoch: u32) {
let _ = try_with_runtime(|inner| inner.unpark_at(pid, epoch)); let _ = try_with_runtime(|inner| inner.unpark_at(pid, epoch));
} }
/// Open a new wait for the CURRENT actor: bump its park-epoch and return // Open a new wait for the current actor and return its wait identity
/// it. Call once per wait, before registering `(pid, epoch)` with any // ("epoch"). Call once per wait, before registering with any waker. Lock-free,
/// waker. Lock-free (one CAS on the own slot word), so it is legal under // so it's legal to call while already holding another internal lock.
/// any lock, including a Channel-class lock.
pub(crate) fn begin_wait() -> u32 { pub(crate) fn begin_wait() -> u32 {
let me = match current_pid() { let me = match current_pid() {
Some(pid) => pid, Some(pid) => pid,
@@ -305,48 +413,47 @@ pub(crate) fn begin_wait() -> u32 {
with_runtime(|inner| inner.begin_wait(me)) with_runtime(|inner| inner.begin_wait(me))
} }
/// Close the current actor's wait WITHOUT parking on it. For the no-park // Close the current actor's wait without parking on it: the no-park exit
/// exit of multi-registration waits (`select` finding an arm ready at // used when a multi-arm `select` finds an arm already ready at registration
/// registration time): bumps the epoch so in-flight wakes die at their CAS, // time. Leftover registrations from the other arms are left to self-clean
/// eats a notification that already landed, then observes a pending stop — // when their wakers try to use them.
/// in that order. After this, leftover registrations are stale-epoch and
/// self-clean at their wakers' failed CAS; nothing can fault the actor's
/// next one-shot park.
pub(crate) fn retire_wait() { pub(crate) fn retire_wait() {
let me = match current_pid() { let me = match current_pid() {
Some(pid) => pid, Some(pid) => pid,
None => panic!("retire_wait() called outside an actor"), None => panic!("retire_wait() called outside an actor"),
}; };
with_runtime(|inner| inner.retire_wait(me)); with_runtime(|inner| inner.retire_wait(me));
// A request_stop that fired before the clear had its notification
// eaten, but it set the stop flag first — observe it here and unwind.
// One that fires after re-notifies a Running word as usual.
crate::preempt::check_cancelled(); crate::preempt::check_cancelled();
} }
/// Request cooperative cancellation of `pid`. /// Ask `pid` to stop cooperatively.
/// ///
/// Sets the actor's stop flag and wakes it so it observes the flag promptly: /// This sets a flag on the target actor and wakes it so it notices promptly;
/// a parked target is re-queued; a running target is marked notified, so its /// the actor itself decides when it's safe to actually unwind, at its next
/// next park returns immediately to an observation point. The actor realises /// natural checkpoint (a blocking call returning, a `check!()`, or an
/// the stop as a controlled unwind at its next observation point /// allocation). Once it does, it terminates as if it panicked, except that
/// (`check!()`/allocation, or the wakeup side of a blocking park), /// [`JoinHandle::join`] reports it as a normal, non-error exit: cooperative
/// terminating with `Outcome::Stopped`. /// stop is a controlled shutdown, not a failure.
/// ///
/// This is best-effort and cooperative: an actor that never reaches an /// This is exactly the mechanism `gen_server` shutdown, supervisor restarts,
/// observation point — a tight loop with no `check!()`, no allocation, and no /// and structured teardown are built from: reach for [`GenServerRef::shutdown`](crate::GenServerRef::shutdown)
/// blocking op — cannot be stopped, exactly as it cannot be preempted. A no-op /// or a [`supervisor`](crate::supervisor) instead of calling this directly
/// if `pid` is already gone. /// where those apply.
///
/// Because it's cooperative, an actor stuck in a tight loop with no
/// blocking call, no [`check!`](crate::check), and no allocation cannot be
/// stopped, for the same reason it cannot be preempted. Calling this on an
/// actor that has already finished is a harmless no-op.
pub fn request_stop<A>(pid: Pid<A>) { pub fn request_stop<A>(pid: Pid<A>) {
let pid = pid.erase(); let pid = pid.erase();
let _ = try_with_runtime(|inner| request_stop_inner(inner, pid)); let _ = try_with_runtime(|inner| request_stop_inner(inner, pid));
} }
/// The core of [`request_stop`], taking the runtime directly so it can be // The core of `request_stop`, taking the runtime directly so it can also be
/// driven from inside the runtime (e.g. the root-exit sweep in // driven from inside the runtime itself (the root-exit sweep) without
/// `finalize_actor`) without re-borrowing the thread-local. Sets the stop flag // re-borrowing the thread-local. Sets the stop flag under the target's lock
/// under the target's cold lock (generation re-verified there; a mismatch or a // (a generation mismatch, or no live actor there, makes it a no-op) and
/// slot with no live actor is a no-op) and wakes it. // wakes the target.
pub(crate) fn request_stop_inner(inner: &RuntimeInner, pid: Pid) { pub(crate) fn request_stop_inner(inner: &RuntimeInner, pid: Pid) {
if let Some(slot) = inner.slot_at(pid) { if let Some(slot) = inner.slot_at(pid) {
{ {
@@ -367,6 +474,10 @@ pub(crate) fn request_stop_inner(inner: &RuntimeInner, pid: Pid) {
// NoPreempt // NoPreempt
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// A guard that disables preemption for its lifetime, restoring the
/// previous setting on drop. Internal-use: application code has no need to
/// disable preemption directly. See [`check!`](crate::check) for the
/// user-facing side of preemption.
pub struct NoPreempt(bool); pub struct NoPreempt(bool);
impl NoPreempt { impl NoPreempt {
@@ -386,6 +497,21 @@ impl Drop for NoPreempt {
// sleep / insert_wait_timer // sleep / insert_wait_timer
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Suspend the calling actor for `duration`. Unlike
/// [`std::thread::sleep`], this parks only the actor, not the underlying OS
/// thread, so every other actor (including others sharing the same OS
/// thread) keeps running normally while this one waits.
///
/// ```
/// use smarm::{run, sleep};
/// use std::time::Duration;
///
/// run(|| {
/// sleep(Duration::from_millis(1));
/// });
/// ```
///
/// Panics if called outside an actor.
pub fn sleep(duration: std::time::Duration) { pub fn sleep(duration: std::time::Duration) {
let me = match current_pid() { let me = match current_pid() {
Some(pid) => pid, Some(pid) => pid,
@@ -403,6 +529,33 @@ pub fn sleep(duration: std::time::Duration) {
park_current(); park_current();
} }
/// Like [`sleep`], but the deadline is always measured in real (wall-clock)
/// time. Ordinary code should use [`sleep`]; this variant exists for smarm's
/// own profiling and measurement tooling, which can otherwise stretch or
/// compress simulated time. Without that tooling active, `sleep_wall` and
/// `sleep` behave identically.
pub fn sleep_wall(duration: std::time::Duration) {
let me = match current_pid() {
Some(pid) => pid,
None => panic!("sleep_wall() called outside an actor"),
};
let _np = NoPreempt::enter();
let epoch = begin_wait();
let deadline = crate::timer::deadline_from_now(duration);
with_runtime(|inner| {
match inner.timers.lock() {
Ok(mut timers) => timers.insert_sleep_wall(deadline, me, epoch),
Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"),
}
});
park_current();
}
// Building block for bounded waits elsewhere in the crate (Mutex::lock_timeout,
// Receiver::recv_timeout, select_timeout): arm a timer that, on expiry, asks
// `target` whether this particular wait is still pending and should be woken
// with a timeout. Not part of the public API; application code wants
// `sleep`, `send_after`, or one of the `*_timeout` methods instead.
pub fn insert_wait_timer( pub fn insert_wait_timer(
deadline: std::time::Instant, deadline: std::time::Instant,
pid: Pid, pid: Pid,
@@ -422,20 +575,22 @@ pub fn insert_wait_timer(
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// send_after / cancel_timer message-delivery timers (Erlang send_after). // send_after / cancel_timer: message-delivery timers, in the same spirit as
// // Erlang's `erlang:send_after/3`. Each schedules `msg` for delivery after a
// Arm a timer that delivers `msg` to an address after `after`, returning a // delay and returns a TimerId; `cancel_timer` can call one off before it
// `TimerId`. The destination is resolved *on fire*, not at arm time: a // fires. The destination is resolved when the timer actually fires, not when
// `Pid<A>` that has since died yields `SendError::Dead`, a `Name<M>` resolves // it's armed, so a `Name`-addressed timer always reaches whoever currently
// to whoever currently holds it (so a restarted server is reached). Either way // holds that name, even if the original holder has since restarted. If the
// a failed resolve / closed inbox is dropped, matching `erlang:send_after`. // destination is gone by fire time, the message is silently dropped, exactly
// `cancel_timer` prevents an as-yet-unfired delivery; it returns whether the // as a live send to a dead address would be.
// timer was still armed.
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Deliver `msg` to the exact actor named by `dest` (identity-bound, no /// Deliver `msg` to the exact actor identified by `dest` after `after` has
/// redirect — see [`send_to`](crate::registry::send_to)) after `after`. /// elapsed. Unlike [`send_after_named`], this targets one specific actor: if
/// Returns a [`TimerId`](crate::timer::TimerId) for [`cancel_timer`]. /// that actor is gone by the time the timer fires, the message is dropped
/// (it is never redirected to a different actor, even one that inherited the
/// same name). Returns a [`TimerId`](crate::timer::TimerId) you can pass to
/// [`cancel_timer`] to call it off early.
pub fn send_after<A: crate::pid::Addressable>( pub fn send_after<A: crate::pid::Addressable>(
after: std::time::Duration, after: std::time::Duration,
dest: Pid<A>, dest: Pid<A>,
@@ -453,9 +608,11 @@ pub fn send_after<A: crate::pid::Addressable>(
}) })
} }
/// Deliver `msg` to whichever actor holds the name `dest` at fire time /// Deliver `msg` to whichever actor holds the name `dest` when the timer
/// (re-resolving [`send`](crate::registry::send) semantics) after `after`. /// fires, not necessarily whoever holds it now: if the named actor restarts
/// Returns a [`TimerId`](crate::timer::TimerId) for [`cancel_timer`]. /// (for example under a supervisor) and re-registers before the deadline,
/// the message reaches the new instance. Returns a
/// [`TimerId`](crate::timer::TimerId) you can pass to [`cancel_timer`].
pub fn send_after_named<M: Send + 'static>( pub fn send_after_named<M: Send + 'static>(
after: std::time::Duration, after: std::time::Duration,
dest: Name<M>, dest: Name<M>,
@@ -475,17 +632,56 @@ pub fn send_after_named<M: Send + 'static>(
}) })
} }
/// Deliver `msg` onto a channel the caller owns after `after`, rather than to a /// Like [`send_after`], but the deadline is always measured in real
/// registry address. The sibling of [`send_after`] used by the gen_server timer /// (wall-clock) time rather than being subject to smarm's own profiling and
/// layer (RFC 015 §5): arming a server timer must land the fire on the loop's /// measurement tooling. Use this for deadlines that need to reflect the
/// own `Sys` channel — giving it the loop's arm position — not in the inbox. /// outside world (a protocol timeout, a wall-clock schedule) rather than
/// /// simulated workload pacing. Without that tooling active, it behaves
/// Same substrate as [`send_after`]: a `Reason::Send` entry, the same `armed` /// identically to [`send_after`].
/// set, the same [`TimerId`](crate::timer::TimerId) for [`cancel_timer`]. Only pub fn send_after_wall<A: crate::pid::Addressable>(
/// the fire thunk differs — `tx.send(msg)` instead of a registry resolve — so a after: std::time::Duration,
/// send onto a channel whose receiver is gone is dropped, exactly as a failed dest: Pid<A>,
/// address resolve is (Erlang `send_after` semantics). `pid` is informational msg: A::Msg,
/// (who armed it); delivery lives entirely in the thunk. ) -> crate::timer::TimerId {
let deadline = crate::timer::deadline_from_now(after);
let fire = Box::new(move || {
let _ = crate::registry::send_to(dest, msg);
});
with_runtime(|inner| {
match inner.timers.lock() {
Ok(mut timers) => timers.insert_send_wall(deadline, dest.erase(), fire),
Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"),
}
})
}
/// Wall-clock-anchored [`send_after_named`]: re-resolving name delivery,
/// with the same real-time deadline guarantee as [`send_after_wall`].
pub fn send_after_named_wall<M: Send + 'static>(
after: std::time::Duration,
dest: Name<M>,
msg: M,
) -> crate::timer::TimerId {
let deadline = crate::timer::deadline_from_now(after);
// Informational only (who armed it); not used for delivery.
let armed_by = current_pid().unwrap_or(Pid::new(0, 0));
let fire = Box::new(move || {
let _ = crate::registry::send(dest, msg);
});
with_runtime(|inner| {
match inner.timers.lock() {
Ok(mut timers) => timers.insert_send_wall(deadline, armed_by, fire),
Err(e) => panic!("smarm: timers lock poisoned (core corrupt): {e}"),
}
})
}
// Deliver `msg` onto a channel the caller already holds, after a delay,
// rather than resolving a registry address at fire time. Used internally by
// gen_server's timer support, which needs the fire to land on the server
// loop's own dedicated channel instead of its public inbox. Otherwise
// identical to `send_after`: same cancellation via `cancel_timer`, and a
// send to a channel whose receiver is gone is silently dropped.
pub(crate) fn send_after_to<T: Send + 'static>( pub(crate) fn send_after_to<T: Send + 'static>(
after: std::time::Duration, after: std::time::Duration,
tx: Sender<T>, tx: Sender<T>,
@@ -504,9 +700,9 @@ pub(crate) fn send_after_to<T: Send + 'static>(
}) })
} }
/// Cancel a timer armed by [`send_after`] / [`send_after_named`]. Returns /// Call off a timer armed by [`send_after`] or [`send_after_named`] before
/// `true` if it was still pending (delivery now prevented), `false` if it had /// it fires. Returns `true` if the timer was still pending and delivery is
/// already fired or been cancelled. /// now prevented, `false` if it had already fired or was already cancelled.
pub fn cancel_timer(id: crate::timer::TimerId) -> bool { pub fn cancel_timer(id: crate::timer::TimerId) -> bool {
with_runtime(|inner| { with_runtime(|inner| {
match inner.timers.lock() { match inner.timers.lock() {
@@ -520,6 +716,22 @@ pub fn cancel_timer(id: crate::timer::TimerId) -> bool {
// block_on_io / wait_readable / wait_writable / read / write // block_on_io / wait_readable / wait_writable / read / write
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Run a blocking closure (blocking file IO, a synchronous library call,
/// anything that isn't itself actor-aware) on a dedicated worker thread,
/// while the calling actor parks and every other actor keeps making
/// progress. When `f` completes, the calling actor resumes with its result.
///
/// Reach for this whenever you need to call into something that would
/// otherwise block the underlying OS thread outright: a blocking C library,
/// a synchronous filesystem call, DNS resolution via the system resolver,
/// and so on. For plain readiness-based network IO on a file descriptor,
/// [`wait_readable`] / [`wait_writable`] are cheaper since they don't need a
/// dedicated thread.
///
/// If `f` panics, that panic is carried over and re-raised in the calling
/// actor, exactly as if the call had been made inline.
///
/// Panics if called outside an actor.
pub fn block_on_io<F, T>(f: F) -> T pub fn block_on_io<F, T>(f: F) -> T
where where
F: FnOnce() -> T + Send + 'static, F: FnOnce() -> T + Send + 'static,
@@ -542,7 +754,14 @@ where
Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"), Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"),
}; };
match io.as_mut() { match io.as_mut() {
Some(io) => io.submit(me, epoch, work), Some(io) => {
// RFC 018: count the op in flight BEFORE submit — the
// pool decrements on completion, and an increment that
// trailed the completion would underflow. Under the io
// lock, so ordered against the same-lock submit.
inner.io_outstanding.fetch_add(1, Ordering::AcqRel);
io.submit(me, epoch, work);
}
None => panic!("io thread not started"), None => panic!("io thread not started"),
} }
}); });
@@ -572,10 +791,19 @@ where
} }
} }
/// Park the calling actor until `fd` becomes readable. Other actors keep
/// running while you wait; when the kernel reports the fd ready, the actor
/// resumes and you perform the actual `read(2)` yourself (see [`read`] for
/// a convenience wrapper that does both steps).
///
/// Only one actor may wait on a given fd for a given direction at a time.
/// Panics if called outside an actor.
pub fn wait_readable(fd: std::os::fd::RawFd) -> std::io::Result<()> { pub fn wait_readable(fd: std::os::fd::RawFd) -> std::io::Result<()> {
wait_fd(fd, true, false) wait_fd(fd, true, false)
} }
/// Park the calling actor until `fd` becomes writable. See [`wait_readable`]
/// for the read-side counterpart; the same notes apply.
pub fn wait_writable(fd: std::os::fd::RawFd) -> std::io::Result<()> { pub fn wait_writable(fd: std::os::fd::RawFd) -> std::io::Result<()> {
wait_fd(fd, false, true) wait_fd(fd, false, true)
} }
@@ -593,18 +821,28 @@ fn wait_fd(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::R
Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"), Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"),
}; };
match io.as_mut() { match io.as_mut() {
Some(io) => io.epoll_register(fd, me, epoch, readable, writable), Some(io) => {
// RFC 018: count the waiter BEFORE the ADD (mirror of
// submit); roll back if the registration fails so a
// rejected wait leaves the verdict counters clean.
inner.io_fd_waiters.fetch_add(1, Ordering::AcqRel);
let r = io.epoll_register(fd, me, epoch, readable, writable);
if r.is_err() {
inner.io_fd_waiters.fetch_sub(1, Ordering::AcqRel);
}
r
}
None => panic!("io thread not started"), None => panic!("io thread not started"),
} }
})?; })?;
// If a terminal stop unwinds us out of the park below, the registration // If a terminal stop unwinds us out of the park below, the registration
// must not outlive us: a stale `waiters` entry fails every future // must not outlive us: a stale registration would fail every future
// `wait_*` on this fd with AlreadyExists, and the kernel-side ADD leaks // `wait_*` on this fd, and the kernel-side registration would leak until
// until the fd happens to be reused. Clean up iff the entry is still // the fd happens to be reused. Clean up only if the entry is still
// ours a `FdReady` racing the stop may have consumed it already (it // ours; a wakeup racing the stop may have already consumed it, possibly
// removes + DELs under the io lock), after which the fd may even carry // leaving a different actor's fresh registration in its place, which
// ANOTHER actor's fresh registration; in that case touch nothing. // must not be disturbed.
struct Dereg { struct Dereg {
fd: std::os::fd::RawFd, fd: std::os::fd::RawFd,
me: Pid, me: Pid,
@@ -618,9 +856,12 @@ fn wait_fd(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::R
Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"), Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"),
}; };
if let Some(io) = io.as_mut() { if let Some(io) = io.as_mut() {
if io.waiters.get(&self.fd) == Some(&(self.me, self.epoch)) { // `cancel_waiter` removes + DELs iff still ours, all
io.waiters.remove(&self.fd); // under the waiters lock (the ADD/DEL serialization);
io.epoll_deregister(self.fd); // decrement only when we actually removed it — a
// FdReady that consumed it already did the decrement.
if io.cancel_waiter(self.fd, self.me, self.epoch) {
inner.io_fd_waiters.fetch_sub(1, Ordering::AcqRel);
} }
} }
}); });
@@ -628,24 +869,28 @@ fn wait_fd(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::R
} }
let guard = Dereg { fd, me, epoch }; let guard = Dereg { fd, me, epoch };
park_current(); park_current();
// Normal wake: the FdReady path removed the entry and DEL'd the fd // Normal wake: the ready-fd path already removed the entry and
// before the unpark, so the guard's check would be a guaranteed no-op — // deregistered from epoll before waking us, so the guard's check on drop
// skip the io lock on the hot path. (Dereg owns nothing; forget leaks // is a guaranteed no-op. Skip it on this hot path (Dereg owns no
// no resource.) // resource itself, so forgetting it leaks nothing).
std::mem::forget(guard); std::mem::forget(guard);
Ok(()) Ok(())
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// FdArm fd readiness as a select arm (RFC 008) // FdArm: fd readiness as a select arm
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// An fd-readiness arm for [`crate::select`] / [`crate::select_timeout`]: /// A file-descriptor readiness condition usable as an arm of
/// ready when the fd is readable (resp. writable), composable with channel /// [`select`](crate::select) / [`select_timeout`](crate::select_timeout),
/// receivers on one wait epoch. Phase-1 rules apply: one waiter per fd at a /// so you can wait on "this fd is readable" alongside ordinary channel
/// time, one direction per arm (duplex on a single fd needs `dup`; epoll /// receivers in the same call. Build one with [`FdArm::readable`] or
/// registrations key on the open file description, so dup'd fds register /// [`FdArm::writable`].
/// independently). ///
/// Only one actor may wait on a given fd for a given direction at a time. A
/// single fd open in both directions needs two `FdArm`s (or `dup` the fd) if
/// you want to wait on both; you cannot wait on read and write readiness
/// with one arm.
pub struct FdArm { pub struct FdArm {
fd: std::os::fd::RawFd, fd: std::os::fd::RawFd,
readable: bool, readable: bool,
@@ -653,10 +898,12 @@ pub struct FdArm {
} }
impl FdArm { impl FdArm {
/// An arm that becomes ready when `fd` is readable.
pub fn readable(fd: std::os::fd::RawFd) -> Self { pub fn readable(fd: std::os::fd::RawFd) -> Self {
FdArm { fd, readable: true, writable: false } FdArm { fd, readable: true, writable: false }
} }
/// An arm that becomes ready when `fd` is writable.
pub fn writable(fd: std::os::fd::RawFd) -> Self { pub fn writable(fd: std::os::fd::RawFd) -> Self {
FdArm { fd, readable: false, writable: true } FdArm { fd, readable: false, writable: true }
} }
@@ -665,14 +912,12 @@ impl FdArm {
impl crate::channel::sealed::Sealed for FdArm {} impl crate::channel::sealed::Sealed for FdArm {}
impl crate::channel::Selectable for FdArm { impl crate::channel::Selectable for FdArm {
/// Ready-now check is a zero-timeout `poll(2)`; if the requested events // Ready-now check is a zero-timeout poll(2); if the requested events are
/// are pending the wait is retired without registering (`Ok(false)`, // already pending, the wait is retired without registering. Otherwise
/// the channel-arm contract). Otherwise register with the io thread — // register with the IO thread. Any registration failure (a closed fd,
/// every failure surfaces as `Err` (EBADF including a closed-fd // too many fds registered, a second waiter already on this fd) is
/// POLLNVAL, EMFILE on the epoll set, AlreadyExists for a second // surfaced as an error rather than silently treated as "always ready",
/// waiter on the fd): the fallible-out, nothing-left-behind rule, in // so a caller never spins on an fd that genuinely can't be registered.
/// deviation from RFC 008's permanently-ready lean, which would spin a
/// consumer whose fd is healthy but unregistrable (EMFILE).
fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool> { fn sel_register(&self, pid: Pid, epoch: u32) -> std::io::Result<bool> {
if poll_events(self.fd, self.readable, self.writable)? { if poll_events(self.fd, self.readable, self.writable)? {
return Ok(false); return Ok(false);
@@ -684,7 +929,14 @@ impl crate::channel::Selectable for FdArm {
}; };
match io.as_mut() { match io.as_mut() {
Some(io) => { Some(io) => {
io.epoll_register(self.fd, pid, epoch, self.readable, self.writable) inner.io_fd_waiters.fetch_add(1, Ordering::AcqRel);
let r = io.epoll_register(
self.fd, pid, epoch, self.readable, self.writable,
);
if r.is_err() {
inner.io_fd_waiters.fetch_sub(1, Ordering::AcqRel);
}
r
} }
None => panic!("io thread not started"), None => panic!("io thread not started"),
} }
@@ -692,20 +944,19 @@ impl crate::channel::Selectable for FdArm {
Ok(true) Ok(true)
} }
/// Classification is the same zero-timeout poll: a pure function of fd // Same zero-timeout poll used for classification: a pure function of
/// state, independent of the registration the cleanup pass removed. An // fd state. An error here (fd closed mid-wait) is reported as ready
/// error here (EBADF: fd closed mid-wait) reports READY — the // rather than pending forever: the caller's own read/write then
/// consumer's read/write surfaces the errno; a dead arm is an event, // surfaces the real error, so a dead fd is an observable event, not a
/// not a hang. // silent hang.
fn sel_ready(&self) -> bool { fn sel_ready(&self) -> bool {
poll_events(self.fd, self.readable, self.writable).unwrap_or(true) poll_events(self.fd, self.readable, self.writable).unwrap_or(true)
} }
/// `wait_fd`'s `Dereg` compare, verbatim: remove the waiters entry and // Mirrors wait_fd's cleanup guard: remove the registration only if it's
/// kernel-side registration iff the entry is still `(pid, epoch)`-ours. // still ours. A wakeup racing this cleanup may have already consumed
/// A `FdReady` racing the wake may have consumed it (it removes + DELs // it, possibly leaving a different actor's fresh registration on the
/// under the io lock), after which the fd may even carry ANOTHER // same fd, which must not be touched.
/// actor's fresh registration; in that case touch nothing.
fn sel_unregister(&self, pid: Pid, epoch: u32) { fn sel_unregister(&self, pid: Pid, epoch: u32) {
with_runtime(|inner| { with_runtime(|inner| {
let mut io = match inner.io.lock() { let mut io = match inner.io.lock() {
@@ -713,9 +964,8 @@ impl crate::channel::Selectable for FdArm {
Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"), Err(e) => panic!("smarm: io lock poisoned (core corrupt): {e}"),
}; };
if let Some(io) = io.as_mut() { if let Some(io) = io.as_mut() {
if io.waiters.get(&self.fd) == Some(&(pid, epoch)) { if io.cancel_waiter(self.fd, pid, epoch) {
io.waiters.remove(&self.fd); inner.io_fd_waiters.fetch_sub(1, Ordering::AcqRel);
io.epoll_deregister(self.fd);
} }
} }
}); });
@@ -726,9 +976,9 @@ impl crate::channel::Selectable for FdArm {
} }
} }
/// Zero-timeout `poll(2)`: are any of the requested events (or ERR/HUP, // Zero-timeout poll(2): are any of the requested events (or an error/hangup
/// which make the consumer's read/write fail loudly rather than park // condition, so the caller's read/write fails loudly instead of parking
/// forever) pending on `fd`? POLLNVAL maps to `Err(EBADF)`. // forever) pending on `fd` right now?
fn poll_events(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::Result<bool> { fn poll_events(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::io::Result<bool> {
let mut events: libc::c_short = 0; let mut events: libc::c_short = 0;
if readable { if readable {
@@ -757,8 +1007,9 @@ fn poll_events(fd: std::os::fd::RawFd, readable: bool, writable: bool) -> std::i
} }
} }
/// Wait until `fd` is readable or `timeout` elapses: `Ok(true)` = ready, /// Like [`wait_readable`], but gives up after `timeout` instead of waiting
/// `Ok(false)` = timed out. A one-arm [`crate::try_select_timeout`]. /// indefinitely: `Ok(true)` means the fd became ready, `Ok(false)` means the
/// timeout elapsed first.
pub fn wait_readable_timeout( pub fn wait_readable_timeout(
fd: std::os::fd::RawFd, fd: std::os::fd::RawFd,
timeout: std::time::Duration, timeout: std::time::Duration,
@@ -767,8 +1018,9 @@ pub fn wait_readable_timeout(
Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some()) Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some())
} }
/// Wait until `fd` is writable or `timeout` elapses: `Ok(true)` = ready, /// Like [`wait_writable`], but gives up after `timeout` instead of waiting
/// `Ok(false)` = timed out. /// indefinitely: `Ok(true)` means the fd became ready, `Ok(false)` means the
/// timeout elapsed first.
pub fn wait_writable_timeout( pub fn wait_writable_timeout(
fd: std::os::fd::RawFd, fd: std::os::fd::RawFd,
timeout: std::time::Duration, timeout: std::time::Duration,
@@ -777,12 +1029,18 @@ pub fn wait_writable_timeout(
Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some()) Ok(crate::channel::try_select_timeout(&[&arm], timeout)?.is_some())
} }
/// Convenience wrapper: park until `fd` is readable, then perform the
/// `read(2)` into `buf`. Equivalent to calling [`wait_readable`] yourself
/// followed by a raw read, provided as a shorthand for the common case.
pub fn read(fd: std::os::fd::RawFd, buf: &mut [u8]) -> std::io::Result<usize> { pub fn read(fd: std::os::fd::RawFd, buf: &mut [u8]) -> std::io::Result<usize> {
wait_readable(fd)?; wait_readable(fd)?;
let n = unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) }; let n = unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) };
if n < 0 { Err(std::io::Error::last_os_error()) } else { Ok(n as usize) } if n < 0 { Err(std::io::Error::last_os_error()) } else { Ok(n as usize) }
} }
/// Convenience wrapper: park until `fd` is writable, then perform the
/// `write(2)` of `buf`. Equivalent to calling [`wait_writable`] yourself
/// followed by a raw write, provided as a shorthand for the common case.
pub fn write(fd: std::os::fd::RawFd, buf: &[u8]) -> std::io::Result<usize> { pub fn write(fd: std::os::fd::RawFd, buf: &[u8]) -> std::io::Result<usize> {
wait_writable(fd)?; wait_writable(fd)?;
let n = unsafe { libc::write(fd, buf.as_ptr() as *const _, buf.len()) }; let n = unsafe { libc::write(fd, buf.as_ptr() as *const _, buf.len()) };
@@ -790,7 +1048,7 @@ pub fn write(fd: std::os::fd::RawFd, buf: &[u8]) -> std::io::Result<usize> {
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// register_supervisor_channel // register_supervisor_channel: internal wiring used by supervisor.rs
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
pub fn register_supervisor_channel(pid: Pid, sender: Sender<Signal>) { pub fn register_supervisor_channel(pid: Pid, sender: Sender<Signal>) {
@@ -807,11 +1065,21 @@ pub fn register_supervisor_channel(pid: Pid, sender: Sender<Signal>) {
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Legacy run() — convenience wrapper // run(): the single-threaded convenience entry point
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Single-threaded runtime entry point (backwards-compatible wrapper). /// Start the smarm runtime on a single OS thread, run `f` as the first
/// Equivalent to `runtime::init(Config::exact(1)).run(f)`. /// (root) actor, and block until every actor it transitively spawned has
/// finished.
///
/// This is the simplest way to get started, and is all you need for most
/// programs: `f` typically calls [`spawn`] to create more actors and waits
/// on their [`JoinHandle`]s. If you want smarm to schedule actors across
/// multiple OS threads instead, use
/// [`crate::runtime::init`] with a
/// [`Config`](crate::runtime::Config) that requests more than one scheduler
/// thread, then call [`Runtime::run`](crate::runtime::Runtime::run) on it;
/// `run` here is exactly that, pinned to one thread.
pub fn run<F: FnOnce() + Send + 'static>(f: F) { pub fn run<F: FnOnce() + Send + 'static>(f: F) {
crate::runtime::init(crate::runtime::Config::exact(1)).run(f); crate::runtime::init(crate::runtime::Config::exact(1)).run(f);
} }
+11 -2
View File
@@ -6,10 +6,19 @@
//! Build the loom models with: `RUSTFLAGS="--cfg loom" cargo test --lib --release` //! Build the loom models with: `RUSTFLAGS="--cfg loom" cargo test --lib --release`
#[cfg(loom)] #[cfg(loom)]
pub(crate) use loom::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; pub(crate) use loom::sync::atomic::{fence, AtomicU64, AtomicUsize, Ordering};
#[cfg(not(loom))] #[cfg(not(loom))]
pub(crate) use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; pub(crate) use std::sync::atomic::{fence, AtomicU64, AtomicUsize, Ordering};
// park.rs condvar-parker (loom + non-Linux builds only; the Linux non-loom
// build parks on a futex and never touches these — gating them identically
// keeps the default build free of unused imports).
#[cfg(loom)]
pub(crate) use loom::sync::{Condvar, Mutex};
#[cfg(all(not(loom), not(target_os = "linux")))]
pub(crate) use std::sync::{Condvar, Mutex};
/// `UnsafeCell` with loom's `with`/`with_mut` access API; pass-through cost /// `UnsafeCell` with loom's `with`/`with_mut` access API; pass-through cost
/// is zero in normal builds (`#[inline]`, newtype over std's cell). /// is zero in normal builds (`#[inline]`, newtype over std's cell).
+147 -14
View File
@@ -102,6 +102,19 @@ pub struct Entry {
seq: u64, seq: u64,
pub pid: Pid, pub pid: Pid,
pub reason: Reason, pub reason: Reason,
/// RFC 007 virtual time: the global delay ledger reading when this entry
/// was (re-)queued. `pop_due` shifts the effective deadline by any delay
/// injected since, so timers dilate together with the causally-delayed
/// workload instead of firing early in virtual terms.
#[cfg(feature = "smarm-causal")]
delay_stamp: u64,
/// RFC 007: a wall-anchored entry opts out of the virtual-time shift —
/// its deadline is honoured in wall time regardless of injected delay.
/// Used by the causal controller's own measurement/cooldown sleeps so
/// experiment windows keep a fixed wall length; ordinary workload timers
/// stay virtual (`false`).
#[cfg(feature = "smarm-causal")]
wall: bool,
} }
impl PartialEq for Entry { impl PartialEq for Entry {
@@ -128,6 +141,14 @@ impl PartialOrd for Entry {
#[derive(Default)] #[derive(Default)]
pub struct Timers { pub struct Timers {
/// RFC 018: the scheduler coordination layer. Attached once at
/// `RuntimeInner::new`; every insert notes its deadline (min-maintained
/// snapshot for the busy-path due-check + the timekeeper re-arm wake)
/// and every pop/clear re-anchors the snapshot to the heap minimum.
/// All calls happen under the timers mutex — the serialization the
/// coordinator's timer protocol mandates. `None` only in unit tests
/// that construct a bare `Timers`.
coord: Option<std::sync::Arc<crate::park::Coordinator>>,
/// Reverse-wrapped so the smallest deadline is at the top. /// Reverse-wrapped so the smallest deadline is at the top.
heap: BinaryHeap<Reverse<Entry>>, heap: BinaryHeap<Reverse<Entry>>,
/// Monotonic counter for the tiebreaker `seq` field (and the `TimerId` of a /// Monotonic counter for the tiebreaker `seq` field (and the `TimerId` of a
@@ -144,7 +165,18 @@ pub struct Timers {
impl Timers { impl Timers {
pub fn new() -> Self { pub fn new() -> Self {
Self { heap: BinaryHeap::new(), next_seq: 0, armed: std::collections::HashSet::new() } Self {
coord: None,
heap: BinaryHeap::new(),
next_seq: 0,
armed: std::collections::HashSet::new(),
}
}
/// Attach the scheduler coordination layer (RFC 018). Called once, at
/// runtime construction, before any scheduler thread exists.
pub(crate) fn attach_coordinator(&mut self, c: std::sync::Arc<crate::park::Coordinator>) {
self.coord = Some(c);
} }
/// Insert a `Sleep` timer. Convenience for the common case. /// Insert a `Sleep` timer. Convenience for the common case.
@@ -152,6 +184,19 @@ impl Timers {
self.insert(deadline, pid, Reason::Sleep { epoch }); self.insert(deadline, pid, Reason::Sleep { epoch });
} }
/// Insert a *wall-anchored* `Sleep` timer: fires at `deadline` in wall
/// time even while causal profiling (feature `smarm-causal`) is injecting
/// virtual delay — it never chases the delay ledger. Without the feature
/// this is identical to [`insert_sleep`](Self::insert_sleep).
///
/// Intended for measurement machinery (the causal controller's window and
/// cooldown sleeps, TSC calibration) whose durations *define* wall time
/// rather than participate in the workload. Workload code should use the
/// ordinary virtual-anchored timers.
pub fn insert_sleep_wall(&mut self, deadline: Instant, pid: Pid, epoch: u32) {
self.push(deadline, pid, Reason::Sleep { epoch }, true);
}
/// Arm a cancellable `send_after` timer: run `fire` at `deadline` unless /// Arm a cancellable `send_after` timer: run `fire` at `deadline` unless
/// [`cancel`](Self::cancel)led first. `pid` is informational only (the /// [`cancel`](Self::cancel)led first. `pid` is informational only (the
/// destination, or who armed it — useful for introspection); it is *not* /// destination, or who armed it — useful for introspection); it is *not*
@@ -163,11 +208,24 @@ impl Timers {
pid: Pid, pid: Pid,
fire: Box<dyn FnOnce() + Send>, fire: Box<dyn FnOnce() + Send>,
) -> TimerId { ) -> TimerId {
let seq = self.next_seq; self.armed.insert(self.next_seq);
self.next_seq = self.next_seq.wrapping_add(1); TimerId(self.push(deadline, pid, Reason::Send { fire }, false))
self.armed.insert(seq); }
self.heap.push(Reverse(Entry { deadline, seq, pid, reason: Reason::Send { fire } }));
TimerId(seq) /// Arm a *wall-anchored* cancellable `send_after` timer (RFC 007): the
/// same contract as [`insert_send`](Self::insert_send), but the entry
/// opts out of the virtual-time shift and fires at its raw deadline
/// regardless of injected delay — the `Send`-reason sibling of
/// [`insert_sleep_wall`](Self::insert_sleep_wall). Without the
/// `smarm-causal` feature this is identical to `insert_send`.
pub fn insert_send_wall(
&mut self,
deadline: Instant,
pid: Pid,
fire: Box<dyn FnOnce() + Send>,
) -> TimerId {
self.armed.insert(self.next_seq);
TimerId(self.push(deadline, pid, Reason::Send { fire }, true))
} }
/// Cancel an armed `send_after` timer. Returns `true` if the timer was /// Cancel an armed `send_after` timer. Returns `true` if the timer was
@@ -179,11 +237,38 @@ impl Timers {
self.armed.remove(&id.0) self.armed.remove(&id.0)
} }
/// Insert an arbitrary timer entry. /// Insert an arbitrary (virtual-anchored) timer entry.
pub fn insert(&mut self, deadline: Instant, pid: Pid, reason: Reason) { pub fn insert(&mut self, deadline: Instant, pid: Pid, reason: Reason) {
self.push(deadline, pid, reason, false);
}
/// Common insertion path. `wall` selects the RFC 007 anchor (see
/// [`insert_sleep_wall`](Self::insert_sleep_wall)); it is accepted — and
/// ignored — without the `smarm-causal` feature so callers don't fork.
/// Returns the entry's `seq`.
fn push(&mut self, deadline: Instant, pid: Pid, reason: Reason, wall: bool) -> u64 {
#[cfg(not(feature = "smarm-causal"))]
let _ = wall;
let seq = self.next_seq; let seq = self.next_seq;
self.next_seq = self.next_seq.wrapping_add(1); self.next_seq = self.next_seq.wrapping_add(1);
self.heap.push(Reverse(Entry { deadline, seq, pid, reason })); self.heap.push(Reverse(Entry {
deadline,
seq,
pid,
reason,
#[cfg(feature = "smarm-causal")]
delay_stamp: crate::causal::global_delay_cycles(),
#[cfg(feature = "smarm-causal")]
wall,
}));
// RFC 018: publish the (possibly new-minimum) deadline to the
// busy-path snapshot and wake the timekeeper if it is parked
// toward a later one. We hold the timers mutex — the mandated
// serialization for both.
if let Some(c) = &self.coord {
c.note_deadline(deadline);
}
seq
} }
pub fn is_empty(&self) -> bool { pub fn is_empty(&self) -> bool {
@@ -196,6 +281,9 @@ impl Timers {
pub fn clear(&mut self) { pub fn clear(&mut self) {
self.heap.clear(); self.heap.clear();
self.armed.clear(); self.armed.clear();
if let Some(c) = &self.coord {
c.refresh_deadline(None);
}
} }
/// Soonest pending deadline, or `None` if the heap is empty. /// Soonest pending deadline, or `None` if the heap is empty.
@@ -210,23 +298,68 @@ impl Timers {
/// one is silently dropped here (its `seq` was already removed from /// one is silently dropped here (its `seq` was already removed from
/// `armed` by [`cancel`](Self::cancel)). Returning it removes it from /// `armed` by [`cancel`](Self::cancel)). Returning it removes it from
/// `armed`, so a later `cancel` of a fired timer reports `false`. /// `armed`, so a later `cancel` of a fired timer reports `false`.
///
/// RFC 007 virtual time (feature `smarm-causal`): before an entry fires,
/// any global delay injected since it was (re-)queued is added to its
/// deadline; an entry whose *effective* deadline hasn't passed is pushed
/// back with the shifted deadline and a fresh stamp, so it keeps chasing
/// delay injected while it waits. Consequences, both benign:
/// [`peek_deadline`](Self::peek_deadline) may under-report (raw deadline
/// earlier than effective), costing at most one spurious scheduler wake
/// per injected chunk; and a shift never converts wall time — with zero
/// debt the path is byte-identical to the featureless one. Wall-anchored
/// entries ([`insert_sleep_wall`](Self::insert_sleep_wall)) are exempt
/// from the shift and always fire at their raw deadline.
pub fn pop_due(&mut self, now: Instant) -> Vec<Entry> { pub fn pop_due(&mut self, now: Instant) -> Vec<Entry> {
let mut out = Vec::new(); let mut out = Vec::new();
#[cfg(feature = "smarm-causal")]
let global = crate::causal::global_delay_cycles();
while let Some(r) = self.heap.peek() { while let Some(r) = self.heap.peek() {
if r.0.deadline <= now { if r.0.deadline > now {
let entry = match self.heap.pop() { break;
}
#[allow(unused_mut)]
let mut entry = match self.heap.pop() {
Some(e) => e.0, Some(e) => e.0,
None => panic!("smarm: timer heap pop after peek returned None (core corrupt)"), None => panic!("smarm: timer heap pop after peek returned None (core corrupt)"),
}; };
if matches!(entry.reason, Reason::Send { .. }) && !self.armed.remove(&entry.seq) { if matches!(entry.reason, Reason::Send { .. }) && !self.armed.contains(&entry.seq) {
// Cancelled before it came due: discard, do not deliver. // Cancelled before it came due: discard, do not deliver.
// (Checked before any shift so a cancelled entry is never
// re-queued just to be discarded later.)
continue; continue;
} }
out.push(entry); #[cfg(feature = "smarm-causal")]
} else { if !entry.wall {
break; let debt = global.saturating_sub(entry.delay_stamp);
if debt > 0 {
let shifted = entry
.deadline
.checked_add(crate::causal::cycles_to_duration(debt))
.unwrap_or(entry.deadline);
if shifted > now {
// Not due in virtual time: re-queue at the shifted
// deadline, stamped, keeping `seq` (and thus `Send`
// cancellation identity) intact.
entry.deadline = shifted;
entry.delay_stamp = global;
self.heap.push(Reverse(entry));
continue;
} }
} }
}
if matches!(entry.reason, Reason::Send { .. }) {
self.armed.remove(&entry.seq);
}
out.push(entry);
}
// RFC 018: re-anchor the busy-path snapshot to the new heap minimum
// (still under the timers mutex). A causal-shift re-queue above went
// through `heap.push` directly, so this peek is the one place the
// snapshot is guaranteed to catch up.
if let Some(c) = &self.coord {
c.refresh_deadline(self.peek_deadline());
}
out out
} }
} }
+1040
View File
File diff suppressed because it is too large Load Diff
+69
View File
@@ -0,0 +1,69 @@
//! RFC 018 scheduler park/wake — observable-behavior guards.
//!
//! These pin the two timer-latency properties the park/wake swap must
//! preserve or introduce:
//!
//! - `sleep_fires_under_saturation`: due timers fire even when every
//! scheduler is busy (nobody parked ⇒ no timekeeper) — the busy-path
//! due-check, ratified design point (a). The old drain phase gave this
//! for free (timers drained every loop iteration); the new design must
//! not lose it.
//! - `submillisecond_sleep_is_prompt`: a sub-ms sleep completes promptly.
//! Under the old wake pipe, `poll_wake`'s `as_millis` truncation turned
//! sub-ms deadlines into 0ms busy-polls (correct wall time, pathological
//! CPU); under park/wake the futex timespec carries full nanosecond
//! precision.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
#[test]
fn sleep_fires_under_saturation() {
let rt = smarm::runtime::init(smarm::runtime::Config::exact(4));
rt.run(|| {
let stop = Arc::new(AtomicBool::new(false));
let mut spinners = Vec::new();
// 8 spinners over 4 schedulers: the run queue never empties, so no
// scheduler ever parks and no timekeeper exists. Only the busy-path
// due-check can fire the sleeper's timer before the spinners quit.
for _ in 0..8 {
let stop = stop.clone();
spinners.push(smarm::spawn(move || {
let t0 = Instant::now();
while !stop.load(Ordering::Relaxed) && t0.elapsed() < Duration::from_secs(5) {
smarm::yield_now();
}
}));
}
let t0 = Instant::now();
smarm::sleep(Duration::from_millis(10));
let dt = t0.elapsed();
stop.store(true, Ordering::Relaxed);
for s in spinners {
let _ = s.join();
}
assert!(
dt < Duration::from_millis(500),
"10ms sleep took {dt:?} under scheduler saturation — busy-path \
timer firing is broken (timekeeper-only firing stalls under load)"
);
});
}
#[test]
fn submillisecond_sleep_is_prompt() {
let rt = smarm::runtime::init(smarm::runtime::Config::exact(2));
rt.run(|| {
// Warm one iteration, then measure.
smarm::sleep(Duration::from_micros(500));
let t0 = Instant::now();
smarm::sleep(Duration::from_micros(500));
let dt = t0.elapsed();
assert!(dt >= Duration::from_micros(400), "woke early: {dt:?}");
assert!(
dt < Duration::from_millis(100),
"500µs sleep took {dt:?} — sub-ms deadline handling is broken"
);
});
}
+32
View File
@@ -485,3 +485,35 @@ fn multi_thread_timer_only_no_pipe_contention() {
SLEEP_MS * 2, SLEEP_MS * 2,
); );
} }
// ---------------------------------------------------------------------------
// Root panic propagation
/// A panic in the root actor escapes `run()` to the caller. Anything else
/// makes every assert inside `run` silently vacuous — found live when a
/// failing-first test passed: the tripped assert was caught by the
/// trampoline, recorded as `Outcome::Panic` on the root slot, and dropped
/// unread with the initial handle.
#[test]
#[should_panic(expected = "root actor panic escapes")]
fn root_panic_escapes_run() {
rt1().run(|| {
panic!("root actor panic escapes");
});
}
/// Teardown completes before the root panic propagates: a caller that
/// catches it can immediately `run()` again on the same `Runtime` (the
/// documented sequential-reuse contract).
#[test]
fn runtime_reusable_after_root_panic() {
let r = rt1();
let caught = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
r.run(|| panic!("boom"));
}));
assert!(caught.is_err(), "root panic must escape run()");
let ran = Arc::new(AtomicBool::new(false));
let ran_t = ran.clone();
r.run(move || ran_t.store(true, Ordering::Relaxed));
assert!(ran.load(Ordering::Relaxed), "runtime unusable after root panic");
}
+7 -1
View File
@@ -108,8 +108,14 @@ fn loser_arm_wake_after_parked_select_stays_precise() {
t0.elapsed() >= Duration::from_millis(40), t0.elapsed() >= Duration::from_millis(40),
"one-shot park returned early: a stale loser-arm wake landed" "one-shot park returned early: a stale loser-arm wake landed"
); );
// Arm 0 is now closed (the sender actor exited after its sends) and
// a closed arm reports ready forever under priority order — observe
// the disconnect and drop it from the set, per the documented
// closed-arm rule.
assert_eq!(select(&[&rxa, &rxb]), 0);
assert!(rxa.try_recv().is_err(), "arm 0 must report disconnect");
// The loser's message was never lost. // The loser's message was never lost.
assert_eq!(select(&[&rxa, &rxb]), 1); assert_eq!(select(&[&rxb]), 0);
assert_eq!(rxb.try_recv().unwrap(), Some(2)); assert_eq!(rxb.try_recv().unwrap(), Some(2));
h.join().unwrap(); h.join().unwrap();
}); });
+169
View File
@@ -0,0 +1,169 @@
//! Reproducer (soak20 signature 2, refcount_test.exs "watcher crash"):
//! `by_name` stores only the slot *index*, so a name whose holder died — never
//! unregistered, since no smarm stop path unregisters (prune is lazy) — and
//! whose slot was then re-tenanted by an unrelated actor reads as *live-held*:
//!
//! - `register` of the name fails `NameTaken { holder: <unrelated tenant> }`,
//! so the bridge's generated `start()` (a `let _ =`) silently no-ops and
//! `start_server/1` reports `:ok` for a server that never came up;
//! - a by-name `call` resolves the tenant's mailbox, misses on the message
//! `TypeId`, and fails `ServerDown` fast — and does NOT prune (only the
//! dead-holder and dangling-name arms prune), so the name never heals
//! while the tenant lives. The wedge is self-sustaining.
//!
//! Wild signature: 110235x fast `{:error, :server_down}` probes over the full
//! 5 s await window after a swallowed restart (200-run width-20 soak, run 59).
//!
//! The test asserts the *contract*: after its holder dies, a name must be
//! re-registrable regardless of what happened to the slot. Red pre-fix.
use smarm::{
call, init, request_stop, whereis, CallError, Config, GenServer, GenServerBuilder,
GenServerName, RegisterError,
};
use std::sync::{Arc, Mutex};
use std::time::Duration;
const TARGET: GenServerName<Target> = GenServerName::new("stale_reuse_target");
/// The named server whose death opens the window. Trivial on purpose.
struct Target;
impl GenServer for Target {
type Call = ();
type Reply = ();
type Cast = ();
type Info = ();
type Timer = ();
fn handle_call(&mut self, _req: ()) {}
fn handle_cast(&mut self, _op: ()) {}
}
/// The unrelated tenant. A *different* server type, so its mailbox holds a
/// different `Envelope` `TypeId` — a same-typed tenant would make the by-name
/// `call` *deliver to the wrong server* instead of failing, which is the same
/// root hole wearing a worse hat.
struct Filler;
impl GenServer for Filler {
type Call = ();
type Reply = ();
type Cast = ();
type Info = ();
type Timer = ();
fn handle_call(&mut self, _req: ()) {}
fn handle_cast(&mut self, _op: ()) {}
}
#[derive(Debug)]
struct Observed {
old_slot: (u32, u32),
tenant_slot: (u32, u32),
/// `whereis` of the dead name after re-tenanting — `Some` is the misread.
whereis_after_reuse: Option<(u32, u32)>,
/// By-name call after re-tenanting — the wild `server_down` fast-fail.
call_after_reuse: Result<(), CallError>,
/// The contract under test: re-registering the dead name.
restart: Result<(), RegisterError>,
}
#[test]
fn dead_name_with_reused_slot_must_be_re_registrable() {
let out: Arc<Mutex<Option<Observed>>> = Arc::new(Mutex::new(None));
let out_w = out.clone();
// A deliberately tiny slab forces prompt slot recycling: with every filler
// held alive, the freed slot is the only *recycled* one, so a filler lands
// on it deterministically well before the slab (a loud panic) runs out.
init(Config::exact(2).max_actors(32)).run(move || {
// 1. Named server up; record its slot.
let target = GenServerBuilder::new(Target)
.named(TARGET)
.start()
.expect("name should be free at test start");
let old_pid = target.pid();
// 2. Kill it WITHOUT unregistering (no stop path does). Death is
// confirmed via the *ref*, never the name — a by-name resolve of a
// dead-but-not-yet-reused holder takes the prune arm and heals the
// name, destroying the precondition.
request_stop(old_pid);
loop {
match target.call(()) {
Err(CallError::ServerDown) => break,
Ok(()) => smarm::sleep(Duration::from_millis(5)),
}
}
drop(target);
// 3. Re-tenant the slot: spawn fillers (all kept alive) until one
// lands on the old index.
let mut fillers = Vec::new();
let mut tenant = None;
for i in 0..24 {
let name: &'static str = Box::leak(format!("stale_filler_{i}").into_boxed_str());
let f = GenServerBuilder::new(Filler)
.named(GenServerName::<Filler>::new(name))
.start()
.expect("filler names are fresh");
let fp = f.pid();
fillers.push(f);
if fp.index() == old_pid.index() {
tenant = Some(fp);
break;
}
}
let tenant = tenant.expect(
"precondition: the freed slot must be re-tenanted within the tiny slab \
(slots are recycled; every filler is held alive)",
);
// 4. Observe the poisoned state through the same paths the bridge uses.
let whereis_after_reuse = whereis(TARGET.as_str()).map(|p| (p.index(), p.generation()));
let call_after_reuse = call(TARGET, ());
let restart = GenServerBuilder::new(Target)
.named(TARGET)
.start()
.map(|_fresh_ref| ());
*out_w.lock().unwrap() = Some(Observed {
old_slot: (old_pid.index(), old_pid.generation()),
tenant_slot: (tenant.index(), tenant.generation()),
whereis_after_reuse,
call_after_reuse,
restart,
});
drop(fillers);
});
let o = out.lock().unwrap().take().expect("run body completed");
eprintln!("observed: {o:?}");
assert!(
o.restart.is_ok(),
"re-registering '{}' after its holder died failed with {:?}: the dead name \
reads as held by the live, unrelated tenant {:?} because by_name kept only \
the slot index (old slot {:?}). This is the silent-no-op start_server path \
of soak20 signature 2.",
TARGET.as_str(),
o.restart,
o.tenant_slot,
o.old_slot,
);
// The healed semantics around the re-register: the stale name reads
// *unbound* (never the tenant), and a by-name call fails ServerDown rather
// than resolving anything of the tenant's.
assert_eq!(
o.whereis_after_reuse, None,
"whereis of a dead name must prune and report unbound, not the slot's new tenant",
);
assert_eq!(
o.call_after_reuse,
Err(CallError::ServerDown),
"a by-name call to a dead name must fail ServerDown",
);
}
+120
View File
@@ -0,0 +1,120 @@
//! Reproducer: a *named* gen_server stopped with `request_stop` while a `call`
//! sits **un-dequeued** in its inbox does NOT release the parked caller with
//! `CallError::ServerDown`. The caller parks forever, contradicting the
//! documented gen_server guarantee ("Any caller currently waiting in `call`
//! sees `Err(ServerDown)`").
//!
//! Root cause (channel.rs): `Receiver::Drop` only flips `receiver_alive = false`
//! and never drains `queue`. The queued `Envelope::Call(_, reply_tx)` therefore
//! survives as long as the channel `Arc<Inner>` does — and for a *named* server
//! the registry holds a `Sender` clone (lazy prune) that keeps the `Arc` alive
//! after the server is gone. So the queued `reply_tx` is never dropped, the
//! caller's `reply_rx` never closes, and `reply_rx.recv()` parks forever.
//!
//! Anonymous servers happen to dodge this: when their last `GenServerRef`
//! drops, every `Sender` drops, the `Arc` refcount hits zero, `Inner` (and its
//! queue) is dropped, and the queued `reply_tx` goes with it — waking the
//! caller. The bug is specific to "a `Sender` outlives the `Receiver`", which a
//! registry entry guarantees for every named server.
use smarm::{
call, channel, init, request_stop, spawn, Config, GenServer, GenServerBuilder, GenServerName,
CallError, Receiver, RecvTimeoutError,
};
use std::sync::{Arc, Mutex};
use std::time::Duration;
const BLOCKER: GenServerName<Blocker> = GenServerName::new("repro_blocker");
/// A server that, on its single cast, parks forever on a gate channel the test
/// never feeds. This deterministically holds the server *inside a handler* (not
/// at the inbox recv), so any subsequent `call` queues behind it and stays
/// un-dequeued — exactly the state `request_stop` then has to clean up.
struct Blocker {
gate: Option<Receiver<()>>,
}
impl GenServer for Blocker {
type Call = ();
type Reply = ();
type Cast = ();
type Info = ();
type Timer = ();
// Trivial + instant: if this ever ran for the queued call, the caller would
// get Ok(()) immediately. It must NOT run — the server is parked on the gate
// when the stop arrives.
fn handle_call(&mut self, _req: ()) {}
// Park forever (until cancelled). recv() on an open channel with no message
// parks the actor; the gate sender is held by the test and never fires.
fn handle_cast(&mut self, _op: ()) {
if let Some(gate) = self.gate.take() {
let _ = gate.recv();
}
}
}
#[test]
fn named_server_request_stop_releases_queued_caller_with_server_down() {
// Final observation, asserted after the run.
// Some(Err(ServerDown)) -> contract honored (fixed)
// None -> caller never released; parked past the 3s
// bound (bug reproduced)
let outcome: Arc<Mutex<Option<Result<(), CallError>>>> = Arc::new(Mutex::new(None));
let outcome_w = outcome.clone();
init(Config::exact(2)).run(move || {
// Gate the server will park on. Held for the whole run so the server's
// gate.recv() parks (rather than seeing Disconnected and returning).
let (gate_tx, gate_rx) = channel::<()>();
// Channel the queued caller reports its result back on.
let (res_tx, res_rx) = channel::<Result<(), CallError>>();
// 1. Start the named server and keep its ref alive.
let server = GenServerBuilder::new(Blocker { gate: Some(gate_rx) })
.named(BLOCKER)
.start()
.expect("name should be free");
let spid = server.pid();
// 2. Send the cast and let the server dequeue it and park on the gate.
server.cast(()).expect("server is live");
smarm::sleep(Duration::from_millis(100));
// 3. A separate caller issues a by-name `call`. The server is parked on
// the gate, so this Call envelope queues un-dequeued; the caller then
// parks on its reply channel.
spawn(move || {
let r = call(BLOCKER, ());
let _ = res_tx.send(r);
});
smarm::sleep(Duration::from_millis(100));
// 4. Stop the server. Its loop unwinds out of the gate.recv() and drops
// the inbox Receiver — at which point the queued caller is *supposed*
// to be released with ServerDown.
request_stop(spid);
// 5. Bounded wait. A correct runtime releases the caller in well under
// 3s; the bug leaves it parked, so we time out.
let observed = match res_rx.recv_timeout(Duration::from_secs(3)) {
Ok(r) => Some(r),
Err(RecvTimeoutError::Timeout) => None,
Err(RecvTimeoutError::Disconnected) => None,
};
*outcome_w.lock().unwrap() = observed;
// Keep the gate sender alive until the very end.
drop(gate_tx);
});
let observed = outcome.lock().unwrap().take();
assert_eq!(
observed,
Some(Err(CallError::ServerDown)),
"queued caller was not released with ServerDown after the named server \
was request_stop'd (None = parked forever => bug reproduced)"
);
}
+61 -2
View File
@@ -401,7 +401,66 @@ fn send_after_to_dead_typed_pid_is_silent() {
assert_eq!(report_rx.recv().unwrap(), 1); // sink has now exited assert_eq!(report_rx.recv().unwrap(), 1); // sink has now exited
let _id = send_after(Duration::from_millis(15), sink, 2); let _id = send_after(Duration::from_millis(15), sink, 2);
sleep(Duration::from_millis(45)); // let it fire against the dead pid sleep(Duration::from_millis(45)); // let it fire against the dead pid
// No panic, and nothing further delivered. // No panic; the sink is gone, so its report sender dropped with it —
assert_eq!(report_rx.try_recv(), Ok(None)); // closed+empty is Err (documented), which also proves nothing
// further was delivered.
assert!(report_rx.try_recv().is_err(), "nothing further delivered");
});
}
// ---------------------------------------------------------------------------
// Wall-anchored send_after (RFC 007 user-facing opt-out). The API exists in
// both feature configs; featureless it is behaviourally identical to
// `send_after` — these tests pin exactly that.
// ---------------------------------------------------------------------------
#[test]
fn armed_wall_send_timer_is_returned_and_fires() {
let mut t = Timers::new();
let now = Instant::now();
let fired = Arc::new(AtomicBool::new(false));
let f = fired.clone();
let _id = t.insert_send_wall(
now + Duration::from_millis(10),
Pid::new(0, 0),
Box::new(move || f.store(true, Ordering::SeqCst)),
);
let mut due = t.pop_due(now + Duration::from_millis(20));
assert_eq!(due.len(), 1, "an armed wall send timer should pop when due");
run_fire(due.pop().unwrap());
assert!(fired.load(Ordering::SeqCst), "running the thunk delivers");
assert!(t.is_empty());
}
use smarm::send_after_named_wall;
#[test]
fn send_after_named_wall_delivers_after_the_delay() {
const WPING: Name<u64> = Name::new("send_after_wall_ping");
run(|| {
let (tx, rx) = channel::<u64>();
register(WPING, tx).unwrap();
let t0 = Instant::now();
let _id = send_after_named_wall(Duration::from_millis(30), WPING, 99);
assert_eq!(rx.recv().unwrap(), 99);
assert!(
t0.elapsed() >= Duration::from_millis(25),
"delivered too early: {:?}",
t0.elapsed()
);
});
}
#[test]
fn send_after_named_wall_cancels() {
const WC: Name<u64> = Name::new("send_after_wall_cancel");
run(|| {
let (tx, rx) = channel::<u64>();
register(WC, tx).unwrap();
let id = send_after_named_wall(Duration::from_millis(50), WC, 7);
assert!(cancel_timer(id), "cancel before fire returns true");
sleep(Duration::from_millis(90));
assert_eq!(rx.try_recv(), Ok(None), "cancelled wall timer delivered");
}); });
} }