smarm — Stubborn Marks Actor Runtime Machinery

Green-Thread Actor Runtime

Erlang's isolation model. Rust's zero-copy ownership. No function colouring.

smarm is a concurrent runtime for Rust. Each actor is a green thread with its own mmap'd stack. N OS threads share a run queue backed by a fixed slot slab (16,384 slots, ~4 MiB, allocated once) with lock-free slot lookup. Actors communicate exclusively via message passing (owned values over channels); no shared mutable state without an explicit Arc<Mutex<T>>.

Preemption is allocator-driven: every Nth heap allocation, smarm reads RDTSC and yields the actor if its timeslice has expired. No OS signals, no separate timer thread for scheduling. Since v0.3 the monolithic Mutex<SharedState> is gone: scheduling state lives in one atomic word per slot, the loom-checked transitions of which close every lost-wakeup window by construction. On top sits an OTP layer: gen_server, supervisors with restart strategies, monitors, links, and a named-pid registry.

vs async/await

No function colouring. No Box<dyn Future>. No poll state machines. Just plain Rust functions that block.

vs OS threads

64 KB stacks instead of 8 MB. Context switch in ~10–20 ns (6 GPR saves + ret) instead of kernel mode.

vs Erlang BEAM

Zero-copy ownership via Rust's type system. No GC pause. Message passing is a move, not a clone — with gen_server, supervision, links and monitors all the same.

Architecture

Module Map

22 source modules, four layers. Layer 0 has no smarm dependencies. Layer 1 is new in the v0.5–v0.8 line and is the heart of the runtime: two small modules whose every transition is model-checked with loom. Layer 2 is the machinery, with runtime.rs as the hub. Layer 3 is the public surface plus the OTP layer built on it.

LAYER 0 — PRIMITIVES (no smarm deps) stack mmap + guard context naked asm CSW preempt alloc hook + stop pid (index, gen) timer min-heap + epoch raw_mutex futex, no poison sync_shim std / loom LAYER 1 — THE LOOM-CHECKED PROTOCOL CORE slot_state (gen|epoch|state) word, all CAS run_queue rq-mutex | rq-mpmc | rq-striped LAYER 2 — RUNTIME MACHINERY actor trampoline channel MPSC + select mutex timeout, FIFO io epoll + pool monitor one-shot Down link 2-way + trap registry name ↔ pid trace perfetto runtime.rs — the hub slot slab · spawn/finalize · schedule_loop · unpark LAYER 3 — PUBLIC API + OTP LAYER scheduler spawn · park · stop · join lib.rs re-exports + GlobalAlloc gen_server call / cast / info / down supervisor strategies + intensity
stack
Layer 0 · primitive

mmap a contiguous region, mprotect the bottom page to PROT_NONE as a guard. Overflow → SIGSEGV. New in v0.8: a stack pool in RuntimeInner recycles stacks across spawns (cap: threads × 4), avoiding mmap/munmap churn.

context
Layer 0 · primitive

Two #[naked] assembly functions (switch_to_actor, switch_to_scheduler). Save 6 callee-saved GPRs, swap rsp, restore, ret. Thread-locals hold each side's saved stack pointer. XMM registers intentionally not saved — the compiler spills them at Rust call sites.

preempt
Layer 0 · primitive

Implements GlobalAlloc. Every Nth alloc (or check!()), read RDTSC; timeslice expired → switch_to_scheduler(). Also hosts cooperative cancellation: check_cancelled() reads the on-CPU actor's stop flag and raises the StopSentinel panic so the stack unwinds cleanly.

pid
Layer 0 · primitive

Pid(u32 index, u32 generation). Index = slot in the slab; generation bumps on reclaim and is never reused. A stale Pid fails the generation check atomically with any transition it attempts — ABA is structurally impossible.

timer
Layer 0 · primitive

Min-heap by deadline. Reason::Sleep and Reason::WaitTimeout entries now carry the wait's park-epoch: a stale entry (wait already satisfied) fails the epoch match and is a guaranteed no-op rather than a heuristic one.

raw_mutex
Layer 0 · primitive

Three-state futex mutex (Drepper's mutex3) that cannot poison. The guard enters NoPreempt, which both bounds lock hold times and structurally prevents the stop sentinel from unwinding a critical section. Used for slot cold data, free list, stack pool, registry.

sync_shim
Layer 0 · primitive

std vs loom::sync indirection for the two model-checked modules. Build the models with RUSTFLAGS="--cfg loom" cargo test --lib --release. Everything else uses std directly — context switches and futexes aren't loom-able.

slot_state
Layer 1 · loom-checked core

The per-slot state machine as a standalone unit: one AtomicU64 packing (gen | park-epoch | state), every transition a CAS on the whole word. Loom proves lost-wakeup, double-enqueue, stale-epoch and ABA unreachable. See the Slot State section.

run_queue
Layer 1 · loom-checked core

The global queue behind a compile-time choice: rq-mutex (default), rq-mpmc (Vyukov ring), rq-striped (M rings). All satisfy the same contract: push is infallible, a pid is in the queue at most once. See Run Queue.

actor
Layer 2 · machinery

Owns the Stack; defines the trampoline every first ret lands in. Runs the closure inside catch_unwind; outcome is Exit, Panic(payload), or — new — Stopped when the unwind was the StopSentinel from cooperative cancellation.

runtime
Layer 2 · the hub

Owns RuntimeInner: the fixed slot slab, free list, run queue, timers, IO handle, live_actors counter, per-thread stats. Hosts spawn, finalize_actor, the unpark protocol, and schedule_loop. The old Mutex<SharedState> no longer exists.

channel
Layer 2 · machinery

Unbounded MPSC, plus — new — recv_timeout, select, and select_timeout over any set of Selectable arms with ready-index priority in declaration order. A closed arm counts as ready-forever, which gen_server leans on for shutdown.

mutex
Layer 2 · machinery

Actor-aware mutex with mandatory timeout (default 30s). FIFO waiter queue, WaitTimeout timer entry, park. Timer and grant now race through the epoch-matched unpark — exactly one of them wakes the waiter, the other is a proven no-op.

io
Layer 2 · machinery

Two background OS threads: epoll thread (EPOLLONESHOT, pushes FdReady) and pool thread (blocking closures in catch_unwind, pushes Blocking). Completions and waiter registrations now carry epochs; the v0.3 fd-leak-on-death gap is closed by an unwind guard plus a belt-and-braces EPOLL_CTL_DEL.

monitor · link
Layer 2 · machinery

monitor(pid): unidirectional, one-shot Down on a private channel; demonitor takes it back; dead targets yield an immediate NoProc. link(pid): bidirectional and persistent — abnormal death propagates as a transitive stop cascade, or as an ExitSignal message if the peer called trap_exit.

registry
Layer 2 · machinery

Erlang-style name registry: a bimap (name ↔ pid) under one RawMutex, O(1) both directions. Cleanup is lazy: liveness is checked against the generation-checked slot word on contact, so dead bindings behave as absent — zero coupling to the actor lifecycle.

trace
Layer 2 · machinery

Per-event tracing behind --features smarm-trace, zero cost without it. MPSC to a dedicated drain thread that batches to disk; output is Chrome-trace JSON viewable in Perfetto. record() disables preemption (it can allocate).

scheduler · lib
Layer 3 · public facade

The public surface: spawn, spawn_under, yield_now, sleep, request_stop, join, IO waits, NoPreempt. lib.rs re-exports everything and installs the PreemptingAllocator as #[global_allocator].

gen_server · supervisor
Layer 3 · OTP

Erlang's two workhorses, built on channels + select + spawn — not on runtime internals. gen_server: call/cast/info/down over one actor. supervisor: OneForOne / OneForAll / RestForOne strategies with a restart-intensity cap (default 3 per 5s).

Dependency Graph

Who Imports What

The shape survives the v0.8 rewrite: runtime.rs is the hub. What changed is what feeds it — the scheduling state itself (slot_state) and the queue (run_queue) are now standalone, loom-checked inputs rather than fields of a big struct. The OTP layer deliberately sits on the public surface: gen_server and the supervisor use channels, select, monitors and spawn the same way user code would.

stack + stack pool context switch fns preempt stop flag actor trampoline timer min-heap io epoll + pool slot_state the word run_queue 3 variants runtime.rs slot slab · schedule_loop channel + select mutex lock_timeout registry name ↔ pid begin_wait · park · unpark_at (through scheduler's facade) scheduler.rs / lib.rs public surface · GlobalAlloc OTP layer gen_server · supervisor monitor · link built on channels, select, spawn — not on internals
⚠

Circular dependency, still intentional: channel and mutex call scheduler::unpark_at(), which calls into runtime — and runtime's schedule_loop resumes actors that run channel/mutex code. It works because an unpark is now a single CAS on the slot word plus an enqueue: it never blocks, and preemption is disabled while any smarm-internal lock (RawMutex guard, queue mutex) is held.

Initialisation

What Happens When You Call run(f)

Starting from user code calling smarm::run(|| { ... }). The single-threaded run() is a wrapper around runtime::init(Config::exact(1)).run(f).

1

Install panic hook (once)

A OnceLock guard installs a custom panic hook that suppresses output inside actor context. Without this, concurrent actor panics can deadlock Rust's default backtrace printer (non-reentrant internal lock). The previous hook is chained for panics outside actors.

2

Allocate the slot slab runtime.rs

Box<[Slot]> with max_actors entries (default 16,384, ~4 MiB) is allocated once and never moves — that's what makes lock-free slot lookup sound. Every index goes onto the free list. Exhausting the slab is a loud panic naming the Config::max_actors knob.

3

Start IoThread io.rs

Creates a wake pipe (O_NONBLOCK), an epollfd, and a shutdown pipe registered in the epollfd. Spawns the epoll thread (epoll_wait loop) and the pool thread (blocking-work mpsc receiver). Both share a completion VecDeque behind a mutex.

4

Install RUNTIME thread-local runtime.rs

Arc<RuntimeInner> is cloned into the calling thread's RUNTIME thread-local. This makes with_runtime() work on the calling thread immediately — needed for the next step.

5

Spawn initial actor scheduler.rs

scheduler::spawn(f) pops a slot index from the free list, grabs a stack from the pool (or mmaps one), writes the initial register frame, stores the closure in the slot, bumps live_actors, publishes Queued and enqueues the pid. Details in Spawn.

6

Spawn N−1 OS scheduler threads, enter schedule_loop on thread 0

Each extra thread clones Arc<RuntimeInner>, sets its thread-locals, and enters schedule_loop. Thread 0 is the calling thread and blocks in the loop until the program is done.

7

Termination & shutdown — counter-based

No slot-table scan under a big lock anymore. A thread exits when the queue pops empty and live_actors == 0 and no IO is outstanding. live_actors is incremented before each spawn's enqueue and decremented as the very last step of finalize_actor — strictly after every wakeup finalize produces — so live == 0 proves no work can ever appear again, and every thread independently reaches the same verdict. Then OS threads are joined and IoThread::drop() tears down both background threads.

Core Mechanism

The Scheduler Cycle

This is the heartbeat of the runtime, drawn the way biochemists draw the Krebs cycle — because it is one: a closed loop of state transformations that regenerates its own starting point, fed from outside by unparks and drained by finalization. The OS thread's execution context is the energy currency: at station 4 it is handed to the actor (switch_to_actor) and at station 5 the actor hands it back — the bottom arc is dashed because, for that stretch, the scheduler doesn't exist on this thread at all.

schedule_loop one revolution ≈ one resume 1 · drain (try-lock) pop due timers · drain IO one winner per round 2 · pop run queue None: idle-sleep on wake fd live==0 ∧ io==0 → AllDone 3 · claim slot CAS Queued → Running stale gen → skip pid 4 · arm & hand off load sp · take closure (1st) arm timeslice · preempt on 5 · take back preempt off · store sp read YieldIntent / done flag 6 · settle the pid Yield → re-queue · Park → CAS done → finalize_actor run queue rq-mutex | rq-mpmc | rq-striped a pid is here at most once pop() unpark_at(pid,e) unparks from anywhere channel send · mutex grant Down / ExitSignal · joiner wake epoch-matched: at most one wake lands per wait due work timer min-heap (Sleep · WaitTimeout) · completions: FdReady · Blocking{result} Parked(epoch) off the cycle entirely Park: CAS unpark_at Yield: enqueue finalize_actor (leaves cycle) Down → monitors ExitSignal / stop cascade → links Signal → supervisor · wake joiners stack → pool · slot → Vacant(gen+1) live_actors −= 1 (last) done actor runs on its own stack until: timeslice/yield_now → Yield recv/lock/sleep/wait_fd → Park return/panic/stop → done flag switch_to_actor() switch_to_scheduler() rsp swap · 6 GPRs each way

Two details worth pausing on. First, the drain is try-lock: one winner per revolution pops due timers and IO completions; losers skip straight to the queue, so the pure-compute hot path never contends a global lock just to learn there's nothing to drain. Second, the claim is a CAS, Queued → Running on the slot word. The only way it fails is a stale generation (the actor died and the slot was recycled while the pid sat in the queue), in which case the pid is simply skipped — nothing else can move a queued actor's word, because wakes no-op on Queued.

The Yield Sources

Source Intent set Who re-queues Notes
yield_now() Yield Scheduler immediately yield_return: Running → Queued, pushed to queue tail
Allocator preemption Yield Scheduler immediately RDTSC check in maybe_preempt() (every Nth alloc or check!())
channel::recv() (empty) Park send() → unpark_at(pid, e) begin_wait opens the epoch, then the receiver registers
select(arms) Park First ready arm's sender Loser arms hold a consumed epoch — proven no-ops, not pending wakes
mutex::lock() (contended) Park Guard drop or timeout timer Grant and timer race through the epoch — exactly one lands
sleep(d) / recv_timeout Park Timer heap → drain step Entry carries the epoch; a satisfied wait's entry is a strict no-op
wait_readable/writable(fd) Park epoll thread → completions → drain EPOLLONESHOT; waiter registered as (pid, epoch)
ServerRef::call() Park Server's reply send (or its death) Just recv() on a fresh one-shot reply channel
request_stop(pid) target — (unwinds) n/a — StopSentinel panic Raised at the next check_cancelled(): alloc, check!(), or wake from a park
The Loom-Checked Heart

Slot State & the Park-Epoch

Everything the scheduler needs to know about an actor's schedulability lives in one atomic u64 per slot: (generation << 32) | (park-epoch << 8) | state. Every transition is a CAS on the whole packed word, which buys two things at once: the generation check is atomic with the transition (a stale pid can never act on a recycled slot), and an unpark racing the prep-to-park window becomes a state — RunningNotified — resolved by the scheduler's park-return CAS rather than a flag read under a lock.

ONE ATOMIC u64 PER SLOT — EVERY TRANSITION IS A CAS ON THE WHOLE WORD generation (32b) park-epoch (24b) state (8b) checked atomically: no ABA wait identity Vacant in free list Queued in run queue, once Running on an OS thread RunningNotified wake landed mid-run Parked off the run queue Done return · panic · stop spawn: publish e = 0 pop: try_claim yield_return begin_wait: e+1, then register (pid, e) with wakers park_return: e preserved unpark mid-run: e consumed (e+1) park_return here: re-queue, no park unpark_at(e): e+1, waker enqueues set_done: e=0 reclaim: gen+1, stale pids die here every successful wake consumes the epoch ⇒ at most one wake lands per wait, by construction loom model-checks these transitions: lost wakeup, double enqueue, stale epoch, ABA — all unreachable

The park-epoch (middle 24 bits) is the actor's wait identity. A waiting actor calls begin_wait once per wait — bump, get e — before registering (pid, e) with any waker. Wakes are epoch-matched: unpark_at(pid, e) lands only if the word still carries e, and every successful wake consumes the epoch. So at most one wake can ever land per wait, by construction. This is what lets a select register with several channels and park once: the winning arm's wake bumps the epoch, and every loser arm's later wake fails the match and no-ops — instead of leaving a phantom notification that would fault the actor's next one-shot park in sleep, lock_timeout, or block_on_io.

The only wildcard wake (no epoch) is request_stop, which is terminal — control never returns to the code that parked. A no-park exit (a select arm ready at registration time) retires the wait explicitly: bump the epoch, then clear_notify eats anything that already landed, then re-check the stop flag so a terminal wake can't be swallowed.

🔬

slot_state.rs compiles its atomics from sync_shim, so loom model-checks the production transitions directly (RUSTFLAGS="--cfg loom"). The shipped theorems: no lost wakeup (park vs unpark), at-most-one enqueue (two racing unparkers), consumed-epoch wakes never land (select's loser arms), the retire eats late arm notifications, stale-generation unparks never touch a reused slot, and unpark-vs-claim coalesces. Each is an exhaustive interleaving proof, not a stress test.

Scheduling

Run Queue: Three Variants, One Contract

The v0.3 doc listed the single global Mutex<SharedState> as the primary scalability ceiling. v0.8's answer is to make the queue its own module behind a compile-time feature choice, all three variants loom-checked against the same contract — and to shrink what the queue mutex covers to only the pop/push itself; the slot's own atomics carry everything needed to resume.

rq-mutex default

A VecDeque behind a RawMutex. Innermost lock in the ordering — nothing else is ever acquired under it. Simple, predictable, and fine until queue ops dominate.

rq-mpmc

A Vyukov-style bounded MPMC ring with per-cell sequence counters. Lock-free push/pop; falls back to an overflow side-list to keep push infallible.

rq-striped

M independent rings; producers and consumers hash/rotate across stripes to spread contention. A stepping stone toward per-thread deques with work stealing.

the contract

Push is infallible. A pid is in the queue at most once. The only pushes are paired 1:1 with successful transitions into Queued; only the scheduler transitions Queued → Running, paired 1:1 with pops. Queue ops require preemption disabled (debug-asserted).

📎

Termination doesn't lean on pop-None being a fence (with the rings it's only a snapshot). The argument is counter-first: every queue entry's target stays Queued — hence counted in live_actors — until that very entry is popped, so live == 0 by itself implies the queue is empty forever.

Spawn Mechanics

New Actor From First Resume

Spawning is still the trickiest part of the runtime: an actor's first resume is fundamentally different from subsequent ones because we can't "call" into a new stack — we have to ret into it.

1

scheduler::spawn(f) called

Pops a slot index from the free list (RawMutex<Vec<u32>>) — the slab is fixed, so slot exhaustion panics loudly naming Config::max_actors. The slot's current generation becomes the new Pid(index, gen). A Stack comes from the stack pool if one is cached, else a fresh 64 KB mmap + guard page.

2

Initial stack frame written context::init_actor_stack()

Starting from top & ~15 − 8 (aligned), pushes downward: the trampoline address as the ret target, then 6 zero words for the callee-saved registers. The resulting rsp is stored in the slot's sp atomic. No actual function call has happened yet.

high addr ← top
  top-8:  &trampoline   ← will be popped by 'ret'
  top-16: 0             ← rbx
  top-24: 0             ← rbp
  top-32: 0             ← r12
  top-40: 0             ← r13
  top-48: 0             ← r14
  top-56: 0             ← r15  ← initial rsp stored here
3

Closure stored in the slot

The Box<dyn FnOnce() + Send> goes into the slot's closure: AtomicPtr — swap-to-take, no map, no lock. (v0.3 kept a pending_closures map inside SharedState; that's gone with the struct.) It can't ride the actor's stack because nothing can pass it via a register during first resume.

4

Count it live, then publish

live_actors += 1 before the actor becomes poppable — the termination argument needs every enqueue to target a counted actor. Then publish_queued(gen): a plain Release store Vacant → Queued with epoch 0 (the spawner owns the vacant slot exclusively, so no CAS needed). This store is the moment the actor exists to pops, unparks and stops. Finally the pid is enqueued.

5

Scheduler pops the pid, claims, prepares first resume

try_claim CASes Queued → Running. The scheduler loads sp from the slot, sees take_closure() return Some (it's the first resume) and moves the box into the trampoline's thread-local. Then: bind the stop flag, reset the timeslice, enable preemption, switch_to_actor().

6

First context switch lands in trampoline()

switch_to_actor() saves the scheduler's GPRs, loads the slot's sp as the new rsp, pops the 6 zero words, then rets — popping the trampoline address and jumping to it. We're now on the actor's stack. The trampoline takes the closure from the thread-local and calls it inside catch_unwind(AssertUnwindSafe(f)).

7

Actor returns → trampoline classifies the outcome

Ok(()) → Exit. Err(payload) → Panic(payload) — unless the payload is the StopSentinel, in which case the outcome is Stopped: a cooperative cancellation, not a crash, and links/supervisors treat it as such. The done flag is set, one last switch_to_scheduler(), and the scheduler runs finalize_actor: Down to monitors, ExitSignal or stop cascade to links, Signal to the supervisor, epoch-matched joiner wakes, stack back to the pool, slot reclaimed at gen+1, and live_actors −= 1 last of all.

Preemption & Cancellation

Allocator-Driven Timeslicing

How it works

The PreemptingAllocator is installed as the process's #[global_allocator]. Its alloc(), alloc_zeroed(), and realloc() all call maybe_preempt() before delegating to the system allocator.

maybe_preempt() decrements a thread-local counter. Every 128 allocations (default), it reads RDTSC. If rdtsc() − timeslice_start > 300_000 cycles (~100µs at 3 GHz) and PREEMPTION_ENABLED == true, it calls switch_to_scheduler().

The check!() macro calls the same maybe_preempt() — the explicit preemption point for tight no-alloc loops, since stable Rust offers no transparent way to interrupt them.

Cooperative cancellation rides the same rail

request_stop(pid) sets a flag in the target's Arc<AtomicBool> and fires a wildcard unpark. The flag is observed by check_cancelled() — called from maybe_preempt() and from the wake side of every park — which raises the StopSentinel panic. The trampoline's catch_unwind recognises it and reports Outcome::Stopped, distinct from a user Panic: the stack unwinds, Drops run, links and supervisors see a stop, not a crash.

Invariant: preemption off while holding smarm-internal locks

A timeslice switch while holding an internal lock would suspend the actor with the lock held, stalling every OS thread that touches it. With the big SharedState mutex gone, the rule is enforced structurally:

  • PREEMPTION_ENABLED = false in the scheduler loop before/after switch_to_actor()
  • every RawMutex guard (slot cold data, free list, stack pool, registry) enters NoPreempt on construction
  • run-queue ops require preemption disabled — debug-asserted in run_queue.rs
  • trace::record() also disables preemption (it can allocate)

Disabling preemption also gates the stop sentinel: with PREEMPTION_ENABLED == false, maybe_preempt neither yields nor raises the sentinel, so no allocation inside a critical section can unwind it. That's why RawMutex can afford to be non-poisoning.

⚠

Known gap, unchanged: tight no-alloc loops are invisible without explicit check!() calls — to preemption and to cancellation. Documented and by design; such loops are uncommon in message-passing workloads.

// preempt.rs — simplified
pub fn maybe_preempt() {
    ALLOC_COUNT.with(|c| {
        let n = c.get();
        if n == 0 {
            c.set(ACTIVE_ALLOC_INTERVAL.with(|i| i.get()));  // reset counter
            if PREEMPTION_ENABLED.with(|e| e.get()) {
                check_cancelled();  // may raise StopSentinel → Outcome::Stopped
                let elapsed = rdtsc() - TIMESLICE_START.with(|s| s.get());
                if elapsed > ACTIVE_TIMESLICE_CYCLES.with(|i| i.get()) {
                    unsafe { switch_to_scheduler() };  // YieldIntent::Yield
                }
            }
        } else {
            c.set(n - 1);
        }
    });
}
IO Architecture

Two Background Threads, One Wake Pipe

Actor calls wait_readable(fd) or block_on_io(f) → begin_wait: epoch e → register (pid, e), park epoll thread epoll_wait(-1) loop EPOLLONESHOT per fd on ready: push FdReady write wake_pipe on shutdown pipe: exit pool thread mpsc::recv() loop catch_unwind(closure) push Blocking result write wake_pipe tx drop → exit completions Arc<Mutex<VecDeque>> FdReady { fd, events } Blocking { pid, e, result } drained by schedule_loop scheduler poll(wake_fd) drain completions FdReady → resolve waiters[fd] unpark_at(pid, e) Blocking → stash in cold, unpark_at(pid, e) epoll_ctl ADD submit(closure) drain wake pipe write

Both completion kinds now carry the waiter's park-epoch, so the wake side is the same protocol as everything else. For Blocking, the result is stashed under the slot's cold lock before the unpark_at — and if the actor died with the op in flight (a stop), the generation check fails and the result is discarded instead of being written into a recycled slot. This path closed a latent lost wakeup: the v0.3 code set the result for a still-Running actor without flagging it; under v0.8 that interleaving lands as RunningNotified and the upcoming park re-queues.

📎

epoll_ctl ADD/DEL is called by the scheduler thread directly on the epollfd — legal per epoll_ctl(2) even while the epoll thread is inside epoll_wait. Avoids a second command channel. The v0.3 fd-leak on actor-death-during-wait is fixed: an unwind guard in wait_fd deregisters on the stop path, and registration does a belt-and-braces EPOLL_CTL_DEL first (harmless ENOENT if absent).

OTP — Request / Reply

gen_server: One Object Above, One Actor Below

The mental model you program against is a synchronous object: a state value with handlers, processing one message at a time — no locks, no races, because there's no concurrency inside the plane. The implementation is one actor, one inbox channel, and a select loop. The diagram maps each promise of plane A to the mechanism in plane B that pays for it.

PLANE A — THE MODEL YOU PROGRAM AGAINST caller call(req) → blocks, gets Reply back cast(msg) → fire&forget GenServer owned state, one message at a time handle_call(Call) → Reply handle_cast(Cast) handle_info(Info) · handle_down(Down) init(ctx) · terminate() no locks, no races: it's sequential the world out-of-band events → Info watched pid dies → Down (system messages first) call reply cast call() = send + park each call carries a fresh one-shot reply_tx ONE shared inbox: calls ∥ casts ordered dispatch by Envelope arm separate channels, selected first PLANE B — WHAT ACTUALLY RUNS: ONE ACTOR + CHANNELS caller actor tx.send(Envelope::Call (req, reply_tx)) reply_rx.recv() → parks recv_timeout = call_timeout inbox channel Envelope::Call(req, reply_tx) | Envelope::Cast(msg) closed ⇒ graceful shutdown server actor — server_loop select priority (rebuilt per turn): Down arms › Watcher arm › infos › inbox dispatch → handle_call / cast / info / down reply_tx.send(reply) — failed send is fine Terminate drop guard: terminate() on every exit (close · panic · stop) one-shot reply channel drop = ServerDown to caller out-of-band channels info Receivers (fixed at start) + Monitor rx, fed at runtime via Watcher wakes the parked caller

The arm priority is the semantics

Calls and casts travel the same inbox as Envelope variants — that's what makes them mutually ordered, like Erlang. Everything out-of-band rides separate channels composed at the wait: select priority is Down arms › Watcher arm › info channels (declaration order) › inbox, rebuilt each turn. A hot inbox can't starve a death notice or a system message; conversely a hot info channel can starve the inbox — deliberately. A closed info arm is silently dropped from the set; a closed inbox (every ServerRef gone) is graceful shutdown.

Death needs no monitor

Server death detection falls out of channel closure. Already dead → the inbox is closed and send fails → ServerDown. Dies mid-call (handler panic or request_stop) → the reply sender drops as the server's stack unwinds → the parked caller's recv errors → ServerDown. So call_timeout is exactly recv_timeout on the reply channel — nothing registered, nothing to leak on the timeout path by construction. One Erlang-faithful sharp edge: a timed-out request stays in the inbox and will still be handled; only the reply is discarded. Design idempotent calls accordingly.

📎

terminate() is wired through a drop guard, so it runs on every exit path — clean close, panic, stop — not just the happy one. Keep it cheap and non-panicking: it may run mid-unwind, where a second panic aborts the process. The loop also calls check!() every iteration, so a server whose arms are never empty stays preemptible and cancellable.

OTP — Failure As Data

Monitors, Links, Supervisors, Registry

All four are built on the same two primitives — the slot's cold lifecycle data, walked by finalize_actor, and the epoch-matched unpark — and all four follow Erlang's semantics closely enough that the BEAM chapter of your brain transfers.

monitor — watching without touching

monitor(pid) returns a Monitor: a private Receiver<Down> plus the identity for demonitor. Unidirectional, one-shot — exactly one Down ever; the channel closes after. DownReason is deliberately payload-free: a panic payload has one owner, the joiner. Monitoring an already-dead pid yields an immediate NoProc Down rather than silence.

link — fate-sharing, opt-out via trap

link(pid) is bidirectional and persistent. Abnormal death (panic or stop — never a normal return) propagates: a non-trapping peer is request_stop'd, and because its own links are walked when it finalizes, the stop cascades transitively — "let it crash." A peer that called trap_exit() instead receives an ExitSignal message and keeps running. Linking a dead pid behaves as an immediate NoProc death.

supervisor — policy over mechanism

A supervisor is just an actor running OneForOne::new().child(spec)…run(). Strategies: OneForOne (restart the dead child), OneForAll (restart everything), RestForOne (restart it and everyone after it). Restart::{Permanent, Transient, Temporary} per child. The intensity cap (default 3 restarts per 5s, Erlang's default) stops a crash-looping child from burning the tree — tripping it kills the supervisor, escalating upward.

registry — names with lazy hygiene

register(name, pid) / whereis(name) / name_of(pid): a bimap under one RawMutex, O(1) both ways. No hook in finalize_actor — every operation checks the bound pid's liveness via the generation-checked slot word, so a stale binding is detectable, never misdirected: dead bindings behave as absent and are pruned on contact. Zero coupling to the actor lifecycle, at the cost of a dead name lingering until touched.

// the whole OTP loop, in user space
spawn(|| {
    OneForOne::new()
        .strategy(Strategy::RestForOne)
        .intensity(3, Duration::from_secs(5))
        .child(ChildSpec::new(Restart::Permanent, || db_writer()))
        .child(ChildSpec::new(Restart::Transient, || batch_job()))
        .run();   // parks on its Signal channel; restarts per policy
});
Key Gotchas

Things That Would Bite You

Lost-wakeup window — closed

The v0.3 doc's headline gotcha is now a theorem instead of a flag. An unpark racing the prep-to-park window lands as the RunningNotified state; the scheduler's park-return CAS re-queues instead of parking. Loom proves parked-forever unreachable across all interleavings — including the Blocking-IO path the old pending_unpark bool silently missed.

Global state mutex — gone

The single Mutex<SharedState> that capped scalability no longer exists. Slot lookup is lock-free against a fixed slab; the queue mutex (if rq-mutex) covers only push/pop; rq-mpmc / rq-striped remove even that. Timers and IO sit behind their own try-locked drain.

std::thread::sleep inside an actor

Still blocks the entire OS scheduler thread, starving every actor on it. There's no detection. Use smarm::sleep(d). Same family: any blocking syscall outside block_on_io / wait_readable.

Panics in terminate()

gen_server's terminate() runs from a drop guard, possibly mid-unwind. A panic inside it during an unwind is a double panic → process abort, no supervision tree to save you. Keep it cheap, non-blocking, non-panicking.

Cold locks are leaf locks

Per-slot RawMutex cold data follows a strict leaf rule: never hold two cold locks at once — finalize_actor's link cascade locks peers one at a time. Overall order: io → (cold | free | stack_pool) → run queue innermost; timers independent. Internal code only, but if you're hacking on smarm, this is the map.

Timed-out calls are still handled

call_timeout giving up does not cancel the request — it's already in the server's inbox and will run; only the reply is discarded (the abandoned reply channel makes the server's send fail harmlessly). Erlang behaves identically. Make calls idempotent or live with it.

Stale timer entries — now provably harmless

A granted lock or satisfied wait still leaves its timer entry in the heap (no cancellation). But the entry carries the wait's epoch, and the epoch was consumed by the real wake — so the late fire is a proven no-op, not a "should be fine." Cost unchanged: ~32 bytes per stale entry, bounded.

Stops are observation-point cancellation

request_stop is cooperative: the sentinel is raised at the next allocation, check!(), or park wake. An actor spinning in a no-alloc loop without check!() is unstoppable, exactly as it is unpreemptible. Holding a RawMutex guard also defers the sentinel (by design — see Preemption).

XMM registers not saved in context switch

Intentional and correct. XMM0–15 are caller-saved in SysV AMD64 ABI. Every yield passes through a Rust call site, so the compiler has already spilled live XMM values to the actor's stack before the naked asm runs; they're restored from its own stack on resume.

panic = unwind is required

The trampoline uses catch_unwind to intercept actor panics — and cooperative stops are implemented as an unwind (StopSentinel). With panic = abort, both become process death: no Outcome::Panic, no Stopped, no supervision. Documented; the profile is set in Cargo.toml.