Markk116/smarm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(runtime): phase 2 — fixed slab, per-slot packed-atomic state machine, raw cold locks

Browse Source 
The slot table split (ROADMAP_v0.5 phase 2): slot lookup is lock-free, the
run path (yield/park/unpark/pop/resume) takes zero locks beyond the queue
mutex itself, and SharedState shrinks to { run_queue }.

Core pieces:

- src/raw_mutex.rs: hand-rolled 3-state futex mutex (Drepper mutex3),
  non-poisoning; the guard enters NoPreempt, which both bounds hold times and
  structurally closes the unwind-under-lock hole (the stop sentinel shares
  the gate). Used for per-slot cold data, the free list, and the stack pool.

- Fixed slab Box<[Slot]> (default max_actors = 16_384, ~4 MiB, align(128)
  against false sharing). Slots never move -> stable addresses, lock-free
  index. Exhaustion panics loudly, naming Config::max_actors(n) as the fix.
  Unbounded/segmented slab stays deferred (see ROADMAP).

- Per-slot AtomicU64 packing (generation << 32 | state), states
  Vacant/Queued/Running/RunningNotified/Parked/Done. Every transition CASes
  the packed word, so the generation check is atomic with the transition: no
  ABA, no spurious unparks on recycled slots. RunningNotified replaces the
  pending_unpark bool (the lost-wakeup window is a state, not a flag) and
  uniformly fixes a LATENT LOST WAKEUP in the old Blocking-IO completion
  path, which set the result for a still-Running actor without flagging it.

- Invariant: a pid is in the run queue at most once (pushes pair 1:1 with
  transitions into Queued; only the scheduler does Queued->Running). This is
  what makes phase 3's bounded rings sound.

- sp -> relaxed AtomicUsize; stop flag + first-resume closure as AtomicPtrs
  (closure double-boxed for a thin pointer, swap-to-take): the resume path is
  fully atomic.

- finalize_actor: Done published under the dying slot's cold lock (join's
  check-or-register is linearized by it); link cascade locks peers ONE AT A
  TIME (cold locks are leaves) with the acyclicity argument written at the
  site; link() registers on the target first, then self (stale self-entries
  are benign, every walk re-verifies the peer's word).

- Termination by counters: live_actors incremented in spawn pre-enqueue,
  decremented at the very END of finalize after all wakeup enqueues; exit on
  io_out == 0 (read before the queue lock, phase-1 ordering) && queue empty
  && live == 0. Soundness note at the site: any enqueue targets a live actor.

- spawn boxes the closure and acquires the stack BEFORE any runtime lock: no
  allocation ever happens under a global lock anymore.

- with_runtime/try_with_runtime now enter NoPreempt for their full span.
  This fixes a bug the rework exposed: install_actor allocated with
  preemption enabled while the RUNTIME RefCell borrow was live; a timeslice
  preemption there migrates the actor across OS threads and the borrow guard
  increments one thread's RefCell count and decrements another's — underflow
  to 'permanently mutably borrowed', cascading panics (caught by stress
  suite: deterministic non-unwinding-panic abort in lost_wakeup_many_pairs).
  The old code was safe only by accident; now it's structural.

- Behavior note: request_stop on a RUNNING target now marks it
  RunningNotified, so its next park returns immediately to an observation
  point — faster stop observation; parked/queued/done semantics unchanged.

Validated: 22 suites green in release (1/2/8-thread oversubscribed on the
1-core sandbox) and in debug with all debug_asserts live; stress suite x5.
This commit is contained in:
Claude
2026-06-09 20:08:12 +00:00
parent 453f6f491f
commit 5e0c9d45da
7 changed files with 1070 additions and 578 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/raw_mutex.rs
+228
View File
@@ -0,0 +1,228 @@
//! A minimal futex-based mutex that cannot poison.
//!
//! `std::sync::Mutex` poisons on unwind, turning one panic into a cascade of
//! `lock().unwrap()` panics in every later user. The runtime's internal
//! critical sections must never unwind anyway (the stop sentinel is gated
//! behind `PREEMPTION_ENABLED`, and the guard below disables preemption), so
//! poisoning buys nothing and costs a failure mode. This mutex has no poison
//! state by construction.
//!
//! Two further properties the runtime wants:
//!
//! - **The guard enters `NoPreempt`.** A timeslice switch while holding an OS
//! mutex would suspend the actor with the lock held, stalling every other
//! OS thread that touches it until the actor is resumed. Disabling
//! preemption for the (short) critical section keeps lock hold times
//! bounded. It also closes the unwind hole structurally: with
//! `PREEMPTION_ENABLED` false, `maybe_preempt` neither yields nor raises
//! the stop sentinel, so no allocation inside the critical section can
//! unwind it.
//! - **No std machinery.** One `AtomicU32` and two futex syscalls; friendlier
//! to an eventual embedded port than `std::sync::Mutex` (swap the futex for
//! a spin or WFE backend).
//!
//! Algorithm: the classic three-state futex mutex (Drepper, "Futexes Are
//! Tricky", mutex3). 0 = unlocked, 1 = locked, 2 = locked with (possible)
//! waiters. Uncontended lock/unlock is one CAS / one swap, no syscall.
//!
//! Lock-order position: slot cold locks, the free list, and the stack pool
//! are all `RawMutex`es and all *leaves among themselves* — never hold two at
//! once. Holding a `RawMutex` while taking the run-queue mutex (`with_shared`)
//! is permitted (e.g. unpark from inside a cold section); the reverse —
//! taking any `RawMutex` from inside `with_shared` — is forbidden.
use std::cell::UnsafeCell;
use std::ops::{Deref, DerefMut};
use std::sync::atomic::{AtomicU32, Ordering};
const UNLOCKED: u32 = 0;
const LOCKED: u32 = 1;
const CONTENDED: u32 = 2;
/// How many `pause` spins to burn before falling back to the futex. Critical
/// sections under this lock are tens of nanoseconds (push to a Vec, clone a
/// sender), so a short spin almost always avoids the syscall.
const SPIN_LIMIT: u32 = 64;
pub(crate) struct RawMutex<T> {
state: AtomicU32,
data: UnsafeCell<T>,
}
// SAFETY: standard mutex argument — exclusive access to `data` is mediated by
// `state`; `T: Send` suffices for both because `&RawMutex` only ever hands out
// access to one thread at a time.
unsafe impl<T: Send> Send for RawMutex<T> {}
unsafe impl<T: Send> Sync for RawMutex<T> {}
impl<T> RawMutex<T> {
pub(crate) const fn new(data: T) -> Self {
Self {
state: AtomicU32::new(UNLOCKED),
data: UnsafeCell::new(data),
}
}
#[inline]
pub(crate) fn lock(&self) -> RawMutexGuard<'_, T> {
// Enter NoPreempt *before* acquiring, so a preemption can't fire
// between acquisition and guard construction.
let prev_preempt = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
if self
.state
.compare_exchange(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_err()
{
self.lock_slow();
}
RawMutexGuard { m: self, prev_preempt }
}
#[cold]
fn lock_slow(&self) {
// Bounded spin first: the expected hold time is far below the cost of
// a futex round trip.
let mut spins = 0;
loop {
let s = self.state.load(Ordering::Relaxed);
if s == UNLOCKED
&& self
.state
.compare_exchange_weak(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_ok()
{
return;
}
spins += 1;
if spins >= SPIN_LIMIT {
break;
}
std::hint::spin_loop();
}
// Futex path. Mark contended and sleep until woken; on wake, retake
// by swapping to CONTENDED (we cannot know whether other waiters
// remain, so we must conservatively keep the contended marker).
while self.state.swap(CONTENDED, Ordering::Acquire) != UNLOCKED {
futex_wait(&self.state, CONTENDED);
}
}
#[inline]
fn unlock(&self) {
if self.state.swap(UNLOCKED, Ordering::Release) == CONTENDED {
futex_wake(&self.state, 1);
}
}
}
pub(crate) struct RawMutexGuard<'a, T> {
m: &'a RawMutex<T>,
prev_preempt: bool,
}
impl<T> Deref for RawMutexGuard<'_, T> {
type Target = T;
#[inline]
fn deref(&self) -> &T {
// SAFETY: guard existence implies exclusive ownership of the lock.
unsafe { &*self.m.data.get() }
}
}
impl<T> DerefMut for RawMutexGuard<'_, T> {
#[inline]
fn deref_mut(&mut self) -> &mut T {
// SAFETY: as above, plus &mut self.
unsafe { &mut *self.m.data.get() }
}
}
impl<T> Drop for RawMutexGuard<'_, T> {
#[inline]
fn drop(&mut self) {
self.m.unlock();
// Restore preemption only after the lock is released.
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(self.prev_preempt));
}
}
// ---------------------------------------------------------------------------
// futex (x86-64 Linux; master is x86-only, see arm-port branch)
// ---------------------------------------------------------------------------
fn futex_wait(state: &AtomicU32, expected: u32) {
// SAFETY: `state` is a valid, aligned u32 for the duration of the call.
// Spurious wakeups and EAGAIN (value already changed) are both handled by
// the caller's retry loop.
unsafe {
libc::syscall(
libc::SYS_futex,
state.as_ptr(),
libc::FUTEX_WAIT | libc::FUTEX_PRIVATE_FLAG,
expected,
std::ptr::null::<libc::timespec>(),
);
}
}
fn futex_wake(state: &AtomicU32, n: i32) {
// SAFETY: as above.
unsafe {
libc::syscall(
libc::SYS_futex,
state.as_ptr(),
libc::FUTEX_WAKE | libc::FUTEX_PRIVATE_FLAG,
n,
);
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[test]
fn uncontended_lock_unlock() {
let m = RawMutex::new(0u64);
for _ in 0..1000 {
*m.lock() += 1;
}
assert_eq!(*m.lock(), 1000);
}
#[test]
fn contended_counter_is_exact() {
const THREADS: usize = 8;
const PER: u64 = 50_000;
let m = Arc::new(RawMutex::new(0u64));
let hs: Vec<_> = (0..THREADS)
.map(|_| {
let m = m.clone();
std::thread::spawn(move || {
for _ in 0..PER {
*m.lock() += 1;
}
})
})
.collect();
for h in hs {
h.join().unwrap();
}
assert_eq!(*m.lock(), THREADS as u64 * PER);
}
#[test]
fn no_poison_on_unwind() {
let m = Arc::new(RawMutex::new(0u64));
let m2 = m.clone();
let _ = std::thread::spawn(move || {
let _g = m2.lock();
panic!("unwind while holding");
})
.join();
// A std Mutex would now be poisoned; this one just works.
*m.lock() += 1;
assert_eq!(*m.lock(), 1);
}
}