Markk116/smarm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(runtime): phase 2 — fixed slab, per-slot packed-atomic state machine, raw cold locks

Browse Source 
The slot table split (ROADMAP_v0.5 phase 2): slot lookup is lock-free, the
run path (yield/park/unpark/pop/resume) takes zero locks beyond the queue
mutex itself, and SharedState shrinks to { run_queue }.

Core pieces:

- src/raw_mutex.rs: hand-rolled 3-state futex mutex (Drepper mutex3),
  non-poisoning; the guard enters NoPreempt, which both bounds hold times and
  structurally closes the unwind-under-lock hole (the stop sentinel shares
  the gate). Used for per-slot cold data, the free list, and the stack pool.

- Fixed slab Box<[Slot]> (default max_actors = 16_384, ~4 MiB, align(128)
  against false sharing). Slots never move -> stable addresses, lock-free
  index. Exhaustion panics loudly, naming Config::max_actors(n) as the fix.
  Unbounded/segmented slab stays deferred (see ROADMAP).

- Per-slot AtomicU64 packing (generation << 32 | state), states
  Vacant/Queued/Running/RunningNotified/Parked/Done. Every transition CASes
  the packed word, so the generation check is atomic with the transition: no
  ABA, no spurious unparks on recycled slots. RunningNotified replaces the
  pending_unpark bool (the lost-wakeup window is a state, not a flag) and
  uniformly fixes a LATENT LOST WAKEUP in the old Blocking-IO completion
  path, which set the result for a still-Running actor without flagging it.

- Invariant: a pid is in the run queue at most once (pushes pair 1:1 with
  transitions into Queued; only the scheduler does Queued->Running). This is
  what makes phase 3's bounded rings sound.

- sp -> relaxed AtomicUsize; stop flag + first-resume closure as AtomicPtrs
  (closure double-boxed for a thin pointer, swap-to-take): the resume path is
  fully atomic.

- finalize_actor: Done published under the dying slot's cold lock (join's
  check-or-register is linearized by it); link cascade locks peers ONE AT A
  TIME (cold locks are leaves) with the acyclicity argument written at the
  site; link() registers on the target first, then self (stale self-entries
  are benign, every walk re-verifies the peer's word).

- Termination by counters: live_actors incremented in spawn pre-enqueue,
  decremented at the very END of finalize after all wakeup enqueues; exit on
  io_out == 0 (read before the queue lock, phase-1 ordering) && queue empty
  && live == 0. Soundness note at the site: any enqueue targets a live actor.

- spawn boxes the closure and acquires the stack BEFORE any runtime lock: no
  allocation ever happens under a global lock anymore.

- with_runtime/try_with_runtime now enter NoPreempt for their full span.
  This fixes a bug the rework exposed: install_actor allocated with
  preemption enabled while the RUNTIME RefCell borrow was live; a timeslice
  preemption there migrates the actor across OS threads and the borrow guard
  increments one thread's RefCell count and decrements another's — underflow
  to 'permanently mutably borrowed', cascading panics (caught by stress
  suite: deterministic non-unwinding-panic abort in lost_wakeup_many_pairs).
  The old code was safe only by accident; now it's structural.

- Behavior note: request_stop on a RUNNING target now marks it
  RunningNotified, so its next park returns immediately to an observation
  point — faster stop observation; parked/queued/done semantics unchanged.

Validated: 22 suites green in release (1/2/8-thread oversubscribed on the
1-core sandbox) and in debug with all debug_asserts live; stress suite x5.
This commit is contained in:
Claude
2026-06-09 20:08:12 +00:00
parent 453f6f491f
commit 5e0c9d45da
7 changed files with 1070 additions and 578 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/scheduler.rs
+105 -114
View File
@@ -21,18 +21,39 @@ use std::sync::Arc;
// ---------------------------------------------------------------------------
/// Borrow the current runtime. Panics if called outside `Runtime::run()`.
///
/// The whole span runs with preemption disabled. Two reasons, both load-
/// bearing:
///
/// - The `RUNTIME` thread-local borrow is live across `f`. A preemption-
/// driven context switch inside `f` can resume the actor on a DIFFERENT OS
/// thread; the borrow guard would then increment this thread's RefCell but
/// decrement the other's — a count underflow that leaves that thread's
/// RefCell permanently "mutably borrowed". Holding a thread-local guard
/// across a potential switch point is the one unforgivable sin of green
/// threads; disabling preemption makes "no switch inside `f`" structural
/// instead of an accident of which bodies happen to allocate.
/// - `f` is runtime bookkeeping. Suspending an actor halfway through it (or
/// unwinding via the stop sentinel, which shares the gate) is never wanted.
pub(crate) fn with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> R {
RUNTIME.with(|r| {
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| {
let b = r.borrow();
let inner = b.as_ref().expect("smarm: not inside Runtime::run()");
f(inner)
})
});
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
result
}
/// Borrow the runtime if present; returns `None` otherwise.
/// Used on cleanup paths (channel Drop during teardown).
/// Same preemption gate as [`with_runtime`]; same reasons.
pub(crate) fn try_with_runtime<R>(f: impl FnOnce(&Arc<RuntimeInner>) -> R) -> Option<R> {
RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)))
let prev = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
let result = RUNTIME.with(|r| r.borrow().as_ref().map(|inner| f(inner)));
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(prev));
result
}
// ---------------------------------------------------------------------------
@@ -54,22 +75,30 @@ impl JoinHandle {
pub fn join(mut self) -> Result<(), JoinError> {
use crate::actor::Outcome;
use crate::runtime::State; // need State visibility
let me = current_pid().expect("join() called outside an actor");
loop {
// Check-Done-or-register-waiter is atomic under the target's cold
// lock; finalize publishes Done and takes the waiter list under
// the same lock, so we either see the outcome or are woken.
let outcome = with_runtime(|inner| {
inner.with_shared(|s| {
let slot = s.slot_mut(self.pid)
.expect("join: target slot has been reused");
if matches!(slot.state, State::Done) {
Some(slot.outcome.take().expect("Done slot must have outcome"))
} else {
slot.waiters.push(me);
None
}
})
let slot = inner.slot_at(self.pid)
.expect("join: pid index out of range");
let mut cold = slot.cold.lock();
let w = slot.word();
// Our outstanding handle pins the slot: it cannot be
// reclaimed (generation cannot change) while we hold it.
assert_eq!(
crate::runtime::word_gen(w), self.pid.generation(),
"join: target slot has been reused"
);
if crate::runtime::word_state(w) == crate::runtime::ST_DONE {
Some(cold.outcome.take().expect("Done slot must have outcome"))
} else {
cold.waiters.push(me);
None
}
});
match outcome {
@@ -95,20 +124,24 @@ impl JoinHandle {
fn decrement_handle_count(&mut self) {
with_runtime(|inner| {
inner.with_shared(|s| {
let should_reclaim = match s.slot_mut(self.pid) {
Some(slot) => {
slot.outstanding_handles =
slot.outstanding_handles.saturating_sub(1);
matches!(slot.state, crate::runtime::State::Done)
&& slot.outstanding_handles == 0
let should_reclaim = match inner.slot_at(self.pid) {
Some(slot) => {
let mut cold = slot.cold.lock();
if slot.generation() == self.pid.generation() {
cold.outstanding_handles = cold.outstanding_handles.saturating_sub(1);
cold.outstanding_handles == 0
&& crate::runtime::word_state(slot.word())
== crate::runtime::ST_DONE
} else {
false
}
None => false,
};
if should_reclaim {
crate::runtime::reclaim_slot(s, self.pid);
}
})
None => false,
};
if should_reclaim {
// Re-verified inside; benign if finalize's reclaim won a race.
crate::runtime::reclaim_slot(inner, self.pid);
}
});
}
}
@@ -129,51 +162,28 @@ impl Drop for JoinHandle {
// ---------------------------------------------------------------------------
pub fn spawn(f: impl FnOnce() + Send + 'static) -> JoinHandle {
let parent = current_pid()
.or_else(|| with_runtime(|inner| inner.with_shared(|s| s.root_pid)))
.expect("spawn() before run()");
let parent = current_pid().unwrap_or_else(|| {
// Outside an actor but inside run(): the initial spawn. with_runtime
// panics with "not inside Runtime::run()" if there's no runtime at all.
with_runtime(|_| crate::runtime::ROOT_PID)
});
spawn_under(parent, f)
}
pub fn spawn_under(supervisor: Pid, f: impl FnOnce() + Send + 'static) -> JoinHandle {
// Try to reuse a stack from the pool; fall back to a fresh mmap if empty.
// Allocation happens before taking the shared lock so any syscall doesn't
// stall other scheduler threads.
let stack = with_runtime(|inner| inner.stack_pool.lock().unwrap().pop())
// Stack + closure boxing happen before ANY runtime lock is taken: no
// syscall and no allocation ever stalls another scheduler thread.
let stack = with_runtime(|inner| inner.stack_pool.lock().pop())
.unwrap_or_else(|| {
crate::stack::Stack::new(crate::runtime::ACTOR_STACK_SIZE)
.expect("stack allocation failed")
});
let sp = init_actor_stack(stack.top(), crate::actor::trampoline);
let closure: crate::runtime::Closure = Box::new(f);
let pid = with_runtime(|inner| {
inner.with_shared(|s| {
let (idx, gen) = s.allocate_slot();
let pid = Pid::new(idx, gen);
let slot = &mut s.slots[idx as usize];
slot.actor = Some(crate::actor::Actor {
pid,
stack,
sp,
supervisor,
stop: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)),
trap: None,
});
slot.state = crate::runtime::State::Runnable;
slot.outstanding_handles = 1;
slot.outcome = None;
slot.waiters.clear();
slot.supervisor_channel = None;
slot.monitors.clear();
slot.links.clear();
slot.pending_unpark = false;
slot.pending_io_result = None;
slot.pending_closure = Some(Box::new(f) as crate::runtime::Closure);
s.run_queue.push_back(pid);
crate::te!(crate::trace::Event::Spawn { parent: supervisor, child: pid });
crate::te!(crate::trace::Event::Enqueue(pid));
pid
})
let idx = inner.allocate_slot(); // panics loudly on slab exhaustion
crate::runtime::install_actor(inner, idx, sp, stack, supervisor, closure)
});
JoinHandle { pid, consumed: false }
@@ -207,39 +217,19 @@ pub fn park_current() {
}
pub fn unpark(pid: Pid) {
let result = try_with_runtime(|inner| {
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
match slot.state {
crate::runtime::State::Parked => {
// Actor is suspended — safe to re-queue immediately.
slot.state = crate::runtime::State::Runnable;
s.run_queue.push_back(pid);
crate::te!(crate::trace::Event::UnparkDirect(pid));
crate::te!(crate::trace::Event::Enqueue(pid));
}
crate::runtime::State::Runnable => {
// Actor is still running (between registering its
// parked_receiver and calling park_current). Set the
// flag; the scheduler will re-queue after the Park
// yield instead of sleeping.
slot.pending_unpark = true;
crate::te!(crate::trace::Event::UnparkDeferred(pid));
}
crate::runtime::State::Done => {}
}
}
})
});
let _ = result;
// The whole protocol lives on the slot's packed word (gen + state in one
// CAS) — see Slot::unpark_action. No runtime lock unless we enqueue.
let _ = try_with_runtime(|inner| inner.unpark(pid));
}
/// Request cooperative cancellation of `pid`.
///
/// Sets the actor's stop flag and, if it is parked, wakes it so it observes
/// the flag promptly. The actor realises the stop as a controlled unwind at
/// its next observation point (`check!()`/allocation, or the wakeup side of a
/// blocking park), terminating with `Outcome::Stopped`.
/// Sets the actor's stop flag and wakes it so it observes the flag promptly:
/// a parked target is re-queued; a running target is marked notified, so its
/// next park returns immediately to an observation point. The actor realises
/// the stop as a controlled unwind at its next observation point
/// (`check!()`/allocation, or the wakeup side of a blocking park),
/// terminating with `Outcome::Stopped`.
///
/// This is best-effort and cooperative: an actor that never reaches an
/// observation point — a tight loop with no `check!()`, no allocation, and no
@@ -247,21 +237,19 @@ pub fn unpark(pid: Pid) {
/// if `pid` is already gone.
pub fn request_stop(pid: Pid) {
let _ = try_with_runtime(|inner| {
inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
if let Some(actor) = slot.actor.as_ref() {
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
}
// Wake a parked target so it reaches its post-park observation
// point now. A Runnable target will observe on its next yield;
// a Done target has nothing to stop.
if matches!(slot.state, crate::runtime::State::Parked) {
slot.state = crate::runtime::State::Runnable;
s.run_queue.push_back(pid);
crate::te!(crate::trace::Event::Enqueue(pid));
if let Some(slot) = inner.slot_at(pid) {
{
let cold = slot.cold.lock();
// Verify under the cold lock: generation can't change while
// we hold it (reclaim takes the same lock).
if slot.generation() == pid.generation() {
if let Some(actor) = cold.actor.as_ref() {
actor.stop.store(true, std::sync::atomic::Ordering::Relaxed);
}
}
}
})
inner.unpark(pid);
}
});
}
@@ -333,13 +321,13 @@ where
});
park_current();
}
let result = with_runtime(|inner| inner.with_shared(|s| {
s.slot_mut(me)
.expect("block_on_io: own slot vanished")
.pending_io_result
let result = with_runtime(|inner| {
let slot = inner.slot_at(me).expect("block_on_io: own slot vanished");
let mut cold = slot.cold.lock();
cold.pending_io_result
.take()
.expect("block_on_io: resumed without a result")
}));
});
match result {
Ok(any) => *any.downcast::<T>().expect("block_on_io: type mismatch"),
Err(payload) => std::panic::resume_unwind(payload),
@@ -382,13 +370,16 @@ pub fn write(fd: std::os::fd::RawFd, buf: &[u8]) -> std::io::Result<usize> {
// ---------------------------------------------------------------------------
pub fn register_supervisor_channel(pid: Pid, sender: Sender<Signal>) {
with_runtime(|inner| inner.with_shared(|s| {
if let Some(slot) = s.slot_mut(pid) {
slot.supervisor_channel = Some(sender);
} else {
panic!("register_supervisor_channel: pid {:?} not found", pid);
}
}));
with_runtime(|inner| {
let slot = inner.slot_at(pid)
.unwrap_or_else(|| panic!("register_supervisor_channel: pid {:?} not found", pid));
let mut cold = slot.cold.lock();
assert_eq!(
slot.generation(), pid.generation(),
"register_supervisor_channel: pid {:?} not found", pid
);
cold.supervisor_channel = Some(sender);
});
}
// ---------------------------------------------------------------------------