Markk116/smarm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(scheduler): RFC 005 wake slot — per-scheduler capacity-one wake cache

Browse Source 
A thread-local Cell<Option<Pid>> per scheduler, checked before the shared
run queue. Runtime-selected via Config { wake_slot: bool }, default OFF
until the slot shootout accepts it (one binary benches both arms).

Push policy: slot-eligible iff the wake originates from actor context
(current_pid().is_some()) — the slot push replaces run_queue.push at the
tail of the unpark protocol's Parked → Queued CAS, so at-most-once-enqueued
holds verbatim as (slot ⊕ shared queue). Scheduler-context wakes (timer/IO
drain) and spawns always go shared (spawns never reach unpark_inner at all).
Displacement is Go semantics: newest wake takes the slot, occupant pushed
shared — moved, never copied.

Pop order: slot, then shared. Slot-popped actors skip reset_timeslice() and
inherit the waker's remaining slice; a handoff chain is bounded by one
slice, after which the preempt-yield re-enqueue goes shared (a yield is not
a wake) — the one-slice starvation bound, zero new counters. Idle and
AllDone are only reachable with an empty local slot by pop order; an
occupied slot elsewhere holds a Queued (live) actor, so the counter-first
termination argument is untouched.

Observability: per-thread slot_hits / slot_displacements (reset at run()
start so post-run stats() reads are per-run), SlotPush/SlotPop trace events.

Bench plan (roadmap v0.9 item 2): rq_runtime gains the slot on/off
dimension (SMARM_BENCH_SLOT, default "0 1") — ping-pong-pairs is the
target metric, yield-storm the regression guard, spawn-storm the
neutrality check. RQCSV grows a slot column; RQSLOT lines carry the
counters; bench_rq.sh aggregates both. Tests pin the push policy through
the counters (actor-context hits, spawn/join bypass, displacement,
default-off, per-run reset).
This commit is contained in:
Claude
2026-06-11 20:20:07 +02:00
parent f09e992f32
commit 2708042990
6 changed files with 519 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/run_queue.rs
+6 -2
View File
@@ -25,8 +25,12 @@
//! - **Occupancy is bounded by `max_actors`.** A pid is in the queue at most
//! once (pushes pair 1:1 with transitions into `Queued`; only the
//! scheduler transitions `Queued → Running` — see the state-machine docs
//! in `runtime.rs`), and at most `max_actors` actors exist. The bounded
//! rings are sized ≥ `max_actors`, so **`push` is infallible**; a full
//! in `runtime.rs`), and at most `max_actors` actors exist. With the RFC
//! 005 wake slot enabled the invariant reads "in (slot ⊕ shared queue) at
//! most once" — a slot push *replaces* the queue push at the same
//! protocol point, and a displacement moves the occupant, never copies
//! it — so the bound holds verbatim. The bounded rings are sized ≥
//! `max_actors`, so **`push` is infallible**; a full
//! ring is an invariant violation and panics loudly rather than spinning.
//! - **Preemption must be disabled around every push/pop** (debug-asserted).
//! For the mutex variant this is the usual no-switch/no-unwind-under-lock
src/runtime.rs
+204 -65
View File
@@ -154,6 +154,7 @@ pub struct Config {
timeslice_cycles: u64,
stack_pool_cap: usize,
max_actors: usize,
wake_slot: bool,
}
impl Config {
@@ -166,6 +167,7 @@ impl Config {
timeslice_cycles: crate::preempt::DEFAULT_TIMESLICE_CYCLES,
stack_pool_cap: n * 4,
max_actors: DEFAULT_MAX_ACTORS,
wake_slot: false,
}
}
@@ -182,6 +184,7 @@ impl Config {
timeslice_cycles: crate::preempt::DEFAULT_TIMESLICE_CYCLES,
stack_pool_cap: max * 4,
max_actors: DEFAULT_MAX_ACTORS,
wake_slot: false,
}
}
@@ -226,6 +229,18 @@ impl Config {
self
}
/// Enable the per-scheduler wake slot (RFC 005): a thread-local,
/// capacity-one wake cache checked before the shared run queue. A wake
/// performed from actor context parks the woken pid in the waking
/// thread's slot; it is resumed next on that core and inherits the
/// remainder of the waker's timeslice. Scheduler-context wakes
/// (timer/IO drain) and spawns always go to the shared queue.
/// Default: `false` (off until the slot shootout accepts it).
pub fn wake_slot(mut self, on: bool) -> Self {
self.wake_slot = on;
self
}
/// The number of scheduler threads this config resolves to.
pub fn resolved_thread_count(&self) -> usize {
if let Some(e) = self.exact {
@@ -249,6 +264,7 @@ impl Default for Config {
timeslice_cycles: crate::preempt::DEFAULT_TIMESLICE_CYCLES,
stack_pool_cap: avail * 4,
max_actors: DEFAULT_MAX_ACTORS,
wake_slot: false,
}
}
}
@@ -264,6 +280,10 @@ pub struct SchedulerStats {
pub current_pid_index: AtomicU32,
/// Snapshot of run queue length maintained on every push/pop.
pub run_queue_len: AtomicU64,
/// RFC 005: wakes resumed from this thread's wake slot.
pub slot_hits: AtomicU64,
/// RFC 005: slot occupants displaced to the shared queue by a newer wake.
pub slot_displacements: AtomicU64,
}
impl SchedulerStats {
@@ -271,6 +291,8 @@ impl SchedulerStats {
Self {
current_pid_index: AtomicU32::new(u32::MAX),
run_queue_len: AtomicU64::new(0),
slot_hits: AtomicU64::new(0),
slot_displacements: AtomicU64::new(0),
}
}
}
@@ -305,6 +327,23 @@ impl RuntimeStats {
pub fn sleeping_count(&self) -> u32 {
self.inner.sleeping.load(Ordering::Relaxed)
}
/// RFC 005: total wakes resumed from a wake slot, summed across
/// scheduler threads. Counters are reset at the start of each `run()`,
/// so after a run this reads that run's total.
pub fn slot_hits(&self) -> u64 {
self.inner.stats.iter()
.map(|s| s.slot_hits.load(Ordering::Relaxed))
.sum()
}
/// RFC 005: total slot occupants displaced to the shared queue, summed
/// across scheduler threads. Reset at the start of each `run()`.
pub fn slot_displacements(&self) -> u64 {
self.inner.stats.iter()
.map(|s| s.slot_displacements.load(Ordering::Relaxed))
.sum()
}
}
// ---------------------------------------------------------------------------
@@ -460,6 +499,9 @@ pub(crate) struct RuntimeInner {
/// Preemption knobs, written into each scheduler thread's locals on startup.
pub(crate) alloc_interval: u32,
pub(crate) timeslice_cycles: u64,
/// RFC 005: whether actor-context wakes route through the per-scheduler
/// wake slot. Read-only after init; one predictable branch per wake.
pub(crate) wake_slot: bool,
/// The name <-> pid registry (bidirectional). RawMutex Leaf: never held
/// with any other lock; liveness checks under it read only the atomic
/// slot word.
@@ -477,6 +519,7 @@ impl RuntimeInner {
timeslice_cycles: u64,
stack_pool_cap: usize,
max_actors: usize,
wake_slot: bool,
) -> Arc<Self> {
let stats = (0..thread_count).map(|_| SchedulerStats::new()).collect();
let slots: Box<[Slot]> = (0..max_actors).map(|_| Slot::vacant()).collect();
@@ -496,6 +539,7 @@ impl RuntimeInner {
sleeping: AtomicU32::new(0),
alloc_interval,
timeslice_cycles,
wake_slot,
registry: RawMutex::new(crate::registry::Registry::new()),
stack_pool: RawMutex::new(Vec::new()),
stack_pool_cap,
@@ -568,7 +612,20 @@ impl RuntimeInner {
match slot.word.unpark(pid.generation(), want) {
Unpark::Enqueue => {
crate::te!(crate::trace::Event::UnparkDirect(pid));
self.enqueue(pid);
// RFC 005: a wake from ACTOR context is slot-eligible —
// the woken actor's message bytes are hot in this core's
// cache. Scheduler-context wakes (timer/IO drain, where
// current_pid is None) have no locality to exploit,
// arrive in bursts that would thrash the slot, and
// concentrate on the drain winner by construction:
// shared queue, always. Spawns never come through here
// (install_actor enqueues directly), so they bypass the
// slot by construction.
if self.wake_slot && crate::actor::current_pid().is_some() {
self.slot_push(pid);
} else {
self.enqueue(pid);
}
}
Unpark::Notified => {
crate::te!(crate::trace::Event::UnparkDeferred(pid));
@@ -578,6 +635,38 @@ impl RuntimeInner {
}
}
/// RFC 005: park `pid` in this thread's wake slot instead of the shared
/// queue. Replaces `enqueue` at the tail of the wake protocol's
/// `Parked → Queued` CAS, so the caller has just transitioned the pid
/// into `Queued` — same precondition, same invariant, different home.
/// Displacement: the NEW wake takes the slot (newest is hottest; the old
/// occupant was about to lose its locality window anyway) and the old
/// occupant is pushed to the shared queue.
fn slot_push(&self, pid: Pid) {
debug_assert!(
!crate::preempt::PREEMPTION_ENABLED.with(|c| c.get()),
"slot_push with preemption enabled — a switch mid-op could \
migrate the actor and split the slot access across threads"
);
debug_assert!(
self.slot_at(pid).map(|s| s.word.load()).is_some_and(|w| {
crate::slot_state::word_gen(w) == pid.generation()
&& crate::slot_state::word_state(w) == crate::slot_state::ST_QUEUED
}),
"slot_push of a pid not in (gen, Queued)"
);
let displaced = WAKE_SLOT.with(|s| s.replace(Some(pid)));
crate::te!(crate::trace::Event::SlotPush(pid));
if let Some(old) = displaced {
SCHED_SLOT.with(|s| {
self.stats[s.get()]
.slot_displacements
.fetch_add(1, Ordering::Relaxed)
});
self.enqueue(old);
}
}
/// Allocate the next process-unique `MonitorId`. Lock-free; monitors are a
/// cold path but there is no reason to serialize id minting under any lock.
pub(crate) fn alloc_monitor_id(&self) -> MonitorId {
@@ -621,6 +710,7 @@ pub fn init(config: Config) -> Runtime {
config.timeslice_cycles,
config.stack_pool_cap,
config.max_actors,
config.wake_slot,
),
thread_count: n,
}
@@ -673,6 +763,13 @@ impl Runtime {
);
*self.inner.io.lock().unwrap() = Some(IoThread::start().expect("failed to start IO thread"));
// RFC 005: slot counters reset at the START of a run (not the end),
// so `stats()` read after `run()` returns reports that run's totals.
for stat in &self.inner.stats {
stat.slot_hits.store(0, Ordering::Relaxed);
stat.slot_displacements.store(0, Ordering::Relaxed);
}
// Spawn the initial actor through the public spawn path (which
// requires a running runtime in the thread-local).
RUNTIME.with(|r| *r.borrow_mut() = Some(self.inner.clone()));
@@ -750,6 +847,17 @@ thread_local! {
/// This scheduler thread's index into RuntimeInner::stats.
static SCHED_SLOT: Cell<usize> = const { Cell::new(0) };
/// RFC 005: the per-scheduler wake slot — a capacity-one wake cache
/// checked before the shared run queue. Holds a pid in state
/// `(gen, Queued)` exactly as a shared-queue entry would; the
/// at-most-once-enqueued invariant reads "in (slot ⊕ shared queue) at
/// most once". All access is from the owning thread with preemption
/// disabled (the existing queue-op contract), so plain Cell ops suffice:
/// no atomics, nothing to steal, nothing to model. Empty whenever the
/// thread reaches the idle or termination path (pop order drains it
/// first), so it never holds a pid across the end of a run.
static WAKE_SLOT: Cell<Option<Pid>> = const { Cell::new(None) };
/// What the actor wants when it yields back to the scheduler.
static YIELD_INTENT: Cell<YieldIntent> = const { Cell::new(YieldIntent::Yield) };
}
@@ -1060,8 +1168,9 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
} // drain_guard drops here
// ----------------------------------------------------------------
// 2. Pop a runnable pid. The queue mutex covers ONLY the pop; the
// slot's own atomics carry everything needed to resume.
// 2. Pop a runnable pid. Pop order (RFC 005): wake slot first, then
// shared queue. The queue mutex covers ONLY the pop; the slot's
// own atomics carry everything needed to resume.
// ----------------------------------------------------------------
enum Pop {
Got(Pid),
@@ -1069,76 +1178,95 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
AllDone,
}
// Read IO liveness BEFORE the queue lock (phase-1 ordering: a
// completion resurrects an actor only via the drain path, whose
// enqueue would be visible under the queue lock we take next).
let (io_out, io_fd) = match inner.io.lock().unwrap().as_ref() {
Some(io) => (io.outstanding + io.waiters.len() as u32, Some(io.wake_fd())),
None => (0, None),
// 2a. RFC 005: drain this thread's wake slot before touching the
// shared queue. Two consequences fall out of slot-first order:
// the idle path below is only reachable with an empty slot, and so
// is AllDone — an occupied slot on ANOTHER thread holds a Queued
// (hence live) actor, so `live_actors > 0` and termination cannot
// fire; the counter-first argument is untouched.
let slot_pid = if inner.wake_slot {
WAKE_SLOT.with(|s| s.take())
} else {
None
};
let from_slot = slot_pid.is_some();
stats.run_queue_len.store(inner.run_queue.len(), Ordering::Relaxed);
let pop = match inner.run_queue.pop() {
Some(pid) => Pop::Got(pid),
None => {
// Termination does not lean on pop-None being a fence (with
// the ring queues it is only a snapshot). The argument is
// counter-first: every queue entry's target stays `Queued` —
// hence un-finalized, hence counted live — until that very
// entry is popped. So `live == 0` (Acquire, pairing with
// finalize's Release decrement, which strictly follows all
// wakeup enqueues) by itself implies no entry is in, or can
// ever again enter, the queue: enqueues only target live
// actors, and a spawner is itself live. The pop-None above
// is then just the cheap fast-path filter; io_out was read
// before it per the phase-1 ordering. `live == 0` is also
// final — no spawn can resurrect the count — so every
// scheduler thread independently reaches this same verdict.
let live = inner.live_actors.load(Ordering::Acquire);
if live == 0 && io_out == 0 {
Pop::AllDone
} else {
Pop::Idle { io_outstanding: io_out, wake_fd: io_fd }
let pid = if let Some(pid) = slot_pid {
stats.slot_hits.fetch_add(1, Ordering::Relaxed);
crate::te!(crate::trace::Event::SlotPop(pid));
pid
} else {
// Read IO liveness BEFORE the queue lock (phase-1 ordering: a
// completion resurrects an actor only via the drain path, whose
// enqueue would be visible under the queue lock we take next).
let (io_out, io_fd) = match inner.io.lock().unwrap().as_ref() {
Some(io) => (io.outstanding + io.waiters.len() as u32, Some(io.wake_fd())),
None => (0, None),
};
stats.run_queue_len.store(inner.run_queue.len(), Ordering::Relaxed);
let pop = match inner.run_queue.pop() {
Some(pid) => Pop::Got(pid),
None => {
// Termination does not lean on pop-None being a fence (with
// the ring queues it is only a snapshot). The argument is
// counter-first: every queue entry's target stays `Queued` —
// hence un-finalized, hence counted live — until that very
// entry is popped. So `live == 0` (Acquire, pairing with
// finalize's Release decrement, which strictly follows all
// wakeup enqueues) by itself implies no entry is in, or can
// ever again enter, the queue: enqueues only target live
// actors, and a spawner is itself live. The pop-None above
// is then just the cheap fast-path filter; io_out was read
// before it per the phase-1 ordering. `live == 0` is also
// final — no spawn can resurrect the count — so every
// scheduler thread independently reaches this same verdict.
let live = inner.live_actors.load(Ordering::Acquire);
if live == 0 && io_out == 0 {
Pop::AllDone
} else {
Pop::Idle { io_outstanding: io_out, wake_fd: io_fd }
}
}
}
};
};
let pid = match pop {
Pop::Got(pid) => pid,
Pop::AllDone => {
// Remaining timer entries are orphaned (no live actor can be
// woken by them — e.g. a sleeper cancelled out of its sleep);
// they must not keep the runtime alive. Drop them on the way out.
inner.timers.lock().unwrap().clear();
return;
}
Pop::Idle { io_outstanding, wake_fd } => {
// Something is still in flight. Sleep on the appropriate
// source to avoid hammering the queue mutex; retry on wake.
let next_deadline = inner.timers.lock().unwrap().peek_deadline();
match (next_deadline, wake_fd) {
(Some(deadline), fd_opt) => {
let now = std::time::Instant::now();
if deadline > now {
let timeout = deadline - now;
match fd_opt {
Some(fd) => {
crate::io::poll_wake(fd, Some(timeout));
crate::io::drain_wake_pipe(fd);
match pop {
Pop::Got(pid) => pid,
Pop::AllDone => {
// Remaining timer entries are orphaned (no live actor can be
// woken by them — e.g. a sleeper cancelled out of its sleep);
// they must not keep the runtime alive. Drop them on the way out.
inner.timers.lock().unwrap().clear();
return;
}
Pop::Idle { io_outstanding, wake_fd } => {
// Something is still in flight. Sleep on the appropriate
// source to avoid hammering the queue mutex; retry on wake.
let next_deadline = inner.timers.lock().unwrap().peek_deadline();
match (next_deadline, wake_fd) {
(Some(deadline), fd_opt) => {
let now = std::time::Instant::now();
if deadline > now {
let timeout = deadline - now;
match fd_opt {
Some(fd) => {
crate::io::poll_wake(fd, Some(timeout));
crate::io::drain_wake_pipe(fd);
}
None => thread::sleep(timeout),
}
None => thread::sleep(timeout),
}
}
(None, Some(fd)) if io_outstanding > 0 => {
crate::io::poll_wake(fd, None);
crate::io::drain_wake_pipe(fd);
}
_ => {
thread::sleep(std::time::Duration::from_micros(100));
}
}
(None, Some(fd)) if io_outstanding > 0 => {
crate::io::poll_wake(fd, None);
crate::io::drain_wake_pipe(fd);
}
_ => {
thread::sleep(std::time::Duration::from_micros(100));
}
continue;
}
continue;
}
};
@@ -1172,7 +1300,18 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
crate::preempt::set_current_stop(stop_flag);
reset_actor_done();
YIELD_INTENT.with(|c| c.set(YieldIntent::Yield));
crate::preempt::reset_timeslice();
// RFC 005 timeslice inheritance: a slot-popped actor does NOT get a
// fresh slice — it inherits the waker's remaining one (this thread's
// TIMESLICE_START/ALLOC_COUNT carry over from the waker's run, with
// only scheduler bookkeeping in between). A chain of slot handoffs
// is therefore collectively bounded by one slice, after which
// preemption fires and the preempt-yield re-enqueue goes to the
// SHARED queue (a yield is not a wake — never slot-eligible). The
// shared queue is thus consulted at least once per slice per
// scheduler: the one-slice starvation bound, zero new counters.
if !from_slot {
crate::preempt::reset_timeslice();
}
PREEMPTION_ENABLED.with(|c| c.set(true));
crate::te!(crate::trace::Event::Resume(pid));
src/trace.rs
+5
View File
@@ -58,6 +58,9 @@ mod inner {
// Queue
Enqueue(Pid),
Dequeue(Pid),
// RFC 005 wake slot
SlotPush(Pid), // actor-context wake parked in the waking thread's slot
SlotPop(Pid), // scheduler resumed a pid from its own slot
}
// -----------------------------------------------------------------------
@@ -237,6 +240,8 @@ mod inner {
Event::RecvWake(p) => ("recv_wake".into(), p.index()),
Event::Enqueue(p) => ("enqueue".into(), p.index()),
Event::Dequeue(p) => ("dequeue".into(), p.index()),
Event::SlotPush(p) => ("slot_push".into(), p.index()),
Event::SlotPop(p) => ("slot_pop".into(), p.index()),
}
}