feat(park): scheduler coordination layer — parkers, idle mask, wake protocol (RFC 018)
Schedulers get an IO-agnostic sleep/wake primitive of their own: one futex Parker per scheduler thread (permit semantics, std::thread::park shaped — closes the check-then-park race), an AtomicU64 idle mask with a set-bit → re-check → wait park protocol, wake_one (highest-bit LIFO, CAS-clear before unpark: exactly one wakeup per call by construction), wake_all for the terminal path, and the timekeeper role — at most one parked scheduler holds the timer deadline, with an atomic armed-deadline snapshot for the busy-path due-check and an insert-side re-arm wake. Deadlines travel as nanosecond timespecs end to end; the wake pipe's as_millis truncation is unrepresentable here. The Dekker publish/re-check shape is resolved by the same-location-RMW handshake (AcqRel), not SeqCst loads; loom verifies exactly this in four models (no-lost-wake, chain propagation, timekeeper handoff, termination), run with LOOM_MAX_PREEMPTIONS=3 — unbounded exploration is impractical for the looped models. Loom/non-Linux builds park on a Mutex+Condvar via sync_shim. Standalone until the runtime swap (next commit): nothing outside tests constructs a Coordinator yet, hence the temporary dead_code allow in lib.rs.
This commit is contained in:
smarm
@@ -32,6 +32,10 @@ pub mod introspect;
|
||||
#[cfg(feature = "observer")]
|
||||
pub mod observer;
|
||||
pub mod runtime;
|
||||
// TEMPORARY dead_code allow: park is standalone until the RFC 018 runtime
|
||||
// swap (next commit) wires it into schedule_loop/enqueue; the allow dies there.
|
||||
#[allow(dead_code)]
|
||||
pub(crate) mod park;
|
||||
pub(crate) mod raw_mutex;
|
||||
pub(crate) mod slot_state;
|
||||
pub(crate) mod sync_shim;
|
||||
|
||||
+851
@@ -0,0 +1,851 @@
|
||||
//! Scheduler park/wake coordination layer (RFC 018).
|
||||
//!
|
||||
//! Schedulers never touch an fd to sleep: they park on a per-thread
|
||||
//! [`Parker`] and are woken through an idle-mask protocol the runtime owns
|
||||
//! outright. IO backends (epoll today, io_uring later) are *producers*
|
||||
//! behind a two-call contract — make actors runnable, then wake — which is
|
||||
//! what makes backend selection tractable (RFC 018 §step-back).
|
||||
//!
|
||||
//! Three pieces, all in [`Coordinator`]:
|
||||
//!
|
||||
//! - **Parker** (one per scheduler): permit semantics, `std::thread::park`
|
||||
//! shaped — an unpark delivered before the park sets a permit; the next
|
||||
//! park consumes it and returns immediately. This single property closes
|
||||
//! the check-then-park race. Linux: `futex(2)` `FUTEX_WAIT`/`FUTEX_WAKE`
|
||||
//! (private) with a nanosecond-precision relative `timespec` — the
|
||||
//! `as_millis` truncation defect of the retired wake pipe is
|
||||
//! unrepresentable here. Loom / non-Linux: `Mutex<bool>` + `Condvar`
|
||||
//! (the loom models run against this build).
|
||||
//! - **Idle mask**: an `AtomicU64` bitmask of parked scheduler ids
|
||||
//! (construction asserts ≤ 64 schedulers). Park protocol: set own bit,
|
||||
//! run the caller's mandatory post-publish re-check, then wait. A
|
||||
//! producer that published work before observing our bit has left us
|
||||
//! work the re-check finds; one that observes the bit wakes us.
|
||||
//!
|
||||
//! The publish/re-check pair is a store-buffer (Dekker) shape, and it is
|
||||
//! resolved by the **same-location-RMW handshake**, not by `SeqCst`
|
||||
//! loads: every mask read on the producer side ([`Coordinator::wake_one`],
|
||||
//! [`Coordinator::idle_mask`]) is an atomic RMW (`fetch_or(0)`). An RMW
|
||||
//! always reads the *latest* value in the mask's modification order, so
|
||||
//! a producer can never miss a bit that a consumer's `fetch_or(bit)`
|
||||
//! has already written; conversely, a consumer whose `fetch_or(bit)`
|
||||
//! lands *after* the producer's release-RMW reads-from it, acquires the
|
||||
//! producer's prior work publication, and its re-check finds the work.
|
||||
//! Both directions hold under plain acquire/release — the loom models
|
||||
//! verify exactly this (loom models `SeqCst` as `AcqRel`, so a
|
||||
//! plain-load design would — and did — fail model 1/2 with a lost wake).
|
||||
//! - **Timekeeper**: at most one parked scheduler holds the timer
|
||||
//! deadline (RFC 018 §timers) so a timer expiry wakes one scheduler,
|
||||
//! not a herd. The role is an atomic `(holder id, armed deadline)`
|
||||
//! pair; a timer insertion with an earlier deadline wakes the holder to
|
||||
//! re-peek. Arm / insert-check MUST be serialized by the caller (the
|
||||
//! timers mutex in the runtime) — the atomics exist so the *busy-path
|
||||
//! due-check* (one Relaxed load, [`Coordinator::armed_deadline_nanos`])
|
||||
//! and the wake stay lock-free. All races are biased over-wake: a
|
||||
//! spurious permit costs one failed pop; a missed wake would cost a
|
||||
//! stranded actor, and is unrepresentable under the serialization rule.
|
||||
//!
|
||||
//! Every wake here is *at most one* futex round-trip and wakes *exactly
|
||||
//! one* scheduler by construction (`wake_one` CASes a bit clear before
|
||||
//! unparking its owner) — there is no shared level-triggered anything
|
||||
//! left to herd on.
|
||||
//!
|
||||
//! Standalone until the runtime swap (RFC 018 commit 2): nothing outside
|
||||
//! tests constructs a [`Coordinator`] yet.
|
||||
|
||||
use crate::sync_shim::{AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
/// Sentinel for "no timekeeper" in the holder word.
|
||||
const NO_TIMEKEEPER: u64 = u64::MAX;
|
||||
/// Sentinel for "no armed deadline" in the deadline word. Also what the
|
||||
/// busy-path due-check compares against: `now_nanos < armed` is one branch.
|
||||
pub(crate) const NO_DEADLINE: u64 = u64::MAX;
|
||||
|
||||
/// Outcome of a [`Coordinator::park`] call.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(crate) enum ParkResult {
|
||||
/// A permit was consumed (wake delivered before or during the park).
|
||||
Woken,
|
||||
/// The deadline passed with no wake. Only possible when a deadline was
|
||||
/// supplied (and never under loom, which has no time — see `park`).
|
||||
TimedOut,
|
||||
/// The post-publish re-check found work; the thread never blocked.
|
||||
/// A permit may still be pending (a racing `wake_one` picked us after
|
||||
/// the bit was set) — it will surface as one spurious `Woken` on a
|
||||
/// later park. Benign: over-wake by design.
|
||||
WorkFound,
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Parker — permit-semantics thread parking
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Linux, non-loom: futex on a state word.
|
||||
///
|
||||
/// States: EMPTY (no permit, nobody waiting), PARKED (a thread is, or is
|
||||
/// about to be, in `futex_wait`), NOTIFIED (permit pending). The classic
|
||||
/// std-parker protocol: `unpark` swaps to NOTIFIED and futex-wakes iff it
|
||||
/// displaced PARKED; `park` CASes EMPTY→PARKED, waits, and consumes
|
||||
/// NOTIFIED on every exit path.
|
||||
#[cfg(all(target_os = "linux", not(loom)))]
|
||||
mod parker {
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
const EMPTY: u32 = 0;
|
||||
const PARKED: u32 = 1;
|
||||
const NOTIFIED: u32 = 2;
|
||||
|
||||
pub(super) struct Parker {
|
||||
state: AtomicU32,
|
||||
}
|
||||
|
||||
impl Parker {
|
||||
pub(super) fn new() -> Self {
|
||||
Self { state: AtomicU32::new(EMPTY) }
|
||||
}
|
||||
|
||||
/// Returns `true` = woken (permit consumed), `false` = timed out.
|
||||
pub(super) fn park(&self, deadline: Option<Instant>) -> bool {
|
||||
// Fast path: consume a pending permit without blocking.
|
||||
if self
|
||||
.state
|
||||
.compare_exchange(EMPTY, PARKED, Ordering::AcqRel, Ordering::Acquire)
|
||||
.is_err()
|
||||
{
|
||||
// Only NOTIFIED can be here (one thread parks at a time).
|
||||
self.state.store(EMPTY, Ordering::Release);
|
||||
return true;
|
||||
}
|
||||
loop {
|
||||
let timeout = match deadline {
|
||||
None => None,
|
||||
Some(d) => {
|
||||
let now = Instant::now();
|
||||
if d <= now {
|
||||
// Deadline passed: cancel the park. The swap
|
||||
// races a concurrent unpark — if it delivered
|
||||
// NOTIFIED first, report Woken (never lose a
|
||||
// permit).
|
||||
return self.state.swap(EMPTY, Ordering::AcqRel) == NOTIFIED;
|
||||
}
|
||||
Some(d - now)
|
||||
}
|
||||
};
|
||||
futex_wait(&self.state, PARKED, timeout);
|
||||
if self
|
||||
.state
|
||||
.compare_exchange(NOTIFIED, EMPTY, Ordering::AcqRel, Ordering::Acquire)
|
||||
.is_ok()
|
||||
{
|
||||
return true;
|
||||
}
|
||||
// Spurious wake or timeout with state still PARKED: loop —
|
||||
// the deadline check at the top decides.
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn unpark(&self) {
|
||||
if self.state.swap(NOTIFIED, Ordering::AcqRel) == PARKED {
|
||||
futex_wake(&self.state, 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `FUTEX_WAIT` with a *relative* nanosecond timeout (`CLOCK_MONOTONIC`
|
||||
/// per futex(2) for relative waits). No millisecond conversion anywhere:
|
||||
/// the timespec carries the full sub-ms remainder (RFC 018 kills the
|
||||
/// `as_millis` truncation structurally).
|
||||
fn futex_wait(word: &AtomicU32, expected: u32, timeout: Option<std::time::Duration>) {
|
||||
let ts;
|
||||
let ts_ptr: *const libc::timespec = match timeout {
|
||||
Some(d) => {
|
||||
ts = libc::timespec {
|
||||
tv_sec: d.as_secs() as libc::time_t,
|
||||
tv_nsec: d.subsec_nanos() as libc::c_long,
|
||||
};
|
||||
&ts
|
||||
}
|
||||
None => std::ptr::null(),
|
||||
};
|
||||
// Errors (EAGAIN: word changed; ETIMEDOUT; EINTR) all mean "return
|
||||
// and let the caller's state machine decide" — deliberately ignored.
|
||||
unsafe {
|
||||
libc::syscall(
|
||||
libc::SYS_futex,
|
||||
word.as_ptr(),
|
||||
libc::FUTEX_WAIT | libc::FUTEX_PRIVATE_FLAG,
|
||||
expected,
|
||||
ts_ptr,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn futex_wake(word: &AtomicU32, n: u32) {
|
||||
unsafe {
|
||||
libc::syscall(
|
||||
libc::SYS_futex,
|
||||
word.as_ptr(),
|
||||
libc::FUTEX_WAKE | libc::FUTEX_PRIVATE_FLAG,
|
||||
n,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Loom / non-Linux: `Mutex<bool>` permit + `Condvar` — loom's own model
|
||||
/// of a parker, and the portable fallback. Under loom the deadline is
|
||||
/// ignored (loom has no time); the models exercise wake paths only.
|
||||
#[cfg(any(loom, not(target_os = "linux")))]
|
||||
mod parker {
|
||||
use crate::sync_shim::{Condvar, Mutex};
|
||||
use std::time::Instant;
|
||||
|
||||
pub(super) struct Parker {
|
||||
permit: Mutex<bool>,
|
||||
cv: Condvar,
|
||||
}
|
||||
|
||||
impl Parker {
|
||||
pub(super) fn new() -> Self {
|
||||
Self { permit: Mutex::new(false), cv: Condvar::new() }
|
||||
}
|
||||
|
||||
/// Returns `true` = woken (permit consumed), `false` = timed out.
|
||||
pub(super) fn park(&self, deadline: Option<Instant>) -> bool {
|
||||
let mut permit = match self.permit.lock() {
|
||||
Ok(g) => g,
|
||||
Err(_) => panic!("smarm: parker permit lock poisoned (core corrupt)"),
|
||||
};
|
||||
loop {
|
||||
if *permit {
|
||||
*permit = false;
|
||||
return true;
|
||||
}
|
||||
#[cfg(loom)]
|
||||
{
|
||||
// Loom has no clock: block until a wake. Models must
|
||||
// deliver one (a park nobody wakes is a real deadlock
|
||||
// and loom reports it as such).
|
||||
let _ = deadline;
|
||||
permit = match self.cv.wait(permit) {
|
||||
Ok(g) => g,
|
||||
Err(_) => panic!("smarm: parker cv poisoned (core corrupt)"),
|
||||
};
|
||||
}
|
||||
#[cfg(not(loom))]
|
||||
{
|
||||
match deadline {
|
||||
None => {
|
||||
permit = match self.cv.wait(permit) {
|
||||
Ok(g) => g,
|
||||
Err(_) => panic!("smarm: parker cv poisoned (core corrupt)"),
|
||||
};
|
||||
}
|
||||
Some(d) => {
|
||||
let now = Instant::now();
|
||||
if d <= now {
|
||||
return false;
|
||||
}
|
||||
permit = match self.cv.wait_timeout(permit, d - now) {
|
||||
Ok((g, _)) => g,
|
||||
Err(_) => panic!("smarm: parker cv poisoned (core corrupt)"),
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn unpark(&self) {
|
||||
let mut permit = match self.permit.lock() {
|
||||
Ok(g) => g,
|
||||
Err(_) => panic!("smarm: parker permit lock poisoned (core corrupt)"),
|
||||
};
|
||||
*permit = true;
|
||||
self.cv.notify_one();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
use parker::Parker;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Coordinator — idle mask + wake protocol + timekeeper
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub(crate) struct Coordinator {
|
||||
parkers: Box<[Parker]>,
|
||||
/// Bit `i` set = scheduler `i` is parked or committed to parking (set
|
||||
/// before the re-check; cleared by `wake_one`'s CAS or by the parker
|
||||
/// itself on return). AcqRel same-location-RMW handshake — see module
|
||||
/// docs (no SeqCst needed: every producer-side read is an RMW).
|
||||
idle: AtomicU64,
|
||||
/// Timekeeper holder id, or `NO_TIMEKEEPER`. Written under the
|
||||
/// caller's timer serialization (arm/disarm/insert-check); read
|
||||
/// lock-free by the insert wake path.
|
||||
tk_holder: AtomicU64,
|
||||
/// Armed deadline as nanos since `origin`, or `NO_DEADLINE`. The
|
||||
/// busy-path due-check reads this Relaxed.
|
||||
tk_armed: AtomicU64,
|
||||
/// Time origin for the nanos encoding.
|
||||
origin: Instant,
|
||||
}
|
||||
|
||||
impl Coordinator {
|
||||
pub(crate) fn new(schedulers: usize) -> Self {
|
||||
assert!(
|
||||
(1..=64).contains(&schedulers),
|
||||
"smarm: scheduler count must be 1..=64 (idle mask is one u64); got {schedulers}"
|
||||
);
|
||||
Self {
|
||||
parkers: (0..schedulers).map(|_| Parker::new()).collect(),
|
||||
idle: AtomicU64::new(0),
|
||||
tk_holder: AtomicU64::new(NO_TIMEKEEPER),
|
||||
tk_armed: AtomicU64::new(NO_DEADLINE),
|
||||
origin: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Encode a deadline for the armed snapshot / busy-path compare.
|
||||
/// Saturating: a deadline at-or-before `origin` encodes as 0 (always
|
||||
/// due), one beyond ~584 years as `NO_DEADLINE - 1`.
|
||||
pub(crate) fn deadline_nanos(&self, deadline: Instant) -> u64 {
|
||||
let nanos = deadline
|
||||
.checked_duration_since(self.origin)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
if nanos >= NO_DEADLINE as u128 {
|
||||
NO_DEADLINE - 1
|
||||
} else {
|
||||
nanos as u64
|
||||
}
|
||||
}
|
||||
|
||||
/// Park scheduler `id` until a wake, the deadline, or a positive
|
||||
/// re-check. Protocol: (1) publish own idle bit (SeqCst), (2) run
|
||||
/// `recheck` — it MUST re-read the work source with an ordering that
|
||||
/// pairs with the producer's publish (SeqCst load, or take the mutex
|
||||
/// the producer publishes under); a `true` aborts the park, (3) block.
|
||||
pub(crate) fn park(
|
||||
&self,
|
||||
id: usize,
|
||||
deadline: Option<Instant>,
|
||||
recheck: impl FnOnce() -> bool,
|
||||
) -> ParkResult {
|
||||
debug_assert!(id < self.parkers.len(), "park: scheduler id out of range");
|
||||
let bit = 1u64 << id;
|
||||
// (1) publish. AcqRel RMW: the acquire half is the handshake — if
|
||||
// this lands after a producer's mask RMW in modification order, we
|
||||
// read-from it and the producer's earlier work publication is
|
||||
// visible to the re-check below (see module docs).
|
||||
let prev = self.idle.fetch_or(bit, Ordering::AcqRel);
|
||||
debug_assert_eq!(prev & bit, 0, "park: idle bit already set for this id");
|
||||
// (2) the mandatory post-publish re-check.
|
||||
if recheck() {
|
||||
self.idle.fetch_and(!bit, Ordering::AcqRel);
|
||||
return ParkResult::WorkFound;
|
||||
}
|
||||
// (3) block. The permit closes the window between the re-check and
|
||||
// the futex wait: a wake_one that picked us in that window has
|
||||
// already CASed our bit clear and set the permit.
|
||||
let woken = self.parkers[id].park(deadline);
|
||||
// Clear own bit — a no-op when a waker already CASed it clear.
|
||||
self.idle.fetch_and(!bit, Ordering::AcqRel);
|
||||
if woken {
|
||||
ParkResult::Woken
|
||||
} else {
|
||||
ParkResult::TimedOut
|
||||
}
|
||||
}
|
||||
|
||||
/// Wake exactly one parked scheduler, if any: pick the highest set idle
|
||||
/// bit (LIFO-ish — warmest cache), CAS it clear, deliver a permit.
|
||||
/// Empty mask = no-op (everyone is awake and will find work by
|
||||
/// popping). Returns whether a scheduler was woken.
|
||||
pub(crate) fn wake_one(&self) -> bool {
|
||||
// RMW read, not a load: reads the latest mask by modification-order
|
||||
// coherence, closing the Dekker race with a parking consumer (see
|
||||
// module docs). The release side of the RMW is what a later-parking
|
||||
// consumer's fetch_or acquires to make its re-check sound.
|
||||
let mut mask = self.idle.fetch_or(0, Ordering::AcqRel);
|
||||
loop {
|
||||
if mask == 0 {
|
||||
return false;
|
||||
}
|
||||
let id = 63 - mask.leading_zeros() as usize; // highest set bit
|
||||
let bit = 1u64 << id;
|
||||
// The CAS is the exactly-one guarantee: whoever clears the bit
|
||||
// owns the wake; a racing wake_one retries on the observed value
|
||||
// (coherence: a failed CAS can never read older than `mask`).
|
||||
match self.idle.compare_exchange(
|
||||
mask,
|
||||
mask & !bit,
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
) {
|
||||
Ok(_) => {
|
||||
self.parkers[id].unpark();
|
||||
return true;
|
||||
}
|
||||
Err(m) => mask = m,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Terminal wake (replaces the AllDone wake-pipe byte): clear the mask
|
||||
/// and deliver a permit to *every* parker, parked or not. A permit set
|
||||
/// on a busy scheduler costs one spurious park return — nothing at the
|
||||
/// terminal boundary. Idempotent.
|
||||
pub(crate) fn wake_all(&self) {
|
||||
self.idle.store(0, Ordering::Release);
|
||||
for p in self.parkers.iter() {
|
||||
p.unpark();
|
||||
}
|
||||
}
|
||||
|
||||
/// Latest idle mask (RMW read — same handshake as `wake_one`; used by
|
||||
/// the chain rule: "queue non-empty and mask non-zero ⇒ wake_one
|
||||
/// again", where missing a just-parked sibling would strand its item).
|
||||
pub(crate) fn idle_mask(&self) -> u64 {
|
||||
self.idle.fetch_or(0, Ordering::AcqRel)
|
||||
}
|
||||
|
||||
// ----- timekeeper -----
|
||||
|
||||
/// Try to take the timekeeper role for scheduler `id` with `deadline`.
|
||||
/// MUST be called under the caller's timer serialization (the timers
|
||||
/// mutex), with `deadline` the heap minimum peeked under that same
|
||||
/// hold — this is what makes the insert-check race-free. Returns
|
||||
/// whether the role was taken (false = someone else holds it; park
|
||||
/// with no deadline).
|
||||
pub(crate) fn try_arm_timer(&self, id: usize, deadline: Instant) -> bool {
|
||||
debug_assert!(id < self.parkers.len(), "try_arm_timer: id out of range");
|
||||
if self
|
||||
.tk_holder
|
||||
.compare_exchange(NO_TIMEKEEPER, id as u64, Ordering::SeqCst, Ordering::SeqCst)
|
||||
.is_err()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
// Holder-then-deadline order: an insert-check that sees the holder
|
||||
// with the deadline still NO_DEADLINE compares `new < MAX` = true
|
||||
// and over-wakes — the benign direction. (Under the mandated timer
|
||||
// serialization this interleaving cannot occur anyway.)
|
||||
self.tk_armed.store(self.deadline_nanos(deadline), Ordering::SeqCst);
|
||||
true
|
||||
}
|
||||
|
||||
/// Release the timekeeper role (the holder, on wake, before it
|
||||
/// re-peeks/fires). Callable without the timer serialization: a
|
||||
/// racing insert may wake a no-longer-holder — over-wake, benign.
|
||||
pub(crate) fn disarm_timer(&self, id: usize) {
|
||||
debug_assert_eq!(
|
||||
self.tk_holder.load(Ordering::SeqCst),
|
||||
id as u64,
|
||||
"disarm_timer by a non-holder"
|
||||
);
|
||||
// Deadline first: a concurrent insert-check then sees NO_DEADLINE
|
||||
// and skips the (now pointless) wake instead of waking a stale
|
||||
// holder id. Either order is correct; this one wastes less.
|
||||
self.tk_armed.store(NO_DEADLINE, Ordering::SeqCst);
|
||||
self.tk_holder.store(NO_TIMEKEEPER, Ordering::SeqCst);
|
||||
}
|
||||
|
||||
/// Insert-side re-arm check: if `deadline` is earlier than the armed
|
||||
/// snapshot, wake the timekeeper to re-peek. MUST be called under the
|
||||
/// same timer serialization as `try_arm_timer` (see there).
|
||||
pub(crate) fn timer_inserted(&self, deadline: Instant) {
|
||||
if self.deadline_nanos(deadline) < self.tk_armed.load(Ordering::SeqCst) {
|
||||
let holder = self.tk_holder.load(Ordering::SeqCst);
|
||||
if holder != NO_TIMEKEEPER {
|
||||
// Direct unpark, not wake_one: the wake targets the
|
||||
// timekeeper specifically (it must re-peek the heap). Its
|
||||
// idle bit stays set until it returns from park — a
|
||||
// concurrent wake_one may pick it too; over-wake, benign.
|
||||
self.parkers[holder as usize].unpark();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The armed-deadline snapshot (nanos since origin; `NO_DEADLINE` =
|
||||
/// none). One Relaxed load — the busy-path due-check (RFC 018 design
|
||||
/// point (a)) reads this every scheduler-loop iteration.
|
||||
pub(crate) fn armed_deadline_nanos(&self) -> u64 {
|
||||
self.tk_armed.load(Ordering::Relaxed)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Unit tests (std build)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[cfg(all(test, not(loom)))]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering as O};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
#[test]
|
||||
fn permit_before_park_returns_immediately() {
|
||||
let c = Coordinator::new(1);
|
||||
// Deliver the wake first (nobody parked: wake_one no-ops on the
|
||||
// mask, so use the timekeeper-direct path? No — permit semantics
|
||||
// are the parker's own; exercise via wake_all which permits all).
|
||||
c.wake_all();
|
||||
let t0 = Instant::now();
|
||||
let r = c.park(0, None, || false);
|
||||
assert_eq!(r, ParkResult::Woken);
|
||||
assert!(t0.elapsed() < Duration::from_millis(100), "park blocked despite permit");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn submillisecond_deadline_is_honored() {
|
||||
// Regression for the retired as_millis truncation: a 500µs deadline
|
||||
// must neither busy-return instantly forever nor round to 0/∞.
|
||||
let c = Coordinator::new(1);
|
||||
let t0 = Instant::now();
|
||||
let r = c.park(0, Some(t0 + Duration::from_micros(500)), || false);
|
||||
let dt = t0.elapsed();
|
||||
assert_eq!(r, ParkResult::TimedOut);
|
||||
assert!(dt >= Duration::from_micros(400), "woke too early: {dt:?}");
|
||||
assert!(dt < Duration::from_millis(50), "overslept: {dt:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recheck_true_aborts_park_and_clears_bit() {
|
||||
let c = Coordinator::new(2);
|
||||
let r = c.park(1, None, || true);
|
||||
assert_eq!(r, ParkResult::WorkFound);
|
||||
assert_eq!(c.idle_mask(), 0, "bit not cleared after WorkFound");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recheck_observes_own_bit_published() {
|
||||
let c = Coordinator::new(2);
|
||||
let seen = std::cell::Cell::new(0u64);
|
||||
let r = c.park(1, None, || {
|
||||
seen.set(c.idle_mask());
|
||||
true
|
||||
});
|
||||
assert_eq!(r, ParkResult::WorkFound);
|
||||
assert_eq!(seen.get() & 0b10, 0b10, "bit not published before re-check");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wake_one_wakes_exactly_one_of_n() {
|
||||
const N: usize = 4;
|
||||
let c = Arc::new(Coordinator::new(N));
|
||||
let woken = Arc::new(AtomicUsize::new(0));
|
||||
let mut ts = Vec::new();
|
||||
for id in 0..N {
|
||||
let c = c.clone();
|
||||
let woken = woken.clone();
|
||||
ts.push(std::thread::spawn(move || {
|
||||
let r = c.park(id, None, || false);
|
||||
assert_eq!(r, ParkResult::Woken);
|
||||
woken.fetch_add(1, O::SeqCst);
|
||||
}));
|
||||
}
|
||||
// Wait until all four are published idle.
|
||||
let t0 = Instant::now();
|
||||
while c.idle_mask().count_ones() != N as u32 {
|
||||
assert!(t0.elapsed() < Duration::from_secs(5), "threads never parked");
|
||||
std::thread::yield_now();
|
||||
}
|
||||
assert!(c.wake_one());
|
||||
// Exactly one wakes; give the others a beat to (incorrectly) wake.
|
||||
let t0 = Instant::now();
|
||||
while woken.load(O::SeqCst) == 0 {
|
||||
assert!(t0.elapsed() < Duration::from_secs(5), "wake_one woke nobody");
|
||||
std::thread::yield_now();
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(100));
|
||||
assert_eq!(woken.load(O::SeqCst), 1, "wake_one woke more than one");
|
||||
assert_eq!(c.idle_mask().count_ones(), (N - 1) as u32);
|
||||
c.wake_all();
|
||||
for t in ts {
|
||||
match t.join() {
|
||||
Ok(()) => {}
|
||||
Err(p) => std::panic::resume_unwind(p),
|
||||
}
|
||||
}
|
||||
assert_eq!(woken.load(O::SeqCst), N);
|
||||
assert_eq!(c.idle_mask(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wake_one_prefers_highest_bit() {
|
||||
let c = Arc::new(Coordinator::new(3));
|
||||
let woken_id = Arc::new(AtomicUsize::new(usize::MAX));
|
||||
let mut ts = Vec::new();
|
||||
for id in 0..3 {
|
||||
let c = c.clone();
|
||||
let woken_id = woken_id.clone();
|
||||
ts.push(std::thread::spawn(move || {
|
||||
if c.park(id, None, || false) == ParkResult::Woken {
|
||||
let _ = woken_id.compare_exchange(usize::MAX, id, O::SeqCst, O::SeqCst);
|
||||
}
|
||||
}));
|
||||
}
|
||||
let t0 = Instant::now();
|
||||
while c.idle_mask() != 0b111 {
|
||||
assert!(t0.elapsed() < Duration::from_secs(5));
|
||||
std::thread::yield_now();
|
||||
}
|
||||
assert!(c.wake_one());
|
||||
let t0 = Instant::now();
|
||||
while woken_id.load(O::SeqCst) == usize::MAX {
|
||||
assert!(t0.elapsed() < Duration::from_secs(5));
|
||||
std::thread::yield_now();
|
||||
}
|
||||
assert_eq!(woken_id.load(O::SeqCst), 2, "LIFO-ish: highest bit first");
|
||||
c.wake_all();
|
||||
for t in ts {
|
||||
let _ = t.join();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wake_one_on_empty_mask_is_noop() {
|
||||
let c = Coordinator::new(2);
|
||||
assert!(!c.wake_one());
|
||||
assert_eq!(c.idle_mask(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn timekeeper_arm_is_exclusive_and_snapshot_readable() {
|
||||
let c = Coordinator::new(2);
|
||||
let d2 = Instant::now() + Duration::from_secs(10);
|
||||
let d1 = Instant::now() + Duration::from_secs(1);
|
||||
assert_eq!(c.armed_deadline_nanos(), NO_DEADLINE);
|
||||
assert!(c.try_arm_timer(0, d2));
|
||||
assert!(!c.try_arm_timer(1, d1), "second arm must fail while held");
|
||||
assert_eq!(c.armed_deadline_nanos(), c.deadline_nanos(d2));
|
||||
c.disarm_timer(0);
|
||||
assert_eq!(c.armed_deadline_nanos(), NO_DEADLINE);
|
||||
assert!(c.try_arm_timer(1, d1), "role must be re-takeable after disarm");
|
||||
c.disarm_timer(1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn earlier_insert_wakes_timekeeper() {
|
||||
let c = Coordinator::new(2);
|
||||
let far = Instant::now() + Duration::from_secs(60);
|
||||
let near = Instant::now() + Duration::from_millis(1);
|
||||
assert!(c.try_arm_timer(0, far));
|
||||
// Holder not yet parked: the wake must land as a permit.
|
||||
c.timer_inserted(near);
|
||||
let t0 = Instant::now();
|
||||
let r = c.park(0, Some(far), || false);
|
||||
assert_eq!(r, ParkResult::Woken, "re-arm wake lost");
|
||||
assert!(t0.elapsed() < Duration::from_secs(5), "slept toward the stale deadline");
|
||||
c.disarm_timer(0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn later_insert_does_not_wake_timekeeper() {
|
||||
let c = Coordinator::new(2);
|
||||
let near = Instant::now() + Duration::from_millis(20);
|
||||
let far = Instant::now() + Duration::from_secs(60);
|
||||
assert!(c.try_arm_timer(0, near));
|
||||
c.timer_inserted(far); // later than armed: no wake
|
||||
let r = c.park(0, Some(near), || false);
|
||||
assert_eq!(r, ParkResult::TimedOut, "spurious wake for a later insert");
|
||||
c.disarm_timer(0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[should_panic(expected = "1..=64")]
|
||||
fn more_than_64_schedulers_asserts() {
|
||||
let _ = Coordinator::new(65);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// loom models — RUSTFLAGS="--cfg loom" cargo test --lib --release park
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[cfg(all(test, loom))]
|
||||
mod loom_tests {
|
||||
use super::*;
|
||||
use loom::sync::atomic::{AtomicU64 as LAtomicU64, Ordering as O};
|
||||
use loom::sync::{Arc, Mutex as LMutex};
|
||||
use loom::thread;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
/// RFC 018 loom model 1 — no lost wake.
|
||||
/// producer{publish item; wake_one} ∥ consumer{set bit; re-check; park}:
|
||||
/// the consumer always observes the item or a permit; it can never
|
||||
/// sleep past a published item (loom's deadlock detector is the
|
||||
/// assertion — a consumer parked forever fails the model).
|
||||
#[test]
|
||||
fn no_lost_wake() {
|
||||
loom::model(|| {
|
||||
let c = Arc::new(Coordinator::new(1));
|
||||
let item = Arc::new(LAtomicU64::new(0));
|
||||
|
||||
let prod = {
|
||||
let c = c.clone();
|
||||
let item = item.clone();
|
||||
thread::spawn(move || {
|
||||
item.store(1, O::SeqCst);
|
||||
c.wake_one();
|
||||
})
|
||||
};
|
||||
|
||||
// Consumer: loop until the item is popped. Guarded load+CAS,
|
||||
// not a blind swap — a swap writes 0 even when empty, and
|
||||
// coherence allows that write to land after the producer's
|
||||
// store in modification order, destroying the item (a model
|
||||
// bug loom caught in an earlier draft of this test).
|
||||
loop {
|
||||
if item.load(O::SeqCst) == 1
|
||||
&& item.compare_exchange(1, 0, O::SeqCst, O::SeqCst).is_ok()
|
||||
{
|
||||
break;
|
||||
}
|
||||
let _ = c.park(0, None, || item.load(O::SeqCst) == 1);
|
||||
}
|
||||
prod.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
/// RFC 018 loom model 2 — chain propagation.
|
||||
/// Two items, two sleepers, ONE producer wake: the chain rule (a woken
|
||||
/// consumer that sees surplus work and a non-empty mask wakes again)
|
||||
/// must get both items consumed with no further producer action.
|
||||
#[test]
|
||||
fn chain_propagation() {
|
||||
loom::model(|| {
|
||||
let c = Arc::new(Coordinator::new(2));
|
||||
let items = Arc::new(LAtomicU64::new(0));
|
||||
let consumed = Arc::new(LAtomicU64::new(0));
|
||||
|
||||
let mut hs = Vec::new();
|
||||
for id in 0..2usize {
|
||||
let c = c.clone();
|
||||
let items = items.clone();
|
||||
let consumed = consumed.clone();
|
||||
hs.push(thread::spawn(move || loop {
|
||||
if consumed.load(O::SeqCst) == 2 {
|
||||
c.wake_all(); // release a sibling still parked
|
||||
break;
|
||||
}
|
||||
let cur = items.load(O::SeqCst);
|
||||
if cur > 0
|
||||
&& items
|
||||
.compare_exchange(cur, cur - 1, O::SeqCst, O::SeqCst)
|
||||
.is_ok()
|
||||
{
|
||||
consumed.fetch_add(1, O::SeqCst);
|
||||
// THE CHAIN RULE: surplus + idle sibling ⇒ wake.
|
||||
if items.load(O::SeqCst) > 0 && c.idle_mask() != 0 {
|
||||
c.wake_one();
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let _ = c.park(id, None, || {
|
||||
items.load(O::SeqCst) > 0 || consumed.load(O::SeqCst) == 2
|
||||
});
|
||||
}));
|
||||
}
|
||||
|
||||
// Producer (main): two items, ONE wake.
|
||||
items.store(2, O::SeqCst);
|
||||
c.wake_one();
|
||||
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
assert_eq!(consumed.load(O::SeqCst), 2);
|
||||
assert_eq!(items.load(O::SeqCst), 0);
|
||||
});
|
||||
}
|
||||
|
||||
/// RFC 018 loom model 3 — timekeeper handoff.
|
||||
/// An earlier-deadline insert racing the parking timekeeper: the
|
||||
/// earlier deadline is always honored — either the timekeeper armed it
|
||||
/// directly (insert landed first under the timers lock) or the insert
|
||||
/// wakes the timekeeper to re-peek. A timekeeper sleeping toward the
|
||||
/// stale later deadline would deadlock the model (loom has no time).
|
||||
#[test]
|
||||
fn timekeeper_handoff() {
|
||||
loom::model(|| {
|
||||
let origin = Instant::now();
|
||||
let d_far = origin + Duration::from_secs(60);
|
||||
let d_near = origin + Duration::from_secs(1);
|
||||
|
||||
let c = Arc::new(Coordinator::new(1));
|
||||
// The timers-mutex stand-in: heap min under a lock.
|
||||
let heap_min = Arc::new(LMutex::new(d_far));
|
||||
|
||||
let tk = {
|
||||
let c = c.clone();
|
||||
let heap_min = heap_min.clone();
|
||||
thread::spawn(move || {
|
||||
// Peek + arm under the lock (the serialization rule).
|
||||
let armed = {
|
||||
let g = heap_min.lock().unwrap();
|
||||
let min = *g;
|
||||
assert!(c.try_arm_timer(0, min));
|
||||
min
|
||||
};
|
||||
if armed == d_far {
|
||||
// Insert hadn't landed: it MUST wake us. Parking
|
||||
// toward d_far with no wake = model deadlock.
|
||||
let r = c.park(0, Some(armed), || false);
|
||||
assert_eq!(r, ParkResult::Woken, "re-arm wake lost");
|
||||
}
|
||||
// Woken (or armed the near deadline directly): re-peek.
|
||||
c.disarm_timer(0);
|
||||
let g = heap_min.lock().unwrap();
|
||||
assert_eq!(*g, d_near, "earlier deadline not visible on re-peek");
|
||||
})
|
||||
};
|
||||
|
||||
// Inserter (main): publish the earlier deadline under the lock,
|
||||
// then the insert-check.
|
||||
{
|
||||
let mut g = heap_min.lock().unwrap();
|
||||
*g = d_near;
|
||||
c.timer_inserted(d_near);
|
||||
}
|
||||
|
||||
tk.join().unwrap();
|
||||
});
|
||||
}
|
||||
|
||||
/// RFC 018 loom model 4 — termination.
|
||||
/// The AllDone verdict: producer flips done and wake_all()s; consumers
|
||||
/// must never park forever past done (park's re-check + wake_all's
|
||||
/// permits close every interleaving; a stuck consumer = loom deadlock).
|
||||
#[test]
|
||||
fn termination_no_park_past_done() {
|
||||
loom::model(|| {
|
||||
let c = Arc::new(Coordinator::new(2));
|
||||
let done = Arc::new(LAtomicU64::new(0));
|
||||
|
||||
let mut hs = Vec::new();
|
||||
for id in 0..2usize {
|
||||
let c = c.clone();
|
||||
let done = done.clone();
|
||||
hs.push(thread::spawn(move || loop {
|
||||
if done.load(O::SeqCst) == 1 {
|
||||
break;
|
||||
}
|
||||
let _ = c.park(id, None, || done.load(O::SeqCst) == 1);
|
||||
}));
|
||||
}
|
||||
|
||||
done.store(1, O::SeqCst);
|
||||
c.wake_all();
|
||||
|
||||
for h in hs {
|
||||
h.join().unwrap();
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,15 @@ pub(crate) use loom::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
|
||||
#[cfg(not(loom))]
|
||||
pub(crate) use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
|
||||
|
||||
// park.rs condvar-parker (loom + non-Linux builds only; the Linux non-loom
|
||||
// build parks on a futex and never touches these — gating them identically
|
||||
// keeps the default build free of unused imports).
|
||||
#[cfg(loom)]
|
||||
pub(crate) use loom::sync::{Condvar, Mutex};
|
||||
|
||||
#[cfg(all(not(loom), not(target_os = "linux")))]
|
||||
pub(crate) use std::sync::{Condvar, Mutex};
|
||||
|
||||
/// `UnsafeCell` with loom's `with`/`with_mut` access API; pass-through cost
|
||||
/// is zero in normal builds (`#[inline]`, newtype over std's cell).
|
||||
#[cfg(loom)]
|
||||
|
||||
Reference in New Issue
Block a user