feat(runtime,io): driver-enqueues + park/wake idle path — retire the wake pipe

The swap (RFC 018). Schedulers no longer sleep on a shared level-triggered
wake pipe — the herd source that made the default 8-thread config 7x
slower than 2 threads (E1). They park on per-thread futex parkers via the
coordination layer; IO backends become producers behind a two-call
contract (make runnable, then the enqueue tail wakes exactly one parked
scheduler).

Deleted: the drain lock and the one-winner phase-1 drain; the shared
completions VecDeque; the wake pipe fds, poll_wake, drain_wake_pipe,
wake_scheduler, the FdReady/Blocking Completion enum; the 100us idle nap;
the per-pop io.lock liveness read; io.rs's as_millis timeout truncation.

Added:
- enqueue wake tail (fixes the silent enqueue): wake_one_if_idle, a fence
  + one Relaxed mask load when everyone is busy — the pure-compute hot
  path pays almost nothing.
- driver-enqueues: the pool thread stashes its result in the slot,
  decrements io_outstanding, unparks; the epoll thread removes+DELs the
  waiter under the waiters lock and unparks. Both reach the runtime via a
  Weak (no Arc cycle). The waiters map moves behind its own Arc<Mutex> so
  the epoll thread never takes the runtime io lock (teardown holds it
  while joining that thread).
- io_outstanding / io_fd_waiters atomics: the termination verdict reads
  two atomics instead of taking io.lock on every pop.
- timekeeper idle path: at most one parked scheduler holds the timer
  deadline (an expiry wakes one, not a herd); everyone else parks
  indefinitely and is woken by the enqueue tail.
- busy-path timer due-check (ratified design point (a)): under saturation
  nobody parks and no timekeeper exists, yet due timers must still fire —
  one Relaxed load of the earliest-deadline snapshot per loop, clock read
  only when a timer is armed. Maintained under the timers mutex.
- chain rule: a scheduler that pops with more work queued and a sibling
  parked wakes one, so surplus runs in parallel rather than behind it.

tests/park_wake.rs pins the two new observable properties: timers fire
under full scheduler saturation, and sub-ms sleeps are prompt (the
as_millis truncation regression). Full suite + all loom models green;
clippy --lib clean.
This commit is contained in:
smarm
2026-07-24 09:12:12 +02:00
parent 2854b560d6
commit d4839f1d81
7 changed files with 522 additions and 455 deletions
+37 -1
View File
@@ -141,6 +141,14 @@ impl PartialOrd for Entry {
#[derive(Default)]
pub struct Timers {
/// RFC 018: the scheduler coordination layer. Attached once at
/// `RuntimeInner::new`; every insert notes its deadline (min-maintained
/// snapshot for the busy-path due-check + the timekeeper re-arm wake)
/// and every pop/clear re-anchors the snapshot to the heap minimum.
/// All calls happen under the timers mutex — the serialization the
/// coordinator's timer protocol mandates. `None` only in unit tests
/// that construct a bare `Timers`.
coord: Option<std::sync::Arc<crate::park::Coordinator>>,
/// Reverse-wrapped so the smallest deadline is at the top.
heap: BinaryHeap<Reverse<Entry>>,
/// Monotonic counter for the tiebreaker `seq` field (and the `TimerId` of a
@@ -157,7 +165,18 @@ pub struct Timers {
impl Timers {
pub fn new() -> Self {
Self { heap: BinaryHeap::new(), next_seq: 0, armed: std::collections::HashSet::new() }
Self {
coord: None,
heap: BinaryHeap::new(),
next_seq: 0,
armed: std::collections::HashSet::new(),
}
}
/// Attach the scheduler coordination layer (RFC 018). Called once, at
/// runtime construction, before any scheduler thread exists.
pub(crate) fn attach_coordinator(&mut self, c: std::sync::Arc<crate::park::Coordinator>) {
self.coord = Some(c);
}
/// Insert a `Sleep` timer. Convenience for the common case.
@@ -242,6 +261,13 @@ impl Timers {
#[cfg(feature = "smarm-causal")]
wall,
}));
// RFC 018: publish the (possibly new-minimum) deadline to the
// busy-path snapshot and wake the timekeeper if it is parked
// toward a later one. We hold the timers mutex — the mandated
// serialization for both.
if let Some(c) = &self.coord {
c.note_deadline(deadline);
}
seq
}
@@ -255,6 +281,9 @@ impl Timers {
pub fn clear(&mut self) {
self.heap.clear();
self.armed.clear();
if let Some(c) = &self.coord {
c.refresh_deadline(None);
}
}
/// Soonest pending deadline, or `None` if the heap is empty.
@@ -324,6 +353,13 @@ impl Timers {
}
out.push(entry);
}
// RFC 018: re-anchor the busy-path snapshot to the new heap minimum
// (still under the timers mutex). A causal-shift re-queue above went
// through `heap.push` directly, so this peek is the one place the
// snapshot is guaranteed to catch up.
if let Some(c) = &self.coord {
c.refresh_deadline(self.peek_deadline());
}
out
}
}