feat(runtime,io): driver-enqueues + park/wake idle path — retire the wake pipe
The swap (RFC 018). Schedulers no longer sleep on a shared level-triggered wake pipe — the herd source that made the default 8-thread config 7x slower than 2 threads (E1). They park on per-thread futex parkers via the coordination layer; IO backends become producers behind a two-call contract (make runnable, then the enqueue tail wakes exactly one parked scheduler). Deleted: the drain lock and the one-winner phase-1 drain; the shared completions VecDeque; the wake pipe fds, poll_wake, drain_wake_pipe, wake_scheduler, the FdReady/Blocking Completion enum; the 100us idle nap; the per-pop io.lock liveness read; io.rs's as_millis timeout truncation. Added: - enqueue wake tail (fixes the silent enqueue): wake_one_if_idle, a fence + one Relaxed mask load when everyone is busy — the pure-compute hot path pays almost nothing. - driver-enqueues: the pool thread stashes its result in the slot, decrements io_outstanding, unparks; the epoll thread removes+DELs the waiter under the waiters lock and unparks. Both reach the runtime via a Weak (no Arc cycle). The waiters map moves behind its own Arc<Mutex> so the epoll thread never takes the runtime io lock (teardown holds it while joining that thread). - io_outstanding / io_fd_waiters atomics: the termination verdict reads two atomics instead of taking io.lock on every pop. - timekeeper idle path: at most one parked scheduler holds the timer deadline (an expiry wakes one, not a herd); everyone else parks indefinitely and is woken by the enqueue tail. - busy-path timer due-check (ratified design point (a)): under saturation nobody parks and no timekeeper exists, yet due timers must still fire — one Relaxed load of the earliest-deadline snapshot per loop, clock read only when a timer is armed. Maintained under the timers mutex. - chain rule: a scheduler that pops with more work queued and a sibling parked wakes one, so surplus runs in parallel rather than behind it. tests/park_wake.rs pins the two new observable properties: timers fire under full scheduler saturation, and sub-ms sleeps are prompt (the as_millis truncation regression). Full suite + all loom models green; clippy --lib clean.
This commit is contained in:
+37
-1
@@ -141,6 +141,14 @@ impl PartialOrd for Entry {
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct Timers {
|
||||
/// RFC 018: the scheduler coordination layer. Attached once at
|
||||
/// `RuntimeInner::new`; every insert notes its deadline (min-maintained
|
||||
/// snapshot for the busy-path due-check + the timekeeper re-arm wake)
|
||||
/// and every pop/clear re-anchors the snapshot to the heap minimum.
|
||||
/// All calls happen under the timers mutex — the serialization the
|
||||
/// coordinator's timer protocol mandates. `None` only in unit tests
|
||||
/// that construct a bare `Timers`.
|
||||
coord: Option<std::sync::Arc<crate::park::Coordinator>>,
|
||||
/// Reverse-wrapped so the smallest deadline is at the top.
|
||||
heap: BinaryHeap<Reverse<Entry>>,
|
||||
/// Monotonic counter for the tiebreaker `seq` field (and the `TimerId` of a
|
||||
@@ -157,7 +165,18 @@ pub struct Timers {
|
||||
|
||||
impl Timers {
|
||||
pub fn new() -> Self {
|
||||
Self { heap: BinaryHeap::new(), next_seq: 0, armed: std::collections::HashSet::new() }
|
||||
Self {
|
||||
coord: None,
|
||||
heap: BinaryHeap::new(),
|
||||
next_seq: 0,
|
||||
armed: std::collections::HashSet::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Attach the scheduler coordination layer (RFC 018). Called once, at
|
||||
/// runtime construction, before any scheduler thread exists.
|
||||
pub(crate) fn attach_coordinator(&mut self, c: std::sync::Arc<crate::park::Coordinator>) {
|
||||
self.coord = Some(c);
|
||||
}
|
||||
|
||||
/// Insert a `Sleep` timer. Convenience for the common case.
|
||||
@@ -242,6 +261,13 @@ impl Timers {
|
||||
#[cfg(feature = "smarm-causal")]
|
||||
wall,
|
||||
}));
|
||||
// RFC 018: publish the (possibly new-minimum) deadline to the
|
||||
// busy-path snapshot and wake the timekeeper if it is parked
|
||||
// toward a later one. We hold the timers mutex — the mandated
|
||||
// serialization for both.
|
||||
if let Some(c) = &self.coord {
|
||||
c.note_deadline(deadline);
|
||||
}
|
||||
seq
|
||||
}
|
||||
|
||||
@@ -255,6 +281,9 @@ impl Timers {
|
||||
pub fn clear(&mut self) {
|
||||
self.heap.clear();
|
||||
self.armed.clear();
|
||||
if let Some(c) = &self.coord {
|
||||
c.refresh_deadline(None);
|
||||
}
|
||||
}
|
||||
|
||||
/// Soonest pending deadline, or `None` if the heap is empty.
|
||||
@@ -324,6 +353,13 @@ impl Timers {
|
||||
}
|
||||
out.push(entry);
|
||||
}
|
||||
// RFC 018: re-anchor the busy-path snapshot to the new heap minimum
|
||||
// (still under the timers mutex). A causal-shift re-queue above went
|
||||
// through `heap.push` directly, so this peek is the one place the
|
||||
// snapshot is guaranteed to catch up.
|
||||
if let Some(c) = &self.coord {
|
||||
c.refresh_deadline(self.peek_deadline());
|
||||
}
|
||||
out
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user