fix(runtime): wake idle sibling schedulers at termination
An idle scheduler thread blocks in poll_wake on a snapshot of peek_deadline / io_outstanding. Only the io threads ever write the wake pipe (on completion push); enqueue does not. Mid-flight that is masked — the thread that caused an enqueue is awake and processes it — but at termination the snapshot can go terminally stale, two ways: - Stale io_outstanding (infinite hang): an actor parked in wait_readable is request_stop'ped. Cancellation deregisters the waiter and produces NO completion, so the wake pipe is never written. A sibling blocked on io_outstanding > 0 with no timers pending sits in poll(-1) forever. - Orphaned timer deadline (finite stall): an actor cancelled out of a long sleep leaves its timer entry behind. Clearing the wheel at AllDone doesn't help a sibling that already blocked on that deadline; it sleeps it out in full. In both cases the remaining actors finish on the other scheduler thread, which reaches the live==0 && io_out==0 verdict and returns without waking the blocked one. Runtime::run then stalls or hangs in its worker join. Found via urus's graceful-shutdown tests (~10% flake, one test per variant); confirmed by gdb dumps of hung processes showing run() in JoinHandle::join over a sibling in poll(wake_fd, 59750ms) resp. poll(-1). Fix: Io::wake() writes the wake pipe directly; the AllDone arm calls it after the timer clear. One byte wakes every poller; each re-runs the verdict, independently reaches AllDone, and re-wakes — idempotent. tests/terminal_wake.rs reproduces both variants deterministically: the root busy-spins (no timer entries, occupies one scheduler thread) so the sibling settles into the stale idle wait before the stop is issued. Both hang without the fix and pass in <0.5s with it. Known residual gap, deliberately unfixed: the timers-pending / no-io-subsystem idle branch blocks in thread::sleep and has no wake mechanism at all — the same stall exists for runtimes that never initialize io. Roadmap candidate alongside cross-thread unpark and entry-side check_cancelled in park_current (the lossy-QUEUED-stop bug).
This commit is contained in:
@@ -1238,6 +1238,17 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
|
||||
// woken by them — e.g. a sleeper cancelled out of its sleep);
|
||||
// they must not keep the runtime alive. Drop them on the way out.
|
||||
inner.timers.lock().unwrap().clear();
|
||||
// Terminal wake: a sibling scheduler may be blocked in its
|
||||
// idle wait on a snapshot that is now terminally stale — an
|
||||
// orphaned long deadline (it would sleep it out in full) or
|
||||
// a stale `io_outstanding > 0` from a stop-cancelled waiter
|
||||
// (it would block in poll(-1) forever; cancellation produces
|
||||
// no completion, so nothing else writes the wake pipe).
|
||||
// One byte wakes every poller; each re-runs the verdict,
|
||||
// reaches AllDone itself, and re-wakes — idempotent.
|
||||
if let Some(io) = inner.io.lock().unwrap().as_ref() {
|
||||
io.wake();
|
||||
}
|
||||
return;
|
||||
}
|
||||
Pop::Idle { io_outstanding, wake_fd } => {
|
||||
|
||||
Reference in New Issue
Block a user