feat(scheduler,supervisor,gen_server): graceful shutdown — request_shutdown, child Shutdown policy, handle_shutdown

Lift OTP's `exit(Pid, shutdown)` + child-spec `shutdown` wholesale.

scheduler / runtime
- `request_shutdown(pid)`: the polite stop. A target trapping exits gets an
  `ExitSignal { reason: DownReason::Shutdown }` on its trap inbox and keeps
  running; a non-trapping target is stopped as by `request_stop`, which is
  now documented as the hard stop (`exit(Pid, kill)`). Dead pid: no-op.
- `RuntimeHandle::request_shutdown` for the off-runtime (signal thread) path;
  `from == ROOT_PID` there.
- `DownReason::Shutdown` — appears only in ExitSignal, never in Down (a
  complying target exits *normally*).

supervisor
- `ChildSpec::shutdown(Shutdown::{BrutalKill, Timeout(d), Infinity})`,
  default Timeout(5s). Every supervisor-initiated stop (ordered shutdown and
  OneForAll/RestForOne sibling cycling) is: request_shutdown → await the
  child's Signal up to the grace → request_stop → await. Sequential, reverse
  start order.
- The supervisor traps exits; a Shutdown ExitSignal runs the ordered
  shutdown and `run()` returns normally, so `request_shutdown(root_sup)`
  tears a whole tree down top-down with each child's grace period.
- FIX: a hard `request_stop` on a supervisor previously orphaned its
  children (the ordered shutdown lived after the loop, and the unwind
  skipped it). `Live` (the by_pid map) now carries a drop guard that
  fire-and-forget hard-stops live children when unwinding.

gen_server
- `GenServerCtx::trap_exit()` opt-in in `init`; the trap inbox becomes arm 0
  of the loop's select. Shutdown ExitSignal → `handle_shutdown() ->
  ShutdownAction::{Exit, Continue}` (default Exit: loop breaks, `terminate`
  runs on the normal path and may block). Other ExitSignals →
  `handle_exit(sig)`.
- `GenServerCtx::stop_handle() -> StopHandle`, `stop()` ends the server
  after the current message with a *normal* exit — the missing
  `{stop, normal, State}`; `request_stop(self_pid())` was the only self-exit
  and it is abnormal (Transient restarts it).
- `GenServerRef::shutdown()` / `gen_server::shutdown(name)` now go through
  `request_shutdown`.

Tests: tests/shutdown.rs, tests/supervisor_shutdown.rs,
tests/gen_server_shutdown.rs. Full suite green; fmt + clippy --lib clean.
This commit is contained in:
claude
2026-08-19 06:34:48 +00:00
parent 1002777ef3
commit 9c8f59ca53
9 changed files with 1058 additions and 100 deletions
+216
View File
@@ -0,0 +1,216 @@
//! gen_server graceful shutdown.
//!
//! - A server that does not opt in (`ctx.trap_exit()` in `init`) is stopped
//! outright by `request_shutdown`, exactly as by `request_stop`.
//! - A trapping server receives the request as `handle_shutdown`. The default
//! returns `ShutdownAction::Exit`: the loop breaks and `terminate` runs on
//! the normal (non-unwind) path, so it may block. `Continue` keeps the loop
//! dispatching; the state later ends itself with a `StopHandle` — the only
//! way for a gen_server to exit *normally* on its own (`request_stop` on
//! self is an abnormal `Stopped`, which `Transient` restarts).
//! - Other exit signals (linked peers dying) reach a trapping server via
//! `handle_exit`.
use smarm::gen_server::{
start, GenServer, GenServerBuilder, GenServerCtx, GenServerRef, ShutdownAction, StopHandle,
};
use smarm::supervisor::{ChildSpec, OneForOne, Restart};
use smarm::{link, monitor, request_shutdown, run, self_pid, sleep, spawn, DownReason, ExitSignal};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
#[derive(Default, Clone)]
struct Log {
events: Arc<Mutex<Vec<&'static str>>>,
}
impl Log {
fn push(&self, e: &'static str) {
self.events.lock().unwrap().push(e);
}
fn get(&self) -> Vec<&'static str> {
self.events.lock().unwrap().clone()
}
}
/// A server with configurable shutdown behaviour.
struct Srv {
log: Log,
trap: bool,
action: ShutdownAction,
stop: Option<StopHandle<Srv>>,
exits: Arc<Mutex<Vec<ExitSignal>>>,
}
impl Srv {
fn new(log: &Log, trap: bool, action: ShutdownAction) -> Self {
Srv {
log: log.clone(),
trap,
action,
stop: None,
exits: Default::default(),
}
}
}
enum Cast {
Note(&'static str),
StopNow,
}
impl GenServer for Srv {
type Call = ();
type Reply = ();
type Cast = Cast;
type Info = ();
type Timer = ();
fn init(&mut self, ctx: &GenServerCtx<Self>) {
if self.trap {
ctx.trap_exit();
}
self.stop = Some(ctx.stop_handle());
}
fn handle_call(&mut self, _: ()) {}
fn handle_cast(&mut self, c: Cast) {
match c {
Cast::Note(s) => self.log.push(s),
Cast::StopNow => self.stop.as_ref().unwrap().stop(),
}
}
fn handle_shutdown(&mut self) -> ShutdownAction {
self.log.push("handle_shutdown");
self.action
}
fn handle_exit(&mut self, sig: ExitSignal) {
self.log.push("handle_exit");
self.exits.lock().unwrap().push(sig);
}
fn terminate(&mut self) {
// Allowed to block on the graceful path.
if self.trap {
sleep(Duration::from_millis(10));
}
self.log.push("terminate");
}
}
fn spawn_settled<G: GenServer>(state: G) -> GenServerRef<G> {
let r = start(state);
sleep(Duration::from_millis(20)); // let init (trap_exit) run
r
}
#[test]
fn non_trapping_server_is_stopped_outright() {
let log = Log::default();
let l = log.clone();
run(move || {
let r = spawn_settled(Srv::new(&l, false, ShutdownAction::Exit));
let mon = monitor(r.pid());
request_shutdown(r.pid());
let d = mon.rx.recv().unwrap();
assert_eq!(d.reason, DownReason::Stopped);
});
assert_eq!(log.get(), vec!["terminate"]);
}
#[test]
fn trapping_server_exits_normally_via_handle_shutdown() {
let log = Log::default();
let l = log.clone();
run(move || {
let r = spawn_settled(Srv::new(&l, true, ShutdownAction::Exit));
let mon = monitor(r.pid());
request_shutdown(r.pid());
let d = mon.rx.recv().unwrap();
assert_eq!(d.reason, DownReason::Exit);
});
assert_eq!(log.get(), vec!["handle_shutdown", "terminate"]);
}
#[test]
fn continue_keeps_dispatching_until_stop_handle() {
let log = Log::default();
let l = log.clone();
run(move || {
let r = spawn_settled(Srv::new(&l, true, ShutdownAction::Continue));
let mon = monitor(r.pid());
request_shutdown(r.pid());
sleep(Duration::from_millis(20));
r.cast(Cast::Note("after-shutdown-request")).unwrap();
r.cast(Cast::StopNow).unwrap();
let d = mon.rx.recv().unwrap();
assert_eq!(d.reason, DownReason::Exit);
});
assert_eq!(
log.get(),
vec!["handle_shutdown", "after-shutdown-request", "terminate"]
);
}
#[test]
fn stop_handle_is_a_normal_exit_that_transient_does_not_restart() {
let starts = Arc::new(AtomicUsize::new(0));
let s = starts.clone();
run(move || {
let s2 = s.clone();
let sup = spawn(move || {
let s3 = s2.clone();
OneForOne::new()
.child(ChildSpec::new(Restart::Transient, move || {
s3.fetch_add(1, Ordering::SeqCst);
let log = Log::default();
let r = GenServerBuilder::new(Srv::new(&log, false, ShutdownAction::Exit))
.under(self_pid())
.start();
r.cast(Cast::StopNow).unwrap();
// Block until the server is gone; a bare spawn parent
// returning would not itself end the server.
let mon = monitor(r.pid());
let _ = mon.rx.recv();
}))
.run();
});
sup.join().unwrap(); // returns only if the child was not restarted forever
});
assert_eq!(starts.load(Ordering::SeqCst), 1);
}
#[test]
fn linked_peer_death_reaches_handle_exit() {
let log = Log::default();
let l = log.clone();
let alive = Arc::new(AtomicBool::new(false));
let a = alive.clone();
run(move || {
let r = spawn_settled(Srv::new(&l, true, ShutdownAction::Exit));
let pid = r.pid();
let peer = spawn(move || {
link(pid);
panic!("peer dies");
});
let _ = peer.join();
sleep(Duration::from_millis(20));
r.cast(Cast::Note("still-serving")).unwrap();
sleep(Duration::from_millis(20));
a.store(true, Ordering::SeqCst);
let mon = monitor(r.pid());
drop(r); // inbox closes → clean exit
let _ = mon.rx.recv(); // don't let the root-exit sweep race terminate
});
assert!(alive.load(Ordering::SeqCst));
assert_eq!(log.get(), vec!["handle_exit", "still-serving", "terminate"]);
}
#[test]
fn gen_server_ref_shutdown_is_graceful_for_a_trapping_server() {
let log = Log::default();
let l = log.clone();
run(move || {
let r = spawn_settled(Srv::new(&l, true, ShutdownAction::Exit));
r.shutdown();
});
assert_eq!(log.get(), vec!["handle_shutdown", "terminate"]);
}
+122
View File
@@ -0,0 +1,122 @@
//! Graceful shutdown — `request_shutdown` (OTP `exit(Pid, shutdown)`).
//!
//! `request_stop` is `exit(Pid, kill)`: an uncatchable unwind at the target's
//! next observation point. `request_shutdown` is the polite form:
//! - a target that is NOT trapping exits is stopped exactly as by
//! `request_stop` (OTP's rule: don't trap, you die);
//! - a target that IS trapping receives an `ExitSignal { reason: Shutdown }`
//! on its trap inbox and keeps running — it is expected to wind down and
//! exit normally on its own.
use smarm::{monitor, request_shutdown, run, self_pid, sleep, spawn, trap_exit, DownReason, Pid};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{mpsc, Arc};
use std::thread;
use std::time::Duration;
const WATCHDOG: Duration = Duration::from_secs(10);
#[test]
fn request_shutdown_stops_a_non_trapping_actor() {
run(|| {
let h = spawn(|| sleep(Duration::from_secs(3600)));
let mon = monitor(h.pid());
request_shutdown(h.pid());
let down = mon.rx.recv().expect("down");
assert_eq!(down.reason, DownReason::Stopped);
});
}
#[test]
fn request_shutdown_is_a_message_to_a_trapping_actor() {
let unwound = Arc::new(AtomicBool::new(false));
let u = unwound.clone();
run(move || {
struct Unwound(Arc<AtomicBool>);
impl Drop for Unwound {
fn drop(&mut self) {
if std::thread::panicking() {
self.0.store(true, Ordering::SeqCst);
}
}
}
let (tx, rx) = smarm::channel::<(Pid, DownReason)>();
let (ready_tx, ready_rx) = smarm::channel::<()>();
let h = spawn(move || {
let _g = Unwound(u);
let inbox = trap_exit();
let _ = ready_tx.send(());
let sig = inbox.recv().expect("exit signal");
let _ = tx.send((sig.from, sig.reason));
// Keep doing work after the request: shutdown is advisory.
sleep(Duration::from_millis(20));
});
// Trapping is set by the target itself; a request that beats it is a
// plain stop (same window as OTP's exit-before-process_flag).
ready_rx.recv().expect("ready");
let me = self_pid();
let mon = monitor(h.pid());
request_shutdown(h.pid());
let (from, reason) = rx.recv().expect("relayed");
assert_eq!(from, me);
assert_eq!(reason, DownReason::Shutdown);
let down = mon.rx.recv().expect("down");
assert_eq!(
down.reason,
DownReason::Exit,
"target exited normally, not stopped"
);
});
assert!(
!unwound.load(Ordering::SeqCst),
"trapping target must not be unwound"
);
}
#[test]
fn request_shutdown_on_dead_pid_is_a_no_op() {
run(|| {
let h = spawn(|| {});
let pid = h.pid();
let _ = h.join();
request_shutdown(pid); // must not panic
});
}
#[test]
fn handle_request_shutdown_from_foreign_thread() {
let rt = smarm::init(smarm::Config::exact(2));
let handle = rt.handle();
let (pid_tx, pid_rx) = mpsc::channel::<Pid>();
let requester = thread::spawn(move || {
let pid = pid_rx.recv().expect("pid");
thread::sleep(Duration::from_millis(50));
handle.request_shutdown(pid);
});
let (done_tx, done_rx) = mpsc::channel();
thread::spawn(move || {
rt.run(move || {
let (tx, rx) = smarm::channel::<DownReason>();
let (ready_tx, ready_rx) = smarm::channel::<()>();
let h = spawn(move || {
let inbox = trap_exit();
let _ = ready_tx.send(());
let sig = inbox.recv().expect("exit signal");
let _ = tx.send(sig.reason);
});
ready_rx.recv().expect("ready");
pid_tx.send(h.pid()).expect("send pid");
let reason = rx.recv().expect("relayed");
assert_eq!(reason, DownReason::Shutdown);
let _ = h.join();
});
let _ = done_tx.send(());
});
done_rx
.recv_timeout(WATCHDOG)
.expect("run did not return: foreign-thread request_shutdown never reached the target");
requester.join().expect("requester thread");
}
+303
View File
@@ -0,0 +1,303 @@
//! Supervisor shutdown — the OTP child-spec `shutdown` policy.
//!
//! A supervisor traps exits. A `request_shutdown` reaching it (from its parent
//! supervisor, or from the app via `request_shutdown`/`RuntimeHandle`) runs
//! the ordered shutdown: children are stopped in reverse start order, each
//! per its `Shutdown` policy — `request_shutdown`, wait up to the timeout for
//! its termination signal, `request_stop` if it overstays — and then `run()`
//! returns normally. Every supervisor-initiated child stop (ordered shutdown,
//! OneForAll/RestForOne sibling cycling) goes through the same policy.
//!
//! A *hard* `request_stop` on a supervisor unwinds it; a drop guard then
//! hard-stops its live children so the subtree is never orphaned.
use smarm::supervisor::{ChildSpec, OneForOne, Restart, Shutdown, Strategy};
use smarm::{
monitor, request_shutdown, request_stop, run, sleep, spawn, trap_exit, DownReason, JoinHandle,
};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
/// A child that traps exits, records the order it was shut down in, and exits
/// normally on the request (after `delay`). Ignores the request if `comply`
/// is false — a straggler that must be hard-stopped.
fn polite_child(
tag: usize,
log: &Arc<Mutex<Vec<usize>>>,
delay: Duration,
comply: bool,
) -> impl Fn() + Send + Sync + 'static {
let log = log.clone();
move || {
let inbox = trap_exit();
loop {
let sig = match inbox.recv() {
Ok(s) => s,
Err(_) => return,
};
if sig.reason == DownReason::Shutdown {
log.lock().unwrap().push(tag);
if comply {
sleep(delay);
return;
}
// Not complying: keep running until hard-stopped.
loop {
sleep(Duration::from_millis(5));
}
}
}
}
}
/// Spawn `sup`, let its children reach `trap_exit`, return the handle.
fn spawn_settled(sup: OneForOne) -> JoinHandle {
let h = spawn(move || sup.run());
sleep(Duration::from_millis(30));
h
}
#[test]
fn shutdown_stops_children_in_reverse_order_and_returns_normally() {
let log = Arc::new(Mutex::new(Vec::new()));
let l = log.clone();
run(move || {
let sup = OneForOne::new()
.child(ChildSpec::new(
Restart::Permanent,
polite_child(1, &l, Duration::ZERO, true),
))
.child(ChildSpec::new(
Restart::Permanent,
polite_child(2, &l, Duration::ZERO, true),
))
.child(ChildSpec::new(
Restart::Permanent,
polite_child(3, &l, Duration::ZERO, true),
));
let h = spawn_settled(sup);
let mon = monitor(h.pid());
request_shutdown(h.pid());
let down = mon.rx.recv().expect("down");
assert_eq!(
down.reason,
DownReason::Exit,
"supervisor exits normally after shutdown"
);
});
assert_eq!(*log.lock().unwrap(), vec![3, 2, 1]);
}
#[test]
fn non_trapping_child_is_simply_stopped() {
let dropped = Arc::new(AtomicBool::new(false));
let d = dropped.clone();
run(move || {
struct G(Arc<AtomicBool>);
impl Drop for G {
fn drop(&mut self) {
self.0.store(true, Ordering::SeqCst);
}
}
let sup = OneForOne::new().child(ChildSpec::new(Restart::Permanent, move || {
let _g = G(d.clone());
loop {
sleep(Duration::from_millis(5));
}
}));
let h = spawn_settled(sup);
request_shutdown(h.pid());
h.join().expect("sup");
});
assert!(dropped.load(Ordering::SeqCst));
}
#[test]
fn straggler_is_hard_stopped_after_timeout() {
let log = Arc::new(Mutex::new(Vec::new()));
let l = log.clone();
run(move || {
let sup = OneForOne::new().child(
ChildSpec::new(
Restart::Permanent,
polite_child(1, &l, Duration::ZERO, false),
)
.shutdown(Shutdown::Timeout(Duration::from_millis(50))),
);
let h = spawn_settled(sup);
let t0 = Instant::now();
request_shutdown(h.pid());
h.join().expect("sup");
let took = t0.elapsed();
assert!(
took >= Duration::from_millis(50),
"returned before the grace period: {took:?}"
);
assert!(
took < Duration::from_secs(2),
"did not fall back to a hard stop: {took:?}"
);
});
assert_eq!(
*log.lock().unwrap(),
vec![1],
"the straggler did receive the request"
);
}
#[test]
fn infinity_waits_for_a_slow_but_compliant_child() {
let log = Arc::new(Mutex::new(Vec::new()));
let l = log.clone();
let finished = Arc::new(AtomicBool::new(false));
let f = finished.clone();
run(move || {
let f2 = f.clone();
let l2 = l.clone();
let sup = OneForOne::new().child(
ChildSpec::new(Restart::Permanent, move || {
let inbox = trap_exit();
let _ = inbox.recv();
l2.lock().unwrap().push(1);
sleep(Duration::from_millis(150));
f2.store(true, Ordering::SeqCst); // only reached if not hard-stopped
})
.shutdown(Shutdown::Infinity),
);
let h = spawn_settled(sup);
request_shutdown(h.pid());
h.join().expect("sup");
});
assert!(
finished.load(Ordering::SeqCst),
"Infinity must not hard-stop a compliant child"
);
}
#[test]
fn brutal_kill_skips_the_request() {
let log = Arc::new(Mutex::new(Vec::new()));
let l = log.clone();
run(move || {
let sup = OneForOne::new().child(
ChildSpec::new(
Restart::Permanent,
polite_child(1, &l, Duration::ZERO, true),
)
.shutdown(Shutdown::BrutalKill),
);
let h = spawn_settled(sup);
request_shutdown(h.pid());
h.join().expect("sup");
});
assert!(
log.lock().unwrap().is_empty(),
"a BrutalKill child never sees the request"
);
}
#[test]
fn hard_stop_of_supervisor_does_not_orphan_children() {
let alive = Arc::new(AtomicUsize::new(0));
let a = alive.clone();
run(move || {
struct Alive(Arc<AtomicUsize>);
impl Drop for Alive {
fn drop(&mut self) {
self.0.fetch_sub(1, Ordering::SeqCst);
}
}
let mk = |a: Arc<AtomicUsize>| {
move || {
a.fetch_add(1, Ordering::SeqCst);
let _g = Alive(a.clone());
loop {
sleep(Duration::from_millis(5));
}
}
};
let sup = OneForOne::new()
.child(ChildSpec::new(Restart::Permanent, mk(a.clone())))
.child(ChildSpec::new(Restart::Permanent, mk(a.clone())));
let h = spawn_settled(sup);
assert_eq!(a.load(Ordering::SeqCst), 2);
let mon = monitor(h.pid());
request_stop(h.pid());
let _ = mon.rx.recv();
sleep(Duration::from_millis(50));
assert_eq!(
a.load(Ordering::SeqCst),
0,
"children orphaned by a hard supervisor stop"
);
});
}
#[test]
fn nested_shutdown_reaches_grandchildren() {
let log = Arc::new(Mutex::new(Vec::new()));
let l = log.clone();
run(move || {
let l_inner = l.clone();
let inner = move || {
OneForOne::new()
.child(ChildSpec::new(
Restart::Permanent,
polite_child(10, &l_inner, Duration::ZERO, true),
))
.child(ChildSpec::new(
Restart::Permanent,
polite_child(11, &l_inner, Duration::ZERO, true),
))
.run()
};
let sup = OneForOne::new()
.child(ChildSpec::new(
Restart::Permanent,
polite_child(1, &l, Duration::ZERO, true),
))
.child(ChildSpec::new(Restart::Permanent, inner).shutdown(Shutdown::Infinity));
let h = spawn_settled(sup);
request_shutdown(h.pid());
h.join().expect("sup");
});
assert_eq!(*log.lock().unwrap(), vec![11, 10, 1]);
}
#[test]
fn sibling_cycling_uses_graceful_shutdown() {
// OneForAll: when child A dies, sibling B (trapping) must receive a
// Shutdown request rather than a bare stop.
let log = Arc::new(Mutex::new(Vec::new()));
let l = log.clone();
let a_runs = Arc::new(AtomicUsize::new(0));
let ar = a_runs.clone();
run(move || {
let ar2 = ar.clone();
let sup = OneForOne::new()
.strategy(Strategy::OneForAll)
.intensity(5, Duration::from_secs(60))
.child(ChildSpec::new(Restart::Transient, move || {
let n = ar2.fetch_add(1, Ordering::SeqCst) + 1;
sleep(Duration::from_millis(30));
if n == 1 {
panic!("first run dies");
}
// Second run: park until shut down.
let inbox = trap_exit();
let _ = inbox.recv();
}))
.child(ChildSpec::new(
Restart::Permanent,
polite_child(2, &l, Duration::ZERO, true),
));
let h = spawn(move || sup.run());
sleep(Duration::from_millis(150));
request_shutdown(h.pid());
h.join().expect("sup");
});
// B was shut down once by the cycle and once by the final shutdown.
assert_eq!(*log.lock().unwrap(), vec![2, 2]);
assert_eq!(a_runs.load(Ordering::SeqCst), 2);
}