feat(runtime,stack): sampled stack high-water + MADV_FREE shrink at actor-park (RFC 019 §§2–3)
hwm: AtomicUsize lands beside sp on the slot: the single context-save site min-updates it (one branch + at most one Relaxed store into the line the sp store just dirtied), install resets it to the fresh top. Advisory by construction — correctness never depends on it. The mod-doc ordering chain gains a line: hwm piggybacks the existing Relaxed-store-before-Release pattern and adds no edges. Shrink hook in the YieldIntent::Park arm only, before the park_return Release transition — the owned window (obligation 1's assert-comment at the site): after the sp store, before Parked is published, scheduler on its own stack, actor saved and unstealable. It runs on both arms of the park_return race (a consumed unpark flag means one wasted-but-harmless madvise). The preempt/yield path deliberately never checks: §4's bounded, self-healing leak under saturation, when syscalls are least affordable. SHRINK_THRESHOLD = 256 KiB and SHRINK_COOLDOWN = 64 parks are pub constants with the ratified doc rationale, not Config fields. The freed span is shrink_range(hwm, sp, page): whole pages of [hwm, sp − 1-page redzone), rounded inward, checked arithmetic — adversarial inputs collapse to None (obligation 2). MADV_FREE marks lazily; the kernel's reclaim-under-pressure IS the hysteresis, cancel-on-write is the safety net. parks_since_shrink + shrink_count ride the slot for the cooldown and the future introspect surface. Tests: 7 adversarial shrink_range units (inverted/empty spans, redzone underflow, unaligned ends, sp-crossing sweep); integration — 8 MiB reserve, ~3 MiB spike sampled via yield-at-depth, parks gated on introspected Parked state past the cooldown, then ≥ 2 MiB LazyFree asserted inside the stack's smaps range with live data intact; and the inverse guard — a shallow never-spiking actor ends at exactly 0 LazyFree (also proves the parser isn't vacuously zero via the first test).
This commit is contained in:
Claude (sandbox)
+112
-1
@@ -65,6 +65,8 @@
|
||||
//! word stores are `Release`, loads are `Acquire`. The chain that matters:
|
||||
//! the park path stores `sp` (Relaxed) *before* its Release transition; any
|
||||
//! later Acquire transition/load of the word therefore observes that `sp`.
|
||||
//! RFC 019's `hwm` (and the shrink that reads it) piggybacks this exact
|
||||
//! pattern in the same pre-Release window and adds no edges.
|
||||
//! The run-queue mutex independently provides the same edges today; the
|
||||
//! word's own ordering is what phase 3's lock-free queue will rely on.
|
||||
//!
|
||||
@@ -422,6 +424,23 @@ pub const DEFAULT_STACK_RESERVE: usize = 64 * 1024;
|
||||
/// page so unprobed C frames cannot leap it. See [`Config::stack_guard`].
|
||||
pub const DEFAULT_STACK_GUARD: usize = 64 * 1024;
|
||||
|
||||
/// RFC 019 §3: minimum releasable span (`sp − hwm` at park) before the
|
||||
/// park-path shrink spends a syscall. A constant, not a `Config` field
|
||||
/// (ratified): nobody tunes this well and the measured stakes are low — a
|
||||
/// threshold-sized `MADV_FREE` costs ~3 µs against a ~100 ns park, paid
|
||||
/// only on spike-recovery parks, which are rare by construction and *were*
|
||||
/// the spike. Steady-state actors never reach the syscall: their check is
|
||||
/// two Relaxed loads and a compare on a line the context-save just wrote.
|
||||
pub const SHRINK_THRESHOLD: usize = 256 * 1024;
|
||||
|
||||
/// RFC 019 §3: parks between shrinks of one actor. Guards a few-µs cost, so
|
||||
/// it can be coarse (parks, not wall time); the kernel's
|
||||
/// reclaim-under-pressure-only handling of `MADV_FREE` is the real release
|
||||
/// hysteresis — re-touched-before-pressure pages cost a 0.24 µs/page
|
||||
/// cancel-write and no fault. A constant, not `Config` (ratified, same
|
||||
/// rationale as [`SHRINK_THRESHOLD`]).
|
||||
pub const SHRINK_COOLDOWN: u32 = 64;
|
||||
|
||||
pub(crate) type Closure = Box<dyn FnOnce() + Send>;
|
||||
|
||||
/// Lifecycle data, mutated only under the slot's cold [`RawMutex`]. Everything
|
||||
@@ -463,6 +482,22 @@ pub(crate) struct Slot {
|
||||
/// Release transition out of Running; read after the Acquire transition
|
||||
/// Queued→Running. Relaxed is sufficient — ordering rides on `word`.
|
||||
sp: AtomicUsize,
|
||||
/// RFC 019: sampled stack high-water — the minimum `sp` ever stored above,
|
||||
/// i.e. the deepest excursion *observed at a switch point*. Advisory:
|
||||
/// correctness never depends on it; its one job is "is a shrink worth a
|
||||
/// syscall?". Declared adjacent to `sp` so the min-update dirties the
|
||||
/// line the context-save just wrote. Same single-writer Relaxed
|
||||
/// discipline as `sp`; reset to the fresh `sp` at install.
|
||||
hwm: AtomicUsize,
|
||||
/// RFC 019: parks since the last shrink (or install). Counted on every
|
||||
/// pass through the Park arm by the owning scheduler thread; the shrink
|
||||
/// fires only once this clears [`SHRINK_COOLDOWN`] *and* the releasable
|
||||
/// span clears [`SHRINK_THRESHOLD`]. Single-writer Relaxed.
|
||||
parks_since_shrink: AtomicU32,
|
||||
/// RFC 019: shrinks performed on this incarnation (introspection lands
|
||||
/// with the RFC's introspect surface; the counter exists from birth so
|
||||
/// tests can rely on install resetting it). Single-writer Relaxed.
|
||||
shrink_count: AtomicU32,
|
||||
/// Pointer into the actor's `Arc<AtomicBool>` stop flag. Set at spawn,
|
||||
/// nulled at finalize. The box outlives every read: it is only ever read
|
||||
/// on the resume path while the actor cannot be finalized (it is on-CPU).
|
||||
@@ -534,6 +569,9 @@ impl Slot {
|
||||
Self {
|
||||
word: StateWord::new(),
|
||||
sp: AtomicUsize::new(0),
|
||||
hwm: AtomicUsize::new(0),
|
||||
parks_since_shrink: AtomicU32::new(0),
|
||||
shrink_count: AtomicU32::new(0),
|
||||
stop_ptr: AtomicPtr::new(std::ptr::null_mut()),
|
||||
closure: AtomicPtr::new(std::ptr::null_mut()),
|
||||
overruns: AtomicU64::new(0),
|
||||
@@ -1356,6 +1394,49 @@ pub const ROOT_PID: Pid = Pid::new(u32::MAX, u32::MAX);
|
||||
// Spawn-side slot installation
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Stack shrink — RFC 019 §3 (park path only)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// The per-park shrink check. Called from the `YieldIntent::Park` arm inside
|
||||
/// the owned window (see the assert-comment at the call site). Fast path —
|
||||
/// no spike since the last shrink — is two Relaxed loads, a compare, and the
|
||||
/// park counter bump, all on the slot line the context-save just wrote.
|
||||
///
|
||||
/// On a shrink: `MADV_FREE` the whole pages of `[hwm, sp − redzone)` (the
|
||||
/// inward-rounded range from [`crate::stack::shrink_range`]), then reset
|
||||
/// `hwm = sp` and the park counter. MADV_FREE only *marks*: the kernel
|
||||
/// reclaims under pressure, skips re-dirtied pages, and refaults zero pages
|
||||
/// for writes after reclaim — so an over-eager mark costs a cancel-write,
|
||||
/// never data.
|
||||
fn maybe_shrink_stack(slot: &Slot) {
|
||||
let parks = slot.parks_since_shrink.load(Ordering::Relaxed).saturating_add(1);
|
||||
slot.parks_since_shrink.store(parks, Ordering::Relaxed);
|
||||
|
||||
let sp = slot.sp.load(Ordering::Relaxed);
|
||||
let hwm = slot.hwm.load(Ordering::Relaxed);
|
||||
if sp.wrapping_sub(hwm) < SHRINK_THRESHOLD || sp < hwm {
|
||||
return; // common case: nothing worth a syscall
|
||||
}
|
||||
if parks < SHRINK_COOLDOWN {
|
||||
return;
|
||||
}
|
||||
let page = crate::stack::page_size();
|
||||
if let Some((addr, len)) = crate::stack::shrink_range(hwm, sp, page) {
|
||||
// Advisory: on the (kernel-config) chance MADV_FREE is unsupported,
|
||||
// failing silently degrades to "never shrinks", which is correct.
|
||||
unsafe {
|
||||
libc::madvise(addr as *mut libc::c_void, len, libc::MADV_FREE);
|
||||
}
|
||||
slot.hwm.store(sp, Ordering::Relaxed);
|
||||
slot.parks_since_shrink.store(0, Ordering::Relaxed);
|
||||
slot.shrink_count.store(
|
||||
slot.shrink_count.load(Ordering::Relaxed).saturating_add(1),
|
||||
Ordering::Relaxed,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Stack acquisition / recycling — RFC 019 pool rule
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -1428,6 +1509,11 @@ pub(crate) fn install_actor(
|
||||
cold.pending_io_result = None;
|
||||
}
|
||||
slot.sp.store(sp, Ordering::Relaxed);
|
||||
// RFC 019: a fresh incarnation starts with its high-water at the fresh
|
||||
// top-of-stack `sp` and its shrink bookkeeping zeroed.
|
||||
slot.hwm.store(sp, Ordering::Relaxed);
|
||||
slot.parks_since_shrink.store(0, Ordering::Relaxed);
|
||||
slot.shrink_count.store(0, Ordering::Relaxed);
|
||||
slot.store_closure(closure);
|
||||
slot.reset_counters();
|
||||
inner.live_actors.fetch_add(1, Ordering::Relaxed);
|
||||
@@ -1926,7 +2012,15 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
|
||||
crate::preempt::clear_current_slot();
|
||||
|
||||
let intent = YIELD_INTENT.with(|c| c.get());
|
||||
slot.sp.store(get_actor_sp(), Ordering::Relaxed);
|
||||
let saved_sp = get_actor_sp();
|
||||
slot.sp.store(saved_sp, Ordering::Relaxed);
|
||||
// RFC 019 §2: sampled high-water — one branch + at most one store
|
||||
// into the line the store above just dirtied. Relaxed and advisory;
|
||||
// it piggybacks the existing Relaxed-store-before-Release pattern
|
||||
// (mod docs, "Memory ordering") and adds no edges.
|
||||
if saved_sp < slot.hwm.load(Ordering::Relaxed) {
|
||||
slot.hwm.store(saved_sp, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
if is_actor_done() {
|
||||
crate::te!(crate::trace::Event::Done(pid));
|
||||
@@ -1948,6 +2042,23 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
|
||||
inner.enqueue(pid);
|
||||
}
|
||||
YieldIntent::Park => {
|
||||
// RFC 019 §3 shrink window (correctness obligation 1):
|
||||
// this site sits after the `sp` store above and before
|
||||
// the `park_return` Release transition below publishes
|
||||
// Parked — the scheduler is on its own stack and the
|
||||
// actor is saved but not yet stealable, so the madvise
|
||||
// races nothing (belt). MADV_FREE's cancel-on-write is
|
||||
// the suspenders: even a racing writer could lose
|
||||
// nothing written after the mark, and everything below
|
||||
// live `sp` is dead by definition. Runs on BOTH arms of
|
||||
// the park_return race — a consumed unpark flag means a
|
||||
// wasted-but-harmless madvise on a rare window.
|
||||
//
|
||||
// This is the ONLY shrink site: the preempt/yield path
|
||||
// deliberately never checks (§4's bounded leak under
|
||||
// saturation — syscalls must not fire when scheduler
|
||||
// cycles are scarcest).
|
||||
maybe_shrink_stack(slot);
|
||||
if slot.word.park_return(gen) {
|
||||
// RFC 007 audit: an in-site park drops its sample
|
||||
// tail (nothing flushes it; on_resume re-arms).
|
||||
|
||||
Reference in New Issue
Block a user