feat(runtime,stack): sampled stack high-water + MADV_FREE shrink at actor-park (RFC 019 §§2–3)

hwm: AtomicUsize lands beside sp on the slot: the single context-save
site min-updates it (one branch + at most one Relaxed store into the
line the sp store just dirtied), install resets it to the fresh top.
Advisory by construction — correctness never depends on it. The mod-doc
ordering chain gains a line: hwm piggybacks the existing
Relaxed-store-before-Release pattern and adds no edges.

Shrink hook in the YieldIntent::Park arm only, before the park_return
Release transition — the owned window (obligation 1's assert-comment at
the site): after the sp store, before Parked is published, scheduler on
its own stack, actor saved and unstealable. It runs on both arms of the
park_return race (a consumed unpark flag means one wasted-but-harmless
madvise). The preempt/yield path deliberately never checks: §4's
bounded, self-healing leak under saturation, when syscalls are least
affordable.

SHRINK_THRESHOLD = 256 KiB and SHRINK_COOLDOWN = 64 parks are pub
constants with the ratified doc rationale, not Config fields. The freed
span is shrink_range(hwm, sp, page): whole pages of [hwm, sp − 1-page
redzone), rounded inward, checked arithmetic — adversarial inputs
collapse to None (obligation 2). MADV_FREE marks lazily; the kernel's
reclaim-under-pressure IS the hysteresis, cancel-on-write is the safety
net. parks_since_shrink + shrink_count ride the slot for the cooldown
and the future introspect surface.

Tests: 7 adversarial shrink_range units (inverted/empty spans, redzone
underflow, unaligned ends, sp-crossing sweep); integration — 8 MiB
reserve, ~3 MiB spike sampled via yield-at-depth, parks gated on
introspected Parked state past the cooldown, then ≥ 2 MiB LazyFree
asserted inside the stack's smaps range with live data intact; and the
inverse guard — a shallow never-spiking actor ends at exactly 0
LazyFree (also proves the parser isn't vacuously zero via the first
test).
This commit is contained in:
Claude (sandbox)
2026-08-08 14:30:32 +00:00
parent 3cb64eefc2
commit 8225716b11
3 changed files with 361 additions and 2 deletions
+112 -1
View File
@@ -65,6 +65,8 @@
//! word stores are `Release`, loads are `Acquire`. The chain that matters: //! word stores are `Release`, loads are `Acquire`. The chain that matters:
//! the park path stores `sp` (Relaxed) *before* its Release transition; any //! the park path stores `sp` (Relaxed) *before* its Release transition; any
//! later Acquire transition/load of the word therefore observes that `sp`. //! later Acquire transition/load of the word therefore observes that `sp`.
//! RFC 019's `hwm` (and the shrink that reads it) piggybacks this exact
//! pattern in the same pre-Release window and adds no edges.
//! The run-queue mutex independently provides the same edges today; the //! The run-queue mutex independently provides the same edges today; the
//! word's own ordering is what phase 3's lock-free queue will rely on. //! word's own ordering is what phase 3's lock-free queue will rely on.
//! //!
@@ -422,6 +424,23 @@ pub const DEFAULT_STACK_RESERVE: usize = 64 * 1024;
/// page so unprobed C frames cannot leap it. See [`Config::stack_guard`]. /// page so unprobed C frames cannot leap it. See [`Config::stack_guard`].
pub const DEFAULT_STACK_GUARD: usize = 64 * 1024; pub const DEFAULT_STACK_GUARD: usize = 64 * 1024;
/// RFC 019 §3: minimum releasable span (`sp − hwm` at park) before the
/// park-path shrink spends a syscall. A constant, not a `Config` field
/// (ratified): nobody tunes this well and the measured stakes are low — a
/// threshold-sized `MADV_FREE` costs ~3 µs against a ~100 ns park, paid
/// only on spike-recovery parks, which are rare by construction and *were*
/// the spike. Steady-state actors never reach the syscall: their check is
/// two Relaxed loads and a compare on a line the context-save just wrote.
pub const SHRINK_THRESHOLD: usize = 256 * 1024;
/// RFC 019 §3: parks between shrinks of one actor. Guards a few-µs cost, so
/// it can be coarse (parks, not wall time); the kernel's
/// reclaim-under-pressure-only handling of `MADV_FREE` is the real release
/// hysteresis — re-touched-before-pressure pages cost a 0.24 µs/page
/// cancel-write and no fault. A constant, not `Config` (ratified, same
/// rationale as [`SHRINK_THRESHOLD`]).
pub const SHRINK_COOLDOWN: u32 = 64;
pub(crate) type Closure = Box<dyn FnOnce() + Send>; pub(crate) type Closure = Box<dyn FnOnce() + Send>;
/// Lifecycle data, mutated only under the slot's cold [`RawMutex`]. Everything /// Lifecycle data, mutated only under the slot's cold [`RawMutex`]. Everything
@@ -463,6 +482,22 @@ pub(crate) struct Slot {
/// Release transition out of Running; read after the Acquire transition /// Release transition out of Running; read after the Acquire transition
/// Queued→Running. Relaxed is sufficient — ordering rides on `word`. /// Queued→Running. Relaxed is sufficient — ordering rides on `word`.
sp: AtomicUsize, sp: AtomicUsize,
/// RFC 019: sampled stack high-water — the minimum `sp` ever stored above,
/// i.e. the deepest excursion *observed at a switch point*. Advisory:
/// correctness never depends on it; its one job is "is a shrink worth a
/// syscall?". Declared adjacent to `sp` so the min-update dirties the
/// line the context-save just wrote. Same single-writer Relaxed
/// discipline as `sp`; reset to the fresh `sp` at install.
hwm: AtomicUsize,
/// RFC 019: parks since the last shrink (or install). Counted on every
/// pass through the Park arm by the owning scheduler thread; the shrink
/// fires only once this clears [`SHRINK_COOLDOWN`] *and* the releasable
/// span clears [`SHRINK_THRESHOLD`]. Single-writer Relaxed.
parks_since_shrink: AtomicU32,
/// RFC 019: shrinks performed on this incarnation (introspection lands
/// with the RFC's introspect surface; the counter exists from birth so
/// tests can rely on install resetting it). Single-writer Relaxed.
shrink_count: AtomicU32,
/// Pointer into the actor's `Arc<AtomicBool>` stop flag. Set at spawn, /// Pointer into the actor's `Arc<AtomicBool>` stop flag. Set at spawn,
/// nulled at finalize. The box outlives every read: it is only ever read /// nulled at finalize. The box outlives every read: it is only ever read
/// on the resume path while the actor cannot be finalized (it is on-CPU). /// on the resume path while the actor cannot be finalized (it is on-CPU).
@@ -534,6 +569,9 @@ impl Slot {
Self { Self {
word: StateWord::new(), word: StateWord::new(),
sp: AtomicUsize::new(0), sp: AtomicUsize::new(0),
hwm: AtomicUsize::new(0),
parks_since_shrink: AtomicU32::new(0),
shrink_count: AtomicU32::new(0),
stop_ptr: AtomicPtr::new(std::ptr::null_mut()), stop_ptr: AtomicPtr::new(std::ptr::null_mut()),
closure: AtomicPtr::new(std::ptr::null_mut()), closure: AtomicPtr::new(std::ptr::null_mut()),
overruns: AtomicU64::new(0), overruns: AtomicU64::new(0),
@@ -1356,6 +1394,49 @@ pub const ROOT_PID: Pid = Pid::new(u32::MAX, u32::MAX);
// Spawn-side slot installation // Spawn-side slot installation
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// ---------------------------------------------------------------------------
// Stack shrink — RFC 019 §3 (park path only)
// ---------------------------------------------------------------------------
/// The per-park shrink check. Called from the `YieldIntent::Park` arm inside
/// the owned window (see the assert-comment at the call site). Fast path —
/// no spike since the last shrink — is two Relaxed loads, a compare, and the
/// park counter bump, all on the slot line the context-save just wrote.
///
/// On a shrink: `MADV_FREE` the whole pages of `[hwm, sp − redzone)` (the
/// inward-rounded range from [`crate::stack::shrink_range`]), then reset
/// `hwm = sp` and the park counter. MADV_FREE only *marks*: the kernel
/// reclaims under pressure, skips re-dirtied pages, and refaults zero pages
/// for writes after reclaim — so an over-eager mark costs a cancel-write,
/// never data.
fn maybe_shrink_stack(slot: &Slot) {
let parks = slot.parks_since_shrink.load(Ordering::Relaxed).saturating_add(1);
slot.parks_since_shrink.store(parks, Ordering::Relaxed);
let sp = slot.sp.load(Ordering::Relaxed);
let hwm = slot.hwm.load(Ordering::Relaxed);
if sp.wrapping_sub(hwm) < SHRINK_THRESHOLD || sp < hwm {
return; // common case: nothing worth a syscall
}
if parks < SHRINK_COOLDOWN {
return;
}
let page = crate::stack::page_size();
if let Some((addr, len)) = crate::stack::shrink_range(hwm, sp, page) {
// Advisory: on the (kernel-config) chance MADV_FREE is unsupported,
// failing silently degrades to "never shrinks", which is correct.
unsafe {
libc::madvise(addr as *mut libc::c_void, len, libc::MADV_FREE);
}
slot.hwm.store(sp, Ordering::Relaxed);
slot.parks_since_shrink.store(0, Ordering::Relaxed);
slot.shrink_count.store(
slot.shrink_count.load(Ordering::Relaxed).saturating_add(1),
Ordering::Relaxed,
);
}
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Stack acquisition / recycling — RFC 019 pool rule // Stack acquisition / recycling — RFC 019 pool rule
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -1428,6 +1509,11 @@ pub(crate) fn install_actor(
cold.pending_io_result = None; cold.pending_io_result = None;
} }
slot.sp.store(sp, Ordering::Relaxed); slot.sp.store(sp, Ordering::Relaxed);
// RFC 019: a fresh incarnation starts with its high-water at the fresh
// top-of-stack `sp` and its shrink bookkeeping zeroed.
slot.hwm.store(sp, Ordering::Relaxed);
slot.parks_since_shrink.store(0, Ordering::Relaxed);
slot.shrink_count.store(0, Ordering::Relaxed);
slot.store_closure(closure); slot.store_closure(closure);
slot.reset_counters(); slot.reset_counters();
inner.live_actors.fetch_add(1, Ordering::Relaxed); inner.live_actors.fetch_add(1, Ordering::Relaxed);
@@ -1926,7 +2012,15 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
crate::preempt::clear_current_slot(); crate::preempt::clear_current_slot();
let intent = YIELD_INTENT.with(|c| c.get()); let intent = YIELD_INTENT.with(|c| c.get());
slot.sp.store(get_actor_sp(), Ordering::Relaxed); let saved_sp = get_actor_sp();
slot.sp.store(saved_sp, Ordering::Relaxed);
// RFC 019 §2: sampled high-water — one branch + at most one store
// into the line the store above just dirtied. Relaxed and advisory;
// it piggybacks the existing Relaxed-store-before-Release pattern
// (mod docs, "Memory ordering") and adds no edges.
if saved_sp < slot.hwm.load(Ordering::Relaxed) {
slot.hwm.store(saved_sp, Ordering::Relaxed);
}
if is_actor_done() { if is_actor_done() {
crate::te!(crate::trace::Event::Done(pid)); crate::te!(crate::trace::Event::Done(pid));
@@ -1948,6 +2042,23 @@ fn schedule_loop(inner: &Arc<RuntimeInner>, slot_idx: usize) {
inner.enqueue(pid); inner.enqueue(pid);
} }
YieldIntent::Park => { YieldIntent::Park => {
// RFC 019 §3 shrink window (correctness obligation 1):
// this site sits after the `sp` store above and before
// the `park_return` Release transition below publishes
// Parked — the scheduler is on its own stack and the
// actor is saved but not yet stealable, so the madvise
// races nothing (belt). MADV_FREE's cancel-on-write is
// the suspenders: even a racing writer could lose
// nothing written after the mark, and everything below
// live `sp` is dead by definition. Runs on BOTH arms of
// the park_return race — a consumed unpark flag means a
// wasted-but-harmless madvise on a rare window.
//
// This is the ONLY shrink site: the preempt/yield path
// deliberately never checks (§4's bounded leak under
// saturation — syscalls must not fire when scheduler
// cycles are scarcest).
maybe_shrink_stack(slot);
if slot.word.park_return(gen) { if slot.word.park_return(gen) {
// RFC 007 audit: an in-site park drops its sample // RFC 007 audit: an in-site park drops its sample
// tail (nothing flushes it; on_resume re-arms). // tail (nothing flushes it; on_resume re-arms).
+97 -1
View File
@@ -110,10 +110,106 @@ impl Drop for Stack {
} }
} }
fn page_size() -> usize { pub(crate) fn page_size() -> usize {
unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize } unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize }
} }
fn round_up(n: usize, align: usize) -> usize { fn round_up(n: usize, align: usize) -> usize {
(n + align - 1) & !(align - 1) (n + align - 1) & !(align - 1)
} }
/// The whole-page span the park-path shrink may `MADV_FREE` (RFC 019 §3):
/// `[page_up(hwm), page_down(sp − redzone))`, or `None` if no full page fits.
///
/// `hwm` is the sampled high-water (deepest observed `sp`); everything in
/// `[hwm, sp)` is below the live frame and dead by definition. One page of
/// redzone stays resident under live `sp` — it covers the SysV 128-byte red
/// zone plus spill margin with room to spare. Rounding is inward on both
/// ends so the result can never touch the redzone, cross `sp`, or dip below
/// `hwm`; all arithmetic is checked so adversarial inputs (`sp < redzone`,
/// `hwm ≥ sp`, values near the address-space edges) collapse to `None`
/// rather than a wild or negative-length range.
pub(crate) fn shrink_range(hwm: usize, sp: usize, page: usize) -> Option<(usize, usize)> {
debug_assert!(page.is_power_of_two());
if hwm >= sp {
return None;
}
let redzone = page;
let end = sp.checked_sub(redzone)? & !(page - 1); // page_down(sp − redzone)
let start = hwm.checked_add(page - 1)? & !(page - 1); // page_up(hwm)
if end > start {
Some((start, end - start))
} else {
None
}
}
#[cfg(test)]
mod tests {
use super::shrink_range;
const PG: usize = 4096;
#[test]
fn empty_and_inverted_spans_are_none() {
assert_eq!(shrink_range(0x8000_0000, 0x8000_0000, PG), None); // hwm == sp
assert_eq!(shrink_range(0x8000_1000, 0x8000_0000, PG), None); // hwm > sp
}
#[test]
fn span_smaller_than_redzone_plus_page_is_none() {
let sp = 0x8000_0000;
// Everything within redzone+1 page of sp: no full page clears both
// the redzone and the page_up(hwm) rounding.
assert_eq!(shrink_range(sp - PG, sp, PG), None);
assert_eq!(shrink_range(sp - 2 * PG + 1, sp, PG), None);
}
#[test]
fn exact_two_pages_frees_one() {
let sp = 0x8000_0000;
let hwm = sp - 2 * PG;
// [hwm, hwm+PG) frees; [sp−PG, sp) is redzone.
assert_eq!(shrink_range(hwm, sp, PG), Some((hwm, PG)));
}
#[test]
fn unaligned_ends_round_inward() {
let sp = 0x8000_0123; // live sp mid-page
let hwm = 0x7f00_0abc; // high-water mid-page
let (start, len) = shrink_range(hwm, sp, PG).unwrap();
assert_eq!(start % PG, 0);
assert_eq!(len % PG, 0);
assert!(start >= hwm); // never below the sampled high-water
assert!(start + len <= (sp - PG) & !(PG - 1)); // never into the redzone
}
#[test]
fn result_never_crosses_sp() {
// Sweep hwm across every offset of the page straddling the boundary.
let sp = 0x8000_0000 + 137;
for hwm in (sp - 4 * PG)..(sp) {
if let Some((start, len)) = shrink_range(hwm, sp, PG) {
assert!(start >= hwm);
assert!(start + len + PG <= sp + PG); // end ≤ page_down(sp − PG) < sp
assert!(len > 0);
}
}
}
#[test]
fn underflow_near_zero_is_none() {
assert_eq!(shrink_range(0, PG - 1, PG), None); // sp < redzone
assert_eq!(shrink_range(0, 0, PG), None);
}
#[test]
fn big_span_frees_interior() {
let sp = 0x8000_0000;
let spike = 4 * 1024 * 1024;
let hwm = sp - spike;
let (start, len) = shrink_range(hwm, sp, PG).unwrap();
assert_eq!(start, hwm); // aligned input: starts exactly at hwm
assert_eq!(len, spike - PG); // everything but the redzone page
}
}
+152
View File
@@ -0,0 +1,152 @@
//! RFC 019 commit 3 — park-path stack shrink, observed from the outside.
//!
//! The one integration-level claim of the shrink machinery: an actor that
//! spikes deep, returns shallow, and then parks past the cooldown gets its
//! dead span MADV_FREE'd — visible as `LazyFree` in `/proc/self/smaps`
//! within the stack's address range — while everything live survives.
//!
//! The high-water mark is *sampled* at context-save, so the spike yields
//! once at max depth to guarantee a sample there (in production, preemption
//! provides the quasi-random samples; a test must not rely on luck).
use smarm::runtime::{Config, SHRINK_COOLDOWN, SHRINK_THRESHOLD};
use smarm::{actor_info, channel, spawn, spawn_with, yield_now, ActorState, SpawnOpts};
/// Burn ~`frames` × 4 KiB of stack, yielding once at the bottom so the
/// context-save samples `sp` at max depth.
#[inline(never)]
fn burn_stack_yielding(frames: usize) -> u64 {
let mut local = [0u8; 4096];
local[0] = frames as u8;
let below = if frames == 0 {
yield_now();
0
} else {
burn_stack_yielding(frames - 1)
};
std::hint::black_box(&mut local);
below.wrapping_add(local[0] as u64)
}
/// Sum the `LazyFree:` kB of every smaps mapping intersecting [lo, hi).
fn lazy_free_bytes_in(lo: usize, hi: usize) -> usize {
let smaps = std::fs::read_to_string("/proc/self/smaps").unwrap();
let mut total_kb = 0usize;
let mut in_range = false;
for line in smaps.lines() {
if let Some((range, _)) = line.split_once(' ') {
if let Some((a, b)) = range.split_once('-') {
if let (Ok(start), Ok(end)) =
(usize::from_str_radix(a, 16), usize::from_str_radix(b, 16))
{
in_range = start < hi && end > lo;
continue;
}
}
}
if in_range {
if let Some(rest) = line.strip_prefix("LazyFree:") {
let kb: usize = rest.trim().trim_end_matches(" kB").trim().parse().unwrap();
total_kb += kb;
}
}
}
total_kb * 1024
}
#[test]
fn spike_then_parks_marks_lazyfree_and_keeps_live_data() {
// Single scheduler: the controller can gate on the worker being Parked.
let rt = smarm::runtime::init(Config::exact(1));
rt.run(|| {
let (park_tx, park_rx) = channel::<()>();
let (done_tx, done_rx) = channel::<(usize, u64)>();
let spike = 768 * 4096; // ~3 MiB, well past SHRINK_THRESHOLD
assert!(spike > SHRINK_THRESHOLD);
let worker = spawn_with(
SpawnOpts { stack_reserve: Some(8 * 1024 * 1024), ..SpawnOpts::default() },
move || {
// Live data that must survive the shrink, and an anchor
// address inside the stack for the smaps scan.
let live = [0xA5u8; 64];
let anchor = live.as_ptr() as usize;
// Spike: ~3 MiB deep, sampled at the bottom, unwound.
std::hint::black_box(burn_stack_yielding(768));
// Park past the cooldown. Each recv on the drained inbox is
// one park; the controller sends only when it sees us Parked.
for _ in 0..(SHRINK_COOLDOWN + 8) {
park_rx.recv().unwrap();
}
// Measure from inside: the stack spans ≤ 8 MiB below anchor.
let lazy = lazy_free_bytes_in(anchor - 8 * 1024 * 1024, anchor + 4096);
let checksum = live.iter().map(|&b| b as u64).sum();
done_tx.send((lazy, checksum)).unwrap();
},
);
let wpid = worker.pid();
for _ in 0..(SHRINK_COOLDOWN + 8) {
// Gate: send only once the worker is genuinely parked so every
// round is a real park-on-empty-mailbox.
loop {
match actor_info(wpid) {
Some(info) if info.state == ActorState::Parked => break,
Some(_) => yield_now(),
None => panic!("worker died early"),
}
}
park_tx.send(()).unwrap();
}
let (lazy, checksum) = done_rx.recv().unwrap();
// The spike was ~3 MiB; demand at least 2 MiB marked to leave slack
// for the redzone, rounding, and pages the unwind re-dirtied.
assert!(
lazy >= 2 * 1024 * 1024,
"expected ≥ 2 MiB LazyFree in the stack range, got {} bytes",
lazy
);
assert_eq!(checksum, 64 * 0xA5u64, "live stack data corrupted by shrink");
worker.join().unwrap();
});
}
/// Steady-state actors must never pay the syscall: an actor that parks a lot
/// but never spikes past the threshold ends with zero LazyFree in its stack.
#[test]
fn shallow_actor_never_shrinks() {
let rt = smarm::runtime::init(Config::exact(1));
rt.run(|| {
let (park_tx, park_rx) = channel::<()>();
let (done_tx, done_rx) = channel::<usize>();
let worker = spawn(move || {
let probe = 0u8;
let anchor = &probe as *const u8 as usize;
for _ in 0..(SHRINK_COOLDOWN + 8) {
park_rx.recv().unwrap();
}
done_tx.send(lazy_free_bytes_in(anchor - 64 * 1024, anchor + 4096)).unwrap();
});
let wpid = worker.pid();
for _ in 0..(SHRINK_COOLDOWN + 8) {
loop {
match actor_info(wpid) {
Some(info) if info.state == ActorState::Parked => break,
Some(_) => yield_now(),
None => panic!("worker died early"),
}
}
park_tx.send(()).unwrap();
}
assert_eq!(done_rx.recv().unwrap(), 0, "steady-state actor was shrunk");
worker.join().unwrap();
});
}