fix(causal): flush target-site samples at guard boundaries (RFC 007)
Samples were taken only when maybe_preempt's cold block happened to fire in-site, so the interval between the last check and SiteGuard drop was discarded on every site entry. Measured live on the 24-core box: 22-29us lost per entry, a constant attribution efficiency of ~0.93-0.94, which under-reported every impact (+83.5% where theory says +100%; the observed shortfall fits 1/(1-pct*eff)-1 at both 25% and 50%). SiteGuard enter/drop now call site_transition(): leaving the target site flushes the pending interval into the ledger (sample-only, never spins, so safe under no-preempt regions); entering the target site re-arms the sample clock so pre-site time is never attributed (the symmetric over-attribution). Winner attribution is factored into attribute(), shared by the cold check and the flush, with the same interval clamps. Adds examples/causal_attrib_probe.rs (ground-truth in-site time vs ledger attribution, the probe that confirmed the leak) and the site_boundaries_flush_tail regression test (a site entry that never hits a cold check must still be attributed). Also gates causal_probe on smarm-causal in Cargo.toml - it never was, so featureless builds of the examples were broken.
This commit is contained in:
+62
-10
@@ -145,6 +145,7 @@ mod inner {
|
||||
// field docs for the lifetime argument.
|
||||
let prev = unsafe { (*slot).causal_site() };
|
||||
unsafe { (*slot).set_causal_site(site) };
|
||||
site_transition(slot, prev, site);
|
||||
SiteGuard { slot, prev }
|
||||
}
|
||||
}
|
||||
@@ -152,9 +153,11 @@ mod inner {
|
||||
impl Drop for SiteGuard {
|
||||
fn drop(&mut self) {
|
||||
if !self.slot.is_null() {
|
||||
// SAFETY: as in `enter` — the actor (and thus its slot) is
|
||||
// alive for as long as this guard is on its stack.
|
||||
// SAFETY (both): as in `enter` — the actor (and thus its
|
||||
// slot) is alive for as long as this guard is on its stack.
|
||||
let site = unsafe { (*self.slot).causal_site() };
|
||||
unsafe { (*self.slot).set_causal_site(self.prev) };
|
||||
site_transition(self.slot, site, self.prev);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -255,14 +258,9 @@ mod inner {
|
||||
if last == 0 {
|
||||
return; // unarmed clock: no interval to attribute
|
||||
}
|
||||
let interval = now.saturating_sub(last);
|
||||
if interval == 0 || interval > MAX_SAMPLE_CYCLES {
|
||||
return; // clock hiccup: discard the sample, not the run
|
||||
}
|
||||
let delta = interval.saturating_mul(pct) / 100;
|
||||
GLOBAL_DELAY.fetch_add(delta, Ordering::Relaxed);
|
||||
let mine = unsafe { (*slot).causal_delay() };
|
||||
unsafe { (*slot).set_causal_delay(mine.wrapping_add(delta)) };
|
||||
// SAFETY: `slot` is the on-CPU actor's slot (checked non-null
|
||||
// above); see `check_cancelled` for the lifetime argument.
|
||||
unsafe { attribute(slot, now.saturating_sub(last), pct) };
|
||||
} else {
|
||||
// Not the winner: chase the global ledger by spinning off the
|
||||
// difference, then push the slice start forward so injected
|
||||
@@ -285,6 +283,60 @@ mod inner {
|
||||
}
|
||||
}
|
||||
|
||||
/// Attribute one target-site sample of `interval` cycles at `pct`%:
|
||||
/// grow the global ledger and credit the sampling actor's own ledger by
|
||||
/// the same amount — the credited gap *is* the virtual speedup. Shared
|
||||
/// by the cold check and the guard-boundary flush. Applies the same
|
||||
/// clamps as sampling always has: zero intervals and clock hiccups are
|
||||
/// discarded, not the run.
|
||||
///
|
||||
/// SAFETY: `slot` must point at the on-CPU actor's slot (the
|
||||
/// `check_cancelled` lifetime argument).
|
||||
unsafe fn attribute(slot: *const crate::runtime::Slot, interval: u64, pct: u64) {
|
||||
if interval == 0 || interval > MAX_SAMPLE_CYCLES {
|
||||
return;
|
||||
}
|
||||
let delta = interval.saturating_mul(pct) / 100;
|
||||
GLOBAL_DELAY.fetch_add(delta, Ordering::Relaxed);
|
||||
let mine = (*slot).causal_delay();
|
||||
(*slot).set_causal_delay(mine.wrapping_add(delta));
|
||||
}
|
||||
|
||||
/// Site-boundary hook, called by `SiteGuard` enter/drop when the
|
||||
/// actor's current site changes from `old` to `new`. Sample-only —
|
||||
/// never spins — so it is safe anywhere, including no-preempt regions
|
||||
/// where `check()` cannot run.
|
||||
///
|
||||
/// - Leaving the experiment's target site: flush the pending interval.
|
||||
/// Cold checks only sample when they happen to fire in-site, so the
|
||||
/// tail between the last check and the guard drop was otherwise
|
||||
/// discarded on every site entry — measured live at ~22-29µs/entry,
|
||||
/// ~6-7% of all target time (eff 0.93), which under-reported every
|
||||
/// impact (+83.5% where theory says +100%).
|
||||
/// - Entering the target site: re-arm the sample clock, so time spent
|
||||
/// *before* the site can never be attributed to it by the first
|
||||
/// in-site check (the symmetric over-attribution).
|
||||
#[inline]
|
||||
fn site_transition(slot: *const crate::runtime::Slot, old: u32, new: u32) {
|
||||
let exp = EXPERIMENT.load(Ordering::Relaxed);
|
||||
if exp == 0 || old == new {
|
||||
return;
|
||||
}
|
||||
let target = (exp >> 32) as u32;
|
||||
let pct = exp & 0xffff_ffff;
|
||||
if old == target && new != target {
|
||||
let now = preempt::rdtsc();
|
||||
let last = LAST_SAMPLE_TSC.with(|c| c.replace(now));
|
||||
if last != 0 && pct > 0 {
|
||||
// SAFETY: forwarded from the guard, which holds the on-CPU
|
||||
// actor's slot for its whole life (see `SiteGuard::slot`).
|
||||
unsafe { attribute(slot, now.saturating_sub(last), pct) };
|
||||
}
|
||||
} else if new == target && old != target {
|
||||
LAST_SAMPLE_TSC.with(|c| c.set(preempt::rdtsc()));
|
||||
}
|
||||
}
|
||||
|
||||
/// Resume-path hook (scheduler thread, actor off-CPU). Two duties:
|
||||
///
|
||||
/// - If the last deschedule was a *real park*, time blocked absorbs any
|
||||
|
||||
Reference in New Issue
Block a user