perf(preempt): rdtsc unserialised by default; causal attribution opts into lfence

reset_timeslice paid an lfence pipeline drain on every resume via the
shared rdtsc() helper. Nothing in preempt.rs needs it: the timeslice
arm/expiry compare against a ~1e5-cycle slice, and an early stamp only
makes the slice look more used. The consumer that does need it — causal
site attribution, where a speculative early read misattributes a site's
tail — now calls rdtsc_serialising() explicitly (cold_check sample,
SiteGuard enter/exit).

Measured vs 31dc26a (baseline commit), 24-core box, rq-mpmc default:
- switch_cost, taskset -c 2, 9 interleaved old/new pairs: mean_cyc
  149 -> 117 per roundtrip (-32, -21%); mean_ns 43.2 -> 34.9.
- sweep.py regress + run, 20T, two runs agree:
  yield_in_hot_loop 1T 40171 -> 30277/30497 us (-24%)
  yield_many        1T 12513 ->  9965/10013 us (-20%)
  ping_pong_oneshot unchanged (RFC 005 wake-slot handoffs skip
  reset_timeslice, so that path never paid the lfence).
  All other rows within the box's ~+-15% noise floor.
- 1-core sandbox: 218 -> 206 cyc (understates by ~3x).
- bench binary lfence count 24 -> 4 (survivors = the bench's own
  rdtscp;lfence bracket). Tests green.
Baseline (benches/baseline.json) not re-saved in this commit.
This commit is contained in:
claude-asm-audit
2026-08-21 12:22:46 +00:00
parent 343e53e17b
commit 5ddd122711
2 changed files with 27 additions and 3 deletions
+8 -3
View File
@@ -279,7 +279,9 @@ mod inner {
if slot.is_null() { if slot.is_null() {
return; return;
} }
let now = preempt::rdtsc(); // Serialised: `now` closes an interval attributed to a code site; a
// speculative early read would drop that site's tail (RFC 007).
let now = preempt::rdtsc_serialising();
let last = LAST_SAMPLE_TSC.with(|c| c.replace(now)); let last = LAST_SAMPLE_TSC.with(|c| c.replace(now));
let target_site = (exp >> 32) as u32; let target_site = (exp >> 32) as u32;
let pct = exp & 0xffff_ffff; let pct = exp & 0xffff_ffff;
@@ -374,7 +376,8 @@ mod inner {
let target = (exp >> 32) as u32; let target = (exp >> 32) as u32;
let pct = exp & 0xffff_ffff; let pct = exp & 0xffff_ffff;
if old == target && new != target { if old == target && new != target {
let now = preempt::rdtsc(); // Serialised: guard exit bounds the site's interval exactly.
let now = preempt::rdtsc_serialising();
let last = LAST_SAMPLE_TSC.with(|c| c.replace(now)); let last = LAST_SAMPLE_TSC.with(|c| c.replace(now));
if pct > 0 { if pct > 0 {
if last != 0 { if last != 0 {
@@ -386,7 +389,9 @@ mod inner {
} }
} }
} else if new == target && old != target { } else if new == target && old != target {
LAST_SAMPLE_TSC.with(|c| c.set(preempt::rdtsc())); // Serialised: an early arm would let pre-site work leak into the
// first in-site interval — the over-attribution this guards.
LAST_SAMPLE_TSC.with(|c| c.set(preempt::rdtsc_serialising()));
} }
} }
+19
View File
@@ -198,8 +198,27 @@ pub(crate) fn elapsed_slice_cycles() -> u64 {
rdtsc().saturating_sub(TIMESLICE_START.with(|c| c.get())) rdtsc().saturating_sub(TIMESLICE_START.with(|c| c.get()))
} }
/// Read the TSC, unserialised. The core may execute this before earlier
/// instructions retire (or after later ones start), so a single stamp can
/// land tens of cycles — worst case a stalled load's worth — early or late.
/// That is negligible against every consumer in this module: the timeslice
/// arm/expiry compare against a ~10^5-cycle slice, and an early stamp only
/// makes a slice look *more* used (expires marginally sooner, never later).
/// The `lfence` this used to carry was a pipeline drain paid on every
/// resume; the one place that needs it is causal-site attribution, which
/// opts in via [`rdtsc_serialising`].
#[inline(always)] #[inline(always)]
pub fn rdtsc() -> u64 { pub fn rdtsc() -> u64 {
// SAFETY: x86-64 only (this crate is x86-64 Linux only).
unsafe { core::arch::x86_64::_rdtsc() }
}
/// Read the TSC after all prior instructions have completed locally.
/// Use where a stamp bounds an interval attributed to *code* — a speculative
/// early read would credit the tail of that code to whatever comes next.
/// Costs a pipeline drain; keep it off the per-resume path.
#[inline(always)]
pub fn rdtsc_serialising() -> u64 {
unsafe { unsafe {
// SAFETY: x86-64 only. `lfence` serialises the instruction stream so // SAFETY: x86-64 only. `lfence` serialises the instruction stream so
// we don't measure time before prior instructions retire. // we don't measure time before prior instructions retire.