feat(channel): migrate Inner to RawMutex; two-class lock order (Leaf -> Channel)
Phase 2 surfaced the hole this closes: channel guards were held with preemption enabled, so a timeslice switch inside a channel critical section could release the pthread mutex from a different OS thread (UB). RawMutex disables preemption for the guard span and is cross-thread-release sound by construction; poison goes away with it. The strict single-class leaf rule cannot survive the migration: finalize clones the supervisor/trap senders and monitor() clones the Down sender, all under a cold lock, and those senders live in the slot - the nesting is structural. The leaf check is therefore generalized to two classes: Leaf (cold locks, free list, stack pool) and Channel. Order is Leaf -> Channel, at most one of each; both directions of violation are debug-asserted at the acquisition site. Channel critical sections call only the lock-free unpark protocol, so the order is acyclic. Tests: class-ordering unit tests in raw_mutex (allowed nesting + all three rejected shapes), plus a multi-scheduler integration test driving channels through monitor churn and actor death so any ordering regression trips the debug assert instead of deadlocking.
This commit is contained in:
smarm
+125
-25
@@ -25,12 +25,20 @@
|
||||
//! Tricky", mutex3). 0 = unlocked, 1 = locked, 2 = locked with (possible)
|
||||
//! waiters. Uncontended lock/unlock is one CAS / one swap, no syscall.
|
||||
//!
|
||||
//! Lock-order position: slot cold locks, the free list, and the stack pool
|
||||
//! are all `RawMutex`es and all *leaves among themselves* — never hold two at
|
||||
//! once. Holding a `RawMutex` while pushing to the run queue is permitted
|
||||
//! (e.g. unpark from inside a cold section); the reverse — taking any
|
||||
//! `RawMutex` from inside a run-queue op — cannot arise (queue ops call
|
||||
//! nothing).
|
||||
//! Lock-order position — two classes (see [`LockClass`]):
|
||||
//!
|
||||
//! - **Leaf**: slot cold locks, the free list, the stack pool, the name
|
||||
//! registry. Mutual leaves — never hold two at once.
|
||||
//! - **Channel**: a channel's internal lock. May be acquired *under* a Leaf
|
||||
//! (finalize clones the supervisor/trap senders, and `monitor()` clones the
|
||||
//! Down sender, all under a cold lock — structural, the sender lives in the
|
||||
//! slot), but nothing may be acquired under a Channel lock: channel
|
||||
//! critical sections call only the lock-free unpark protocol.
|
||||
//!
|
||||
//! So the total order is Leaf → Channel, one of each at most. Holding either
|
||||
//! while pushing to the run queue is permitted (unpark from inside a cold or
|
||||
//! channel section); the reverse — taking any `RawMutex` from inside a
|
||||
//! run-queue op — cannot arise (queue ops call nothing).
|
||||
|
||||
use std::cell::UnsafeCell;
|
||||
use std::ops::{Deref, DerefMut};
|
||||
@@ -45,37 +53,78 @@ const CONTENDED: u32 = 2;
|
||||
/// sender), so a short spin almost always avoids the syscall.
|
||||
const SPIN_LIMIT: u32 = 64;
|
||||
|
||||
// The leaf rule, mechanically enforced (debug builds): slot cold locks, the
|
||||
// free list, and the stack pool — i.e. every RawMutex — are mutual leaves.
|
||||
// Holding two at once is a deadlock waiting for the right interleaving, so
|
||||
// fail at the acquisition that violates it, not in the eventual hang.
|
||||
/// Which rung of the two-rung lock order a `RawMutex` occupies. Debug builds
|
||||
/// enforce the order mechanically (see the module docs); release builds carry
|
||||
/// no state and no checks.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(crate) enum LockClass {
|
||||
/// Runtime cold data: slot cold locks, free list, stack pool, registry.
|
||||
/// Mutual leaves among themselves; a Channel lock may be taken under one.
|
||||
Leaf,
|
||||
/// A channel's internal lock. One at a time, nothing acquired under it;
|
||||
/// may itself be acquired under a Leaf.
|
||||
Channel,
|
||||
}
|
||||
|
||||
// The ordering rules, mechanically enforced (debug builds): a deadlock from a
|
||||
// violated order is a hang waiting for the right interleaving, so fail at the
|
||||
// acquisition that violates it, not in the eventual hang.
|
||||
#[cfg(debug_assertions)]
|
||||
thread_local! {
|
||||
static RAW_MUTEXES_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
|
||||
static LEAVES_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
|
||||
static CHANNELS_HELD: std::cell::Cell<u32> = const { std::cell::Cell::new(0) };
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn leaf_check_acquire() {
|
||||
fn order_check_acquire(class: LockClass) {
|
||||
#[cfg(debug_assertions)]
|
||||
RAW_MUTEXES_HELD.with(|c| {
|
||||
debug_assert_eq!(
|
||||
c.get(),
|
||||
0,
|
||||
"leaf rule violated: acquiring a RawMutex while already holding one \
|
||||
(cold locks / free list / stack pool are mutual leaves)"
|
||||
);
|
||||
c.set(c.get() + 1);
|
||||
});
|
||||
match class {
|
||||
LockClass::Leaf => LEAVES_HELD.with(|l| {
|
||||
debug_assert_eq!(
|
||||
l.get(),
|
||||
0,
|
||||
"lock order violated: acquiring a Leaf RawMutex while already \
|
||||
holding one (cold locks / free list / stack pool / registry \
|
||||
are mutual leaves)"
|
||||
);
|
||||
CHANNELS_HELD.with(|c| {
|
||||
debug_assert_eq!(
|
||||
c.get(),
|
||||
0,
|
||||
"lock order violated: acquiring a Leaf RawMutex under a \
|
||||
channel lock (order is Leaf -> Channel, never the reverse)"
|
||||
);
|
||||
});
|
||||
l.set(l.get() + 1);
|
||||
}),
|
||||
LockClass::Channel => CHANNELS_HELD.with(|c| {
|
||||
debug_assert_eq!(
|
||||
c.get(),
|
||||
0,
|
||||
"lock order violated: acquiring a channel lock while already \
|
||||
holding one (channel locks are mutual leaves)"
|
||||
);
|
||||
c.set(c.get() + 1);
|
||||
}),
|
||||
}
|
||||
#[cfg(not(debug_assertions))]
|
||||
let _ = class;
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn leaf_check_release() {
|
||||
fn order_check_release(class: LockClass) {
|
||||
#[cfg(debug_assertions)]
|
||||
RAW_MUTEXES_HELD.with(|c| c.set(c.get() - 1));
|
||||
match class {
|
||||
LockClass::Leaf => LEAVES_HELD.with(|c| c.set(c.get() - 1)),
|
||||
LockClass::Channel => CHANNELS_HELD.with(|c| c.set(c.get() - 1)),
|
||||
}
|
||||
#[cfg(not(debug_assertions))]
|
||||
let _ = class;
|
||||
}
|
||||
|
||||
pub(crate) struct RawMutex<T> {
|
||||
state: AtomicU32,
|
||||
class: LockClass,
|
||||
data: UnsafeCell<T>,
|
||||
}
|
||||
|
||||
@@ -86,9 +135,20 @@ unsafe impl<T: Send> Send for RawMutex<T> {}
|
||||
unsafe impl<T: Send> Sync for RawMutex<T> {}
|
||||
|
||||
impl<T> RawMutex<T> {
|
||||
/// A Leaf-class mutex — the default for runtime cold data.
|
||||
pub(crate) const fn new(data: T) -> Self {
|
||||
Self::with_class(data, LockClass::Leaf)
|
||||
}
|
||||
|
||||
/// A Channel-class mutex — for channel internals only.
|
||||
pub(crate) const fn new_channel(data: T) -> Self {
|
||||
Self::with_class(data, LockClass::Channel)
|
||||
}
|
||||
|
||||
pub(crate) const fn with_class(data: T, class: LockClass) -> Self {
|
||||
Self {
|
||||
state: AtomicU32::new(UNLOCKED),
|
||||
class,
|
||||
data: UnsafeCell::new(data),
|
||||
}
|
||||
}
|
||||
@@ -98,7 +158,7 @@ impl<T> RawMutex<T> {
|
||||
// Enter NoPreempt *before* acquiring, so a preemption can't fire
|
||||
// between acquisition and guard construction.
|
||||
let prev_preempt = crate::preempt::PREEMPTION_ENABLED.with(|c| c.replace(false));
|
||||
leaf_check_acquire();
|
||||
order_check_acquire(self.class);
|
||||
if self
|
||||
.state
|
||||
.compare_exchange(UNLOCKED, LOCKED, Ordering::Acquire, Ordering::Relaxed)
|
||||
@@ -172,7 +232,7 @@ impl<T> Drop for RawMutexGuard<'_, T> {
|
||||
#[inline]
|
||||
fn drop(&mut self) {
|
||||
self.m.unlock();
|
||||
leaf_check_release();
|
||||
order_check_release(self.m.class);
|
||||
// Restore preemption only after the lock is released.
|
||||
crate::preempt::PREEMPTION_ENABLED.with(|c| c.set(self.prev_preempt));
|
||||
}
|
||||
@@ -257,4 +317,44 @@ mod tests {
|
||||
*m.lock() += 1;
|
||||
assert_eq!(*m.lock(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn channel_lock_nests_under_leaf() {
|
||||
// The permitted ordering: Leaf -> Channel (finalize/monitor clone a
|
||||
// sender under a cold lock). Must not trip the order check.
|
||||
let leaf = RawMutex::new(0u64);
|
||||
let chan = RawMutex::new_channel(0u64);
|
||||
let _l = leaf.lock();
|
||||
let _c = chan.lock();
|
||||
}
|
||||
|
||||
#[cfg(debug_assertions)]
|
||||
#[test]
|
||||
#[should_panic(expected = "lock order violated")]
|
||||
fn leaf_under_channel_is_rejected() {
|
||||
let leaf = RawMutex::new(0u64);
|
||||
let chan = RawMutex::new_channel(0u64);
|
||||
let _c = chan.lock();
|
||||
let _l = leaf.lock(); // Channel -> Leaf: forbidden
|
||||
}
|
||||
|
||||
#[cfg(debug_assertions)]
|
||||
#[test]
|
||||
#[should_panic(expected = "lock order violated")]
|
||||
fn two_leaves_are_rejected() {
|
||||
let a = RawMutex::new(0u64);
|
||||
let b = RawMutex::new(0u64);
|
||||
let _ga = a.lock();
|
||||
let _gb = b.lock(); // leaves are mutual: forbidden
|
||||
}
|
||||
|
||||
#[cfg(debug_assertions)]
|
||||
#[test]
|
||||
#[should_panic(expected = "lock order violated")]
|
||||
fn two_channel_locks_are_rejected() {
|
||||
let a = RawMutex::new_channel(0u64);
|
||||
let b = RawMutex::new_channel(0u64);
|
||||
let _ga = a.lock();
|
||||
let _gb = b.lock(); // channel locks are mutual leaves: forbidden
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user