Markk116/smarm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(cluster): RFC 010 c6c — heartbeat send + fixed-timeout liveness + teardown

Browse Source 
The timeout arm of the connection actor's select: HEARTBEAT_INTERVAL (1s)
paces outbound Frame::Heartbeat (first at spawn, so the peer's window
starts fed) and LIVENESS_TIMEOUT (4s = 4 intervals) declares the peer dead
when no inbound frame arrives inside it — any frame resets the window, so
heartbeats keep an idle connection alive and real traffic (c8+) counts for
free. Fire => close + exit; the manager's monitor reaps the table entry as
on every other exit path. Fixed timeout per RFC v2 §5 (control connection,
heartbeats can't queue behind bulk). Intervals are the one-viable-answer
call flagged for veto at diff review.

The pump was made non-blocking to keep the deadlines honest: a plain recv()
blocks into the socket while the buffer holds a partial frame, parking the
actor past its timers. Two additive FramedConn methods (read_once,
next_buffered): exactly one socket read per level-triggered readable wake
(cannot block, cannot strand — leftovers re-signal), then drain every
complete buffered frame. Liveness resets only on complete frames.

No-fd transports (loopback) still get the command-only loop: no readiness
means no timers, same caveat as recv_deadline.

tests/cluster_conn_liveness.rs 3/0, stable over 5 runs (raw far end over
localhost TCP: heartbeats appear unprompted; mute peer still up at half
the window, gone after it; heartbeat-only peer survives 1.5x the window,
then reaped once silenced). Cluster suites regression-clean; clippy --lib
green both configs; fmt clean.
This commit is contained in:
Claude
2026-08-15 06:36:07 +00:00
parent c8ed858e4c
commit ad4958421f
3 changed files with 318 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/cluster/conn.rs
+110 -36
View File
@@ -15,11 +15,17 @@
//! ownership of its [`ConnHandle`] and monitors the actor, so any exit
//! deregisters the connection. The actor itself holds no authority over its
//! own lifetime: it runs until the manager drops its handle (deregistration,
//! `Disconnect`, or manager shutdown) or the connection ends. Heartbeat send
//! and fixed-timeout liveness join the loop in c6c (the timeout arm of the
//! same `select`).
//! `Disconnect`, or manager shutdown), the connection ends, or liveness
//! expires. Heartbeat send and fixed-timeout liveness are the timeout arm of
//! the same `select` (c6c): [`HEARTBEAT_INTERVAL`] paces outbound
//! [`Frame::Heartbeat`](crate::cluster::envelope::Frame::Heartbeat)s, and a
//! [`LIVENESS_TIMEOUT`] window — reset by any inbound frame — tears the
//! connection down when it empties.
use crate::channel::{channel, Receiver, Selectable, Sender};
use std::time::{Duration, Instant};
use crate::channel::{channel, try_select_timeout, Receiver, Selectable, Sender};
use crate::cluster::envelope::Frame;
use crate::cluster::handshake::Peer;
use crate::cluster::manager::{Call, Registered, Reply, MANAGER};
use crate::cluster::transport::FramedConn;
@@ -88,39 +94,81 @@ pub fn spawn_established(framed: FramedConn, peer: Peer) -> Result<Pid, Register
}
}
/// How often this end emits [`Frame::Heartbeat`] on an idle connection. The
/// first one goes out immediately at spawn, so the peer's liveness window
/// starts fed.
pub const HEARTBEAT_INTERVAL: Duration = Duration::from_secs(1);
/// How long the connection may go without a single inbound frame before it
/// is declared dead and torn down. Any inbound frame resets the window —
/// heartbeats keep an idle connection alive, and real traffic (c8+) counts
/// for free. Fixed by design (RFC v2 §5): this is the control connection, a
/// heartbeat can never queue behind bulk traffic, so a fixed timeout is an
/// honest detector.
pub const LIVENESS_TIMEOUT: Duration = Duration::from_secs(4);
fn run(mut framed: FramedConn, _peer: Peer, cmd_rx: Receiver<Cmd>) {
match framed.readable_arm() {
Some(arm) => run_live(&mut framed, arm, &cmd_rx),
None => run_inert(&cmd_rx),
}
framed.close();
}
/// The steady-state loop over an fd-backed connection: one
/// `select_timeout` folds the command inbox, socket readability, and the
/// nearer of the two deadlines (`hb_send`, `liveness`) into a single wait.
fn run_live(framed: &mut FramedConn, arm: crate::scheduler::FdArm, cmd_rx: &Receiver<Cmd>) {
let mut next_hb = Instant::now();
let mut live_until = Instant::now() + LIVENESS_TIMEOUT;
loop {
match framed.readable_arm() {
Some(arm) => {
let arms: [&dyn Selectable; 2] = [&cmd_rx, &arm];
match crate::channel::try_select(&arms) {
Ok(0) => {
if should_stop(&cmd_rx) {
break;
}
}
Ok(_) => {
if pump_readable(&mut framed) {
break;
}
}
// The fd arm failed to register — the connection is gone.
Err(_) => break,
}
let now = Instant::now();
if now >= live_until {
break; // liveness expired: the peer is dead to us
}
if now >= next_hb {
if framed.send(&Frame::Heartbeat).is_err() {
break;
}
None => {
// No fd to select on (loopback): only a command can end the
// wait. Liveness over such a transport is out of scope.
let arms: [&dyn Selectable; 1] = [&cmd_rx];
let _ = crate::channel::select(&arms);
if should_stop(&cmd_rx) {
next_hb = now + HEARTBEAT_INTERVAL;
}
let wait = next_hb.min(live_until).saturating_duration_since(now);
let arms: [&dyn Selectable; 2] = [cmd_rx, &arm];
match try_select_timeout(&arms, wait) {
Ok(Some(0)) => {
if should_stop(cmd_rx) {
break;
}
}
Ok(Some(_)) => match pump_readable(framed) {
Pump::Ended => break,
Pump::Frames(n) => {
if n > 0 {
live_until = Instant::now() + LIVENESS_TIMEOUT;
}
}
},
// A deadline passed; the top of the loop acts on whichever.
Ok(None) => {}
// The fd arm failed to register — the connection is gone.
Err(_) => break,
}
}
}
framed.close();
/// No fd to select on (loopback): only a command can end the wait, and
/// neither heartbeats nor liveness run — a transport that can't report
/// readiness can't be timed either (same caveat as
/// [`FramedConn::recv_deadline`]). Loopback is a test transport; every real
/// connection is fd-backed.
fn run_inert(cmd_rx: &Receiver<Cmd>) {
loop {
let arms: [&dyn Selectable; 1] = [cmd_rx];
let _ = crate::channel::select(&arms);
if should_stop(cmd_rx) {
break;
}
}
}
/// Drain the command arm. Returns `true` when the actor should exit — a
@@ -133,13 +181,39 @@ fn should_stop(cmd_rx: &Receiver<Cmd>) -> bool {
}
}
/// Consume whatever is readable now. Returns `true` when the connection has
/// ended (clean EOF or an unrecoverable stream error). This chunk does not
/// interpret frames; c6c handles heartbeats and resets the liveness timer here.
fn pump_readable(framed: &mut FramedConn) -> bool {
match framed.recv() {
Ok(Some(_frame)) => false,
Ok(None) => true, // clean EOF at a frame boundary
Err(_) => true, // corrupt / truncated / io
/// What one readable wake yielded.
enum Pump {
/// The connection has ended: EOF (clean or mid-frame) or an
/// unrecoverable stream error.
Ended,
/// Still up; this many complete frames were consumed (possibly zero, if
/// the wake delivered only part of a frame). Any nonzero count resets
/// the liveness window.
Frames(usize),
}
/// Consume one readable wake: exactly one socket read (which cannot block
/// after a level-triggered readable indication), then drain every complete
/// frame the buffer now holds. A blocking `recv` here would park the actor
/// past its heartbeat and liveness deadlines whenever a frame arrives split.
/// Frames are not interpreted yet — a heartbeat's entire job is the liveness
/// reset, and everything else waits for c8.
fn pump_readable(framed: &mut FramedConn) -> Pump {
let eof = match framed.read_once() {
Ok(n) => n == 0,
Err(_) => return Pump::Ended,
};
let mut got = 0;
loop {
match framed.next_buffered() {
Ok(Some(_frame)) => got += 1,
Ok(None) => break,
Err(_) => return Pump::Ended, // corrupt stream
}
}
if eof {
Pump::Ended
} else {
Pump::Frames(got)
}
}