feat(cluster): RFC 010 c7b — discovery Strategy, static seeds, connector dial loop
Phase 2 gate: 3-node mesh under the subprocess harness, repeatable (10/10).
Strategy (ratified): push-based, spawned as its own actor by the connector —
it emits Discovery events into a channel whenever it learns something and
may run forever; the connector owns all retry/backoff state. StaticSeeds
announces its list once and exits. Discovery is #[non_exhaustive] and
additive-only (candidates announced, never withdrawn) so expiry can land
later without breaking strategies.
One-viable correction to the ratified Discovery shape, flagged: a candidate
is a (name, addr) PAIR, not a bare address. The dial path and the D7
tie-break are keyed by peer name (the dial intent must be registered before
connecting so a crossing inbound Hello sees it), so an anonymous dial would
reintroduce exactly the simultaneous-connect flap D7 exists to prevent.
Discovery mechanisms know names — that is what they discover.
Connector: plain select-loop actor (the c6 shape) folding cmd inbox,
discovery stream, membership stream, and the earliest retry deadline into
one wait. It tracks who is up by SUBSCRIBING TO MEMBERSHIP like any
consumer — first consumer of c7a's snapshot-then-stream surface, no
privileged channel into the manager. Backoff: 250ms doubling to a 5s cap
(the c6c class of one-viable constants), reset on node_up; node_down
schedules a prompt redial with a fresh sequence. A candidate bearing the
local name is parked (that seed is us); every other failure retries — in
particular NameTaken can be our own ghost at the peer, not yet reaped by
its liveness timer, so it must not park. Dials run inline in the loop, the
acceptor's deliberate serialization (each attempt bounded by the connect +
handshake deadlines).
cluster::start(Config {node_name, meta, listen_addr, strategy}) is now the
integrated node start: supervised manager + acceptor + connector. It
completes the node identity: build_hash = cluster::BUILD_HASH (first
consumer, closing the c6d loose end) and incarnation = self_incarnation()
— unix-epoch MILLIS truncated to u32, not seconds: a supervised
crash-and-restart inside one second is routine, and seconds would collide
the ghost with its successor. Cluster handle: local_addr()/local()/
shutdown(); drop stops acceptor+connector loops, manager subtree detaches
(same split as AcceptorHandle alone).
Roadmap-binding, asserted in review: no consumer touches the connection
table — Manager.conns and ConnEntry stay private; the only exposures are
Call::Peers (sorted names, pre-existing) and the membership surface.
tests/cluster_mesh.rs 2/0, 10/10 flake runs: (1) 3-node mesh forms; kill
one (SIGKILL via Drop, per the retractable-state trap: roles park forever)
=> node_down at both survivors; restart same name => new incarnation at
every observer, distinguishable from the ghost; (2) seed unreachable at
start (pre-reserved closed port; accepted micro steal-window, documented)
then arriving later => edge forms via the retry path. All cluster suites
regression-clean (envelope 15, handshake 11, transport 11, lifecycle 1,
liveness 3, connect 9, two_node 3, membership 4); clippy --lib green both
configs; fmt clean; default build compiles.
This commit is contained in:
+100
-14
@@ -10,21 +10,32 @@
|
||||
|
||||
pub mod conn;
|
||||
pub mod connect;
|
||||
pub mod connector;
|
||||
pub mod discovery;
|
||||
pub mod envelope;
|
||||
pub mod handshake;
|
||||
pub mod manager;
|
||||
pub mod membership;
|
||||
pub mod transport;
|
||||
|
||||
use std::time::Duration;
|
||||
use std::io;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
|
||||
use crate::gen_server::{self, GenServerBuilder};
|
||||
use crate::monitor::monitor;
|
||||
use crate::pg::Incarnation;
|
||||
use crate::scheduler::{sleep, spawn, JoinHandle};
|
||||
use crate::supervisor::{ChildSpec, OneForOne, Restart};
|
||||
|
||||
use envelope::NodeMeta;
|
||||
use handshake::Local;
|
||||
use transport::tcp::TcpTransport;
|
||||
use transport::Transport;
|
||||
|
||||
pub use conn::{spawn_established, ConnHandle};
|
||||
pub use connect::{dial, spawn_acceptor, AcceptorHandle};
|
||||
pub use connector::{spawn_connector, ConnectorHandle};
|
||||
pub use discovery::{Discovery, StaticSeeds, Strategy};
|
||||
pub use manager::{Manager, MANAGER};
|
||||
pub use membership::{subscribe, view, MembershipEvents, NodeEvent, NodeInfo};
|
||||
|
||||
@@ -65,20 +76,67 @@ const fn fold_u32(mut h: u64, v: u32) -> u64 {
|
||||
h
|
||||
}
|
||||
|
||||
/// A running cluster subtree: an explicitly-started supervisor over the
|
||||
/// connection [`Manager`]. Roles will eventually mount this subtree; until the
|
||||
/// role mechanism lands it is started by hand (RFC 010 §7). Dropping the handle
|
||||
/// detaches the subtree, which keeps running for the life of the runtime.
|
||||
pub struct Cluster {
|
||||
_sup: JoinHandle,
|
||||
/// How to run this node: its identity and how it finds peers.
|
||||
pub struct Config {
|
||||
/// This node's claimed name — the mesh-wide identity peers dial by and
|
||||
/// the tie-break input. Must be unique across the mesh.
|
||||
pub node_name: String,
|
||||
/// Metadata offered in this node's `Hello`.
|
||||
pub meta: NodeMeta,
|
||||
/// The control-connection listen address (e.g. `"127.0.0.1:0"`; the
|
||||
/// concrete bound address is [`Cluster::local_addr`]).
|
||||
pub listen_addr: String,
|
||||
/// The peer-discovery strategy — [`StaticSeeds`] until richer ones land.
|
||||
pub strategy: Box<dyn Strategy>,
|
||||
}
|
||||
|
||||
/// Start the cluster subtree and block until the manager is registered and
|
||||
/// ready to answer. The manager is a supervised child (restarted on crash);
|
||||
/// per-peer connection actors are dynamic and monitored by the manager rather
|
||||
/// than statically supervised — a lost connection is re-established by dialing
|
||||
/// (c7), never resurrected onto a stale socket.
|
||||
pub fn start() -> Cluster {
|
||||
/// A running cluster node: the supervised [`Manager`], the acceptor over the
|
||||
/// bound listener, and the connector driving its [`Strategy`]. Roles will
|
||||
/// eventually mount this; until the role mechanism lands it is started by
|
||||
/// hand (RFC 010 §7).
|
||||
///
|
||||
/// Dropping the handle stops the acceptor and connector loops (no new
|
||||
/// connections in either direction) but detaches the manager subtree, which
|
||||
/// — with every established connection — keeps running for the life of the
|
||||
/// runtime, the same split as [`AcceptorHandle`] alone.
|
||||
pub struct Cluster {
|
||||
_sup: JoinHandle,
|
||||
acceptor: AcceptorHandle,
|
||||
connector: ConnectorHandle,
|
||||
local: Local,
|
||||
}
|
||||
|
||||
impl Cluster {
|
||||
/// The concrete bound listen address, dialable as-is.
|
||||
pub fn local_addr(&self) -> &str {
|
||||
self.acceptor.local_addr()
|
||||
}
|
||||
|
||||
/// This node's handshake identity (name, incarnation, build hash, meta).
|
||||
pub fn local(&self) -> &Local {
|
||||
&self.local
|
||||
}
|
||||
|
||||
/// Stop accepting and dialing. Established connections stay up (they
|
||||
/// belong to the manager); tear those down via the manager.
|
||||
pub fn shutdown(&self) {
|
||||
self.acceptor.shutdown();
|
||||
self.connector.shutdown();
|
||||
}
|
||||
}
|
||||
|
||||
/// Start a cluster node: the supervised manager (blocking until it is
|
||||
/// registered and ready to answer), the acceptor bound per
|
||||
/// [`Config::listen_addr`], and the connector running [`Config::strategy`].
|
||||
/// The node's identity is completed here: `incarnation` is
|
||||
/// [`self_incarnation`] and `build_hash` is [`BUILD_HASH`] — c7 is its first
|
||||
/// consumer. Errs only if the listener cannot bind.
|
||||
///
|
||||
/// The manager is a supervised child (restarted on crash); per-peer
|
||||
/// connection actors are dynamic and monitored by the manager rather than
|
||||
/// statically supervised — a lost connection is re-established by the
|
||||
/// connector's dial loop, never resurrected onto a stale socket.
|
||||
pub fn start(config: Config) -> io::Result<Cluster> {
|
||||
let sup = spawn(|| {
|
||||
OneForOne::new()
|
||||
.child(ChildSpec::new(Restart::Permanent, manager_child))
|
||||
@@ -87,7 +145,35 @@ pub fn start() -> Cluster {
|
||||
while gen_server::whereis_server(MANAGER).is_none() {
|
||||
sleep(Duration::from_millis(1));
|
||||
}
|
||||
Cluster { _sup: sup }
|
||||
let local = Local {
|
||||
node_name: config.node_name,
|
||||
incarnation: self_incarnation(),
|
||||
build_hash: BUILD_HASH,
|
||||
meta: config.meta,
|
||||
};
|
||||
let listener = TcpTransport.listen(&config.listen_addr)?;
|
||||
let acceptor = spawn_acceptor(listener, local.clone());
|
||||
let connector = spawn_connector(Box::new(TcpTransport), local.clone(), config.strategy);
|
||||
Ok(Cluster {
|
||||
_sup: sup,
|
||||
acceptor,
|
||||
connector,
|
||||
local,
|
||||
})
|
||||
}
|
||||
|
||||
/// This process's incarnation epoch: milliseconds since the Unix epoch,
|
||||
/// truncated to `u32`. Not a clock — its one job is separating a node from
|
||||
/// its own restart (two starts of the same name land on the same value only
|
||||
/// if they happen within the same millisecond modulo ~49.7 days). Seconds
|
||||
/// would be too coarse: a crash-and-restart inside one second is routine
|
||||
/// under supervision.
|
||||
pub fn self_incarnation() -> Incarnation {
|
||||
let ms = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_millis())
|
||||
.unwrap_or(0);
|
||||
Incarnation::new(ms as u32)
|
||||
}
|
||||
|
||||
/// The supervised manager child body. It *is* the child actor: it starts the
|
||||
|
||||
Reference in New Issue
Block a user