Nothing local is remotely reachable by default (RFC §4). expose(Name<M>)
marks a name remotely addressable and registers M's decoder under
type_hash::<M>(); expose_type::<M>() registers only the decoder (the
reply-to path). exposed_names() is the auditable remote surface.
D3's watchable fold, resolved against the code as it stands (stated in the
module docs): register() ALREADY stamps every named holder watchable ('no
successfully-registered actor can die unflagged', registry.rs), so an
exposed name's holder needs no extra mark — and re-registration after a
holder's death re-stamps the new holder for free, which a per-tenancy mark
taken at expose time could not do. The cluster's own mark_watchable
set-site is therefore the pid crossing the wire (frame serialization, c10)
— the exact analog of the membrane crossing. c8 adds only the name/type
state neither the registry nor slot bits can carry. No new pid registry;
RFC §4 honored.
One-viable calls, flagged:
- State lives on RuntimeInner (the pg pattern: leaf RawMutex field,
cfg-gated behind cluster, zero-cost-when-off per c1) — c9's inbound
decode consults it per frame; manager-held state would serialize every
remote delivery through one gen_server.
- type_hash = FNV-1a 64 (fixed seed: the offset basis) over TypeId: a
constant of the binary — stable across runs of the same build (the scope
the build-hash handshake reduces the mesh to), deliberately not across
builds. Collisions degrade to decode error / refused channel, never a
misroute (the NoChannel guarantee, RFC §3).
- Decoder = decode-and-deliver-to-pid Arc closure capturing M (the one
typed site): decode_payload then send_dyn. Wire-name → pid resolution
stays OUTSIDE — that is c9's single seam, which calls decode_deliver.
Arc so the call happens with the exposure lock RELEASED: send_dyn takes
the registry lock, a mutual Leaf (the runtime asserts on nesting — caught
live by the first test run).
- expose is a name-level fact, valid for an unregistered name (names
late-bind; c9 resolves per delivery).
tests/cluster_expose.rs 5/0 stable x5, purely local per roadmap:
exposed/unexposed lookup + audit listing; decoder registration and the
delivery contract (happy path into a registered String channel; unknown
hash; corrupt bytes; wrong channel refused — never misrouted); distinct
types distinct hashes; expose/bridge-crossing agreement via the shared
watchable observable (terminal_reason after holder death); hash stability
across runs in the same binary via a c4-harness re-exec. Payload types are
std types — the crate's serde is derive-less by design, user crates bring
their own derive.
All cluster suites regression-clean (envelope 15, handshake 11, transport
11, lifecycle 1, liveness 3, connect 9, two_node 3, membership 4, mesh 2);
clippy --lib green both configs; fmt clean; default build compiles.
Phase 2 gate: 3-node mesh under the subprocess harness, repeatable (10/10).
Strategy (ratified): push-based, spawned as its own actor by the connector —
it emits Discovery events into a channel whenever it learns something and
may run forever; the connector owns all retry/backoff state. StaticSeeds
announces its list once and exits. Discovery is #[non_exhaustive] and
additive-only (candidates announced, never withdrawn) so expiry can land
later without breaking strategies.
One-viable correction to the ratified Discovery shape, flagged: a candidate
is a (name, addr) PAIR, not a bare address. The dial path and the D7
tie-break are keyed by peer name (the dial intent must be registered before
connecting so a crossing inbound Hello sees it), so an anonymous dial would
reintroduce exactly the simultaneous-connect flap D7 exists to prevent.
Discovery mechanisms know names — that is what they discover.
Connector: plain select-loop actor (the c6 shape) folding cmd inbox,
discovery stream, membership stream, and the earliest retry deadline into
one wait. It tracks who is up by SUBSCRIBING TO MEMBERSHIP like any
consumer — first consumer of c7a's snapshot-then-stream surface, no
privileged channel into the manager. Backoff: 250ms doubling to a 5s cap
(the c6c class of one-viable constants), reset on node_up; node_down
schedules a prompt redial with a fresh sequence. A candidate bearing the
local name is parked (that seed is us); every other failure retries — in
particular NameTaken can be our own ghost at the peer, not yet reaped by
its liveness timer, so it must not park. Dials run inline in the loop, the
acceptor's deliberate serialization (each attempt bounded by the connect +
handshake deadlines).
cluster::start(Config {node_name, meta, listen_addr, strategy}) is now the
integrated node start: supervised manager + acceptor + connector. It
completes the node identity: build_hash = cluster::BUILD_HASH (first
consumer, closing the c6d loose end) and incarnation = self_incarnation()
— unix-epoch MILLIS truncated to u32, not seconds: a supervised
crash-and-restart inside one second is routine, and seconds would collide
the ghost with its successor. Cluster handle: local_addr()/local()/
shutdown(); drop stops acceptor+connector loops, manager subtree detaches
(same split as AcceptorHandle alone).
Roadmap-binding, asserted in review: no consumer touches the connection
table — Manager.conns and ConnEntry stay private; the only exposures are
Call::Peers (sorted names, pre-existing) and the membership surface.
tests/cluster_mesh.rs 2/0, 10/10 flake runs: (1) 3-node mesh forms; kill
one (SIGKILL via Drop, per the retractable-state trap: roles park forever)
=> node_down at both survivors; restart same name => new incarnation at
every observer, distinguishable from the ghost; (2) seed unreachable at
start (pre-reserved closed port; accepted micro steal-window, documented)
then arriving later => edge forms via the retry path. All cluster suites
regression-clean (envelope 15, handshake 11, transport 11, lifecycle 1,
liveness 3, connect 9, two_node 3, membership 4); clippy --lib green both
configs; fmt clean; default build compiles.
node_up/node_down are derived facts of the manager's own register/remove
events, so the membership state lives in the manager (no cross-actor race
between 'connection exists' and 'node is up'); src/cluster/membership.rs is
the consumer surface: NodeEvent/NodeInfo, subscribe(), view(). The conn
table stays private — no consumer touches it (roadmap-binding).
Ratified semantics: subscribe() is snapshot-then-stream — one NodeUp per
live peer is queued before the subscription joins the list, exact because
gen_server handlers are serialized. Dropped subscribers are pruned on the
next emit (closed channel), no monitor needed.
One-viable call, flagged: NodeId is memoized per (name, incarnation) — a
compact local alias for the wire identity, per pg.rs's framing. A reconnect
blip at the same incarnation keeps its id; a restart (new incarnation) gets
a fresh one, so a ghost and its successor are always distinguishable.
Allocation starts at 1; NodeId(0) stays pg::DEFAULT_NODE_ID (self).
Call::Register now carries the whole handshake Peer (the path already has
it; node_up needs incarnation + meta).
tests/cluster_membership.rs 4/0 stable over 5 runs (live up/down over
localhost TCP with commanded and EOF teardown; late-subscriber snapshot +
view agreement; restart-vs-blip id identity; dead-subscriber pruning). All
cluster suites regression-clean; clippy --lib green both configs; fmt
clean; default build compiles.
cluster::BUILD_HASH, the derived value for LocalNode::build_hash: a
compile-time const, FNV-1a 64 over the build script's input string
(rustc -V + sorted enabled CARGO_FEATURE_* set) with PROTO_VERSION folded
in as a continuation of the same state — a proto bump moves the hash even
on an identical toolchain. build.rs emits only the raw inputs
(SMARM_BUILD_HASH_INPUTS); the hashing lives in src/cluster.rs next to
PROTO_VERSION rather than build.rs parsing it out of a source file. The
env var is emitted unconditionally — one string costs the default build
nothing, and the const itself is behind the cluster feature with the rest
of the module.
Domain is the flagged lean (one-viable-answer, veto at diff review):
toolchain + declared features + proto version. Tightenable later without a
wire change — it is just a u64 on the Hello.
Unit tests: FNV-1a 64 published vectors (empty/'a'/'foobar'); the proto
fold is a continuation of the same FNV state and moves the hash;
BUILD_HASH is const-evaluable and nonzero. All cluster suites
regression-clean (7 files); clippy --lib green both configs; fmt clean;
default build compiles.
Drive the c5 machines as straight-line code on the path (D8): dial_handshake
and accept_handshake do the IO on a shared FramedConn, and a connection actor
is spawned only after a successful handshake. Rejects, tie-break losses (D7),
protocol faults and timeouts are all resolved on the path by closing, so no
actor ever exists for a connection that did not establish. The whole
FramedConn travels into spawn_established, carrying any read-ahead past the
handshake frames.
Handshake deadlines land here rather than in c6c: FramedConn::recv_deadline
enforces them between reads via the connection's fd arm, so a peer that
connects and goes silent cannot wedge the acceptor.
Connection lifetime moves to the manager (pulled forward from c7). The path
registers each established connection and hands over its ConnHandle; the
manager owns it, monitors the actor, and tears the connection down on
Disconnect, on peer close, or at manager shutdown. spawn_established returns
a Pid, so a connection neither outlives nor dies with whichever actor
established it — the ownership that made two-node teardown unorderable.
The manager also tracks in-flight dial intents, monitored so a panicking
dial cannot wedge the tie-break, and answers HelloCtx for the accept path.
Per-peer connection actor as a single select-loop plain actor owning the
whole FramedConn: one select folds its command inbox and the transport's
readable arm, so reads and control share one execution context — no reader
thread, no read/write split. The handshake is bypassed here (c6b wires it);
the actor is spawned already-established and self-registers with the manager.
Manager gen_server: the peer-name -> conn-pid registry and the uniqueness
source the handshake's NameTaken depends on. It monitors each connection, so
the table self-heals on any exit path. Explicit supervision subtree keeps the
manager up; connections are dynamic and monitored, never restarted (c7
re-dials).
Transport gains an additive Conn::readable_arm -> Option<FdArm> (default None;
TCP returns its fd's arm, loopback stays None). Existing c3 transport tests
unchanged.
Lifecycle test over localhost TCP: up reflected in the table, commanded
shutdown reaps exactly one, peer EOF reaps the other.
Frames in, actions out — no IO, no clocks, no actors; the c6 connection
actor will drive it. Initiator (dial: emit Hello, interpret the single
response) and Responder (accept: judge the first frame) as consuming-self
machines; check order proto -> hash -> name -> tie-break. Driver-supplied
HelloCtx carries the two facts the pure machine cannot know (name claimed,
own dial in flight). Tie-break ratified as a wire fact: the smaller name's
dial survives; the losing inbound closes silently (both ends compute the
same verdict, no reject frame needed). Peer's own name offered => NameTaken.
build_hash is config-supplied; derivation lands with c6.
The control-connection abstraction (RFC v2 §5): object-safe Transport/
Listener/Conn over opaque pre-resolved addresses (resolution stays the c9
seam), with FramedConn as the single shared byte->Frame codec feeding
Frame::decode's incremental contract. Nothing forecloses additional
per-peer connections for the jarred bulk plane; the membrane is not a
transport (D2).
TCP parks the calling actor via scheduler fd readiness (MSG_NOSIGNAL
writes, EINPROGRESS dial resolved through SO_ERROR). Loopback is the
shipped in-memory test transport: OS-thread-blocking condvar pipes with
TCP-shaped close semantics, per-instance address registry.
Conformance suite runs the same codec over both impls: roundtrips both
directions, framing across split writes, coalesced frames, peer-close
mid-frame as TruncatedByPeer (not EOF), clean close as Ok(None). Plus
impl-specific establishment/error cases and a 4 MiB cross-buffer TCP
frame under real backpressure.
Frame enum per the RFC inventory; hand-rolled encode/decode with u32 LE
length prefix + u8 tag; strings u16-prefixed, payload blobs u32-prefixed;
MAX_FRAME_LEN cap (control plane never carries bulk, §5). Streaming decode:
Ok(None) = need more bytes, every Err = corruption. postcard confined to
encode_payload/decode_payload — the single codec seam (§2); postcard gains
the alloc feature for to_allocvec (still no_std-aligned, no default
features). DownReason/RejectReason travel as single tag bytes; Down tag 5
is reserved for c11's Disconnected.
Tests: per-frame roundtrip, back-to-back frames, golden heartbeat bytes,
zero-length payload, every-prefix incomplete, unknown frame/enum tags,
length prefix lying long (with and without bytes present) and short,
truncation mid-string, adversarial lengths (u32::MAX, cap+1, zero), UTF-8
corruption, payload seam roundtrip through a real Send frame.
Off by default; the default build stays libc-only. serde (payload contract)
and postcard (payload codec) are optional, no default features. src/cluster.rs
is an intentionally empty cfg-gated stub so the flag's default-build
invariance is reviewable in isolation.
Claude