dump some extra files

2026-03-08 22:34:03 +01:00
parent ed85af4a4f
commit 2358ca29c7
4 changed files with 1461 additions and 0 deletions
@@ -0,0 +1,358 @@
+# PostgreSQL Logical Replication Cache Invalidation System
+
+A high-performance cache invalidation system that streams PostgreSQL logical replication messages, buffers changes per transaction, and emits **transaction-atomic** cache invalidation events via Tokio broadcast channels.
+
+## Key Advantage: Transaction Atomicity
+
+Changes are buffered until commit, so all changes in a transaction are delivered to subscribers in a single `CacheInvalidationEvent`. This guarantees:
+
+- **No partial state**: If a transaction changes 5 rows across 3 tables, subscribers see all 5 changes together
+- **Correct semantics**: Subscribers can invalidate cache atomically and notify clients in a single batch
+- **Simpler logic**: No need to track partial transactions or worry about out-of-order delivery
+- **Crash recovery**: Can checkpoint at commit boundary with full event
+
+## Architecture
+
+### Three Core Components
+
+#### 1. **pg_wal_parser.rs** - Logical Replication Message Parser
+Low-level binary protocol parser implementing the pgoutput format spec.
+
+**Handles:**
+- All PostgreSQL 14+ logical replication message types (Begin, Relation, Insert, Update, Delete, Commit, Truncate, Type, Message)
+- TupleData parsing with column values (Null, UnchangedToasted, Text, Binary)
+- Streaming protocol with relation OID caching
+- Multi-version protocol support via graceful degradation
+
+**Key Types:**
+- `LogicalMessage` - enum of all message types with their data
+- `MessageParser` - stateful parser that maintains relation metadata
+- `ColumnInfo`, `RelationInfo` - schema metadata from Relation messages
+
+#### 2. **cache_invalidation.rs** - Transaction-Aware Event Emitter
+Transforms low-level replication messages into transaction-atomic high-level events.
+
+**Handles:**
+- Per-transaction event buffering (keyed by XID)
+- Tracks current transaction via `Begin`/`Commit` boundary
+- Accumulates Insert/Update/Delete/Truncate changes
+- Emits single atomic event on Commit with all changes
+- Row data extraction with column names and values
+- Primary key identification (via REPLICA IDENTITY flags)
+- Tokio broadcast channel emission
+
+**Key Types:**
+- `CacheInvalidationEvent` - contains XID, LSNs, timestamp, and **all changes in transaction**
+- `Change` - a single table operation (Insert/Update/Delete/Truncate) with row data
+- `CacheValue` - serializable column values
+- `RowData` - structured row with column map and optional keys
+- `TransactionBuffer` - internal state for buffering per XID
+
+**Entry Point:**
+```rust
+let mut emitter = CacheInvalidationEmitter::new(tx);
+emitter.process_raw(&raw_wal_bytes).await?;
+```
+
+#### 3. **example.rs** - pgwire-replication Integration
+Shows how to wire everything together with the pgwire-replication crate.
+
+**Handles:**
+- Connecting to PostgreSQL logical replication
+- Processing XLogData events from replication stream
+- Graceful error handling and LSN checkpointing
+- Example subscriber pattern for transaction-atomic cache invalidation
+- Multiple concurrent subscribers
+
+## Data Flow
+
+```
+PostgreSQL WAL (multi-statement transaction)
+       ↓
+pgwire-replication (ReplicationClient)
+       ↓
+XLogData { bytes, wal_end }
+       ↓
+MessageParser::parse_message() [binary protocol parsing]
+       ↓
+LogicalMessage enum
+       ↓
+CacheInvalidationEmitter::process_message() [buffering per XID]
+       ↓
+Begin: Create buffer for XID
+Insert/Update/Delete: Append to buffer for XID
+Commit: Emit CacheInvalidationEvent with all buffered changes
+       ↓
+CacheInvalidationEvent { xid, commit_lsn, changes: [Change, Change, ...] }
+       ↓
+broadcast channel → multiple subscribers receive transaction atomically
+```
+
+## Example Transaction Flow
+
+PostgreSQL executes:
+```sql
+BEGIN;
+  INSERT INTO users (id, name) VALUES (1, 'alice');
+  INSERT INTO posts (id, user_id, title) VALUES (100, 1, 'Hello');
+  UPDATE posts SET views = 1 WHERE id = 100;
+COMMIT;
+```
+
+Replication protocol sends:
+```
+Begin XID=5000 LSN=0/123456
+  Relation OID=16384 name="users" 
+  Relation OID=16385 name="posts"
+  Insert rel=16384 [1, 'alice']
+  Insert rel=16385 [100, 1, 'Hello']
+  Update rel=16385 key=[100] new=[100, 1, 'Hello', 1]
+Commit XID=5000 LSN=0/789ABC
+```
+
+Emitter produces (single event):
+```rust
+CacheInvalidationEvent {
+    xid: 5000,
+    commit_lsn: 0x789ABC,
+    end_lsn: 0x789ABC,
+    timestamp: 1234567890,
+    changes: [
+        Change {
+            table: "users",
+            schema: "public",
+            operation: Insert,
+            row_data: Some({id: 1, name: 'alice'})
+        },
+        Change {
+            table: "posts",
+            schema: "public",
+            operation: Insert,
+            row_data: Some({id: 100, user_id: 1, title: 'Hello'})
+        },
+        Change {
+            table: "posts",
+            schema: "public",
+            operation: Update,
+            row_data: Some({
+                columns: {id: 100, user_id: 1, title: 'Hello', views: 1},
+                key_values: Some({id: 100})
+            })
+        }
+    ]
+}
+```
+
+Subscribers receive this single event and can:
+- Invalidate cache for users:1, posts:100 atomically
+- Notify WebSocket clients in one batch
+- Checkpoint the LSN
+- Update metrics
+
+## Protocol Details
+
+### Message Format (from PostgreSQL docs)
+
+Each logical replication message starts with a type byte:
+
+| Type | Meaning | When |
+|------|---------|------|
+| B | Begin | Transaction start (1 per txn) |
+| R | Relation | Schema definition (1+ per txn) |
+| I | Insert | New row written to table |
+| U | Update | Row modified (includes old/new) |
+| D | Delete | Row removed (includes key/old) |
+| C | Commit | Transaction committed (1 per txn) |
+| T | Truncate | Table truncated |
+| Y | Type | Custom data type definition |
+| M | Message | Custom pg_logical_emit_message() |
+
+### Transaction Boundaries
+
+- **Begin** message: `XID` field identifies the transaction
+- **Changes** (Insert/Update/Delete/Truncate): Include same `XID`
+- **Commit** message: Completes the transaction (no XID in Commit itself)
+
+The emitter tracks `current_xid` to match Commit to its Begin, ensuring all changes are buffered and emitted together.
+
+## Setup & Usage
+
+### PostgreSQL Setup (one-time)
+
+```sql
+-- Enable logical replication
+ALTER SYSTEM SET wal_level = logical;
+SELECT pg_reload_conf();
+
+-- Create replication slot
+SELECT * FROM pg_create_logical_replication_slot('my_slot', 'pgoutput');
+
+-- Create publication
+CREATE PUBLICATION my_pub FOR TABLE users, posts, comments;
+```
+
+### Rust Code Integration
+
+```rust
+use tokio::sync::broadcast;
+use cache_invalidation::CacheInvalidationEmitter;
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let (tx, mut rx) = broadcast::channel(10000);
+
+    // Spawn WAL listener (uses pgwire-replication)
+    tokio::spawn(async move {
+        let mut client = pgwire_replication::ReplicationClient::connect(config).await?;
+        let mut emitter = CacheInvalidationEmitter::new(tx);
+
+        while let Some(event) = client.recv().await? {
+            match event {
+                pgwire_replication::ReplicationEvent::XLogData { data, wal_end, .. } => {
+                    // Process all messages in data; buffered per transaction
+                    emitter.process_raw(&data).await?;
+                    client.update_applied_lsn(wal_end);
+                }
+                _ => {}
+            }
+        }
+        Ok::<(), Box<dyn std::error::Error>>(())
+    });
+
+    // Subscribe to cache invalidation events
+    loop {
+        match rx.recv().await {
+            Ok(event) => {
+                println!("Transaction XID={} committed with {} changes", 
+                    event.xid, event.changes.len());
+                
+                // Invalidate cache for all changes atomically
+                for change in &event.changes {
+                    match &change.operation {
+                        Operation::Insert => {
+                            // cache.delete_by_table(change.table)
+                        }
+                        Operation::Update => {
+                            if let Some(row) = &change.row_data {
+                                if let Some(keys) = &row.key_values {
+                                    // cache.delete(change.table, keys)
+                                }
+                            }
+                        }
+                        Operation::Delete => {
+                            if let Some(row) = &change.row_data {
+                                if let Some(keys) = &row.key_values {
+                                    // cache.delete(change.table, keys)
+                                }
+                            }
+                        }
+                        Operation::Truncate => {
+                            // cache.clear_table(change.table)
+                        }
+                    }
+                }
+                
+                // Notify WebSocket clients once, with all changes
+                // websocket_broadcast(event)
+            }
+            Err(broadcast::error::RecvError::Lagged(_)) => {
+                eprintln!("Subscriber lagged, reloading cache from DB");
+            }
+            Err(broadcast::error::RecvError::Closed) => break,
+        }
+    }
+
+    Ok(())
+}
+```
+
+## Performance Characteristics
+
+- **Zero-copy parsing**: Uses `bytes::BytesMut` with in-place operations
+- **Async-first**: Compatible with Tokio task scheduling
+- **Streaming**: Processes messages as they arrive, no buffering of raw data
+- **Transaction buffering**: Per-XID HashMap (typically 1-10 concurrent txns)
+- **Memory**: Relation metadata cached by OID + per-txn change buffer
+- **LSN tracking**: Explicit control for checkpointing and crash recovery
+
+## Typical Latencies
+
+- PostgreSQL writes change to WAL: < 1ms
+- WAL transmitted to client: network latency (localhost ~0.1ms)
+- Message parsing: ~10-100µs per message
+- Buffering: ~1µs per change
+- Commit event emission: ~10-100µs
+- **End-to-end (localhost)**: ~1-5ms from commit to subscriber notification
+
+## Transaction Buffering Internals
+
+```rust
+// On Begin: Create buffer
+txn_buffers.insert(xid, TransactionBuffer { xid, changes: Vec::new() });
+current_xid = Some(xid);
+
+// On Insert/Update/Delete/Truncate: Append to buffer
+if let Some(buf) = txn_buffers.get_mut(&xid) {
+    buf.changes.push(change);
+}
+
+// On Commit: Emit and clean up
+if let Some(xid) = current_xid.take() {
+    emit_transaction(xid, commit_lsn, end_lsn, timestamp)?;
+    txn_buffers.remove(&xid);
+}
+```
+
+## Testing
+
+Each module has tests. Run with:
+
+```bash
+cargo test
+```
+
+## Caveats & Notes
+
+### What This Doesn't Do
+- Custom output plugins (uses standard pgoutput only)
+- Two-phase commits (v3+) would need extended buffering
+- Concurrent transaction correlation (assumes sequential Begins/Commits)
+
+### REPLICA IDENTITY Configuration
+
+For UPDATE/DELETE to include key information:
+```sql
+ALTER TABLE users REPLICA IDENTITY USING INDEX users_pkey;
+-- or for full old row:
+ALTER TABLE users REPLICA IDENTITY FULL;
+```
+
+Default is PRIMARY KEY, which is usually what you want.
+
+### Slot Management
+
+Replication slots persist until explicitly dropped. If the stream falls behind, the WAL won't be deleted and disk will fill up:
+
+```sql
+-- Monitor slot lag
+SELECT slot_name, active, restart_lsn, confirmed_flush_lsn
+FROM pg_replication_slots;
+
+-- Drop a slot when done
+SELECT pg_drop_replication_slot('my_slot');
+```
+
+## Future Enhancements
+
+1. **Metrics**: Instrument parsing and emit latency histograms
+2. **Backpressure**: Honor broadcast channel capacity
+3. **Checkpointing**: Persist LSN to disk/Redis for recovery
+4. **Filtering**: Table/operation-level subscription filtering
+5. **Parallel streams**: Support multiple concurrent WAL consumers
+
+## References
+
+- [PostgreSQL Logical Replication Protocol](https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html)
+- [pgoutput Output Plugin](https://www.postgresql.org/docs/current/sql-createpublication.html)
+- [pgwire-replication Crate](https://crates.io/crates/pgwire-replication)
+- [Tokio Broadcast Channel](https://tokio.rs/tokio/tutorial/select#broadcast)