livevue-rs/maybe_sql_integration

PostgreSQL Logical Replication Cache Invalidation System

A high-performance cache invalidation system that streams PostgreSQL logical replication messages, buffers changes per transaction, and emits transaction-atomic cache invalidation events via Tokio broadcast channels.

Key Advantage: Transaction Atomicity

Changes are buffered until commit, so all changes in a transaction are delivered to subscribers in a single CacheInvalidationEvent. This guarantees:

• No partial state: If a transaction changes 5 rows across 3 tables, subscribers see all 5 changes together
• Correct semantics: Subscribers can invalidate cache atomically and notify clients in a single batch
• Simpler logic: No need to track partial transactions or worry about out-of-order delivery
• Crash recovery: Can checkpoint at commit boundary with full event

Architecture

Three Core Components

1. pg_wal_parser.rs - Logical Replication Message Parser

Low-level binary protocol parser implementing the pgoutput format spec.

Handles:

• All PostgreSQL 14+ logical replication message types (Begin, Relation, Insert, Update, Delete, Commit, Truncate, Type, Message)
• TupleData parsing with column values (Null, UnchangedToasted, Text, Binary)
• Streaming protocol with relation OID caching
• Multi-version protocol support via graceful degradation

Key Types:

• LogicalMessage - enum of all message types with their data
• MessageParser - stateful parser that maintains relation metadata
• ColumnInfo, RelationInfo - schema metadata from Relation messages

2. cache_invalidation.rs - Transaction-Aware Event Emitter

Transforms low-level replication messages into transaction-atomic high-level events.

Handles:

• Per-transaction event buffering (keyed by XID)
• Tracks current transaction via Begin/Commit boundary
• Accumulates Insert/Update/Delete/Truncate changes
• Emits single atomic event on Commit with all changes
• Row data extraction with column names and values
• Primary key identification (via REPLICA IDENTITY flags)
• Tokio broadcast channel emission

Key Types:

• CacheInvalidationEvent - contains XID, LSNs, timestamp, and all changes in transaction
• Change - a single table operation (Insert/Update/Delete/Truncate) with row data
• CacheValue - serializable column values
• RowData - structured row with column map and optional keys
• TransactionBuffer - internal state for buffering per XID

Entry Point:

let mut emitter = CacheInvalidationEmitter::new(tx);
emitter.process_raw(&raw_wal_bytes).await?;

3. example.rs - pgwire-replication Integration

Shows how to wire everything together with the pgwire-replication crate.

Handles:

• Connecting to PostgreSQL logical replication
• Processing XLogData events from replication stream
• Graceful error handling and LSN checkpointing
• Example subscriber pattern for transaction-atomic cache invalidation
• Multiple concurrent subscribers

Data Flow

PostgreSQL WAL (multi-statement transaction)
       ↓
pgwire-replication (ReplicationClient)
       ↓
XLogData { bytes, wal_end }
       ↓
MessageParser::parse_message() [binary protocol parsing]
       ↓
LogicalMessage enum
       ↓
CacheInvalidationEmitter::process_message() [buffering per XID]
       ↓
Begin: Create buffer for XID
Insert/Update/Delete: Append to buffer for XID
Commit: Emit CacheInvalidationEvent with all buffered changes
       ↓
CacheInvalidationEvent { xid, commit_lsn, changes: [Change, Change, ...] }
       ↓
broadcast channel → multiple subscribers receive transaction atomically

Example Transaction Flow

PostgreSQL executes:

BEGIN;
  INSERT INTO users (id, name) VALUES (1, 'alice');
  INSERT INTO posts (id, user_id, title) VALUES (100, 1, 'Hello');
  UPDATE posts SET views = 1 WHERE id = 100;
COMMIT;

Replication protocol sends:

Begin XID=5000 LSN=0/123456
  Relation OID=16384 name="users" 
  Relation OID=16385 name="posts"
  Insert rel=16384 [1, 'alice']
  Insert rel=16385 [100, 1, 'Hello']
  Update rel=16385 key=[100] new=[100, 1, 'Hello', 1]
Commit XID=5000 LSN=0/789ABC

Emitter produces (single event):

CacheInvalidationEvent {
    xid: 5000,
    commit_lsn: 0x789ABC,
    end_lsn: 0x789ABC,
    timestamp: 1234567890,
    changes: [
        Change {
            table: "users",
            schema: "public",
            operation: Insert,
            row_data: Some({id: 1, name: 'alice'})
        },
        Change {
            table: "posts",
            schema: "public",
            operation: Insert,
            row_data: Some({id: 100, user_id: 1, title: 'Hello'})
        },
        Change {
            table: "posts",
            schema: "public",
            operation: Update,
            row_data: Some({
                columns: {id: 100, user_id: 1, title: 'Hello', views: 1},
                key_values: Some({id: 100})
            })
        }
    ]
}

Subscribers receive this single event and can:

• Invalidate cache for users:1, posts:100 atomically
• Notify WebSocket clients in one batch
• Checkpoint the LSN
• Update metrics

Protocol Details

Message Format (from PostgreSQL docs)

Each logical replication message starts with a type byte:

Type	Meaning	When
B	Begin	Transaction start (1 per txn)
R	Relation	Schema definition (1+ per txn)
I	Insert	New row written to table
U	Update	Row modified (includes old/new)
D	Delete	Row removed (includes key/old)
C	Commit	Transaction committed (1 per txn)
T	Truncate	Table truncated
Y	Type	Custom data type definition
M	Message	Custom pg_logical_emit_message()

Transaction Boundaries

• Begin message: XID field identifies the transaction
• Changes (Insert/Update/Delete/Truncate): Include same XID
• Commit message: Completes the transaction (no XID in Commit itself)

The emitter tracks current_xid to match Commit to its Begin, ensuring all changes are buffered and emitted together.

Setup & Usage

PostgreSQL Setup (one-time)

-- Enable logical replication
ALTER SYSTEM SET wal_level = logical;
SELECT pg_reload_conf();

-- Create replication slot
SELECT * FROM pg_create_logical_replication_slot('my_slot', 'pgoutput');

-- Create publication
CREATE PUBLICATION my_pub FOR TABLE users, posts, comments;

Rust Code Integration

use tokio::sync::broadcast;
use cache_invalidation::CacheInvalidationEmitter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (tx, mut rx) = broadcast::channel(10000);

    // Spawn WAL listener (uses pgwire-replication)
    tokio::spawn(async move {
        let mut client = pgwire_replication::ReplicationClient::connect(config).await?;
        let mut emitter = CacheInvalidationEmitter::new(tx);

        while let Some(event) = client.recv().await? {
            match event {
                pgwire_replication::ReplicationEvent::XLogData { data, wal_end, .. } => {
                    // Process all messages in data; buffered per transaction
                    emitter.process_raw(&data).await?;
                    client.update_applied_lsn(wal_end);
                }
                _ => {}
            }
        }
        Ok::<(), Box<dyn std::error::Error>>(())
    });

    // Subscribe to cache invalidation events
    loop {
        match rx.recv().await {
            Ok(event) => {
                println!("Transaction XID={} committed with {} changes", 
                    event.xid, event.changes.len());
                
                // Invalidate cache for all changes atomically
                for change in &event.changes {
                    match &change.operation {
                        Operation::Insert => {
                            // cache.delete_by_table(change.table)
                        }
                        Operation::Update => {
                            if let Some(row) = &change.row_data {
                                if let Some(keys) = &row.key_values {
                                    // cache.delete(change.table, keys)
                                }
                            }
                        }
                        Operation::Delete => {
                            if let Some(row) = &change.row_data {
                                if let Some(keys) = &row.key_values {
                                    // cache.delete(change.table, keys)
                                }
                            }
                        }
                        Operation::Truncate => {
                            // cache.clear_table(change.table)
                        }
                    }
                }
                
                // Notify WebSocket clients once, with all changes
                // websocket_broadcast(event)
            }
            Err(broadcast::error::RecvError::Lagged(_)) => {
                eprintln!("Subscriber lagged, reloading cache from DB");
            }
            Err(broadcast::error::RecvError::Closed) => break,
        }
    }

    Ok(())
}

Performance Characteristics

• Zero-copy parsing: Uses bytes::BytesMut with in-place operations
• Async-first: Compatible with Tokio task scheduling
• Streaming: Processes messages as they arrive, no buffering of raw data
• Transaction buffering: Per-XID HashMap (typically 1-10 concurrent txns)
• Memory: Relation metadata cached by OID + per-txn change buffer
• LSN tracking: Explicit control for checkpointing and crash recovery

Typical Latencies

• PostgreSQL writes change to WAL: < 1ms
• WAL transmitted to client: network latency (localhost ~0.1ms)
• Message parsing: ~10-100µs per message
• Buffering: ~1µs per change
• Commit event emission: ~10-100µs
• End-to-end (localhost): ~1-5ms from commit to subscriber notification

Transaction Buffering Internals

// On Begin: Create buffer
txn_buffers.insert(xid, TransactionBuffer { xid, changes: Vec::new() });
current_xid = Some(xid);

// On Insert/Update/Delete/Truncate: Append to buffer
if let Some(buf) = txn_buffers.get_mut(&xid) {
    buf.changes.push(change);
}

// On Commit: Emit and clean up
if let Some(xid) = current_xid.take() {
    emit_transaction(xid, commit_lsn, end_lsn, timestamp)?;
    txn_buffers.remove(&xid);
}

Testing

Each module has tests. Run with:

cargo test

Caveats & Notes

What This Doesn't Do

• Custom output plugins (uses standard pgoutput only)
• Two-phase commits (v3+) would need extended buffering
• Concurrent transaction correlation (assumes sequential Begins/Commits)

REPLICA IDENTITY Configuration

For UPDATE/DELETE to include key information:

ALTER TABLE users REPLICA IDENTITY USING INDEX users_pkey;
-- or for full old row:
ALTER TABLE users REPLICA IDENTITY FULL;

Default is PRIMARY KEY, which is usually what you want.

Slot Management

Replication slots persist until explicitly dropped. If the stream falls behind, the WAL won't be deleted and disk will fill up:

-- Monitor slot lag
SELECT slot_name, active, restart_lsn, confirmed_flush_lsn
FROM pg_replication_slots;

-- Drop a slot when done
SELECT pg_drop_replication_slot('my_slot');

Future Enhancements

1. Metrics: Instrument parsing and emit latency histograms
2. Backpressure: Honor broadcast channel capacity
3. Checkpointing: Persist LSN to disk/Redis for recovery
4. Filtering: Table/operation-level subscription filtering
5. Parallel streams: Support multiple concurrent WAL consumers

References

• PostgreSQL Logical Replication Protocol
• pgoutput Output Plugin
• pgwire-replication Crate
• Tokio Broadcast Channel