# PostgreSQL Logical Replication Cache Invalidation System

A high-performance cache invalidation system that streams PostgreSQL logical replication messages, buffers changes per transaction, and emits **transaction-atomic** cache invalidation events via Tokio broadcast channels.

## Key Advantage: Transaction Atomicity

Changes are buffered until commit, so all changes in a transaction are delivered to subscribers in a single `CacheInvalidationEvent`. This guarantees:

- **No partial state**: If a transaction changes 5 rows across 3 tables, subscribers see all 5 changes together
- **Correct semantics**: Subscribers can invalidate cache atomically and notify clients in a single batch
- **Simpler logic**: No need to track partial transactions or worry about out-of-order delivery
- **Crash recovery**: Can checkpoint at commit boundary with full event

## Architecture

### Three Core Components

#### 1. **pg_wal_parser.rs** - Logical Replication Message Parser
Low-level binary protocol parser implementing the pgoutput format spec.

**Handles:**
- All PostgreSQL 14+ logical replication message types (Begin, Relation, Insert, Update, Delete, Commit, Truncate, Type, Message)
- TupleData parsing with column values (Null, UnchangedToasted, Text, Binary)
- Streaming protocol with relation OID caching
- Multi-version protocol support via graceful degradation

**Key Types:**
- `LogicalMessage` - enum of all message types with their data
- `MessageParser` - stateful parser that maintains relation metadata
- `ColumnInfo`, `RelationInfo` - schema metadata from Relation messages

#### 2. **cache_invalidation.rs** - Transaction-Aware Event Emitter
Transforms low-level replication messages into transaction-atomic high-level events.

**Handles:**
- Per-transaction event buffering (keyed by XID)
- Tracks current transaction via `Begin`/`Commit` boundary
- Accumulates Insert/Update/Delete/Truncate changes
- Emits single atomic event on Commit with all changes
- Row data extraction with column names and values
- Primary key identification (via REPLICA IDENTITY flags)
- Tokio broadcast channel emission

**Key Types:**
- `CacheInvalidationEvent` - contains XID, LSNs, timestamp, and **all changes in transaction**
- `Change` - a single table operation (Insert/Update/Delete/Truncate) with row data
- `CacheValue` - serializable column values
- `RowData` - structured row with column map and optional keys
- `TransactionBuffer` - internal state for buffering per XID

**Entry Point:**
```rust
let mut emitter = CacheInvalidationEmitter::new(tx);
emitter.process_raw(&raw_wal_bytes).await?;
```

#### 3. **example.rs** - pgwire-replication Integration
Shows how to wire everything together with the pgwire-replication crate.

**Handles:**
- Connecting to PostgreSQL logical replication
- Processing XLogData events from replication stream
- Graceful error handling and LSN checkpointing
- Example subscriber pattern for transaction-atomic cache invalidation
- Multiple concurrent subscribers

## Data Flow

```
PostgreSQL WAL (multi-statement transaction)
       ↓
pgwire-replication (ReplicationClient)
       ↓
XLogData { bytes, wal_end }
       ↓
MessageParser::parse_message() [binary protocol parsing]
       ↓
LogicalMessage enum
       ↓
CacheInvalidationEmitter::process_message() [buffering per XID]
       ↓
Begin: Create buffer for XID
Insert/Update/Delete: Append to buffer for XID
Commit: Emit CacheInvalidationEvent with all buffered changes
       ↓
CacheInvalidationEvent { xid, commit_lsn, changes: [Change, Change, ...] }
       ↓
broadcast channel → multiple subscribers receive transaction atomically
```

## Example Transaction Flow

PostgreSQL executes:
```sql
BEGIN;
  INSERT INTO users (id, name) VALUES (1, 'alice');
  INSERT INTO posts (id, user_id, title) VALUES (100, 1, 'Hello');
  UPDATE posts SET views = 1 WHERE id = 100;
COMMIT;
```

Replication protocol sends:
```
Begin XID=5000 LSN=0/123456
  Relation OID=16384 name="users" 
  Relation OID=16385 name="posts"
  Insert rel=16384 [1, 'alice']
  Insert rel=16385 [100, 1, 'Hello']
  Update rel=16385 key=[100] new=[100, 1, 'Hello', 1]
Commit XID=5000 LSN=0/789ABC
```

Emitter produces (single event):
```rust
CacheInvalidationEvent {
    xid: 5000,
    commit_lsn: 0x789ABC,
    end_lsn: 0x789ABC,
    timestamp: 1234567890,
    changes: [
        Change {
            table: "users",
            schema: "public",
            operation: Insert,
            row_data: Some({id: 1, name: 'alice'})
        },
        Change {
            table: "posts",
            schema: "public",
            operation: Insert,
            row_data: Some({id: 100, user_id: 1, title: 'Hello'})
        },
        Change {
            table: "posts",
            schema: "public",
            operation: Update,
            row_data: Some({
                columns: {id: 100, user_id: 1, title: 'Hello', views: 1},
                key_values: Some({id: 100})
            })
        }
    ]
}
```

Subscribers receive this single event and can:
- Invalidate cache for users:1, posts:100 atomically
- Notify WebSocket clients in one batch
- Checkpoint the LSN
- Update metrics

## Protocol Details

### Message Format (from PostgreSQL docs)

Each logical replication message starts with a type byte:

| Type | Meaning | When |
|------|---------|------|
| B | Begin | Transaction start (1 per txn) |
| R | Relation | Schema definition (1+ per txn) |
| I | Insert | New row written to table |
| U | Update | Row modified (includes old/new) |
| D | Delete | Row removed (includes key/old) |
| C | Commit | Transaction committed (1 per txn) |
| T | Truncate | Table truncated |
| Y | Type | Custom data type definition |
| M | Message | Custom pg_logical_emit_message() |

### Transaction Boundaries

- **Begin** message: `XID` field identifies the transaction
- **Changes** (Insert/Update/Delete/Truncate): Include same `XID`
- **Commit** message: Completes the transaction (no XID in Commit itself)

The emitter tracks `current_xid` to match Commit to its Begin, ensuring all changes are buffered and emitted together.

## Setup & Usage

### PostgreSQL Setup (one-time)

```sql
-- Enable logical replication
ALTER SYSTEM SET wal_level = logical;
SELECT pg_reload_conf();

-- Create replication slot
SELECT * FROM pg_create_logical_replication_slot('my_slot', 'pgoutput');

-- Create publication
CREATE PUBLICATION my_pub FOR TABLE users, posts, comments;
```

### Rust Code Integration

```rust
use tokio::sync::broadcast;
use cache_invalidation::CacheInvalidationEmitter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (tx, mut rx) = broadcast::channel(10000);

    // Spawn WAL listener (uses pgwire-replication)
    tokio::spawn(async move {
        let mut client = pgwire_replication::ReplicationClient::connect(config).await?;
        let mut emitter = CacheInvalidationEmitter::new(tx);

        while let Some(event) = client.recv().await? {
            match event {
                pgwire_replication::ReplicationEvent::XLogData { data, wal_end, .. } => {
                    // Process all messages in data; buffered per transaction
                    emitter.process_raw(&data).await?;
                    client.update_applied_lsn(wal_end);
                }
                _ => {}
            }
        }
        Ok::<(), Box<dyn std::error::Error>>(())
    });

    // Subscribe to cache invalidation events
    loop {
        match rx.recv().await {
            Ok(event) => {
                println!("Transaction XID={} committed with {} changes", 
                    event.xid, event.changes.len());
                
                // Invalidate cache for all changes atomically
                for change in &event.changes {
                    match &change.operation {
                        Operation::Insert => {
                            // cache.delete_by_table(change.table)
                        }
                        Operation::Update => {
                            if let Some(row) = &change.row_data {
                                if let Some(keys) = &row.key_values {
                                    // cache.delete(change.table, keys)
                                }
                            }
                        }
                        Operation::Delete => {
                            if let Some(row) = &change.row_data {
                                if let Some(keys) = &row.key_values {
                                    // cache.delete(change.table, keys)
                                }
                            }
                        }
                        Operation::Truncate => {
                            // cache.clear_table(change.table)
                        }
                    }
                }
                
                // Notify WebSocket clients once, with all changes
                // websocket_broadcast(event)
            }
            Err(broadcast::error::RecvError::Lagged(_)) => {
                eprintln!("Subscriber lagged, reloading cache from DB");
            }
            Err(broadcast::error::RecvError::Closed) => break,
        }
    }

    Ok(())
}
```

## Performance Characteristics

- **Zero-copy parsing**: Uses `bytes::BytesMut` with in-place operations
- **Async-first**: Compatible with Tokio task scheduling
- **Streaming**: Processes messages as they arrive, no buffering of raw data
- **Transaction buffering**: Per-XID HashMap (typically 1-10 concurrent txns)
- **Memory**: Relation metadata cached by OID + per-txn change buffer
- **LSN tracking**: Explicit control for checkpointing and crash recovery

## Typical Latencies

- PostgreSQL writes change to WAL: < 1ms
- WAL transmitted to client: network latency (localhost ~0.1ms)
- Message parsing: ~10-100µs per message
- Buffering: ~1µs per change
- Commit event emission: ~10-100µs
- **End-to-end (localhost)**: ~1-5ms from commit to subscriber notification

## Transaction Buffering Internals

```rust
// On Begin: Create buffer
txn_buffers.insert(xid, TransactionBuffer { xid, changes: Vec::new() });
current_xid = Some(xid);

// On Insert/Update/Delete/Truncate: Append to buffer
if let Some(buf) = txn_buffers.get_mut(&xid) {
    buf.changes.push(change);
}

// On Commit: Emit and clean up
if let Some(xid) = current_xid.take() {
    emit_transaction(xid, commit_lsn, end_lsn, timestamp)?;
    txn_buffers.remove(&xid);
}
```

## Testing

Each module has tests. Run with:

```bash
cargo test
```

## Caveats & Notes

### What This Doesn't Do
- Custom output plugins (uses standard pgoutput only)
- Two-phase commits (v3+) would need extended buffering
- Concurrent transaction correlation (assumes sequential Begins/Commits)

### REPLICA IDENTITY Configuration

For UPDATE/DELETE to include key information:
```sql
ALTER TABLE users REPLICA IDENTITY USING INDEX users_pkey;
-- or for full old row:
ALTER TABLE users REPLICA IDENTITY FULL;
```

Default is PRIMARY KEY, which is usually what you want.

### Slot Management

Replication slots persist until explicitly dropped. If the stream falls behind, the WAL won't be deleted and disk will fill up:

```sql
-- Monitor slot lag
SELECT slot_name, active, restart_lsn, confirmed_flush_lsn
FROM pg_replication_slots;

-- Drop a slot when done
SELECT pg_drop_replication_slot('my_slot');
```

## Future Enhancements

1. **Metrics**: Instrument parsing and emit latency histograms
2. **Backpressure**: Honor broadcast channel capacity
3. **Checkpointing**: Persist LSN to disk/Redis for recovery
4. **Filtering**: Table/operation-level subscription filtering
5. **Parallel streams**: Support multiple concurrent WAL consumers

## References

- [PostgreSQL Logical Replication Protocol](https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html)
- [pgoutput Output Plugin](https://www.postgresql.org/docs/current/sql-createpublication.html)
- [pgwire-replication Crate](https://crates.io/crates/pgwire-replication)
- [Tokio Broadcast Channel](https://tokio.rs/tokio/tutorial/select#broadcast)