Migrate Event Ledger V3 from Cassandra to NATS JetStream

Description

Replace Event Ledger V3’s Cassandra persistence with NATS JetStream to simplify deployment and reuse existing NVCF infrastructure.

The main functional benefit is partial-context querying. Callers can provide only the context fields they know, such as a cluster ID, and still retrieve events containing additional deployment or GPU specification context. Cassandra required the complete context because it formed the partition key.

The design must preserve the V3 API, durable event ingestion, event-history queries, and namespace stats. This epic assumes V1/V2 endpoints are retired instead of reimplemented.

Planned work:

• Define ordering, deduplication, retention, and consistency requirements.
• Design isolated, replicated JetStream storage and read projections.
• Implement V3 ingestion and partial-context queries.
• Add capacity, lag, and failure monitoring.
• Dual-write to Cassandra and NATS to validate correctness.
• Move reads to NATS progressively, then remove Cassandra dependencies.

Definition of Done

• V3 API behavior and performance are preserved.
• Partial-context queries return all matching events.
• Duplicate and out-of-order event behavior is documented and tested.
• JetStream has explicit replication, retention, capacity, and workload isolation.
• Restart, reconnect, redelivery, and storage-failure scenarios are tested.
• Dual-write comparison shows no unexplained differences.
• Cutover and rollback procedures are validated.
• Event Ledger’s Cassandra code, configuration, credentials, and schema are removed.

Resources

• Event Ledger
• Current schema

Start by reading src/control-plane-services/event-ledger and the current schema in migrations/cassandra/keyspaces/event_ledger/03_init_tables.up.sql. Break the epic into its stated requirements for ordering, deduplication, retention, queries, monitoring, dual-write validation, and cutover; done means the V3 behavior is preserved and the listed failure scenarios, comparisons, rollback, and Cassandra removal are validated.