hybrid architecture for Temporal

Is your feature request related to a problem? Please describe.
Currently, temporal design focuses on ensuring strong guarantees for workflow execution, which requires each event or activity result to be written immediately to the persistence store. For this reason, Temporal does not batch activities for bulk writes to Cassandra. To handle this, one needs to go for bigger persistence store cluster adding to the cost.

Describe the solution you'd like

1. Redis for Real-Time Workflow Execution State:

Purpose: Redis can be used as a cache for workflow execution state and hot data (i.e., frequently accessed or recently updated data). Since Redis is an in-memory database, it provides low-latency reads and writes, making it ideal for workflows where immediate responsiveness is required.

Use Case: Whenever a workflow is being executed, its current state, progress, and results of activities can be stored in Redis. Redis acts as a working memory, keeping active workflow states in memory for fast access during execution.

Persistence in Redis: Redis persistence methods such as RDB snapshots or AOF can be used to periodically back up data to disk, but Redis itself would not be relied upon for full persistence.

Eviction Policy: Since Redis is memory-constrained, an eviction policy can be applied where old or less frequently used workflow data is offloaded to Cassandra after a certain threshold or TTL (time-to-live) period.

2. Kafka for Event Streaming and Replay:

Purpose: Kafka can be used to stream all workflow events and activity results for asynchronous processing and durable backup. If Redis crashes or the system needs to replay workflow histories, Kafka can be used to recover or reconstruct the exact event sequence.

Use Case: Every change in workflow state (activity completion, task scheduling, etc.) is also streamed to Kafka as an event log. Kafka topics can be partitioned based on workflows or workflow shards to allow parallel processing.

Replay Capability: Kafka acts as a reliable event log for replaying events in case Redis or Cassandra fails. If Redis loses the in-memory state, Temporal can replay workflow events from Kafka to restore the most recent workflow state in Redis.

Fault Tolerance: Kafka provides durable message storage and eventual consistency, ensuring that workflow states can be reconstructed by replaying the Kafka log, even after failures.

3. Cassandra for Long-Term, Durable Storage:

Purpose: Cassandra can be used as the durable, long-term persistence store for workflow history and state. This is critical for fault tolerance, event sourcing, and ensuring that even workflows that last for long periods can be recovered.

Use Case: While Redis holds the real-time working state, Cassandra is responsible for storing the entire history of workflow execution, including every event that happened during the workflow lifecycle.

Batch Writes: Cassandra can handle batch writes of workflow events and state transitions that are periodically written from Redis and Kafka. Redis acts as the in-memory cache, while Cassandra ensures that once a workflow completes or reaches certain checkpoints, its state is persisted for long-term storage.

Scalability: Cassandra provides horizontal scalability, allowing it to store huge amounts of workflow data, making it suitable for storing the event history and workflow results after Redis evicts data or when workflows are completed.

No files, tests, or entry points are named. Start by reviewing Temporal's current workflow execution and persistence architecture, then assess how Redis, Kafka, and Cassandra would interact for state, event replay, and durable history. Done would require an agreed, implementable hybrid architecture and defined consistency and failure behavior.