apache / apache/pekko

Add health check support for Persistence journal and snapshot store

Open
#3,128 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
211
Avg merge
1d 6h
Merged PRs (30d)
89

Description

### Motivation

In cloud-native and Kubernetes deployments, it's critical to verify that persistence backends (journals and snapshot stores) are healthy, connected, and ready to serve reads/writes. Currently, pekko provides `ClusterShardingHealthCheck` for cluster sharding readiness, but there is no equivalent health check mechanism for the persistence subsystem.

Without persistence health checks, operators cannot:
- Verify journal connectivity during pod startup (readiness probes)
- Detect journal degradation or disconnection during runtime (liveness probes)
- Monitor recovery status of persistent actors

### Proposed Enhancement

Add health check support to the persistence subsystem, following the pattern established by `ClusterShardingHealthCheck`:

1. A health check extension for `AsyncWriteJournal` that verifies the journal is reachable and responsive
2. A health check for snapshot store connectivity
3. Integration with Kubernetes-style readiness/liveness probe mechanisms

The akka.net project has implemented this feature and found it valuable for production deployments.

### References

This enhancement was inspired by the Akka.NET project: https://github.com/akkadotnet/akka.net/issues/7840

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing ClusterShardingHealthCheck entry point and the AsyncWriteJournal and snapshot store interfaces named in the proposal. Compare the referenced Akka.NET implementation and determine how persistence connectivity, readiness, liveness, and recovery status should be represented. Done means journal and snapshot health checks are integrated with Kubernetes-style probes, with project tests covering the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.