apache / apache/pekko-management
K8S: Truncating lease name causes uniqueness issues
- Dominant language
- Scala
- Stars
- 33
- Forks
- 19
- Avg merge
- 4h 48m
- Merged PRs (30d)
- 29
Description
Hello everyone.
First of all thank you for the great work maintaining this project !
After using the K8S lease in production environments, it has come to my attention that an issue arises when using singleton lease with long actor system names. This also can affect other leases but is less visible.
## Issue
In [pekko](https://github.com/apache/pekko/blob/0d017f1fce7394d9bb836c4b0b1ab9757b481351/cluster-tools/src/main/scala/org/apache/pekko/cluster/singleton/ClusterSingletonManager.scala#L508), the singleton lease name is configured as:
```scala
private val singletonLeaseName = s"${context.system.name}-singleton-${self.path}"
```
In the [AbstractKubernetesLease](https://github.com/apache/pekko-management/blob/eba70fae9c94ff511e7cc7e244b0637667e63c0d/lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/AbstractKubernetesLease.scala#L77) implementation, this name above is truncated to 63 characters to avoid the 64 character limit of K8S API resource names.
The issue here, is that if the actor system name is long (anonymous actor system in Kubernetes for example, that retrieves the deployment name, which can be very long), it can very easily exceeds 64 characters.
Here is an example `leaseName` computed by Pekko:
test-a05b99e1-3377-464b-95e3-ef17b07027b7-singleton-pekko://test-a05b99e1-3377-464b-95e3-ef17b07027b7/path/to/actor
After treatment by the `AbstractKubernetesLease`, the computed lease name is
test-a05b99e1-3377-464b-95e3-ef17b07027b7-singleton-pekkoans-a0
From there, the problem is pretty obvious, each singleton actor tries to access the same lease resource !
This affects both K8S Lease implementations, the CRD and the Native one, as it is defined in the `AbstractKubernetesLease` that is the base for both implementations.
## Proposed solution
From my perspective, I think the normalization that takes place in the `AbstractKubernetesLease` should not trim, as this can hinder unicity.
A possible solution would be to hash the input leaseName, thus reducing length but conserving unicity.
An extension to the resources metadata in form of labels would allow to keep the full lease name for easy audit of the cluster state.
If this solution meets your requirements, I can happily open a PR.
Thank you for your time !
Contributor guide
Research direction
Start in lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/AbstractKubernetesLease.scala, especially the lease-name normalization around the referenced line. Check how the CRD and Native implementations use this shared name handling and look for related tests. Done means long lease names remain distinct while satisfying Kubernetes resource-name limits, with the behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, scala
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100