Kubernetes loses the leader due to the timeout and doesn't elect new one
- Dominant language
- Java
- Stars
- 302
- Forks
- 232
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 114
Description
### Bug description
We have from time to time a problem with our master election via kubernetes. We are currently on Camel Quarkus 3.8.3 and Quarkus 3.8.6 LTS. There is no special configuration of the master election, just defaults:
```
quarkus.camel.cluster.kubernetes.enabled=true
```
Initially the application works as charm, but later it loses the leadership and there is no pod with it. At this point we do see in Log following messages:
```
pod1 (v26rk):
2024-11-04 13:07:12,199 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional.empty
2024-11-04 13:07:12,200 INFO [org.apa.cam.com.qua.QuartzEndpoint] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) Pausing trigger ...
2024-11-04 13:07:12,200 INFO [org.apa.cam.com.qua.QuartzEndpoint] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) Deleting job ...
```
There is nothing more relevant to kubernetes in pod1 Log. The camel routes are since this moment down.
```
pod2 (t5bt5):
2024-11-04 13:12:13,190 WARN [org.apa.cam.com.kub.clu.loc.KubernetesLeadershipController] (Camel (camel-1) thread #2 - CamelKubernetesLeadershipController) Pod[pod2-985884674-t5bt5] Unable to retrieve the current lease resource my-lease for group my-service from Kubernetes
2024-11-04 13:52:15,345 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional.empty
2024-11-04 13:52:15,355 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional[pod1-985884674-v26rk]
2024-11-04 14:07:12,343 WARN [org.apa.cam.com.kub.clu.loc.KubernetesLeadershipController] (Camel (camel-1) thread #2 - CamelKubernetesLeadershipController) Pod[srv-mdn-patientdelivery-dev-985884674-t5bt5] Unable to retrieve the current lease resource my-lease for group my-service from Kubernetes
2024-11-04 14:07:15,130 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional.empty
...
```
After new deployment everything works again. I have absolutely no idea where the bug occurs, that's the reason why i'm trying to report it here. Any ideas? Would really appreciate.
Thanks a lot
Miro
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported setup with quarkus.camel.cluster.kubernetes.enabled=true and review the leadership logs around lease retrieval failures and Optional.empty notifications. Compare behavior before and after redeployment; done means identifying the timeout or election failure and providing a verified fix with coverage for the lost-leader case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100