apache / apache/camel-quarkus

Kubernetes loses the leader due to the timeout and doesn't elect new one

Open
#6,761 4 comments 0 reactions 0 assignees View on GitHub
area/kubernetes bug
Dominant language
Java
Stars
302
Forks
232
Avg merge
1d 20h
Merged PRs (30d)
114

Description

### Bug description

We have from time to time a problem with our master election via kubernetes. We are currently on Camel Quarkus 3.8.3 and Quarkus 3.8.6 LTS. There is no special configuration of the master election, just defaults:

```
quarkus.camel.cluster.kubernetes.enabled=true
```

Initially the application works as charm, but later it loses the leadership and there is no pod with it. At this point we do see in Log following messages:

```
pod1 (v26rk):
2024-11-04 13:07:12,199 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional.empty
2024-11-04 13:07:12,200 INFO [org.apa.cam.com.qua.QuartzEndpoint] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) Pausing trigger ...
2024-11-04 13:07:12,200 INFO [org.apa.cam.com.qua.QuartzEndpoint] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) Deleting job ...
```

There is nothing more relevant to kubernetes in pod1 Log. The camel routes are since this moment down.

```
pod2 (t5bt5):
2024-11-04 13:12:13,190 WARN [org.apa.cam.com.kub.clu.loc.KubernetesLeadershipController] (Camel (camel-1) thread #2 - CamelKubernetesLeadershipController) Pod[pod2-985884674-t5bt5] Unable to retrieve the current lease resource my-lease for group my-service from Kubernetes
2024-11-04 13:52:15,345 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional.empty
2024-11-04 13:52:15,355 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional[pod1-985884674-v26rk]
2024-11-04 14:07:12,343 WARN [org.apa.cam.com.kub.clu.loc.KubernetesLeadershipController] (Camel (camel-1) thread #2 - CamelKubernetesLeadershipController) Pod[srv-mdn-patientdelivery-dev-985884674-t5bt5] Unable to retrieve the current lease resource my-lease for group my-service from Kubernetes
2024-11-04 14:07:15,130 INFO [org.apa.cam.com.kub.clu.loc.TimedLeaderNotifier] (Camel (camel-1) thread #8 - CamelKubernetesLeaderNotifier) The cluster has a new leader: Optional.empty
...
```

After new deployment everything works again. I have absolutely no idea where the bug occurs, that's the reason why i'm trying to report it here. Any ideas? Would really appreciate.

Thanks a lot
Miro

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported setup with quarkus.camel.cluster.kubernetes.enabled=true and review the leadership logs around lease retrieval failures and Optional.empty notifications. Compare behavior before and after redeployment; done means identifying the timeout or election failure and providing a verified fix with coverage for the lost-leader case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.