[FEATURE] Coordinate terminated Spark Driver Pod cleanup across Kyuubi servers
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched the existing issues and found no issue covering coordinated cleanup across multiple Kyuubi servers.
### Describe the feature
When multiple Kyuubi server instances monitor the same Kubernetes context and namespace, each instance starts a Pod informer and maintains its own terminated-application cleanup cache.
After a Spark Driver Pod reaches a terminal state, every server instance that observed the event independently expires its local cache entry and calls the Kubernetes DELETE API:
https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala#L145-L210
This has two side effects:
1. For one terminated Driver Pod, the API server may receive one DELETE request from every Kyuubi server instance. Only the first request deletes the Pod; later requests normally observe that it is already absent.
2. An empty result from Fabric8 DELETE is currently logged as "Failed to delete pod", although a named-resource HTTP 404 is converted by Fabric8 into an empty result. Real authorization, throttling, network, or server failures are therefore difficult to distinguish from an idempotent no-op.
The DELETE event is also passed to `markApplicationTerminated`:
https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala#L400-L434
The cleanup state does not record that the Pod is already deleting or absent. An existing cleanup trigger is not canceled, and a trigger that has already expired may be created again, causing another redundant DELETE attempt later.
The cleanup responsibility should be coordinated independently from application-state observation.
### Motivation
The number of mutating Kubernetes API requests for terminated Driver Pods currently grows with the number of Kyuubi server replicas. The resulting warning logs are also not actionable because they conflate an already-absent Pod with a real DELETE failure.
A coordinated cleaner would reduce DELETE requests and preserve useful failure diagnostics without changing how individual Kyuubi servers monitor applications.
This is different from #7293, which is about Kubernetes clients and informers not being initialized in some deployments.
### Describe the solution
A possible design is:
1. Elect one cleanup leader for each `KubernetesInfo(context, namespace)` through the existing Kyuubi HA backend (ZooKeeper or etcd).
2. Keep Pod informers on all Kyuubi servers for application-state tracking, audit logging, metadata persistence, cancellation handling, and local `getApplicationInfoByTag` calls. Leadership gates only the cleanup responsibility.
3. Only the cleanup leader schedules or executes deletion of terminated Driver Pods for its context and namespace.
4. When leadership is acquired, list and reconcile existing terminated Driver Pods so that cleanup continues after the previous leader fails or restarts.
5. When leadership is lost, stop issuing cleanup DELETE requests.
6. Treat a Pod with `metadata.deletionTimestamp`, or an observed DELETE event, as already satisfying cleanup and do not enqueue it again.
7. Log an accepted DELETE request as such, treat an empty/not-found result as an idempotent no-op, and reserve warning/error logs for real exceptions.
Contributor guide
Research direction
Start with KubernetesApplicationOperation.scala, especially the cleanup flow around lines 145-210 and markApplicationTerminated around lines 400-434. Read the existing Kyuubi HA backend options, ZooKeeper and etcd, before deciding how leadership and reconciliation fit the current application-state observation. Done means cleanup is coordinated per KubernetesInfo, resumes after leadership changes, avoids redundant deletes, and distinguishes idempotent no-ops from real failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, scala, spark
- Domain
- backend, cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100