apache / apache/texera

Automatically terminate idle Computing Units to reclaim cluster resources

Open
#5,362 1 comment 0 reactions 1 assignee Claimed by @yrenat View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

During the dkNET-AI launch, we noticed that a computing unit keeps running when a user leaves the platform without terminating it. Because CUs are per-user compute pods, these idle CUs hold CPU/memory and pin their EKS nodes causing significant resource underutilization and cost.

We need to (1) define what makes a CU "idle" and (2) add a mechanism that automatically terminates idle CUs.

Based on @chenlica's and my investigation, Kubernetes has no built-in mechanism to terminate a pod for inactivity. It automatically stops pods for health/resource/lifecycle reasons (eviction, OOM, node failure, `activeDeadlineSeconds`), but never simply because a workload is idle.

Related links:
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
https://cloud.google.com/blog/products/containers-kubernetes/scale-to-zero-on-gke-with-keda
https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/
https://kubernetes.io/docs/concepts/workloads/controllers/job/

### Task Type

- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.