caikit / caikit/caikit-ray-backend
[RAY] timeout mechanism for job length or long queues
Open
enhancement
- Dominant language
- Python
- Stars
- 0
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
reflected from private internal issue tracker by @chakrn
When we use caikit-ray-backend to submit a new job, that job can run indefinitely with no timeout. We need a way to have a configurable timeout value and cancel the job after exceeding the time.
After having a quick discussion with Dean, this can probably be done simply by changing the [ray.get()](https://github.com/caikit/caikit-ray-backend/blob/main/caikit_ray_backend/ray_submitter.py#L81-L85) to a ray.wait() (which should've been the case anyway). Then poll for status and kill the job after a certain elapsed time.
Contributor guide
Assessment
This issue has not been assessed yet.