caikit / caikit/caikit-ray-backend

[RAY] timeout mechanism for job length or long queues

Open
#45 1 comment 0 reactions 1 assignee Claimed by @ibm-peach-fish View on GitHub
enhancement
Dominant language
Python
Stars
0
Forks
3
PR merge metrics
No merged PRs in 30d

Description

reflected from private internal issue tracker by @chakrn

When we use caikit-ray-backend to submit a new job, that job can run indefinitely with no timeout. We need a way to have a configurable timeout value and cancel the job after exceeding the time.

After having a quick discussion with Dean, this can probably be done simply by changing the [ray.get()](https://github.com/caikit/caikit-ray-backend/blob/main/caikit_ray_backend/ray_submitter.py#L81-L85) to a ray.wait() (which should've been the case anyway). Then poll for status and kill the job after a certain elapsed time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.