Call sky down when cancelling skypilot experiment task
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 258
- Forks
- 113
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 5
Description
From https://github.com/NVIDIA-NeMo/Run/pull/340#issue-3408416245
There's open questions about how to handle ctrl+c and cancelling sky launch requests through nemo run. Current behavior appears to be nemo experiment cancel just cancels the task, but does not sky down the cluster. When coupled with retry_until_up=True, the sky launch request may keep running indefinitely and will need to be manually terminated with sky down or a sky api cancel.
We may want to have sky down run when nemo experiment cancel is run
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the implementation of nemo experiment cancel and trace how Sky launch requests and clusters are managed. Clarify the intended ctrl+c and cancellation behavior, including retry_until_up=True, then verify that cancellation terminates both the task and any associated launch request or cluster.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100