ContextLab / ContextLab/clustrix

Restore PBS support: removed from v0.2.0 as never verified against a real scheduler

Open
#140 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
4
Avg merge
6h 27m
Merged PRs (30d)
9

Description

PBS support is implemented and removed from v0.2.0 as unverified. It has never been run against a real PBS/Torque scheduler.

## What exists, and where

Introduced in `4b2aba5` (2025-06-24). At `299109f`:

| Code | Location |
|-|-|
| Job submission | `clustrix/executor_schedulers.py:218` `submit_pbs_job` |
| Status polling | `clustrix/executor_scheduler_status.py:489` `_check_pbs_status` |
| Dispatch | `clustrix/executor_core.py:111,547,571` |
| Script generation | `clustrix/utils.py:2772` `_create_pbs_script`; dispatch at `:2509,2546` |
| Tutorial | `docs/source/notebooks/pbs_tutorial.ipynb` |

## Known defect found while auditing, never fixed

PBS was the one scheduler that **never set up its remote environment** -- SLURM and SGE staged and built a venv, PBS did not, so a PBS job ran against whatever interpreter happened to be on the node. That was corrected during the v0.2.0 sweep (all four schedulers now share one staging path), but the fix was itself never exercised against a real PBS queue. So the correction is as unverified as the original.

## Why it is being removed rather than fixed

Not because the code is known to be wrong. Because it has **never been run against the real thing**, and shipping it in the cluster-type dropdown states otherwise. A user who selects it gets a code path no one has ever seen succeed.

v0.2.0 keeps exactly the four backends that have been demonstrated end to end -- `local`, `ssh`, `slurm`, `huggingface` -- and the documentation now says the rest are planned for a future release rather than currently supported.

## Restoring it

Nothing is lost: every line cited above stays reachable in git history at the commits named. Reinstating it means reverting the removal commit and then doing the part that was never done -- running it against real hardware and recording the evidence in this issue.

## Definition of done

- [ ] Backend restored from history
- [ ] A real job submitted, executed and its result returned, with the transcript pasted into this issue
- [ ] Failure paths exercised (job rejected, job killed, node lost)
- [ ] Re-added to `SUPPORTED_CLUSTER_TYPES`, the widget dropdown and the CLI
- [ ] Documentation moved from "planned" to "supported"

Contributor guide

Open the contributing guide

Research direction

Start with the removal commit and the cited history, then review `clustrix/executor_schedulers.py`, `clustrix/executor_scheduler_status.py`, `clustrix/executor_core.py`, and `clustrix/utils.py` to trace PBS submission, polling, dispatch, and script generation. The work is done when a real PBS/Torque job returns its result, the listed failure paths are exercised, the backend is restored in the supported-type and interface lists, and the transcript is recorded in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.