NatLabRockies / NatLabRockies/torc
Make runtime-blocked packing diagnostic resource-aware (not time-only)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19
- Forks
- 6
- Avg merge
- 11h 1m
- Merged PRs (30d)
- 1
Description
Background
PR #374 added a runtime-blocked packing signal (torc workflows diagnose plus a cheap runtime_blocked_ready_jobs field on get_workflow_status). It detects ready jobs whose required runtime exceeds the remaining walltime of active allocations.
Copilot's review correctly pointed out that the signal reasons only about the time dimension, while the actual claim filter matches on CPU, memory, GPU, nodes, and runtime together (src/server/http_server/jobs_transport.rs:2570-2575), plus scheduler_id.
Consequences of the time-only approximation:
- False negative (current): an active unlimited/local worker (
end_time IS NULL) suppresses the signal, but that worker may not satisfy a job's non-time requirements (e.g. a CPU-only local worker can't run a GPU job), so the job is still genuinely runtime-blocked on the bounded allocations. - False positive (latent): the "max remaining walltime" used for the bounded case ignores whether the longest-walltime allocation can even satisfy the job's CPU/mem/GPU/nodes.
A single runtime-only scalar cannot be correct in both directions; resources must enter the computation.
Decision
For now (PR #374) we keep the time-only heuristic as an approximate hint — diagnose is a pointer, and the claim path remains the source of truth on what actually runs. This issue tracks the follow-up to make it precise.
Proposed approach (Option B)
Make the diagnose command resource-aware while keeping the status tripwire cheap (and documented as approximate):
diagnosealready fetches all ready jobs, resource requirements, and active compute nodes, so no new API calls are needed.- For each ready job, count it as runtime-blocked only when no active allocation can run it — i.e. there is no node that satisfies its
num_cpus/memory/num_gpus/num_nodesand whose remaining walltime (∞ forend_time IS NULL) coversruntime_s(+ the startup grace). - Mirror the claim matcher's semantics: per-node vs total capacity for multi-node jobs,
memory_gb(node) vsmemory_bytes(RR) unit conversion, andscheduler_idmatching.
Leave the O(1) get_workflow_status tripwire as-is (cheap, time-only), with its docs/comment noting it is approximate and that diagnose is authoritative.
Risks / notes
- Duplicates subtle matching logic from
jobs_transport; consider extracting a shared helper to avoid drift, or add a test that cross-checksdiagnoseagainst the claim query. - Only matters for workflows that mix unlimited/local and bounded/Slurm workers with heterogeneous resource classes — uncommon, hence deferred.
References
- PR #374 (Copilot review threads on
src/server/api/workflows.rs,src/client/commands/diagnose.rs) - Claim filter:
src/server/http_server/jobs_transport.rs:2570-2575
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the diagnose entry point in src/client/commands/diagnose.rs and compare its current logic with the claim filter at src/server/http_server/jobs_transport.rs:2570-2575. Review the related workflow API references in src/server/api/workflows.rs and PR #374. Done means diagnose considers resource capacity, walltime, and scheduler_id together while the status tripwire remains approximate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, hpc
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100