NatLabRockies / NatLabRockies/torc

Make runtime-blocked packing diagnostic resource-aware (not time-only)

Open
#375 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
19
Forks
6
Avg merge
11h 1m
Merged PRs (30d)
1

Description

Background

PR #374 added a runtime-blocked packing signal (torc workflows diagnose plus a cheap runtime_blocked_ready_jobs field on get_workflow_status). It detects ready jobs whose required runtime exceeds the remaining walltime of active allocations.

Copilot's review correctly pointed out that the signal reasons only about the time dimension, while the actual claim filter matches on CPU, memory, GPU, nodes, and runtime together (src/server/http_server/jobs_transport.rs:2570-2575), plus scheduler_id.

Consequences of the time-only approximation:

  • False negative (current): an active unlimited/local worker (end_time IS NULL) suppresses the signal, but that worker may not satisfy a job's non-time requirements (e.g. a CPU-only local worker can't run a GPU job), so the job is still genuinely runtime-blocked on the bounded allocations.
  • False positive (latent): the "max remaining walltime" used for the bounded case ignores whether the longest-walltime allocation can even satisfy the job's CPU/mem/GPU/nodes.

A single runtime-only scalar cannot be correct in both directions; resources must enter the computation.

Decision

For now (PR #374) we keep the time-only heuristic as an approximate hint — diagnose is a pointer, and the claim path remains the source of truth on what actually runs. This issue tracks the follow-up to make it precise.

Proposed approach (Option B)

Make the diagnose command resource-aware while keeping the status tripwire cheap (and documented as approximate):

  • diagnose already fetches all ready jobs, resource requirements, and active compute nodes, so no new API calls are needed.
  • For each ready job, count it as runtime-blocked only when no active allocation can run it — i.e. there is no node that satisfies its num_cpus / memory / num_gpus / num_nodes and whose remaining walltime (∞ for end_time IS NULL) covers runtime_s (+ the startup grace).
  • Mirror the claim matcher's semantics: per-node vs total capacity for multi-node jobs, memory_gb (node) vs memory_bytes (RR) unit conversion, and scheduler_id matching.

Leave the O(1) get_workflow_status tripwire as-is (cheap, time-only), with its docs/comment noting it is approximate and that diagnose is authoritative.

Risks / notes

  • Duplicates subtle matching logic from jobs_transport; consider extracting a shared helper to avoid drift, or add a test that cross-checks diagnose against the claim query.
  • Only matters for workflows that mix unlimited/local and bounded/Slurm workers with heterogeneous resource classes — uncommon, hence deferred.

References

  • PR #374 (Copilot review threads on src/server/api/workflows.rs, src/client/commands/diagnose.rs)
  • Claim filter: src/server/http_server/jobs_transport.rs:2570-2575

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the diagnose entry point in src/client/commands/diagnose.rs and compare its current logic with the claim filter at src/server/http_server/jobs_transport.rs:2570-2575. Review the related workflow API references in src/server/api/workflows.rs and PR #374. Done means diagnose considers resource capacity, walltime, and scheduler_id together while the status tripwire remains approximate.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, hpc
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.