NatLabRockies / NatLabRockies/torc
Feature: show total compute nodes and aggregate node runtime per workflow in the TUI
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19
- Forks
- 6
- Avg merge
- 11h 1m
- Merged PRs (30d)
- 1
Description
Summary
The TUI can show which compute nodes a workflow used, but not how much compute a workflow
consumed. Proposal: surface a per-workflow node count and aggregate node runtime (node-hours)
in the TUI, plus a per-node Duration column so the aggregate can be broken down.
There are a few open questions at the bottom.
Motivation
Today, answering "how much compute did this workflow burn?" requires leaving the TUI:
- The Compute Nodes tab (
draw_compute_nodes_tableinsrc/tui/ui.rs) lists ID, hostname,
CPUs, memory, GPUs, active flag, and CPU/memory peak+avg — but no duration, even though
ComputeNodeModel.duration_secondsalready exists andtorc compute-nodes listrenders it as
aDurationcolumn. - The Summary tab shows job progress and the runtime-blocked tripwire, but says nothing about
allocations. - The Workflows table shows only ID / Name / User / Description, so there is no way to compare
two workflows' allocation footprints side by side.
So a user must page through the node list, mentally sum durations, and multiply by node count —
for a number torc already has all the inputs for. This matters on HPC because node-hours are the
billed unit; src/client/hpc/profiles.rs already carries an "AU per node-hour" charge factor, so
the concept exists in the codebase but isn't reachable from the monitoring UI.
Concrete questions users want answered at a glance:
- How many allocations has this workflow used in total (not just how many are active)?
- How much wall time did those allocations consume in aggregate?
- Are allocation lifetimes uniform, or is one straggler dominating the bill?
- Is this workflow much more expensive than a comparable one?
Proposed UI surface
1. Compute Nodes tab: add a Duration column
Lowest-effort piece and useful on its own. Renders duration_seconds with the existing
format_secs_short helper, - when unset. Should participate in the existing sort hotkeys
(1 id 2 host 3 cpu 4 mem → add a duration key).
2. Compute Nodes tab: add a footer/aggregate line
A single summary line in the block, e.g.:
Total: 48 nodes (12 active) │ 312.4 node-hours │ shortest 1h48m │ median 6h30m │ longest 9h12m
Median rather than mean: allocation lifetimes are routinely skewed by a few long stragglers or a
handful of nodes that exited almost immediately, and a mean reports a duration that no actual
allocation had. Pairing shortest / median / longest gives the spread in one line and makes
that skew visible instead of averaging it away.
This must reflect the whole workflow, not just the currently loaded page — the compute node
list is paginated (default/max limit 10,000), so summing or sorting app.compute_nodes
client-side would silently under-report on large workflows. See the implementation note below.
3. Summary tab: one allocation line
Add a line near the progress line, e.g.:
Allocations: 48 nodes (12 active) │ 312.4 node-hours │ shortest 1h48m │ median 6h30m │ longest 9h12m
This is the piece most users would actually look at, since the Summary tab is the default view.
If the line is too wide for narrow terminals, the shortest/median/longest group is the part to
drop first — see open question 3.
Key design decision: aggregate server-side
Client-side summing is tempting but wrong here:
- The node list is paginated, so the TUI would need to walk every page just to render a header.
- The median in particular needs the full set ordered, so it cannot be derived from a page at
all — unlike a running sum, there is no partial form to accumulate. - The TUI refreshes on a timer; re-walking all pages per refresh is an unnecessary load
multiplier, exactly the kind of idle-loop cost being scrutinized in #377.
Recommended: compute the aggregate in the server with a single pass over compute_node for the
workflow (COUNT/SUM plus MIN/MAX and an ORDER BY … LIMIT 1 OFFSET n/2 for the median)
and return it alongside existing status data, so the TUI gets it for free on a refresh it already
performs.
In-flight nodes (the one real ambiguity)
duration_seconds is written by the runner when it finishes (job_runner.rs computes it from
start_instant.elapsed()), so active nodes have duration_seconds = None. A naive SUM
therefore reports 0 contribution from every node that is currently running — the worst case being
a workflow that is 100% active and shows 0.0 node-hours.
Proposed rule: for nodes with is_active = true and no duration_seconds, compute elapsed time
as now - start_time (start_time is always present). Report the two parts distinctly so the
number is honest:
312.4 node-hours (48.0 in flight)
This also keeps the value monotonically sensible as a workflow runs, rather than jumping only
when allocations exit. The same effective-duration rule feeds shortest / median / longest, so a
freshly started allocation legitimately shows up as the shortest.
Related wrinkle: ComputeNodeModel also carries num_nodes, so a single record can represent a
multi-node allocation. The aggregate should be SUM(num_nodes * duration), not SUM(duration),
and the "total nodes" count should be SUM(num_nodes) rather than COUNT(*) — otherwise
multi-node Slurm allocations are undercounted. Shortest / median / longest are per-allocation
lifetimes and are not weighted by num_nodes (see open question 7).
Implementation sketch
| Layer | Location | Change |
|---|---|---|
| Server | src/server/api/workflows.rs |
Aggregate COUNT/SUM(num_nodes * duration) plus min/median/max over compute_node; handle active nodes via start_time |
| Model | src/models.rs |
Add total_compute_nodes, active_compute_nodes, total_node_seconds, active_node_seconds, min_node_seconds, median_node_seconds, max_node_seconds to the workflow status response |
| API | api/* |
cd api && bash sync_openapi.sh all --promote + client regen |
| TUI | src/tui/app.rs |
Carry the new fields on WorkflowSummary |
| TUI | src/tui/ui.rs |
Duration column + aggregate line in draw_compute_nodes_table; allocation line in draw_summary |
Suggested build order:
Durationcolumn in the Compute Nodes tab (pure TUI, no API change, immediately useful).- Server aggregate + model/API fields.
- Summary tab allocation line + Compute Nodes footer.
- Workflows-table columns, if wanted.
Acceptance Criteria
- Node count, aggregate runtime, and shortest / median / longest reflect all of a workflow's
compute nodes, not just the currently loaded page. - Multi-node allocations (
num_nodes > 1) are counted by node for totals. - Active nodes contribute their elapsed time and are reported separately from finished nodes.
- Median is defined over the same effective durations as shortest and longest, with a documented
even-count convention (proposal: lower median, so the reported value is always a real
allocation's duration). - A workflow with exactly one compute node reports shortest == median == longest.
- Workflows with zero compute nodes render cleanly (
0 nodes │ 0 node-hours, no-, no panic,
no divide-by-zero in the median). - No additional API requests per TUI refresh beyond those already made.
Open questions
- Charge factors. Should the TUI apply the AU-per-node-hour factor from
src/client/hpc/profiles.rsto show estimated allocation cost, or is raw node-hours enough? - Retention. Are compute node records ever pruned for long-lived or archived workflows? If
so the aggregate is a lower bound and should be labeled as such. - Narrow terminals. Is dropping shortest/median/longest the right degradation when the line
doesn't fit, or should the totals be dropped instead? - Median weighting. Should shortest/median/longest be per-allocation (proposed) or weighted
bynum_nodesso a 100-node allocation counts 100 times in the distribution? - Should the same numbers be available from the CLI status/report output, so the TUI isn't the only
way to get them?
Non-Goals
- Changing how compute nodes are registered or how
duration_secondsis reported. - Adding a new time-series/history view of allocation usage.
- Billing or quota enforcement.
This issue text was generated by Claude Opus 5 and was reviewed/edited before submission
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading src/tui/ui.rs, src/tui/app.rs, src/server/api/workflows.rs, and src/models.rs, then trace the existing workflow status response and compute-node pagination. Resolve the open questions about median semantics, retention, narrow terminals, and charge factors before implementation. Done means all acceptance criteria pass without adding TUI refresh requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100