NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner

Report per-column idle time in end-of-run metrics

Open
#695 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.2k
Forks
211
Avg merge
2d 6h
Merged PRs (30d)
40

Description

Feature Request

Add per-column idle-time metrics to the end-of-run metrics emitted after a create run completes.

Motivation

For model-backed generation columns, it is hard to tell whether total runtime is dominated by model response latency or by time where a column is not actually waiting on an in-flight request. When tuning generation plans, concurrency, scheduling, and model configuration, we should be able to see how much of the overall run was spent with each model-backed column truly idle.

Proposed Behavior

For each generation column that can issue model requests, likely columns using the model mixin / ModelConfig, report metrics such as:

  • request_wait_wall_time_s: wall-clock time during the run where the column had at least one in-flight model request and was waiting on a response.
  • idle_time_s: wall-clock time during the run where the column was not waiting on a model response.
  • idle_pct_of_run: idle_time_s / total_run_wall_time_s.
  • Existing or related request metrics, if available, such as request count and summed request latency, should remain separate from wall-clock wait time.

The important distinction is that request_wait_wall_time_s should avoid double-counting overlapping requests for the same column. If a column has multiple concurrent requests in flight, the wall-clock wait metric should represent the union of those waiting intervals, not the sum of every individual request latency. That makes the idle calculation answer: "during how much of the total run was this column not waiting on a model response?"

A table-like end-of-run summary could look like:

column        model_config    request_wait_wall_s    idle_s    idle_pct_of_run    requests
prompt        gpt-4.1-mini    83.2                   16.8      16.8%              240
label         nemotron        42.5                   57.5      57.5%              120

Acceptance Criteria

  • End-of-run metrics after create include per-column idle metrics for columns that issue model requests.
  • The metrics are available wherever end-of-run metrics are currently surfaced, not only in logs, if a structured metrics object exists.
  • Overlapping in-flight requests for a column are handled as wall-clock intervals rather than double-counted summed latencies.
  • Non-model-backed columns are either omitted from this section or reported with an explicit not-applicable state.
  • Tests cover at least one mocked model-backed column with controlled request timing and one case with overlapping requests.

Open Questions

  • Should idle_time_s use the full run wall-clock duration as its denominator, or only the column's active window between first scheduled work and final completed work?
  • Should a later enhancement split idle time into dependency wait, scheduler/admission wait, local processing time, and no-work-complete time?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the end-of-run metrics emitted after a create run and the model mixin or ModelConfig paths used by generation columns. Inspect existing request metrics and their structured output, then trace how concurrent requests are tracked. Done means per-column wait and idle metrics are surfaced after create, overlapping waits are unioned, and controlled single-column and overlapping-request tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.