influxdata / influxdata/telegraf

inputs.nvidia_smi: nvidia_smi_process points collide into a single series

Open
#19,457 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[[inputs.nvidia_smi]]
```

### Logs from Telegraf

No errors or warnings, the plugin reports success. The points are produced and then collapse downstream.

### System info

Telegraf master (a05ff8b), reproducible on every version since v1.30.1 where the measurement was added in #15023. Affects nvidia-smi schema v12 and v13.

### Steps to reproduce

1. Run `inputs.nvidia_smi` on a host where more than one process of the same name is using the GPUs, for example two training jobs, or one job per card on a multi-GPU box.
2. Look at the `nvidia_smi_process` metrics.

### Expected behavior

One series per process, so that `used_memory` can be tracked per job and per card.

### Actual behavior

`nvidia_smi_process` is tagged only with `name` and `type`. `pid` and `used_memory` are fields, and there is no GPU identifier in the tag set at all. Every process sharing a name and type therefore lands on the same series, at the same timestamp, and processes on different cards collide with each other too. Downstream only one of them survives, and which one is arbitrary.

Feeding a two-GPU nvidia-smi output with three `python` processes through the v13 parser:

```text
tags=map[name:python type:C] fields=map[pid:4937 used_memory:160] ts=1785974620000000000
tags=map[name:python type:C] fields=map[pid:4938 used_memory:2048] ts=1785974620000000000
tags=map[name:python type:C] fields=map[pid:5001 used_memory:8192] ts=1785974620000000000
```

Three points, identical tags, identical timestamp. On a multi-GPU training box that is the normal case rather than an edge case.

### Additional info

The tag set is built in the process loop at the end of `Parse` in both `plugins/inputs/nvidia_smi/schema_v12/parser.go` and `plugins/inputs/nvidia_smi/schema_v13/parser.go`, which are identical here.

Suggested fix is to add the GPU `uuid` as a tag, the same value the parent `nvidia_smi` measurement already carries, and to promote `pid` to a tag. The uuid is stable across reboots and across reordering of the cards in a way the loop index is not.

Two things worth deciding before implementing:

- `pid` as a tag is unbounded cardinality, since pids churn as jobs restart. `inputs.procstat` handles this by making it opt-in through its `tag_with` setting, which may be the better precedent here than tagging unconditionally.
- The measurement has shipped since v1.30.1, so moving `pid` from a field to a tag is a metric format change and needs the treatment in `docs/developers/METRIC_FORMAT_CHANGES.md`. Adding `uuid` is purely additive and could go in on its own.

Also worth picking up while in there: `` carries `gpu_instance_id` and `compute_instance_id`, which the parser currently drops. Those would attribute a process to a MIG instance rather than just to the physical card.

Found while reviewing #19408, which documents this measurement for the first time.

Contributor guide

Open the contributing guide

Research direction

Start with the process loops at the end of Parse in plugins/inputs/nvidia_smi/schema_v12/parser.go and schema_v13/parser.go, then reproduce the three-process v13 example. Review inputs.procstat's tag_with behavior and docs/developers/METRIC_FORMAT_CHANGES.md before choosing the tag treatment. Done means processes on different GPUs no longer collapse, with coverage for the affected schemas and any metric-format change documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.