vectordotdev / vectordotdev/vector
prometheus_remote_write source: skip_nan_values does not cover histogram/summary counts, stale-marker NaN still 400s the whole request
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
skip_nan_values: true (added in #23774) filters NaN f64 sample values, but the histogram/summary count conversion still rejects the whole write request when a count sample is NaN:
WARN source{component_id=prometheus_remote_write_source component_type=prometheus_remote_write}:
Received bad request. error=Could not decode write request:
expected value in range [0, 18446744073709551615], found: NaN
error_code=http_response_400 http_code=400
Observed on vector 0.57.0 with skip_nan_values: true active, receiving from VictoriaMetrics vmagent.
The path is try_f64_to_u64() in lib/prometheus-parser/src/lib.rs, used for histogram bucket / summary counts during aggregation. It is unconditional; skip_nan_values never reaches it.
NaN counts are not malformed input: the remote-write 1.0 spec requires senders to mark stale series with a NaN value, so any scrape target disappearing can put NaN on a _count/_bucket series. Senders such as vmagent treat the 400 as non-retryable and drop the entire block, so every unrelated sample batched alongside is silently lost.
Expected: with skip_nan_values: true, NaN counts (at minimum stale markers) are skipped like other NaN samples instead of failing the request.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/prometheus-parser/src/lib.rs at try_f64_to_u64(), then trace its histogram bucket and summary count callers to see where skip_nan_values is available. Use the linked remote-write stale-marker specification as the behavior reference; done means NaN counts are skipped when skip_nan_values is enabled instead of rejecting the whole request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, rust
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100