vectordotdev / vectordotdev/vector

prometheus_remote_write source: skip_nan_values does not cover histogram/summary counts, stale-marker NaN still 400s the whole request

Open
#26,228 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

skip_nan_values: true (added in #23774) filters NaN f64 sample values, but the histogram/summary count conversion still rejects the whole write request when a count sample is NaN:

WARN source{component_id=prometheus_remote_write_source component_type=prometheus_remote_write}:
  Received bad request. error=Could not decode write request:
  expected value in range [0, 18446744073709551615], found: NaN
  error_code=http_response_400 http_code=400

Observed on vector 0.57.0 with skip_nan_values: true active, receiving from VictoriaMetrics vmagent.

The path is try_f64_to_u64() in lib/prometheus-parser/src/lib.rs, used for histogram bucket / summary counts during aggregation. It is unconditional; skip_nan_values never reaches it.

NaN counts are not malformed input: the remote-write 1.0 spec requires senders to mark stale series with a NaN value, so any scrape target disappearing can put NaN on a _count/_bucket series. Senders such as vmagent treat the 400 as non-retryable and drop the entire block, so every unrelated sample batched alongside is silently lost.

Expected: with skip_nan_values: true, NaN counts (at minimum stale markers) are skipped like other NaN samples instead of failing the request.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/prometheus-parser/src/lib.rs at try_f64_to_u64(), then trace its histogram bucket and summary count callers to see where skip_nan_values is available. Use the linked remote-write stale-marker specification as the behavior reference; done means NaN counts are skipped when skip_nan_values is enabled instead of rejecting the whole request.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, rust
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.