vectordotdev / vectordotdev/vector

`datadog_metrics`: sink fails with `Some(BadRequest)` and drops events

Open
#26,133 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

sink: datadog_metrics
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

Problem

The datadog_metrics sink fails with Some(BadRequest) and drops the events in the request:

ERROR Service call failed. No retries or retries exhausted.
      error=Some(BadRequest) error_type="request_failed" stage="sending"
      component_kind="sink" component_type="datadog_metrics"
ERROR Events dropped intentional=false count=1
      reason="Service call failed. No retries or retries exhausted."

Version

0.56.0

Reproduction

Expose a histogram whose observations all fall above its top bucket, so every one of them ends up
in the implicit +Inf bucket (pip install prometheus_client):

# fixture.py
import time

from prometheus_client import CollectorRegistry, Histogram, start_http_server

registry = CollectorRegistry()
# The only bucket is le=1, and every observation is far above it.
h = Histogram("test_overflow", "demo", buckets=(1.0,), registry=registry)

start_http_server(19092, registry=registry)
while True:
    h.observe(5000)
    time.sleep(1)

which exposes:

test_overflow_bucket{le="1.0"} 0.0
test_overflow_bucket{le="+Inf"} 4.0
test_overflow_count 4.0
test_overflow_sum 20000.0
# repro.yaml
sources:
  prom:
    type: prometheus_scrape
    endpoints: ["http://127.0.0.1:19092/metrics"]
    scrape_interval_secs: 3

sinks:
  dd:
    type: datadog_metrics
    inputs: [prom]
    default_api_key: "${DD_API_KEY}"
python3 fixture.py &
DD_API_KEY=<key> vector --config repro.yaml

Every scrape after the first produces one such failure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running fixture.py with repro.yaml and observe the datadog_metrics sink handling of the Prometheus +Inf bucket. Trace the sink's request and error path for histogram observations above the top bucket; done means repeated scrapes no longer produce Some(BadRequest) or drop events.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.