vectordotdev / vectordotdev/vector
`datadog_metrics`: sink fails with `Some(BadRequest)` and drops events
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
Problem
The datadog_metrics sink fails with Some(BadRequest) and drops the events in the request:
ERROR Service call failed. No retries or retries exhausted.
error=Some(BadRequest) error_type="request_failed" stage="sending"
component_kind="sink" component_type="datadog_metrics"
ERROR Events dropped intentional=false count=1
reason="Service call failed. No retries or retries exhausted."
Version
0.56.0
Reproduction
Expose a histogram whose observations all fall above its top bucket, so every one of them ends up
in the implicit +Inf bucket (pip install prometheus_client):
# fixture.py
import time
from prometheus_client import CollectorRegistry, Histogram, start_http_server
registry = CollectorRegistry()
# The only bucket is le=1, and every observation is far above it.
h = Histogram("test_overflow", "demo", buckets=(1.0,), registry=registry)
start_http_server(19092, registry=registry)
while True:
h.observe(5000)
time.sleep(1)
which exposes:
test_overflow_bucket{le="1.0"} 0.0
test_overflow_bucket{le="+Inf"} 4.0
test_overflow_count 4.0
test_overflow_sum 20000.0
# repro.yaml
sources:
prom:
type: prometheus_scrape
endpoints: ["http://127.0.0.1:19092/metrics"]
scrape_interval_secs: 3
sinks:
dd:
type: datadog_metrics
inputs: [prom]
default_api_key: "${DD_API_KEY}"
python3 fixture.py &
DD_API_KEY=<key> vector --config repro.yaml
Every scrape after the first produces one such failure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running fixture.py with repro.yaml and observe the datadog_metrics sink handling of the Prometheus +Inf bucket. Trace the sink's request and error path for histogram observations above the top bucket; done means repeated scrapes no longer produce Some(BadRequest) or drop events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100