elastic / elastic/integrations
Exclude ingest/API error documents from dashboards, transforms, and detection rule metrics
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## Summary
When an API request fails during data collection (for example, due to authentication, connectivity, rate limiting, or other API errors), the generated error document is currently indexed and counted as an event by downstream assets.
As a result, dashboards, transforms, detection rules, and other assets that rely on document counts treat these ingest error documents as valid events, leading to inaccurate metrics.
### Problem
For example:
* A dashboard metric such as **Total Events** shows a count of `1` even when no actual events were collected successfully.
* Transforms and detection rules also process these error documents, resulting in misleading counts and potentially incorrect behavior.
This gives users the impression that events were ingested successfully when, in reality, only an API error occurred.
### Proposed Solution
Introduce a standardized way to identify ingest/API error documents (for example, using a field such as `error.type: ingest_error` or an equivalent global marker).
Dashboards, transforms, detection rules, and other assets should then exclude documents marked as ingest errors from event-based metrics and processing.
### Expected Outcome
* API/ingest error documents remain available for troubleshooting.
* Event count metrics reflect only successfully ingested events.
* Dashboards, transforms, detection rules, and other assets provide accurate and consistent results across integrations.
* A common, reusable solution can be adopted across integrations instead of implementing asset-specific workarounds.
Contributor guide
Research direction
The issue names no files, tests, or entry points; begin by tracing how API/ingest error documents are created and then consumed by dashboards, transforms, and detection rules. Done means error documents remain available for troubleshooting while event-based counts and processing consistently exclude them across the named downstream assets.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100