elastic / elastic/integrations

Exclude ingest/API error documents from dashboards, transforms, and detection rule metrics

Open
#20,409 4 comments 0 reactions 0 assignees View on GitHub
bug Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

## Summary

When an API request fails during data collection (for example, due to authentication, connectivity, rate limiting, or other API errors), the generated error document is currently indexed and counted as an event by downstream assets.

As a result, dashboards, transforms, detection rules, and other assets that rely on document counts treat these ingest error documents as valid events, leading to inaccurate metrics.

### Problem

For example:

* A dashboard metric such as **Total Events** shows a count of `1` even when no actual events were collected successfully.
* Transforms and detection rules also process these error documents, resulting in misleading counts and potentially incorrect behavior.

This gives users the impression that events were ingested successfully when, in reality, only an API error occurred.

### Proposed Solution

Introduce a standardized way to identify ingest/API error documents (for example, using a field such as `error.type: ingest_error` or an equivalent global marker).

Dashboards, transforms, detection rules, and other assets should then exclude documents marked as ingest errors from event-based metrics and processing.

### Expected Outcome

* API/ingest error documents remain available for troubleshooting.
* Event count metrics reflect only successfully ingested events.
* Dashboards, transforms, detection rules, and other assets provide accurate and consistent results across integrations.
* A common, reusable solution can be adopted across integrations instead of implementing asset-specific workarounds.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points; begin by tracing how API/ingest error documents are created and then consumed by dashboards, transforms, and detection rules. Done means error documents remain available for troubleshooting while event-based counts and processing consistently exclude them across the named downstream assets.

Written by the indexing model from the issue text.

Assessment

Domain
data, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.