elastic / elastic/beats

[Filebeat] Add separate error metrics for errors on the input and output sides of file ingestion

Open
#47,015 2 comments 0 reactions 0 assignees View on GitHub
enhancement Filebeat Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

**Describe the enhancement:**

Currently we only have a total error counter `processing_errors_total` which accounts for both errors while reading from the file:

https://github.com/elastic/beats/blob/ca2d2629672042e42105cc2d41d78f6d96b15c8c/filebeat/input/filestream/input.go#L678

and while sending events to the output:

https://github.com/elastic/beats/blob/ca2d2629672042e42105cc2d41d78f6d96b15c8c/filebeat/input/filestream/input.go#L728

Users would like to see the separation of processing errors by "happened on the input side" and "happened on the output side":

* `processing_errors_input`
* `processing_errors_output`

## Definition of Done

* Filebeat produces `processing_errors_input` and `processing_errors_output` as described above
* The [dashboard](https://github.com/elastic/integrations/blob/34a7ece515bf5a5a2607d950370a2c2a27fa5704/packages/elastic_agent/kibana/dashboard/elastic_agent-ea44c820-95cb-11ee-b517-0550321b54cc.json) in the Elastic Agent integration has the added metrics
* The [documentation](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#_metrics_8) is updated accordingly

**Describe a specific use case for the enhancement or feature:**

The `processing_errors_total` metrics is not helpful for troubleshooting file ingestion errors, we need more granularity.

Contributor guide

Open the contributing guide

Research direction

Start in filebeat/input/filestream/input.go at the processing-error sites around lines 678 and 728, then inspect how the existing processing_errors_total metric is exposed. Update the Elastic Agent dashboard JSON and the Filebeat filestream metrics documentation linked in the issue. Done means both side-specific metrics are produced and represented in the dashboard and docs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.