[Filebeat] Add separate error metrics for errors on the input and output sides of file ingestion
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
**Describe the enhancement:**
Currently we only have a total error counter `processing_errors_total` which accounts for both errors while reading from the file:
https://github.com/elastic/beats/blob/ca2d2629672042e42105cc2d41d78f6d96b15c8c/filebeat/input/filestream/input.go#L678
and while sending events to the output:
https://github.com/elastic/beats/blob/ca2d2629672042e42105cc2d41d78f6d96b15c8c/filebeat/input/filestream/input.go#L728
Users would like to see the separation of processing errors by "happened on the input side" and "happened on the output side":
* `processing_errors_input`
* `processing_errors_output`
## Definition of Done
* Filebeat produces `processing_errors_input` and `processing_errors_output` as described above
* The [dashboard](https://github.com/elastic/integrations/blob/34a7ece515bf5a5a2607d950370a2c2a27fa5704/packages/elastic_agent/kibana/dashboard/elastic_agent-ea44c820-95cb-11ee-b517-0550321b54cc.json) in the Elastic Agent integration has the added metrics
* The [documentation](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#_metrics_8) is updated accordingly
**Describe a specific use case for the enhancement or feature:**
The `processing_errors_total` metrics is not helpful for troubleshooting file ingestion errors, we need more granularity.
Contributor guide
Research direction
Start in filebeat/input/filestream/input.go at the processing-error sites around lines 678 and 728, then inspect how the existing processing_errors_total metric is exposed. Update the Elastic Agent dashboard JSON and the Filebeat filestream metrics documentation linked in the issue. Done means both side-specific metrics are produced and represented in the dashboard and docs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100