elastic / elastic/integrations

[All Integrations] Users should be able to do custom processing after a pipeline failure.

Open
#14,848 2 comments 0 reactions 0 assignees View on GitHub
enhancement Integration:fortinet_fortigate Team:Integration-Experience Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

Hello,

Currently when an ingest pipeline processor fails for some reason and it does not have the option `ignore_failure` set to `true`, it will break the processing of further processors, including the custom pipeline for the integration.

When this happens the user has no way to do any processing on this event as all customization needs to be done on the `@custom` ingest pipeline, and this pipeline is skipped when a failure happens.

For example, I have an issue with the Fortigate pipeline where some devices send some events with the wrong format or truncated and this breaks the pipeline, this generate thousands of events per hour that I cannot drop using a custom ingest pipeline as it is ignored.

So the integrations should have a way for the user to capture those failed events and do something with them.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.