[IPTables]: Graceful Grok Error Handling for Custom Log Patterns

### Integration Name

Iptables [iptables]

### Dataset Name

log

### Integration Version

1.20.0

### Agent Version

8.17.4

### OS Version and Architecture

x86_64 RHEL8.6 and RHEL9.2

### User Goal

For integrations where a Grok processor can be used for parsing custom log formats, it would be good to have the ability to have a failure handler that can be used as a second pass for events that don't match the original format.

Under the current configurations, the options when a customer has logs that don't conform to industry standard / are in a custom format, it basically boils down to:

1. Clone the entire pipeline and update the patterns. Update the index template or agent config to use this pipeline instead.
2. Add the pattern to the OOTB pipeline. This will require re-applying this update every time the integration is updated.

A solution could be to copy the existing ‘[logs-iptables.log@custom]’ idea, but make it something like ‘[logs-iptables.log@grok-custom]’ and add that as a failure handler on the primary grok processor. This would allow customers to add their custom log patterns into that @grok-custom pipeline that doesn’t get updated by packages and the @grok-custom pipeline would parse their non-standard logs and then return the event back to the OOTB integration pipeline once the grok has completed.

An example from the customer is as follows (maybe not the best, but it was readily available). The basic premise is that in the default deployment of the iptables integration, there is a grok step in the policy with these different patterns:

`"patterns":` [

"%{IPTABLES_HOSTNAME}%{GREEDYDATA}\\[%{UBIQUITI_LABEL}\\]%{IPTABLES}%{SPACE}",

"%{IPTABLES_ACTION}%{GREEDYDATA}%{IPTABLES}%{SPACE}",

"%{IPTABLES_HOSTNAME}%{SPACE}%{UDM_LOGS}%{IPTABLES_IP_PAYLOAD}",

"%{GREEDYDATA}\\[%{UBIQUITI_LABEL}\\]%{IPTABLES}%{SPACE}",

"%{GREEDYDATA}%{IPTABLES}%{SPACE}"

]`

To make it work with the customers firewalld logs, the pattern list needs to be updated to include a new pattern for the custom firewalld format:

`"%{IPTABLES_HOSTNAME}%{SPACE}Firewalld%{SPACE}%{WORD:network.direction}%{SPACE}%{WORD:event.action}\\:%{SPACE}%{IPTABLES}"`

This means that the firewalld logs are ingested, and all the original iptables enrichments and dashboards all still work ‘out of the box’. Doing it like this though, means that every time the integration is upgraded, the ingest pipeline needs to be updated each time to include the additional pattern.

If it were possible to include a failure handler on this grok step that calls a pipeline named something like ‘logs-iptables.log@grok.custom’ (ignoring non-existent, etc just like the other @custom pipelines) then this would make it easier to apply custom grok patterns in a way that reduces maintenance. All that would need to be done, is create that pipeline and add the custom grok pattern there, so that each time the integration is updated, as with the @custom pipelines, this one wouldn’t need updating again.

This means that for any integration that uses grok, custom grok patterns can be added to suit customer specific environments/formats without having to create/re-engineer the entire functionality of the original pipeline every time new features/fixes are added to the original integration.

### Existing Features

Everytime a custom Grok pattern is added to an integration for custom log formats requires re-application every time the integration is upgraded. Every time the integration is upgraded, the ingest pipeline needs to be updated each time to include the additional pattern.

### What did you see?

Example provided in Goal above.

### Anything else?

This has been raised by an Elastician behalf of a customer.