[Feature Request] Add RE2-based regexp processor for field extraction in Beats

Nobody has claimed this yet.

### Description

I would like to request a generic regexp processor for Beats that can extract named capture groups from a string field and write them into event fields.

Filebeat already supports regular expressions in several places such as multiline, include/exclude filtering, and other matching operations. However, there is currently no generic processor that can use regexp named capture groups to extract structured fields from log messages.

The requested functionality would be similar in purpose to the existing `dissect` processor, but would support variable or optional log formats that cannot be reliably parsed using fixed delimiters.

---

### Use case

We are migrating log collection and parsing from ArcSight FlexAgent to Elastic Agent / Filebeat.

Some of our log formats are simple enough to parse using `dissect`, but others contain optional fields, variable sections, or different message structures.

For these cases, we currently have to perform the parsing in an Elasticsearch ingest pipeline, for example with Grok.

This means the parsing workload is concentrated on Elasticsearch ingest nodes.

We would like to move this deterministic parsing closer to the source and perform it on the Elastic Agent / Filebeat side instead.

This would allow the parsing CPU workload to be distributed across Elastic Agents rather than concentrated on a smaller number of Elasticsearch ingest nodes.

---

### Proposed processor

For example:

```yaml
processors:
- regexp:
field: message
pattern: '^ResourceAudit:(?P[^:]+):(?P[^:]+):(?P.*)$'
target_prefix: arcsight
ignore_failure: true
overwrite_keys: false

Input:

ResourceAudit:allen:10.10.10.1:login

Expected result:

arcsight.user_name: allen
arcsight.source_ip: 10.10.10.1
arcsight.action: login

The processor could use named capture groups from the regexp pattern and write each named capture into the event.

Possible configuration options

A possible configuration could look like this:

processors:
- regexp:
field: message
pattern: ''
target_prefix: ''
ignore_missing: false
ignore_failure: false
overwrite_keys: false

Suggested behavior:

field
Source field to parse.
Default could be message.
pattern
RE2-compatible regular expression.
Named capture groups would become event fields.
target_prefix
Prefix for extracted fields.
ignore_missing
Ignore the event if the source field does not exist.
ignore_failure
Keep the event unchanged if the regexp does not match.
overwrite_keys
Control whether extracted fields are allowed to overwrite existing fields.

Unnamed capture groups could simply be ignored.

Why RE2 / Go regexp instead of Grok

This request is intentionally narrower than adding full Grok support to Beats.

I am not requesting a Grok pattern library such as:

%{IP:source.ip}
%{WORD:user.name}
%{GREEDYDATA:message}

Instead, the proposal is to expose a lightweight field extraction processor based on Go's existing regexp implementation and named capture groups.

For example:

^(?P[^:]+):(?P[^:]+):(?P.*)$

This would keep the implementation relatively small and avoid introducing a separate Grok engine or pattern library.

Go's regexp package already provides APIs such as named subexpressions and submatch extraction, and Beats already uses regular expressions internally.

Why dissect is not always sufficient

The existing dissect processor works very well for fixed-format logs.

For example:

user:ip:action

can easily be parsed with a delimiter-based pattern.

However, some real-world security logs have:

optional fields
variable numbers of fields
different message formats
optional prefixes or suffixes
conditional sections
fields whose boundaries cannot be determined only by delimiters

For these formats, regexp extraction is significantly more flexible.

The desired processing model would therefore be:

simple fixed format
|
v
dissect

variable format
|
v
regexp / RE2

very complex parsing
|
v
Elasticsearch ingest pipeline / Grok

This would provide a useful middle ground between dissect and Elasticsearch ingest-node Grok.

Performance motivation

One of the main motivations is distributed parsing.

Current architecture:

Elastic Agents
|
v
Elasticsearch ingest nodes
|
+-- Grok / regexp parsing
+-- Grok / regexp parsing
+-- Grok / regexp parsing

The parsing CPU cost is concentrated on the Elasticsearch ingest tier.

With an Agent-side regexp processor:

Agent 1 -- regexp parsing --\
Agent 2 -- regexp parsing ----> Elasticsearch
Agent 3 -- regexp parsing --/

the workload can be distributed across the endpoints that are already collecting the logs.

For large SIEM environments with many log sources, this could reduce parsing pressure on Elasticsearch ingest nodes.

Relationship to existing functionality

Beats already has several related capabilities:

regexp-based matching in Filebeat
multiline regexp matching
include/exclude regexp filtering
dissect processor for field extraction
existing processors that already use Go regular expressions internally

The missing capability is a generic processor that combines these concepts:

regexp match
+
named capture groups
+
event field creation
Related historical request

There was also an older request for Filebeat-side Grok parsing:

[https://github.com/elastic/beats/issues/679](https://github.com/elastic/beats/issues/679?utm_source=chatgpt.com)

This proposal is intentionally smaller in scope.

Instead of implementing full Grok functionality, this request proposes a lightweight RE2/Go-regexp based named capture extraction processor.

Summary

The requested feature is essentially:

string field
|
v
RE2-compatible regexp
|
v
named capture groups
|
v
Beat event fields

I believe this would be useful for Elastic Agent / Filebeat users who want to perform lightweight structured parsing at the collection layer without moving all regexp-based parsing to Elasticsearch ingest nodes.

Thank you for considering this enhancement.

Start by reviewing the existing dissect processor, related Beats regexp-based matching, and Go's regexp named-subexpression APIs. Define the processor configuration and behavior for field extraction, missing fields, failed matches, prefixes, and overwrites; done means named captures are written to event fields with focused processor tests.

Written by the indexing model from the issue text.