Suricata Integration — Multi-Data-Stream Architecture

### Integration Name

Suricata [suricata]

### Dataset Name

suricata.eve*

### Integration Version

2.27.0

### Agent Version

9.2-9.3

### OS Version and Architecture

Debian 11-13

### User Goal

Increase usability, efficiency and flexibility of the dataset.
The Suricata integration (v2.27.0) routes ALL EVE event types — alerts, protocol telemetry (dns/http/tls/ssh/flow/etc.), and engine stats — into a single data stream: logs-suricata.eve-*. This creates lifecycle, performance, and field mapping scalability problems at scale. We request the integration adopt a multi-stream design comparable to the Zeek integration.

### Existing Features

COMPARISON — SURICATA vs. ZEEK:
- Suricata: 1 data stream, ~415 defined fields in a single mapping, one ILM policy for everything
- Zeek: 43 data streams (connection, dns, http, ssl, stats, etc.), 57–164 fields each, per-stream ILM

Both integrations monitor the same traffic and produce analogous telemetry categories, yet have fundamentally different data architectures.
PROBLEM 1 — DATA LIFECYCLE:
Alerts need 1–2+ year retention (compliance, forensics). NSM telemetry (90–99% of volume) only needs 30–90 days. Stats are metrics needing different treatment entirely. A single stream forces one ILM policy for all three, meaning either: (a) retain everything at alert-length retention (5–20x storage waste), or (b) delete alerts prematurely. In MSSP environments with 50–200+ sensors, this compounds to terabytes of unnecessary storage monthly.

PROBLEM 2 — SEARCH EFFICIENCY:
Every query against logs-suricata.eve-* scans all event types. SIEM detection rules targeting alerts must scan through vastly more numerous flow/dns/http docs in the same backing indices. Zeek's architecture lets rules target logs-zeek.notice-* directly without touching logs-zeek.connection-* or logs-zeek.dns-*.

PROBLEM 3 — 1000-FIELD MAPPING LIMIT:
The integration defines ~465 fields statically. In production this grows to 600–900+ through:
- Dynamic mapping of unmapped protocol fields (quic, mqtt, pgsql, etc. — each adds 5–20 fields)
- Multi-field type expansion (.keyword/.text sub-fields doubling string field counts)
- Extended EVE logging (custom HTTP headers, JA4+ fingerprints, full TLS cert chains)
- Custom rule metadata extraction beyond the flattened field

Breaching 1000 fields causes some useful nested fields to become non-searchable (this happened with suricata.eve.flow.action for us). In a SOC context, this can have non-obvious effects on reporting and saved searches that worked suddenly stopping to be able to search required keys.

Zeek's largest stream (ssl) has 164 fields — under 17% of the limit — because the multi-stream design naturally isolates field sets.

### What did you see?

REFERENCES:
- github.com/elastic/integrations/tree/main/packages/suricata (1 data_stream: eve)
- github.com/elastic/integrations/tree/main/packages/zeek (43 data_streams)

### Anything else?

PROPOSED SOLUTION:
At minimum, three stream categories:
1. logs-suricata.alert-* — Alerts only. Long retention, lean mapping (~180 fields), SIEM-optimized.

2a. Minimally logs-suricata.nsm-* — Telemetry data stream. Distinct alert vs telemetry retention, mappings, and more efficient search.
2b. Optimally logs-suricata.-* — Per-protocol NSM telemetry (dns, http, tls, flow, etc.). Varied retention, protocol-specific mappings, targeted search.

3. metrics-suricata.stats-* — Engine stats as TSDS time-series with counter/gauge annotations and downsampling support.

CURRENT WORKAROUND:
We've implemented a @custom pipeline using the reroute processor to split into three streams. This approach should work but does not help the community and comes with additional load vs a native solution which could eliminate or distribute the load via earlier processing on the agent.

Ive attached a draft of our workaround (generated using Claude Opus 4.6 with extended thinking).

[suricata-eve-stream-splitting-es9.es.txt](https://github.com/user-attachments/files/26168167/suricata-eve-stream-splitting-es9.es.txt)

ENVIRONMENT:
MSSP operating across ECE and ECK clusters with a sizable fleet of Suricata deployments. Stack version 9.x. Willing to collaborate on design/testing.