elastic / elastic/integrations
[auditd] Field schema normalization: align auditd and auditd_manager namespaces
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Problem
The `auditd` (Auditd Logs) and `auditd_manager` (Auditd Manager) integrations produce incompatible field namespaces:
- `auditd` → `auditd.log.*` (grok/regex ingest pipeline, ~100 fields)
- `auditd_manager` → `auditd.data.*` (go-libaudit agent-side parsing, 300+ fields)
Because field names and values differ between the two, detection rules cannot target both integrations without duplicating every field condition. In practice, the detection-rules team has standardized entirely on `auditd_manager`: approximately 160 rules reference this integration, zero reference `auditd` directly. Customers who cannot install `auditd_manager` (managed environments, restricted installs) get no rule coverage.
## Current state (as of May 2026)
Phase 1 work is underway:
- **elastic/beats#50791**: Adds an `auditd` filestream parser to `libbeat/reader/auditd` using `go-libaudit`/auparse. Replaces the 2,371-line ingest pipeline (4 groks + Painless) with a single auparse call per line. Currently outputs `auditd.log.*` fields.
- **elastic/integrations#19071**: Adds a `use_filebeat_parser` toggle (default `false`) to the auditd data stream, guarding grok processors for pre-parsed events. Beats PR must merge first.
The `false` default ensures no breakage for users on older Filebeat versions that don't have the new parser. Users opt in, validate, then the default flips.
## Goal
Once Phase 1 is stable, evaluate normalizing the `auditd` integration's field output to align with the `auditd_manager` namespace (`auditd.data.*`), so that detection rules can target both integrations with a single field reference.
## Work needed
- [x] Phase 1: Land beats#50791 + integrations#19071 and validate in production
- [ ] Phase 2: Map auparse output fields to `auditd.data.*` namespace (or agree on a shared ECS-aligned namespace)
- [ ] Phase 2: Define deprecation/migration path for existing `auditd.log.*` users (custom rules, dashboards)
- [ ] Phase 2: Update integration manifest with minimum Filebeat version requirement
- [ ] Cross-team: Coordinate with detection-rules team to review ~160 rules once schema is normalized; classify each as auditd-only, auditd_manager-only, or both
## Stakeholders
- **elastic/sec-linux-platform**: owns both integration packages (CODEOWNERS), driving Phase 1 and 2
- **Detection-rules team**: consumer; ~160 rules need review post-normalization
## Related
- elastic/beats#50791
- elastic/integrations#19071
- elastic/beats#50805 (companion docs cleanup: removes misleading `--hostfs` flag documentation from auditbeat as users migrate to the filestream parser)
- elastic/beats#47090 (original go-libaudit tracking issue)
- elastic/integrations#10333 (kv parsing bug, superseded by this work)
Contributor guide
Assessment
This issue has not been assessed yet.