elastic / elastic/integrations
[auditd] SYSCALL records from the file input always classify as process/info; success= is never mapped to event.outcome
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## Summary
The `auditd.log` ingest pipeline has a full syscall-to-ECS mapping table (`unlink` -> `event.action: deleted`, `event.category: file`, `event.type: deletion`, and so on), but it is keyed by syscall **name**. The default file input delivers `syscall=`, and the pipeline has no number-to-name resolution, so every `SYSCALL` record falls into the `*` bucket and gets `event.action: syscall`, `event.category: process`, `event.type: info`. When `log_format = ENRICHED` is used, auditd appends the resolved name as `SYSCALL=unlink`, which the pipeline parses into `auditd.log.SYSCALL` but never uses.
Separately, the `success=yes|no` key on `SYSCALL` records is parsed into `auditd.log.success` (boolean) but is never mapped to `event.outcome`. Only `res=` and `result=` are.
The package's own test fixtures encode this behaviour as expected output, so no test fails today.
## Integration
- Package: `auditd`, data stream `auditd.log`
- Version: 3.24.2 (unchanged since at least 3.23.2; the mapping table dates from the 2021 Beats sync in `250e5ce158`)
## Reproduction
`_simulate` against `logs-auditd.log-3.24.2` with an enriched `unlink` record (`\u001d` is the JSON escape for the Unit Separator that auditd places between the raw and enriched parts of an ENRICHED line; `_simulate` accepts it):
```json
POST _ingest/pipeline/logs-auditd.log-3.24.2/_simulate
{
"docs": [
{
"_source": {
"message": "type=SYSCALL msg=audit(1700000000.000:1001): arch=c000003e syscall=87 success=yes exit=0 a0=7ffd items=2 ppid=1234 pid=5678 auid=1000 uid=0 gid=0 comm=\"rm\" exe=\"/usr/bin/rm\" key=\"delete\"\u001dARCH=x86_64 SYSCALL=unlink AUID=\"someuser\" UID=\"root\""
}
}
]
}
```
Actual:
```json
"event": {
"action": "syscall",
"category": ["process"],
"type": ["info"],
"kind": "event"
}
```
Expected (what the pipeline's own `unlink` entry would produce if the name were used):
```json
"event": {
"action": "deleted",
"category": ["file"],
"type": ["deletion"],
"outcome": "success",
"kind": "event"
}
```
The existing enriched fixture shows the same thing: `test-auditd-enriched.log` line 4 is a `syscall=87 ... SYSCALL=unlink ... success=yes` record, and its expected output is `action: syscall`, `category: process`, `type: info`, no `outcome`. The pre-parsed fixture (`test-auditd-preparsed-syscall.log`, agent-side parser, `syscall: execve` already a name) gets `event.action: executed`, which shows the table works as soon as a name is present.
## Root cause
All links pinned to `main` at `dec7c788`.
1. **Lookup is by name only.** The `script_ecs_mapping` processor looks up `params.syscalls.get(base.syscall)` and falls back to `params.syscalls.get('*')` ([[default.yml L2004-L2010](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L2004-L2010)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L2004-L2010)). `params.syscalls` keys are names (`accept`, `unlink`, `execve`, ...; `unlink` at [[L981-L988](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L981-L988)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L981-L988)); `*` is `process`/`info` ([[L220-L225](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L220-L225)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L220-L225)). The raw file input always yields a number in `auditd.log.syscall`.
2. **Enriched name is parsed but ignored.** `auditd.log.SYSCALL` is a declared field ([[fields.yml L270-L271](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/fields/fields.yml#L270-L271)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/fields/fields.yml#L270-L271)) but nothing in the pipeline reads it. Other enriched keys (`AUID`, `UID`, `OUID`, ...) were wired up in 3.23.1/3.23.2 (#20237, #20238); `SYSCALL` and `ARCH` were not.
3. **`success` is not an outcome source.** `success` is converted to a boolean by the kv script (`possibleBooleanKeys`, [[L174-L176](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L174-L176)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L174-L176)), but `event.outcome` is derived only from `res=` / `result=` / AVC ([[L2320-L2358](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L2320-L2358)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L2320-L2358)). `SYSCALL` records carry neither `res` nor `result`.
4. **Existing numeric checks are arch-unsafe.** The two `network.direction` processors compare `auditd.log.syscall` against x86_64 numbers (`43`, `45`, `47`, `288`, `42`, `44`, `46`) ([[L2100-L2116](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L2100-L2116)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L2100-L2116)). Those numbers mean different syscalls on aarch64 and i386, so they are wrong on non-x86_64 hosts.
## Impact
Every `SYSCALL` event collected through the default file input is classified as generic `process`/`info` with no `event.outcome`, regardless of the syscall. File deletes, renames, chmods, network connects, and module loads are indistinguishable in `event.*`. Detection rules and dashboards keyed on `event.category: file` or `event.type: deletion` never match `auditd.log` data. Users on `log_format = ENRICHED` (the upstream audit-userspace default in [`[init.d/auditd.conf](https://github.com/linux-audit/audit-userspace/blob/master/init.d/auditd.conf)`](https://github.com/linux-audit/audit-userspace/blob/master/init.d/auditd.conf), shipped by RHEL 8+ and derivatives) already send the information needed to fix this on every line.
## Recommended solution
Two small pipeline changes, no new tables. Both are independent of the coalescing work in #20643 and remain useful after it lands because the non-coalescing path stays the default for a while.
### 1. Use the enriched syscall name for the lookup
In `script_ecs_mapping`, resolve a name before the lookup:
```painless
String syscallName = null;
if (base.SYSCALL instanceof String) {
syscallName = base.SYSCALL; // enriched name, e.g. "unlink"
} else if (base.syscall instanceof String && !(base.syscall ==~ /^[0-9]+$/)) {
syscallName = base.syscall; // already a name (agent-side parser path)
}
def acts = params.types.get(base.record_type);
if (acts == null && (syscallName != null || base.syscall != null)) {
acts = syscallName != null ? params.syscalls.get(syscallName) : null;
if (acts == null) acts = params.syscalls.get('*');
}
```
Optionally normalise `auditd.log.syscall` to the name when `auditd.log.SYSCALL` is present. That matches what the agent-side parser (`use_auditd_parser`) and `auditd_manager` (`auditd.data.syscall`) already produce, so queries behave the same across the three paths. Keep the number in `auditd.log.SYSCALL`'s absence; do not invent it. This is a field-value change for enriched users and needs a changelog note; if the team prefers not to change the value, keep the lookup-only change.
Apply the same resolved name to the two `network.direction` processors so they stop depending on x86_64 numbers.
### 2. Map `success` to `event.outcome`
After the existing `res` / `result` processors, add:
```yaml
- set:
field: event.outcome
value: success
if: ctx.event?.outcome == null && ctx.auditd?.log?.success == true
- set:
field: event.outcome
value: failure
if: ctx.event?.outcome == null && ctx.auditd?.log?.success == false
```
`auditd.log.success` is already a boolean at this point, so no string handling is needed. Guarding on `event.outcome == null` keeps `res`/`result`/AVC precedence unchanged.
### 3. Not recommended now: a number-to-name table in the pipeline
A full table would have to be keyed by `arch` as well as number (`unlink` is 87 on x86_64, absent on aarch64 which only has `unlinkat`; the i386 numbers differ again). go-libaudit already ships these per-arch tables and the package exposes them through `use_auditd_parser` (agent-side parsing, Agent >= 9.5.0, [[log.yml.hbs L20-L25](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/agent/stream/log.yml.hbs#L20-L25)](https://github.com/elastic/integrations/blob/dec7c78860faeffd331a39a4dc9d515fe0050b66/packages/auditd/data_stream/log/agent/stream/log.yml.hbs#L20-L25)). Duplicating them in Painless adds a large maintenance surface for users who are not on ENRICHED and cannot upgrade the agent. Document that combination as "syscall classification requires `log_format = ENRICHED` or `use_auditd_parser`" instead.
### Tests
- Regenerate expected output for `test-auditd-enriched.log` and `test-auditd-raw.log` (`elastic-package test pipeline -g`). The enriched fixture line 4 should change to `deleted` / `file` / `deletion` / `success`.
- Add an enriched `SYSCALL` line with `success=no exit=-2` to cover `event.outcome: failure`.
- Add a raw (non-enriched) `SYSCALL` line to pin the unchanged `*` fallback, so a future table change is deliberate.
- Add an aarch64 enriched line (`arch=c00000b7 ... ARCH=aarch64 SYSCALL=connect`) to cover the `network.direction` change.
### Changelog
`bugfix`: "Use the enriched `SYSCALL=` name to classify SYSCALL records and map `success=` to `event.outcome`. Previously all SYSCALL records from the file input were classified as `process`/`info` with no outcome."
## Workaround until fixed
A `logs-auditd.log@custom` pipeline can apply the two `set` processors from step 2 verbatim. Step 1 cannot be replicated in a custom pipeline without copying the mapping table, because Fleet appends the `@custom` call to the end of the integration pipeline, after `script_ecs_mapping` has already written the `*` values.
Contributor guide
Research direction
Start in packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml, especially script_ecs_mapping and the existing outcome processors. Review test-auditd-enriched.log and test-auditd-raw.log, then run `elastic-package test pipeline -g`; done means enriched syscall names produce the mapped event fields, success produces event.outcome, and the new failure, raw, and aarch64 cases are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, linux
- Domain
- observability-sre, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100