elastic / elastic/integrations
aws.cloudtrail: Ingest pipeline & ECS mapping improvements
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 182
Description
## Summary
Improve the `cloudtrail` data stream ingest pipeline (`packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml`, ~2190 lines) so events are fully normalised and enriched, focusing on ECS `related.*` coverage and ECS version currency.
## Current state
- Pipeline already applies geoip to `source.ip` → `source.geo` and `source.as` (tags `geoip_source_ip_to_source_geo_*`, `geoip_source_ip_to_source_as_*`). Enrichment for source IP is present.
- Every processor already carries a `tag:` (99 tags) and `_ingest.on_failure_processor_tag` is surfaced in error handling (10 `on_failure` blocks). The `pipeline-tag` modernisation is already done — do **not** re-do it.
- `user_agent` parsing is present.
- ECS `related.user` (6) and `related.hash` (1) are populated, but **`related.ip` and `related.hosts` are not populated at all** (0 occurrences).
## Proposed changes
### ECS & enrichment
- [ ] Populate `related.ip` — at minimum append `source.ip` (from CloudTrail `sourceIPAddress`) and any resolved destination/target IPs.
- [ ] Populate `related.hosts` where a hostname/host identifier is available (e.g. from `user_identity`/resources or EC2 instance context).
- [ ] Confirm `related.user` covers all principal identities (root, IAM user, assumed role, federated).
- [ ] Verify `source.ip` is only geo-enriched when it is a real IP — `sourceIPAddress` is often an AWS service DNS name (e.g. `cloudtrail.amazonaws.com`); ensure those are not fed to geoip and don't produce failures.
### Robustness (verify, likely already fine)
- [ ] Spot-check the 23 `ignore_missing` usages for redundant `ignore_missing: true` + `if` guards and simplify where found.
- [ ] Confirm partial-failure paths append to `error.message` with the processor tag (already wired in 4 places) across all `on_failure` blocks.
### Exception-handling hygiene
The pipeline has **50 `ignore_failure`**, **10 `script`**, and **2 `grok`** processors. The expensive part of a failure is the exception being *thrown*; `ignore_failure: true` and `on_failure` both catch an already-thrown exception, so converting between them is not a performance win (`on_failure` is marginally heavier — it allocates a wrapper exception and writes failure metadata). Optimise by not throwing, and by matching the handler to intent:
- [ ] Audit the 50 `ignore_failure: true` usages. Where the only expected failure is a missing field, replace with `ignore_missing: true` or an `if` guard so the processor doesn't throw on the common path.
- [ ] For the `script`/`grok` processors that fail on expected input, add `if` guards (or restructure) so the common path doesn't throw. Keep `ignore_failure: true` on these as a last-resort safety net: the guard removes the frequent throw cost, while `ignore_failure` catches the rare unexpected throw so the pipeline continues instead of bubbling to the global `on_failure` handler.
- [ ] Where a failure is genuinely unexpected and a user would want to see it, prefer an `on_failure` that appends to `error.message` over silent `ignore_failure`. Do **not** attach per-doc `error.message` to `grok`/`script` whose failures are expected and frequent — that adds noise and cost.
## Rationale & impact
`related.*` fields power cross-source pivoting in Security; missing `related.ip`/`related.hosts` means CloudTrail events don't join IP/host-centric investigations. An ECS bump keeps mappings current and unlocks newly standardised fields.
## Acceptance criteria
- [ ] Pipeline tests pass and new expectations assert `related.ip`/`related.hosts` population.
- [ ] `elastic-package check` passes with no ECS mapping errors.
- [ ] Changelog entry added and `aws` package version bumped.
## References
- Skills applied from https://github.com/elastic/integration-skills: `ingest-pipelines`, `maintain-integration`, `ecs-field-mappings`.
- ECS: https://www.elastic.co/guide/en/ecs/current/index.html
- SIEM field reference: https://www.elastic.co/guide/en/security/current/siem-field-reference.html
Contributor guide
Assessment
This issue has not been assessed yet.