elastic / elastic/integrations
[ti_*] Adopt ecs@mappings for transform destination indices and remove redundant ECS field definitions
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 182
Description
## Summary
All `ti_*` packages that ship `latest` transforms currently declare ECS fields explicitly in `elasticsearch/transform/*/fields/ecs.yml`. These definitions are now redundant on supported stacks because Fleet composes the `ecs@mappings` component template into every transform destination index template ([elastic/kibana#223878](https://github.com/elastic/kibana/pull/223878), shipped in **8.19.0** and **9.1.0**).
Where a package's ECS definitions are incomplete, the destination index falls back to the catch-all `strings_as_keyword` dynamic template and maps ECS fields with the wrong type. The most visible failure is `host.ip` being mapped as `keyword` instead of `ip`, which breaks ES|QL detection rules that reference `host.ip` across indices (cross-index type unification fails with `verification_exception`).
This was hit in production for `ti_anyrun`: [#18437](https://github.com/elastic/integrations/issues/18437). It was hot-fixed narrowly in [#18616](https://github.com/elastic/integrations/pull/18616) by adding `host.ip: {external: ecs}` back, but that only patches one field. Every other ECS-typed field in every `ti_*` transform destination is exposed to the same class of bug.
This issue tracks moving all `ti_*` packages onto `ecs@mappings` and removing the per-package ECS definitions, following the pattern already applied to the CDR packages in [#17552](https://github.com/elastic/integrations/pull/17552).
## Background
- **Root cause:** transform destination index templates use `dynamic: true` + a catch-all `strings_as_keyword` dynamic template. Historically they did **not** include `ecs@mappings`, so ECS path-based type rules (`*.ip` → `ip`, etc.) were not applied. Any ECS field not explicitly declared in the package's transform `fields/` falls through to `keyword`.
- **Platform fix:** [elastic/kibana#223878](https://github.com/elastic/kibana/pull/223878) adds `STACK_COMPONENT_TEMPLATE_ECS_MAPPINGS` to `composed_of` for all Fleet transform destination index templates. `ecs@mappings` orders `ecs_ip` (matches `ip`, `*.ip`, `*_ip`) **before** the `all_strings_to_keywords` catch-all, and dynamic templates are first-match-wins — so `host.ip` and friends are mapped correctly with no per-package work. Confirmed still present in `kibana` `main` (`x-pack/platform/plugins/shared/fleet/server/services/epm/elasticsearch/transform/install.ts`).
- **Version gate:** the transform `ecs@mappings` change shipped in **8.19.0** and **9.1.0** only. It is **not** in 9.0.x. So a package may only delete its transform ECS definitions once its Kibana constraint guarantees ≥ 8.19.0 / ≥ 9.1.0.
## Required changes per package
For each `ti_*` package with `latest` transforms:
1. **Bump the Kibana constraint to `^8.19.0 || ^9.1.0`** (or stricter) if it isn't already. Constraints that still allow 8.18.x, 8.16.x, 8.17.x, or **any 9.0.x** (including `~9.0.7`) must be raised — those stacks lack the transform `ecs@mappings` fix. Pre-8.19 / pre-9.1 stacks are out of maintenance anyway.
2. **Delete the redundant transform ECS field files**: `elasticsearch/transform/*/fields/ecs.yml` where they only restate standard ECS field types.
3. **Preserve genuine overrides** in `ecs-overridden.yml` (e.g. `constant_keyword` types like `event.module` / `event.dataset`, or any non-ECS-standard typing). Do **not** delete these — only the standard ECS restatements. Keep `external: ecs` entries that exist purely for documentation if the package relies on them.
4. **Revert the `ti_anyrun` hot-fix**: remove the explicit `host.ip: {external: ecs}` added in [#18616](https://github.com/elastic/integrations/pull/18616) as part of the same cleanup, since `ecs@mappings` now covers it.
5. **Bump the package version** and add a `changelog.yml` entry.
6. **Rebuild and validate** with `elastic-package` (transforms install, system tests pass). Existing destination indices are not remapped automatically — note in the changelog / docs that affected users must reinstall the integration (recreate the transform / reindex) for the corrected mapping to apply.
## Per-package task list
Target constraint: `^8.19.0 || ^9.1.0`.
### A. Needs Kibana bump **and** ECS deletion (constraint currently allows pre-8.19 / 9.0.x)
- [ ] `ti_anyrun` — `^8.19.2 || ^9.0.5` → bump; delete `latest_ioc/fields/ecs.yml`; revert #18616 `host.ip`
- [ ] `ti_anomali` — `^8.18.0 || ^9.0.0`
- [ ] `ti_cif3` — `^8.19.2 || ^9.0.5`
- [ ] `ti_crowdstrike` — `^8.18.0 || ^9.0.0` (two transforms: `latest_intel`, `latest_ioc`)
- [ ] `ti_cybersixgill` — `^8.19.2 || ^9.0.5`
- [ ] `ti_cyware_intel_exchange` — `^8.18.0 || ^9.0.0`
- [ ] `ti_domaintools` — `^8.19.2 || ^9.0.5` (6 transforms)
- [ ] `ti_eclecticiq` — `^8.19.2 || ^9.0.5`
- [ ] `ti_flashpoint` — `^8.18.0 || ^9.0.0`
- [ ] `ti_google_threat_intelligence` — `^8.16.0 || ^9.0.0` (10 transforms)
- [ ] `ti_greynoise` — `^8.17.0 || ^9.0.0` (2 transforms)
- [ ] `ti_opencti` — `^8.19.2 || ^9.0.5`
- [ ] `ti_recordedfuture` — `^8.18.0 || ^9.0.0`
- [ ] `ti_ticura` — `^8.19.0 || ^9.0.0` (9.0 branch needs raising to 9.1)
### B. Has `~9.0.7` / `^9.0.x` branch — still needs the 9.0 branch dropped + ECS deletion
- [ ] `ti_eset` — `^8.19.4 || ~9.0.7 || ^9.1.4` (7 transforms) → drop `~9.0.7`
- [ ] `ti_misp` — `^8.19.4 || ~9.0.7 || ^9.1.4` → drop `~9.0.7`
- [ ] `ti_rapid7_threat_command` — `^8.19.4 || ~9.0.7 || ^9.1.4` (3 transforms) → drop `~9.0.7`
- [ ] `ti_threatq` — `^8.19.4 || ~9.0.7 || ^9.1.4` → drop `~9.0.7`
### C. Constraint already safe — ECS deletion only
- [ ] `ti_abusech` — `^8.19.0 || ^9.1.0` (5 transforms)
- [ ] `ti_threatconnect` — `^8.19.2 || ^9.1.2`
- [ ] `ti_custom` — `^8.19.16 || ^9.3.5`
- [ ] `ti_otx` — `^8.19.16 || ^9.3.5` (no transform `ecs.yml` found — verify)
### D. No transform `ecs.yml` found (bump only if needed, for consistency)
- [ ] `ti_maltiverse` — `^8.19.2 || ^9.0.5`
- [ ] `ti_socradar_taxii` — `^8.18.0 || ^9.0.0`
- [ ] `ti_strider` — `^8.17.0 || ^9.0.0`
## Notes / caveats
- **`ecs@mappings` catch-all is intentional.** It also maps unmatched strings to `keyword`; the CDR side now relies on this ([elastic/kibana#254742](https://github.com/elastic/kibana/pull/254742)), so this is stable, established behaviour.
## References
- Bug report: [#18437](https://github.com/elastic/integrations/issues/18437)
- Narrow hot-fix: [#18616](https://github.com/elastic/integrations/pull/18616)
- CDR precedent: [#17552](https://github.com/elastic/integrations/pull/17552)
- Platform enablement: [elastic/kibana#223878](https://github.com/elastic/kibana/pull/223878) (backport [#224546](https://github.com/elastic/kibana/pull/224546))
- Tracking discussion: [elastic/elastic-package#1641](https://github.com/elastic/elastic-package/issues/1641)
Contributor guide
Assessment
This issue has not been assessed yet.