elastic / elastic/integrations
[All]: Update all integration dynamic mappings with additional field sets based on current ECS documentation
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
### Integration Name
All [All]
### Dataset Name
_No response_
### Integration Version
1.18.1
### Agent Version
8.16.3
### OS Version and Architecture
RHEL 8
### User Goal
This is for all integrations as a request to update the dynamic mappings template to copy certain fields to related pre-fixed fields. This is to ensure that what is included in the integrations is also mapped to ECS fields per our ECS documentation.
### Existing Features
Several customers have brought this item to our attention and that per our ECS documentation the fields listed above are to be included in the related fields. This affects Dashboards and searches for them as they can not search the related fields as they do not exist nor contain the appropriate field values. The fields in question are as such: related.hash, related.hosts, related.ip, related.user.
Have also noticed that not all .ip fields are copied to related.ip
### What did you see?
[client.address](https://www.elastic.co/guide/en/ecs/current/ecs-client.html#field-client-address
)- Some event client addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.
Then it should be duplicated to .ip or .domain, depending on which one it is.
[destination.address](https://www.elastic.co/guide/en/ecs/current/ecs-destination.html#field-destination-address)
- Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.
Then it should be duplicated to .ip or .domain, depending on which one it is.
[server.address](https://www.elastic.co/guide/en/ecs/current/ecs-server.html#field-server-address)
- Some event server addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.
Then it should be duplicated to .ip or .domain, depending on which one it is.
[source.address](https://www.elastic.co/guide/en/ecs/current/ecs-source.html#field-source-address)
- Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.
Then it should be duplicated to .ip or .domain, depending on which one it is.
[related fields](https://www.elastic.co/guide/en/ecs/current/ecs-related.html)
Some pieces of information can be seen in many places in an ECS event. To facilitate searching for them, store an array of all seen values to their corresponding field in related..
A concrete example is IP addresses, which can be under host, observer, source, destination, client, server, and network.forwarded_ip. If you append all IPs to related.ip, you can then search for a given IP trivially, no matter where it appeared, by querying related.ip:192.0.2.15.
In screenshot below: host.ip is copied to related.ip, but source.ip & destination.ip are not included if the value is different from related.ip (which seems to only contain host.ip)
.
### Anything else?
Request is to simply add entries into the existing ECS dynamic mappings that are included in each Integration to reflect the related fields appropriately.
Contributor guide
Research direction
Start by locating the existing ECS dynamic mappings included in each integration and compare them with the linked ECS documentation. Update the mappings for related.hash, related.hosts, related.ip, related.user, and the address-derived IP or domain fields, then verify the generated mappings across the affected integrations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- handlebars
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100