elastic / elastic/integrations

[All]: Update all integration dynamic mappings with additional field sets based on current ECS documentation

Open
#13,245 1 comment 0 reactions 0 assignees View on GitHub
Integration:All needs:triage Stalled
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

### Integration Name

All [All]

### Dataset Name

_No response_

### Integration Version

1.18.1

### Agent Version

8.16.3

### OS Version and Architecture

RHEL 8

### User Goal

This is for all integrations as a request to update the dynamic mappings template to copy certain fields to related pre-fixed fields. This is to ensure that what is included in the integrations is also mapped to ECS fields per our ECS documentation.

### Existing Features

Several customers have brought this item to our attention and that per our ECS documentation the fields listed above are to be included in the related fields. This affects Dashboards and searches for them as they can not search the related fields as they do not exist nor contain the appropriate field values. The fields in question are as such: related.hash, related.hosts, related.ip, related.user.

Have also noticed that not all .ip fields are copied to related.ip

### What did you see?

[client.address](https://www.elastic.co/guide/en/ecs/current/ecs-client.html#field-client-address
)- Some event client addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

[destination.address](https://www.elastic.co/guide/en/ecs/current/ecs-destination.html#field-destination-address)
- Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

[server.address](https://www.elastic.co/guide/en/ecs/current/ecs-server.html#field-server-address)
- Some event server addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

[source.address](https://www.elastic.co/guide/en/ecs/current/ecs-source.html#field-source-address)
- Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field.

Then it should be duplicated to .ip or .domain, depending on which one it is.

[related fields](https://www.elastic.co/guide/en/ecs/current/ecs-related.html)
Some pieces of information can be seen in many places in an ECS event. To facilitate searching for them, store an array of all seen values to their corresponding field in related..

A concrete example is IP addresses, which can be under host, observer, source, destination, client, server, and network.forwarded_ip. If you append all IPs to related.ip, you can then search for a given IP trivially, no matter where it appeared, by querying related.ip:192.0.2.15.

In screenshot below: host.ip is copied to related.ip, but source.ip & destination.ip are not included if the value is different from related.ip (which seems to only contain host.ip)

Image

Image

.

Image

### Anything else?

Request is to simply add entries into the existing ECS dynamic mappings that are included in each Integration to reflect the related fields appropriately.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing ECS dynamic mappings included in each integration and compare them with the linked ECS documentation. Update the mappings for related.hash, related.hosts, related.ip, related.user, and the address-derived IP or domain fields, then verify the generated mappings across the affected integrations.

Written by the indexing model from the issue text.

Assessment

Tech stack
handlebars
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.