elastic / elastic/package-spec
[Discuss] Supporting ES|QL LOOKUP JOIN on integration data
- Dominant language
- Go
- Stars
- 20
- Forks
- 93
- Avg merge
- 15h 10m
- Merged PRs (30d)
- 12
Description
ES|QL is adding a new `LOOKUP JOIN` feature (https://github.com/elastic/elasticsearch/issues/116208) that will enable joining data at query time. This feature will be useful in several integrations. For example, the `crowdstrike.fdr` integration receives two distinct types of data: endpoint events containing a host ID and host metadata events mapping the host ID to metadata about the host. To make these two types of data useful, they need to be joined.
To use the `LOOKUP JOIN` feature, packages must meet the following requirements:
1. Write data to a regular index that has `index.mode: lookup`. Enrichment data, such as documents mapping host IDs to host metadata, would be directed to this index by the agent.
2. Be able to evolve the mappings of this enrichment data index (e.g., handle scenarios where you need to change the mapping of a field).
_Note: This feature is still in development, and I haven't used it yet, so some of my assumptions here might be incorrect._
Contributor guide
Assessment
This issue has not been assessed yet.