elastic / elastic/package-spec

[Discuss] Supporting ES|QL LOOKUP JOIN on integration data

Open
#873 0 comments 0 reactions 1 assignee Claimed by @andrewkroh View on GitHub
discuss
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

ES|QL is adding a new `LOOKUP JOIN` feature (https://github.com/elastic/elasticsearch/issues/116208) that will enable joining data at query time. This feature will be useful in several integrations. For example, the `crowdstrike.fdr` integration receives two distinct types of data: endpoint events containing a host ID and host metadata events mapping the host ID to metadata about the host. To make these two types of data useful, they need to be joined.

To use the `LOOKUP JOIN` feature, packages must meet the following requirements:

1. Write data to a regular index that has `index.mode: lookup`. Enrichment data, such as documents mapping host IDs to host metadata, would be directed to this index by the agent.
2. Be able to evolve the mappings of this enrichment data index (e.g., handle scenarios where you need to change the mapping of a field).

_Note: This feature is still in development, and I haven't used it yet, so some of my assumptions here might be incorrect._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.