elastic / elastic/integrations
`latest` transforms should require their `unique_key` fields to exist, rather than filtering on `error.message`
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Summary
`latest` transforms build their composite aggregation with [`missing_bucket: true`](https://github.com/elastic/elasticsearch/blob/59d78137a3a0cf13fa9b062a40634ac9d87c9678/x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/transforms/latest/Latest.java#L58) on every `unique_key` source, so a document that is missing one or more key fields is **not** skipped — it lands in a bucket with null key components and produces a record in the destination index. Most of our `latest` transforms don't guard against this, and the ones that do use a filter that is simultaneously too narrow and too broad.
Proposal: `latest` transform source queries should assert their own precondition by requiring the `unique_key` fields that are genuinely mandatory to exist, and should stop using `must_not: exists error.message` as a proxy for that.
## Why this happens
From `Latest.createCompositeAggregation` in `x-pack/plugin/transform/.../transforms/latest/Latest.java`:
```java
List> sources = config.getUniqueKey()
.stream()
.map(field -> new TermsValuesSourceBuilder(field).field(field).missingBucket(true))
.collect(toList());
```
This is hardcoded and not configurable from the transform config, so the source query is the only place it can be handled. Note the contrast with `pivot` transforms, where `TermsGroupSource` defaults `missing_bucket` to `false` and documents missing a `group_by` field are dropped. The problem is specific to `latest`. (Easy to confirm on any transform with `_preview` against an index containing a document that lacks a key field.)
## Two consequences today
**1. Transforms with no source query write junk records.** Roughly 64 of 138 `latest` transforms in the repo have no `source.query` at all. Any document missing a key field becomes a destination record with null key components. The most common source of such documents is API/collection error documents, which carry `error.message` and little else — `event.dataset` and `data_stream.*` are present, so if any of those are part of the `unique_key`, the document forms a valid-looking bucket and gets written as though it were a real record. The result is a fabricated entity in a `*_latest` index, which then feeds dashboards, CDR views, and detection rules that read the destination index rather than the source data stream.
**2. Transforms that filter `error.message` drop legitimate records.** Roughly 46 `latest` transforms use:
```yaml
must_not:
- exists:
field: error.message
```
This is a blacklist standing in for a precondition, and it fails in both directions:
- Too narrow: it only catches documents that happen to carry `error.message`. A document missing a key field for any other reason still produces a null-keyed record.
- Too broad: a document where a single processor failed non-fatally — say a `convert` to IP on malformed source data — is a perfectly good record with all its key fields present, but it carries `error.message` and is therefore excluded from the pivot entirely. That entity silently disappears from the latest index and from every asset that reads it. A missing count is visible; a missing asset is not.
## Proposed change
Express the precondition positively. For example, for a transform with `unique_key: [vulnerability.id, resource.id, package.name]`:
```yaml
source:
index:
- "logs-.-*"
query:
bool:
filter:
- exists:
field: vulnerability.id
- exists:
field: resource.id
- exists:
field: package.name
must_not:
- terms:
_tier:
- data_frozen
- data_cold
```
and remove the `must_not: exists error.message` clause. Removing it matters as much as adding the `exists` filters — leaving it in place keeps the over-broad drop described above.
This is strictly stronger than filtering error documents: it excludes anything that cannot be keyed, whatever the reason, rather than only the subset we remembered to mark. It also needs no new fields, works on existing and historical data, and is independent of agent or stack version.
## This is not a mechanical find-and-replace
Each transform needs a judgement call on which keys are genuinely mandatory:
- If a legitimate record can arrive without one of its key fields — an OS-level finding with no associated package, for instance — then requiring `exists` on that field drops real data. Today `missing_bucket: true` keeps such a record with a null key component, so adding the filter is a behaviour change for that case, not just a cleanup.
- Where a key field is legitimately optional, require only the mandatory subset. It is still a large improvement, and it is also a signal that the `unique_key` may be wrong for those records — a key that can be absent can't uniquely identify anything.
- Skip `constant_keyword` fields such as `data_stream.namespace`; an `exists` on those is noise.
- Any change bumps `fleet_transform_version`, which reinstalls and restarts the transform, so destination indices get rebuilt.
## Enforcement
Unlike most conventions, this one is mechanically checkable from the transform's own config: `elastic-package` could compare the `exists` clauses in `source.query` against the declared `unique_key` and flag any key that isn't required, with an explicit annotation to opt a field out where it's legitimately optional. Worth considering as a validation rule so new transforms don't reintroduce the gap.
## Scope
Approximate counts from the repo, from a coarse grep, so treat as indicative:
- 145 transforms total, 138 of them `latest`
- 64 `latest` transforms with no `source.query` at all
- 46 using `must_not: exists error.message`
- ~16 already requiring at least one field with `exists`
## Related
#20409 covers marking API/collection error documents so that assets can exclude them. This issue is independent and can be fixed first.
For transforms, requiring the `unique_key` fields subsumes error-document exclusion entirely, because those documents have none of the keys.
For dashboards, which rule applies depends on what the panel counts, not on which index it queries:
- Panels that read a transform's destination index — filtering `labels.is_transform_source: "false"`, or the negated `labels.is_ioc_transform_source: "true"` variant used by the TI packages — need nothing at all. That field is a mapping-level `constant_keyword`, so every document in the source data stream carries `"true"` and every document in the destination carries `"false"`, regardless of what the ingest pipeline did. Those panels cannot see source-stream error documents; their correctness rests entirely on the transform's source query, which makes the fix in this issue sufficient for them.
- Panels that count entities against a source data stream — a "Total Devices" metric over `logs-*.device`, say — are better served by requiring the identifying field to exist than by excluding error documents, for the same reason as the transforms: it excludes anything that isn't an entity, not just the subset we remembered to mark.
- Panels that genuinely count events, and detection rules over event streams, are what the marker in #20409 is for.
Contributor guide
Research direction
Read x-pack/plugin/transform/.../transforms/latest/Latest.java, then inspect the repository's latest transform YAML configurations and their unique_key and source.query entries. Use _preview against data missing a key field to observe current behavior. Done means applicable mandatory keys are positively required, error.message proxy filters are removed, and fleet_transform_version changes are handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java, yaml
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100