elastic / elastic/integrations
`ess_billing` billing stream generates duplicate records due to `total_ecu` in `_id` fingerprint
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Summary
Re-ingesting historical billing data (triggered by a cursor reset, a fresh
deploy, or a policy re-creation that changes the input id) produces duplicate
records in two patterns:
1. Same deployment + same billing window → different `_id` + different ECU value
(upstream-revised data creates a new document instead of overwriting).
2. Same `_id` → appears in multiple backing indices simultaneously (Elasticsearch
cannot de-dup across data stream backing indices on re-ingest).
## Root cause: `total_ecu` is a mutable field included in the `_id` fingerprint
The ingest pipeline uses a `fingerprint` processor to generate `_id`:
```yaml
# data_stream/billing/elasticsearch/ingest_pipeline/default.yml
- fingerprint:
fields:
- ess.billing.deployment_id
- ess.billing.from
- ess.billing.to
- ess.billing.sku
- ess.billing.total_ecu # <-- mutable: revised by the API after initial publish
target_field: _id
```
The ESS billing API revises total_ecu after initial publish (finalization).
When historical data is re-fetched, even for the exact same
deployment_id + from + to + sku window, a revised ECU value produces a
different hash → a new document is written rather than overwriting the original.
Root cause: large lookbehind amplifies the problem on re-ingest
When the CEL cursor is absent (state reset, fresh deploy, or policy re-creation
with a new input id), lookbehind controls how far back data is fetched.
The default is 365 days; users can set it higher. This means a single
cursor-reset event re-ingests a large volume of history in one burst.
On re-ingest, documents already written live in older (possibly frozen/snapshot)
backing indices. New copies land in the current write index, so both are
queryable. Elasticsearch only de-dups by _id within a single index, not
across backing indices of a data stream.
Proposed fix
Remove total_ecu from the fingerprint. The natural identity of a billing
line item is deployment_id + from + to + sku — ECU is a measurement that
can legitimately change. With total_ecu excluded, a re-fetched record with
a revised ECU will produce the same _id and overwrite the stale document
rather than creating a duplicate.
- fingerprint:
fields:
- ess.billing.deployment_id
- ess.billing.from
- ess.billing.to
- ess.billing.sku
# total_ecu removed — mutable field, changes on API finalization
target_field: _id
Note: records already written with the old fingerprint will not be
retroactively de-duped. Those would require a one-time reindex or manual
cleanup. This fix prevents the problem from recurring going forward.
Contributor guide
Assessment
This issue has not been assessed yet.