elastic / elastic/integrations

`ess_billing` billing stream generates duplicate records due to `total_ecu` in `_id` fingerprint

Open
#19,549 1 comment 2 reactions 0 assignees View on GitHub
bug Integration:ess_billing Team:Obs-InfraObs
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Summary

Re-ingesting historical billing data (triggered by a cursor reset, a fresh
deploy, or a policy re-creation that changes the input id) produces duplicate
records in two patterns:

1. Same deployment + same billing window → different `_id` + different ECU value
(upstream-revised data creates a new document instead of overwriting).
2. Same `_id` → appears in multiple backing indices simultaneously (Elasticsearch
cannot de-dup across data stream backing indices on re-ingest).

## Root cause: `total_ecu` is a mutable field included in the `_id` fingerprint

The ingest pipeline uses a `fingerprint` processor to generate `_id`:

```yaml
# data_stream/billing/elasticsearch/ingest_pipeline/default.yml
- fingerprint:
fields:
- ess.billing.deployment_id
- ess.billing.from
- ess.billing.to
- ess.billing.sku
- ess.billing.total_ecu # <-- mutable: revised by the API after initial publish
target_field: _id
```

The ESS billing API revises total_ecu after initial publish (finalization).
When historical data is re-fetched, even for the exact same
deployment_id + from + to + sku window, a revised ECU value produces a
different hash → a new document is written rather than overwriting the original.

Root cause: large lookbehind amplifies the problem on re-ingest

When the CEL cursor is absent (state reset, fresh deploy, or policy re-creation
with a new input id), lookbehind controls how far back data is fetched.
The default is 365 days; users can set it higher. This means a single
cursor-reset event re-ingests a large volume of history in one burst.

On re-ingest, documents already written live in older (possibly frozen/snapshot)
backing indices. New copies land in the current write index, so both are
queryable. Elasticsearch only de-dups by _id within a single index, not
across backing indices of a data stream.

Proposed fix

Remove total_ecu from the fingerprint. The natural identity of a billing
line item is deployment_id + from + to + sku — ECU is a measurement that
can legitimately change. With total_ecu excluded, a re-fetched record with
a revised ECU will produce the same _id and overwrite the stale document
rather than creating a duplicate.

- fingerprint:
fields:
- ess.billing.deployment_id
- ess.billing.from
- ess.billing.to
- ess.billing.sku
# total_ecu removed — mutable field, changes on API finalization
target_field: _id

Note: records already written with the old fingerprint will not be
retroactively de-duped. Those would require a one-time reindex or manual
cleanup. This fix prevents the problem from recurring going forward.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.