elastic / elastic/integrations

ess_billing billing stream stalls permanently after agent runs continuously past req.to > now boundary

Open
#19,548 1 comment 1 reaction 0 assignees View on GitHub
bug Integration:ess_billing Team:Obs-InfraObs
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Summary

The `ess_billing` billing data stream stops collecting data and never self-recovers
while the agent process stays running. The integration appears healthy (no errors,
CEL executions continue on schedule) but makes 0 HTTP requests and publishes
0 events indefinitely. A process restart resolves it.

## Steps to reproduce

1. Deploy `ess_billing` with `interval: 24h`.
2. Let the agent run continuously until the backfill completes and the cursor
reaches yesterday.
3. On the next execution the CEL program computes `from = yesterday 00:00,
to = today 00:00`. Depending on wall-clock time, `to` may equal or exceed
`now`, which triggers the short-circuit branch (`req.to > now`).
4. That branch returns `events: [], cursor: { last_to: req.to }, want_more: false`.
Filebeat only persists the cursor when events are ACKed — so the in-memory
cursor advances to `req.to` (a future/edge-of-now value) but **the on-disk
cursor is not updated**.
5. On every subsequent execution: `from` recomputes from the in-memory cursor
(future), `to = from + 24h` (further future), `req.to > now` is still true,
0 HTTP requests are made, cursor advances in memory only.
6. The gap between `req.to` and `now` stays roughly constant and positive —
the condition is self-perpetuating and never heals while the process runs.

## Observed symptoms

- CEL input executes on schedule but makes 0 HTTP requests and publishes 0 events.
- No errors in agent logs — the integration appears healthy.
- On-disk cursor is frozen at the last date for which real events were ACKed.
- A process restart restores collection (reloads on-disk cursor = past date →
`req.to < now` → real API call is made).

## Root cause

The `req.to > now` guard in `cel.yml.hbs` is correct as a stop condition, but
returning a future cursor value in that branch causes Filebeat's ACK-gated
cursor persistence to permanently desync in-memory vs on-disk state. The
in-memory future cursor keeps the input parked indefinitely.

Relevant branch in
[`data_stream/billing/agent/stream/cel.yml.hbs`](https://github.com/elastic/integrations/blob/main/packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs):

```cel
(req.to > now) ?
{
"events": [],
"cursor": { "last_to": req.to }, // advances in-memory cursor to future
"want_more": false,
}

The same pattern exists in both the add_tags and !add_tags branches.

Proposed fix

Do not advance the cursor when the window has not yet elapsed. Return the
current last_to unchanged so the on-disk cursor stays at the last
successfully fetched day:

(req.to > now) ?
{
"events": [],
"cursor": { "last_to": req.from }, // keep cursor where it is
"want_more": false,
}

The next execution (24h later) will then recompute a window that has actually
elapsed and issue a real API request.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.