apache / apache/iggy

connectors: fix elasticsearch_source cursor overrides runtime state on open

Open
#3,899 0 comments 0 reactions 0 assignees View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Split out of the #3795 review at hubcio's suggestion.

With `[state] enabled = true`, `elasticsearch_source` persists its own cursor at close and applies it at open, overriding the state the runtime restored. A runtime-side guarantee therefore cannot cover this source: whatever position the runtime hands back, the plugin's own file wins.

That matters for the guarantee #3795 documents. After a dropped batch the runtime's contract is "resume from the last delivered batch's state", and this source can resume from somewhere else entirely.

It is opt-in and off by default, so it is not a live regression, but the two mechanisms should not both own the cursor. Either the plugin defers to the runtime state when one is present, or the plugin-side option is documented as replacing the runtime's guarantee rather than supplementing it.

Happy to take it once the direction is settled.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the elasticsearch_source open and close cursor handling alongside the runtime state restoration described in #3795. Confirm which state is applied after a dropped batch and settle whether runtime state or the plugin cursor owns recovery when both are present. Done means the ownership rule is decided and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.