connectors: fix elasticsearch_source cursor overrides runtime state on open
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
Split out of the #3795 review at hubcio's suggestion.
With `[state] enabled = true`, `elasticsearch_source` persists its own cursor at close and applies it at open, overriding the state the runtime restored. A runtime-side guarantee therefore cannot cover this source: whatever position the runtime hands back, the plugin's own file wins.
That matters for the guarantee #3795 documents. After a dropped batch the runtime's contract is "resume from the last delivered batch's state", and this source can resume from somewhere else entirely.
It is opt-in and off by default, so it is not a live regression, but the two mechanisms should not both own the cursor. Either the plugin defers to the runtime state when one is present, or the plugin-side option is documented as replacing the runtime's guarantee rather than supplementing it.
Happy to take it once the direction is settled.
Contributor guide
Research direction
Start by tracing the elasticsearch_source open and close cursor handling alongside the runtime state restoration described in #3795. Confirm which state is applied after a dropped batch and settle whether runtime state or the plugin cursor owns recovery when both are present. Done means the ownership rule is decided and covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100