apache / apache/iggy

connectors: no send acknowledgment from runtime to source plugins — state advances and rows are deleted before delivery is confirmed

Closed
#3,635 9 comments 0 reactions 1 assignee Claimed by @rohankumardubey View on GitHub
bug connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Bug description

## Description

The connector runtime gives source plugins no way to know whether the messages they produced were actually delivered to Iggy. The runtime polls the source, sends the batch via the producer, and persists the source's saved state, but the send result is never fed back to the plugin, and offset advancement is not conditional on a successful send.

This is the source-side counterpart of #2927 (sink `consume()` return value discarded) and #2928 (offsets committed before sink processing). Same root cause: no delivery acknowledgment across the runtime/plugin boundary, just in the opposite direction.

Raised during review of the MySQL source (#3445), where it shows up twice:

1. In `delete` tracking mode, rows are deleted (or marked) in the source database as part of producing the batch before the runtime has sent anything. If the send fails, the row is gone from MySQL and never reached Iggy. Permanent data loss.
2. The tracking offset advances once the batch is handed to the runtime. A send failure doesn't roll it back, so the next poll starts past the failed batch.

`postgres_source` has the same behavior — this is not specific to the MySQL connector.

## Impact

At-least-once delivery isn't achievable for source connectors right now — same conclusion as #2928 on the sink side. For sources doing a destructive read (delete mode), this isn't "duplicates possible" anymore, it's actual data loss: the record's gone from the source DB and never landed in Iggy, and nothing gets logged about it.
Affects any polling source that tracks its own position, not just the SQL connectors.

## Related

- #2927 — sink consume() return value discarded (same gap, sink direction)
- #2928 — offsets committed before sink processing
- #2940 — replay-safe progress contract discussion
- #3445 — MySQL polling source PR where this was flagged in review

### Affected area / component

Connectors

### Deployment

None

### Versions

_No response_

### Hardware / environment

_No response_

### Sample code

_No response_

### Logs

_No response_

### Iggy server config

_No response_

### Reproduction

_No response_

### Contribution

- [ ] I'm willing to submit a pull request to fix this bug

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.