apache / apache/iggy

Create Pareto Sink Connector Test Suite

Open
#2,893 1 comment 0 reactions 1 assignee Claimed by @amuldotexe View on GitHub
connectors design test
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

We already have sink-specific integration tests, but most of them are still connector by connector.

The harder sink problems seem to be the same across connectors, and they are mostly about behavior at the sync boundary.

The same questions keep showing up:
- when do we call a write successful?
- what happens if only part of a batch gets written?
- what can be replayed, duplicated, or lost after restart?
- what makes replay safe?

We have already seen parts of this in a few places:
- #2592 and #2595 added e2e tests for Elasticsearch and Iceberg sinks
- #2815 had to make MongoDB sink failure behavior much more explicit

This makes me think it may be useful to have a small shared sink test suite on top of the current harness.

By Pareto I just mean: not a giant conformance system, just a small set of tests that catch most of the common mistakes.

A practical first version could check things like:
- partial external failure is not reported as full success
- restart behavior is clear and tested
- duplicate/loss windows are explicit
- deterministic write identity is tested where it matters

This would not replace connector-specific happy-path tests. It would just give new sink connectors a small common baseline.

## Done when

- there is a reusable sink-side test helper or suite on top of the current harness
- at least one existing sink connector is wired into it
- there is a short note showing how a new sink connector plugs into it

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.