apache / apache/iceberg

Multi-topic → multi-table routing writes all records to all tables when schemas are identical

Open
#15,584 1 comment 0 reactions 0 assignees View on GitHub
bug stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.6.1

### Query engine

Kafka Connect

### Please describe the bug 🐞

## Summary

When configuring a single Iceberg Kafka Connect Sink connector to route **multiple topics to multiple tables** (without `iceberg.tables.route-field`), records from all topics get written to **all** tables if the target tables share the same `id-columns` name and nearly identical schemas.

---

## Environment

| | |
|---|---|
| Connector | Iceberg Sink Connector (Tabular) |
| Catalog | AWS Glue |
| Storage | Amazon S3 |
| Source | Debezium MySQL CDC (Amazon MSK) |
| `tasks.max` | `1` |
| `iceberg.tables.route-field` | not set |

---

## Steps to Reproduce

### ✅ Works — different `id-columns` & different schemas

```json
"topics": "db.schema_a.table_a,db.schema_a.table_b",
"iceberg.tables": "catalog.table_a,catalog.table_b",
"iceberg.table.catalog.table_a.id-columns": "fund_balance_id",
"iceberg.table.catalog.table_b.id-columns": "loan_application_id"
```

→ Each topic's records are correctly routed to their respective Iceberg tables. ✅

---

### ❌ Fails — same `id-columns` name & nearly identical schemas

```json
"topics": "db.schema_b.table_c,db.schema_b.table_d",
"iceberg.tables": "catalog.table_c,catalog.table_d",
"iceberg.table.catalog.table_c.id-columns": "id",
"iceberg.table.catalog.table_d.id-columns": "id"
```

→ All records from **both topics** get written to **both tables**. ❌

---

## Expected Behavior

Records from `db.schema_b.table_c` should be written **only** to `catalog.table_c`, and records from `db.schema_b.table_d` should be written **only** to `catalog.table_d` — based on topic-to-table name matching (last segment).

## Actual Behavior

- Both tables end up with **identical row counts**
- Records from the wrong topic appear in each table with **NULL columns**
- The connector appears to broadcast records to all registered tables instead of routing by topic

---

## Analysis

The two cases differ in the following ways:

- **`id-columns`**: different per table (`fund_balance_id` / `loan_application_id`) vs. same (`id` / `id`)
- **Table schemas**: completely different structures vs. nearly identical (both have `id`, `modified_datetime`, etc.)
- **Topic suffix → table name matching**: looks correct in both cases
- **`iceberg.control.topic`**: unique per connector in both cases

**Hypothesis:** The routing logic may use schema-based matching (e.g. field name overlap) as a fallback or in conjunction with topic name matching. When two tables share the same `id-columns` name and overlapping field names, the connector may fail to discriminate records by topic and instead broadcasts to all candidate tables.

---

## Workaround

Splitting into **1 connector per topic/table** resolves the issue immediately:

```json
// Connector A
"topics": "db.schema_b.table_c",
"iceberg.tables": "catalog.table_c",
"iceberg.table.catalog.table_c.id-columns": "id"

// Connector B
"topics": "db.schema_b.table_d",
"iceberg.tables": "catalog.table_d",
"iceberg.table.catalog.table_d.id-columns": "id"
```

---

## Questions / Things to Clarify

1. Is multi-topic → multi-table routing **without `route-field`** officially supported? Or is 1 connector per table the intended pattern?
2. Could having the **same `id-columns` name** (`id`) across multiple tables cause the routing to break?
3. Is there a known issue where routing breaks when target tables share **nearly identical schemas**?
4. Is **topic-name-based routing** (matching last topic segment to table name) supported without SMT?

---

## Willingness to Contribute

If this is confirmed as a bug, **I'm happy to investigate the root cause and submit a fix.** It would help to know:

- Which part of the codebase handles topic → table routing resolution
- Whether schema-based matching plays any role in the routing decision
- Any existing tests covering multi-topic / multi-table scenarios with identical `id-columns`

Happy to discuss further before diving in. Thanks!

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by locating the topic-to-table routing resolution in the Iceberg Kafka Connect sink and review any existing multi-topic/multi-table tests. Reproduce the configuration with identical id-columns and nearly identical schemas; done means each topic's records reach only its matching table and a regression test covers the case.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, kafka, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.