apache / apache/iceberg

Iceberg Kafka Connect task cannot recover after failure

Open
#18,067 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Query engine

_No response_

### Question

I have encountered an issue with the Iceberg connector for Kafka Connect that I haven't been able to figure out how to resolve.

When I start the connector from a completely clean state (with the Kafka Connect cluster configuration, connector status, offsets, and the control/source topics all empty), it works as expected.

However, if the task encounters an error and stops working, I cannot get it to recover afterward. Even restarting the Kafka Connect cluster (running in a Kubernetes pod) does not resolve the issue. The task continues to log the following messages:

```text
[2026-09-11 14:03:52,943] INFO [connector-name|task-0] [Consumer clientId=45e600cb-3e05-4461-97b7-e88e4cbe7237, groupId=group-id-prefix779a2a31-e1c0-44d7-a0e7-1cd777609fa4] JoinGroup failed: The coordinator is not aware of this member. Need to re-join the group. Sent generation was Generation{generationId=-1, memberId='45e600cb-3e05-4461-97b7-e88e4cbe7237-1f4d28a5-c66a-4a92-93fd-2fe4bb187ae8', protocol='null'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:689)

[2026-09-11 14:03:52,943] INFO [connector-name|task-0] [Consumer clientId=45e600cb-3e05-4461-97b7-e88e4cbe7237, groupId=group-id-prefix779a2a31-e1c0-44d7-a0e7-1cd777609fa4] Resetting generation and member id due to: encountered UNKNOWN_MEMBER_ID from JOIN_GROUP response (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1062)

[2026-09-11 14:03:52,943] INFO [connector-name|task-0] [Consumer clientId=45e600cb-3e05-4461-97b7-e88e4cbe7237, groupId=group-id-prefix779a2a31-e1c0-44d7-a0e7-1cd777609fa4] Request joining group due to: encountered UNKNOWN_MEMBER_ID from JOIN_GROUP response (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1109)

[2026-09-11 14:03:52,943] INFO [connector-name|task-0] [Consumer clientId=45e600cb-3e05-4461-97b7-e88e4cbe7237, groupId=group-id-prefix779a2a31-e1c0-44d7-a0e7-1cd777609fa4] Request joining group due to: rebalance failed due to 'The coordinator is not aware of this member.' (UnknownMemberIdException) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1109)

[2026-09-11 14:03:52,944] INFO [connector-name|task-0] [Consumer clientId=45e600cb-3e05-4461-97b7-e88e4cbe7237, groupId=group-id-prefix779a2a31-e1c0-44d7-a0e7-1cd777609fa4] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:611)
```

The connector only starts working again if I remove it from the cluster and clear all the Kafka Connect internal topics involved.

### Environment

* **Iceberg Kafka Connect:** 1.11.0
* **Catalog:** Nessie
* **Table storage:** Dell ECS
* **Deployment:** Kubernetes
* **Connectivity:** I have verified connectivity between all deployed components, and there don't appear to be any connectivity issues.

### Steps to reproduce

1. Start Kafka Connect and the Iceberg connector from a clean state.
2. Let the connector process messages normally.
3. Cause the task to encounter an error and stop.
4. Restart the Kafka Connect pod/cluster.
5. The task does not recover and continuously logs `UnknownMemberIdException` / `The coordinator is not aware of this member`.
6. The connector only starts working again after removing it and clearing the Kafka Connect internal topics.

Has anyone encountered this behavior before?

Is there any additional Kafka Connect state or configuration that I should clear or reset to allow the task to recover without deleting the internal topics?

Any suggestions on what could cause the consumer group to remain in this state would be greatly appreciated.

Thanks in advance!

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka Connect task logs showing UnknownMemberIdException and the reproduction steps in this issue. Investigate the connector's consumer-group state and the Kafka Connect internal topics after a task failure and pod restart. Done means identifying a reproducible cause and a recovery path that does not require deleting the connector and all internal topics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, kubernetes
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.