debezium / debezium/dbz

Retry Failed Snapshot Chunks before Failing/Retrying the Entire Snapshot.

Open
#2,297 6 comments 0 reactions 0 assignees View on GitHub
component/core-library type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Bug report

**What Debezium connector do you use and what version?**

`3.5.0.Final`

---

**What is the connector configuration?**

For a large table, use relatively high number of chunks.
```
snapshot.max.threads=1
snapshot.max.threads.multiplier=50
```

---

**What is the captured database version and mode of deployment?**

_E.g. on-premises, with a specific cloud provider, etc._

`Postgres`

---

**What behavior do you expect?**

When a single chunk fails due to for example connection errors, it should be retried before failing the whole snapshot. This is important for large tables where connection failures are likely during a snapshot and where retrying the whole snapshot for a chunk failure would be expensive. This may introduce inconsistency across snapshot chunks because each chunk may have been taken in a separate transaction. However, since the starting offset of the snapshot should have been saved, subsequent cdc streaming should make stream all changes that happened since that point. This makes it possible to construct a valid state of each row where the `LSN`(or other offsets depending on the database) for events are used to decide which event is most updated.

This issue is similar to [issue#1977](https://github.com/debezium/dbz/issues/1977)

---

**What behavior do you see?**

If a single chunk fails all of the snapshot fails

---

**Do you see the same behaviour using the latest released Debezium version?**

_Ideally, also verify with the latest Alpha/Beta/CR version._

Yes

---

**Do you have the connector logs, ideally from start till finish?**

_You might be asked later to provide [DEBUG/TRACE](https://debezium.io/documentation/reference/stable/operations/logging.html) level log._

---

**How to reproduce the issue using our [tutorial](https://github.com/debezium/debezium-examples/tree/main/tutorial) deployment?**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.