cockroachdb / cockroachdb/cockroach

changefeedccl: rebucket changefeed files during backfill

Open
#104,776 1 comment 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

By default, we place files in new folder every day when running a changefeed to cloud storage. However, we do not start doing this until our first checkpoint. Backfilling very large tables can take multiple days. We also do not have good observability to show you that we are making progress on a backfill. This can lead to user confusion since both the storage bucket and the db console will look like we are not making progress over multiple days.

The only way to see that we are making progress is to dig into the backfill_pending_ranges metric or to click through 30+ next pages tabs in s3 to see the updates for the next day. We should instead rebucket while performing the backfill.

Jira issue: CRDB-28718

Epic CRDB-9196

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.