cockroachdb / cockroachdb/cockroach
changefeedccl: rebucket changefeed files during backfill
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
By default, we place files in new folder every day when running a changefeed to cloud storage. However, we do not start doing this until our first checkpoint. Backfilling very large tables can take multiple days. We also do not have good observability to show you that we are making progress on a backfill. This can lead to user confusion since both the storage bucket and the db console will look like we are not making progress over multiple days.
The only way to see that we are making progress is to dig into the backfill_pending_ranges metric or to click through 30+ next pages tabs in s3 to see the updates for the next day. We should instead rebucket while performing the backfill.
Jira issue: CRDB-28718
Epic CRDB-9196
Contributor guide
Assessment
This issue has not been assessed yet.