cockroachdb / cockroachdb/cockroach
cdc: avoid duplicate schema registrations
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When we have a large CRDB cluster starting up an Avro changefeed, each processor will post the same schema registration to the same endpoint at about the same time. Users should have an alternative to provisioning a schema registry endpoint that can handle such spiky traffic.
Fixes:
- https://github.com/cockroachdb/cockroach/pull/99833
- https://github.com/cockroachdb/cockroach/pull/100844
- https://github.com/cockroachdb/cockroach/pull/100843
- Reduces duplicate calls to ~the number of nodes, so in a cluster with high amounts of parallelism per node that's about an 8x reduction.
- Will be in v22.2.9
- Will be in v23.1.0.
- https://github.com/cockroachdb/cockroach/pull/98135
- https://github.com/cockroachdb/cockroach/pull/98349
- https://github.com/cockroachdb/cockroach/pull/98392
- https://github.com/cockroachdb/cockroach/pull/98396
- Reduce retries that were driven by spurious errors.
- Released in v22.2.7.
- Will be in v23.1.0.
- https://github.com/cockroachdb/cockroach/pull/99077
- https://github.com/cockroachdb/cockroach/pull/99300
- https://github.com/cockroachdb/cockroach/pull/99505
- This makes the timeout for schema registry call longer and more configurable. It doesn’t address the root cause, but offers a mitigation.
- Released in v22.2.8.
- Will be in v23.1.0.
We can implement this by having the processor that registers the schema tell the others what the ID is. https://github.com/cockroachdb/cockroach/pull/99059 is the beginning of an attempt to implement this by synchronizing using the job_info table.
Another approach (or complementary to above) could be to register schemas before distributing the job so that the IDs can be serialized into the processor specs, but that doesn't prevent potential spikes when a schema change occurs.
Jira issue: CRDB-25769
gz#16064
gz#16384
Epic CRDB-25039
Contributor guide
Assessment
This issue has not been assessed yet.