cockroachdb / cockroachdb/cockroach

cdc: avoid duplicate schema registrations

Open
#99,221 9 comments 0 reactions 0 assignees View on GitHub
A-cdc backport-23.1.x C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When we have a large CRDB cluster starting up an Avro changefeed, each processor will post the same schema registration to the same endpoint at about the same time. Users should have an alternative to provisioning a schema registry endpoint that can handle such spiky traffic.

Fixes:

- https://github.com/cockroachdb/cockroach/pull/99833
- https://github.com/cockroachdb/cockroach/pull/100844
- https://github.com/cockroachdb/cockroach/pull/100843
- Reduces duplicate calls to ~the number of nodes, so in a cluster with high amounts of parallelism per node that's about an 8x reduction.
- Will be in v22.2.9
- Will be in v23.1.0.
- https://github.com/cockroachdb/cockroach/pull/98135
- https://github.com/cockroachdb/cockroach/pull/98349
- https://github.com/cockroachdb/cockroach/pull/98392
- https://github.com/cockroachdb/cockroach/pull/98396
- Reduce retries that were driven by spurious errors.
- Released in v22.2.7.
- Will be in v23.1.0.
- https://github.com/cockroachdb/cockroach/pull/99077
- https://github.com/cockroachdb/cockroach/pull/99300
- https://github.com/cockroachdb/cockroach/pull/99505
- This makes the timeout for schema registry call longer and more configurable. It doesn’t address the root cause, but offers a mitigation.
- Released in v22.2.8.
- Will be in v23.1.0.

We can implement this by having the processor that registers the schema tell the others what the ID is. https://github.com/cockroachdb/cockroach/pull/99059 is the beginning of an attempt to implement this by synchronizing using the job_info table.

Another approach (or complementary to above) could be to register schemas before distributing the job so that the IDs can be serialized into the processor specs, but that doesn't prevent potential spikes when a schema change occurs.

Jira issue: CRDB-25769

gz#16064

gz#16384

Epic CRDB-25039

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.