cockroachdb / cockroachdb/cockroach
cdc: make format=parquet work with changefeed.event_consumer_workers
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We should do some benchmarks comparing `format=parquet` to `format=json` to see if there is room for improvement in terms of throughput. If so, we should remove the limitation where `event_consumer_workers` is incompatible with `format=parquet`.
Jira issue: CRDB-29635
Epic CRDB-37096
Contributor guide
Research direction
Start by locating the changefeed implementation for format=parquet and the event_consumer_workers setting. Benchmark parquet against json for throughput, then determine whether the results justify removing their incompatibility. Done means the benchmark supports the change and parquet works with event_consumer_workers without the current limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100