cockroachdb / cockroachdb/cockroach

cdc: make format=parquet work with changefeed.event_consumer_workers

Open
#106,591 1 comment 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We should do some benchmarks comparing `format=parquet` to `format=json` to see if there is room for improvement in terms of throughput. If so, we should remove the limitation where `event_consumer_workers` is incompatible with `format=parquet`.

Jira issue: CRDB-29635

Epic CRDB-37096

Contributor guide

Open the contributing guide

Research direction

Start by locating the changefeed implementation for format=parquet and the event_consumer_workers setting. Benchmark parquet against json for throughput, then determine whether the results justify removing their incompatibility. Done means the benchmark supports the change and parquet works with event_consumer_workers without the current limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.