vectordotdev / vectordotdev/vector
ClickHouse sink: support arbitrary query-level settings via extra_query_params
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
Summary
The ClickHouse sink's query_settings block currently only exposes 6 hardcoded async insert settings, all passed as HTTP URL query parameters:
[sinks.my_sink.query_settings.async_insert_settings]
enabled = true
wait_for_processing = true
wait_for_processing_timeout = 10
deduplicate = false
max_data_size = 10000
max_query_number = 100
ClickHouse supports hundreds of query-level settings (e.g. deduplicate_blocks_in_dependent_materialized_views, insert_quorum, max_insert_threads, input_format_skip_unknown_fields, max_insert_block_size, etc.) that are passed as HTTP query parameters. There is currently no way to set any of these from the Vector ClickHouse sink.
Proposed solution
Add an extra_query_params map to the ClickHouse sink config that forwards arbitrary key-value pairs as HTTP query parameters on every insert request:
[sinks.my_sink.extra_query_params]
deduplicate_blocks_in_dependent_materialized_views = "0"
insert_quorum = "2"
max_insert_block_size = "1048576"
These would be appended to the request URL alongside the existing async insert params.
Use case
deduplicate_blocks_in_dependent_materialized_views = 0— disable deduplication for dependent materialized views when Vector handles delivery guarantees itself (at-least-once with idempotent writes)insert_quorum— enforce write quorum on replicated ClickHouse clustersmax_insert_threads— control parallelism of insert processing on the server sideinput_format_skip_unknown_fields— tolerate schema evolution without insert failures
Without this, the only alternative is setting these in the ClickHouse user profile for the Vector user, which is less flexible and couples ClickHouse server config to Vector behavior.
Implementation note
The existing set_uri_query in src/sinks/clickhouse/service.rs already appends params individually. An extra_query_params: HashMap<String, String> in QuerySettingsConfig (or as a top-level sink field) would iterate and append each entry the same way.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/sinks/clickhouse/service.rs, reading the existing set_uri_query handling and locating QuerySettingsConfig in the ClickHouse sink configuration. Trace how async insert settings reach each insert request, then verify that configured extra_query_params are appended alongside them on every request while existing settings continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100