vectordotdev / vectordotdev/vector

ClickHouse sink: support arbitrary query-level settings via extra_query_params

Open
#25,937 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

Summary

The ClickHouse sink's query_settings block currently only exposes 6 hardcoded async insert settings, all passed as HTTP URL query parameters:

[sinks.my_sink.query_settings.async_insert_settings]
enabled                     = true
wait_for_processing         = true
wait_for_processing_timeout = 10
deduplicate                 = false
max_data_size               = 10000
max_query_number            = 100

ClickHouse supports hundreds of query-level settings (e.g. deduplicate_blocks_in_dependent_materialized_views, insert_quorum, max_insert_threads, input_format_skip_unknown_fields, max_insert_block_size, etc.) that are passed as HTTP query parameters. There is currently no way to set any of these from the Vector ClickHouse sink.

Proposed solution

Add an extra_query_params map to the ClickHouse sink config that forwards arbitrary key-value pairs as HTTP query parameters on every insert request:

[sinks.my_sink.extra_query_params]
deduplicate_blocks_in_dependent_materialized_views = "0"
insert_quorum = "2"
max_insert_block_size = "1048576"

These would be appended to the request URL alongside the existing async insert params.

Use case

  • deduplicate_blocks_in_dependent_materialized_views = 0 — disable deduplication for dependent materialized views when Vector handles delivery guarantees itself (at-least-once with idempotent writes)
  • insert_quorum — enforce write quorum on replicated ClickHouse clusters
  • max_insert_threads — control parallelism of insert processing on the server side
  • input_format_skip_unknown_fields — tolerate schema evolution without insert failures

Without this, the only alternative is setting these in the ClickHouse user profile for the Vector user, which is less flexible and couples ClickHouse server config to Vector behavior.

Implementation note

The existing set_uri_query in src/sinks/clickhouse/service.rs already appends params individually. An extra_query_params: HashMap<String, String> in QuerySettingsConfig (or as a top-level sink field) would iterate and append each entry the same way.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/sinks/clickhouse/service.rs, reading the existing set_uri_query handling and locating QuerySettingsConfig in the ClickHouse sink configuration. Trace how async insert settings reach each insert request, then verify that configured extra_query_params are appended alongside them on every request while existing settings continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, rust
Domain
backend, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.