ClickHouse / ClickHouse/clickhousectl

Verify source-kind rules before enforcing ClickPipe field mappings

Open
#845 0 comments 0 reactions 0 assignees View on GitHub
cloud documentation
Dominant language
Rust
Stars
74
Forks
5
Avg merge
2d 11h
Merged PRs (30d)
196

Description

## Validated scope and current-PR plan — 2026-09-11

This section records the QA review and the current decision to finalize existing PRs without opening new PRs. It supersedes the proposed routing/fix suggestions in the original report below; the original observations are retained. Central plan: #757.

### Work for the existing PRs

Implementation status: the checked sub-items below are implemented on the existing PR branches and validated locally on combined head `82b374816a917cbac3a68194659beaaa5c7c9087`. Required final-head Cloud CI and merging remain pending; this issue stays open. See #757 for the combined validation and CI status.

- [x] #793 — Bring the already-verified object-storage fieldMappings/destination.columns requirement into update input help. Keep it scoped to object-storage; do not introduce unverified universal PATCH validation.

### Remaining scope

Client-side mapping enforcement remains deferred until the rule is verified for the relevant pipe kinds; #793 explicitly avoided assuming a universal rule. CDC pause guidance is also outside this narrow documentation completion.

This is a partial assignment. Keep the issue open after these PRs land; use sub-item references rather than a whole-issue closing keyword.

### Validation and corrections

Reproduced an HTTP request for {fieldMappings: []} without columns. However #793’s body explicitly says it avoided global validation because source kind is commonly omitted, and only object-storage behavior was verified. #813 validates source ports; it does not own all PATCH invariants. New enforcement must either establish a universal rule or obtain/validate the existing pipe kind. Preserve omission semantics and cover explicit []. CDC pause guidance is also valid new help work; do not silently pause/resume the pipe.

---

## Original QA report

Found in the 2026-09-11 QA sweep of the combined PR stack at [`d2c37f81`](https://github.com/ClickHouse/clickhousectl/commit/d2c37f814fff53fe68d636e34103e2d0550bf84d), top PR #823, installed release build (package version still 0.4.2). Reproduced behaviour only; no implementation included.

Severity: low-medium.

```sh
printf '{"source":{"postgres":{"settings":{"syncIntervalSeconds":30}}}}' | clickhousectl cloud clickpipe update "$SVC" "$CDC" --config-file -
# Error: BAD_REQUEST: Postgres ClickPipe must be paused to edit (succeeds after `clickpipe stop`)
printf '{"fieldMappings":[{"sourceField":"price","destinationField":"price"}]}' | … update … --config-file -
# Error: BAD_REQUEST: Columns are required if field mappings are provided
printf '{"fieldMappings":[]}' | … update … --config-file -
# Error: BAD_REQUEST: Columns are required if field mappings are provided (explicit [] rejected too)
```

The second matches the README text #763 added exactly — README is correct. But it lives only in the README and is not enforced in `validate_clickpipe_patch_required_fields`, which already runs structural PATCH checks for mysql/mongodb/pubsub/bigquery in the same function. Neither the `--config-file` doc nor the 4-line CONTEXT block mentions pausing CDC pipes.

Suggested: add the `fieldMappings ⇒ destination.columns` check (including explicit `[]`) next to the sibling checks; one CONTEXT line: "CDC pipes must be stopped (Paused) before `update`."

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with validate_clickpipe_patch_required_fields and the existing README and --config-file documentation referenced in the report. Read #757, #793, and #813 first to understand the verified source-kind scope and existing work. Done means the remaining documentation or validation scope is completed without assuming a universal rule, while preserving omission semantics and handling explicit empty mappings.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, documentation
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.