ClickHouse / ClickHouse/clickhousectl
Require destination columns before creating an object-storage ClickPipe
- Dominant language
- Rust
- Stars
- 74
- Forks
- 5
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 196
Description
## Validated scope and current-PR plan — 2026-09-11
This section records the QA review and the current decision to finalize existing PRs without opening new PRs. It supersedes the proposed routing/fix suggestions in the original report below; the original observations are retained. Central plan: #757.
### Disposition
Deferred from this existing-PR finalization pass. Keep the issue open; no new PR is requested now. This is not a resolution, a severity downgrade, or a waiver of release acceptance.
### Validation and corrections
Reproduced an actual HTTP request when object-storage --column is omitted, so earlier validation is justified. --column-file is a feature expansion and can be a follow-up rather than blocking that fix. Verify virtual-column handling against real discovered schemas; do not delete a user column solely because its name starts with _. Optional field presence and Nullable(T) are different concepts: the displayed false is not by itself proof of a CLI bug. The renderer forwards field.optional from the API.
---
## Original QA report
Found in the 2026-09-11 QA sweep of the combined PR stack at [`d2c37f81`](https://github.com/ClickHouse/clickhousectl/commit/d2c37f814fff53fe68d636e34103e2d0550bf84d), top PR #823, installed release build (package version still 0.4.2). Reproduced behaviour only; no implementation included.
Severity: high; the single worst first-run experience in the ClickPipes surface.
```sh
clickhousectl cloud clickpipe create object-storage "$SVC" --name qa --source-url https://…/house_0.parquet \
--format Parquet --database default --table t
# Error: BAD_REQUEST: Destination columns are required if source is not a database pipe and must not be provided for database pipes
# exit 1
```
The CLI knows the source kind at parse time yet sends the request and surfaces API prose that names no flag. `--column` is documented as an ordinary optional flag ("Destination columns as name:type pairs"); the `create` CONTEXT block says to run `schema-discover` but not that its output is mandatory input.
There is no bridge between the two commands: `schema-discover` prints 18 fields, which you hand-transcribe into `--column name:type` flags (the working create was 14 flags / 430 characters). And four of those 18 are ClickHouse's object-storage **virtual** columns (`_path`, `_file`, `_size`, `_time`), which the `schema-discover` CONTEXT line ("pass them to `--column name:type`") tells you to include — do that literally and the destination table gets four junk columns. `Optional` also reads `false` for `Nullable(UInt64)`.
Suggested:
1. Local required-check in `validate_object_storage_create_args`: "`--column` is required for object-storage pipes; run `clickpipe schema-discover object-storage …` and pass each field as `--column name:type`" (exit 2).
2. State the requirement on the `--column` doc comment and in the CONTEXT block.
3. A `--column-file ` that accepts `schema-discover --json` output directly, collapsing the flow to two commands.
4. Mark or separate virtual columns in `schema-discover` output; fix the `Optional` column.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the object-storage create command, validate_object_storage_create_args, the --column documentation, and the create CONTEXT block; compare their behavior with schema-discover output. The payload says the work is deferred under plan #757, so confirm the active scope before changing anything; completion would require an agreed validation and documentation path without introducing unwanted virtual columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100