ClickHouse / ClickHouse/clickhousectl

Require destination columns before creating an object-storage ClickPipe

Open
#840 0 comments 0 reactions 0 assignees View on GitHub
bug cloud
Dominant language
Rust
Stars
74
Forks
5
Avg merge
2d 11h
Merged PRs (30d)
196

Description

## Validated scope and current-PR plan — 2026-09-11

This section records the QA review and the current decision to finalize existing PRs without opening new PRs. It supersedes the proposed routing/fix suggestions in the original report below; the original observations are retained. Central plan: #757.

### Disposition

Deferred from this existing-PR finalization pass. Keep the issue open; no new PR is requested now. This is not a resolution, a severity downgrade, or a waiver of release acceptance.

### Validation and corrections

Reproduced an actual HTTP request when object-storage --column is omitted, so earlier validation is justified. --column-file is a feature expansion and can be a follow-up rather than blocking that fix. Verify virtual-column handling against real discovered schemas; do not delete a user column solely because its name starts with _. Optional field presence and Nullable(T) are different concepts: the displayed false is not by itself proof of a CLI bug. The renderer forwards field.optional from the API.

---

## Original QA report

Found in the 2026-09-11 QA sweep of the combined PR stack at [`d2c37f81`](https://github.com/ClickHouse/clickhousectl/commit/d2c37f814fff53fe68d636e34103e2d0550bf84d), top PR #823, installed release build (package version still 0.4.2). Reproduced behaviour only; no implementation included.

Severity: high; the single worst first-run experience in the ClickPipes surface.

```sh
clickhousectl cloud clickpipe create object-storage "$SVC" --name qa --source-url https://…/house_0.parquet \
--format Parquet --database default --table t
# Error: BAD_REQUEST: Destination columns are required if source is not a database pipe and must not be provided for database pipes
# exit 1
```

The CLI knows the source kind at parse time yet sends the request and surfaces API prose that names no flag. `--column` is documented as an ordinary optional flag ("Destination columns as name:type pairs"); the `create` CONTEXT block says to run `schema-discover` but not that its output is mandatory input.

There is no bridge between the two commands: `schema-discover` prints 18 fields, which you hand-transcribe into `--column name:type` flags (the working create was 14 flags / 430 characters). And four of those 18 are ClickHouse's object-storage **virtual** columns (`_path`, `_file`, `_size`, `_time`), which the `schema-discover` CONTEXT line ("pass them to `--column name:type`") tells you to include — do that literally and the destination table gets four junk columns. `Optional` also reads `false` for `Nullable(UInt64)`.

Suggested:
1. Local required-check in `validate_object_storage_create_args`: "`--column` is required for object-storage pipes; run `clickpipe schema-discover object-storage …` and pass each field as `--column name:type`" (exit 2).
2. State the requirement on the `--column` doc comment and in the CONTEXT block.
3. A `--column-file ` that accepts `schema-discover --json` output directly, collapsing the flow to two commands.
4. Mark or separate virtual columns in `schema-discover` output; fix the `Optional` column.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the object-storage create command, validate_object_storage_create_args, the --column documentation, and the create CONTEXT block; compare their behavior with schema-discover output. The payload says the work is deferred under plan #757, so confirm the active scope before changing anything; completion would require an agreed validation and documentation path without introducing unwanted virtual columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, rust
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.