cockroachdb / cockroachdb/cockroach

sql/importer: no way to set crdb_region via parquet IMPORT on REGIONAL BY ROW tables (CSV can)

Open
#172,342 1 comment 0 reactions 0 assignees View on GitHub
A-import A-multiregion C-bug O-agent O-qa T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

On a REGIONAL BY ROW table there is no way to import explicit per-row regions from a parquet file:

- Explicit targeting is rejected: `IMPORT INTO rbr (crdb_region, a, b) PARQUET DATA (...)` fails with `target column "crdb_region" not found in table schema` — while the identical CSV import succeeds and places each row in its named region. Parquet's target-column resolution apparently skips hidden columns.
- With auto-mapping, a `crdb_region` column present in the parquet file is silently ignored and the gateway-region default is applied.

Net: CSV can do explicit per-row region placement, parquet cannot — a format parity gap that matters when migrating multi-region data.

(The default path itself works: importing a file *without* `crdb_region` correctly lands all rows in the gateway region.)

**To Reproduce**

On a multi-region cluster (`cockroach demo --nodes=3 --demo-locality=region=us-east1:region=us-west1:region=europe-west1`):

```sql
CREATE DATABASE mr PRIMARY REGION "us-east1" REGIONS "us-west1", "europe-west1";
USE mr;
CREATE TABLE rbr (a INT PRIMARY KEY, b STRING) LOCALITY REGIONAL BY ROW;
```

```python
import pyarrow as pa, pyarrow.parquet as pq
t = pa.table({"crdb_region": ["us-west1", "europe-west1"], "a": [1, 2], "b": ["x", "y"]})
pq.write_table(t, "rbr.parquet")
```

```sql
IMPORT INTO rbr (crdb_region, a, b) PARQUET DATA ('nodelocal://1/rbr.parquet');
-- ERROR: target column "crdb_region" not found in table schema
IMPORT INTO rbr PARQUET DATA ('nodelocal://1/rbr.parquet');
-- succeeds, but crdb_region column ignored; all rows in gateway region
```

The same explicit-target CSV import succeeds and honors the per-row regions.

**Expected behavior**

Parity with CSV: explicit targeting of `crdb_region` resolves the hidden column and honors file values. Bonus: auto-mapping should either honor a `crdb_region` file column or (in strict mode) call out that it's being ignored, rather than silently dropping it.

**Environment:**

- Current master (26.3 dev), 3-region demo cluster.

Jira issue: CRDB-65569

Epic CRDB-66113

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the documented multi-region demo and the explicit-target and auto-mapped PARQUET imports. Read the SQL IMPORT PARQUET target-column resolution alongside the working CSV path; done means explicit crdb_region targeting honors per-row regions, with coverage for the described behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.