cockroachdb / cockroachdb/cockroach

sql/importer: parquet auto-mapping applies NULL instead of column DEFAULT for columns missing from the file

Open
#172,337 0 comments 0 reactions 0 assignees View on GitHub
A-import branch-release-26.2 branch-release-26.3 C-bug O-agent O-qa T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

In auto-mapping mode (no explicit target column list), a table column that is absent from the parquet file gets `NULL` instead of its `DEFAULT`. Two symptoms, one root cause:

1. `c INT DEFAULT 42` missing from the file → imported rows get `c = NULL`. The same import with an explicit target list that omits `c` (`IMPORT INTO t (a, b) PARQUET DATA ...`) correctly applies the default, as does CSV.
2. Worse, for `b STRING NOT NULL DEFAULT 'dflt'` missing from the file, planning-time validation passes — its missing-column check clearly intends defaults to count (`required table column "b" (non-nullable, no default) not found in Parquet file` only fires when there's *no* default) — but the import then fails per-row with `null value in column "b" violates not-null constraint`. The default is never applied, so the validation and execution behavior contradict each other.

Root cause: file-absent auto-mapped columns are hardcoded to `DNull` ([read_import_parquet.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/importer/read_import_parquet.go), "Set any nil datums to DNull") instead of being routed through default-expression evaluation like the explicit-target-list path.

**To Reproduce**

```python
import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({"a": [1, 2], "b": ["x", "y"]}), "ab.parquet")
```

```sql
CREATE TABLE t (a INT PRIMARY KEY, b STRING, c INT DEFAULT 42);
IMPORT INTO t PARQUET DATA ('nodelocal://1/ab.parquet'); -- succeeded
SELECT * FROM t; -- c is NULL, not 42

CREATE TABLE t2 (a INT PRIMARY KEY, b STRING, c INT DEFAULT 42);
IMPORT INTO t2 (a, b) PARQUET DATA ('nodelocal://1/ab.parquet'); -- succeeded
SELECT * FROM t2; -- c = 42 (correct)

CREATE TABLE t3 (a INT PRIMARY KEY, b STRING, d STRING NOT NULL DEFAULT 'dflt');
IMPORT INTO t3 PARQUET DATA ('nodelocal://1/ab.parquet');
-- passes validation, then fails per-row:
-- null value in column "d" violates not-null constraint
```

**Expected behavior**

Auto-mapped imports should treat file-absent columns exactly like explicit-target-list imports (and CSV): apply the column default, falling back to NULL only for nullable, default-less columns.

**Environment:**

- Reproduced on current master (26.3 dev); behavior shipped with parquet IMPORT in v26.2.0.

**Additional context**

This NULLing behavior is also the mechanism behind the mixed-version LIST data loss in #172329 (a 26.2 node treats a LIST column as "missing" and NULLs it).

Jira issue: CRDB-65564

Epic CRDB-66113

Contributor guide

Open the contributing guide

Research direction

Start in pkg/sql/importer/read_import_parquet.go at the auto-mapping path described by “Set any nil datums to DNull,” then compare it with the explicit-target-list default handling. Add regression coverage for missing columns with defaults and verify that auto-mapped Parquet imports apply defaults while nullable, default-less columns still become NULL.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.