cockroachdb / cockroachdb/cockroach
sql/importer: parquet auto-mapping applies NULL instead of column DEFAULT for columns missing from the file
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
In auto-mapping mode (no explicit target column list), a table column that is absent from the parquet file gets `NULL` instead of its `DEFAULT`. Two symptoms, one root cause:
1. `c INT DEFAULT 42` missing from the file → imported rows get `c = NULL`. The same import with an explicit target list that omits `c` (`IMPORT INTO t (a, b) PARQUET DATA ...`) correctly applies the default, as does CSV.
2. Worse, for `b STRING NOT NULL DEFAULT 'dflt'` missing from the file, planning-time validation passes — its missing-column check clearly intends defaults to count (`required table column "b" (non-nullable, no default) not found in Parquet file` only fires when there's *no* default) — but the import then fails per-row with `null value in column "b" violates not-null constraint`. The default is never applied, so the validation and execution behavior contradict each other.
Root cause: file-absent auto-mapped columns are hardcoded to `DNull` ([read_import_parquet.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/importer/read_import_parquet.go), "Set any nil datums to DNull") instead of being routed through default-expression evaluation like the explicit-target-list path.
**To Reproduce**
```python
import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({"a": [1, 2], "b": ["x", "y"]}), "ab.parquet")
```
```sql
CREATE TABLE t (a INT PRIMARY KEY, b STRING, c INT DEFAULT 42);
IMPORT INTO t PARQUET DATA ('nodelocal://1/ab.parquet'); -- succeeded
SELECT * FROM t; -- c is NULL, not 42
CREATE TABLE t2 (a INT PRIMARY KEY, b STRING, c INT DEFAULT 42);
IMPORT INTO t2 (a, b) PARQUET DATA ('nodelocal://1/ab.parquet'); -- succeeded
SELECT * FROM t2; -- c = 42 (correct)
CREATE TABLE t3 (a INT PRIMARY KEY, b STRING, d STRING NOT NULL DEFAULT 'dflt');
IMPORT INTO t3 PARQUET DATA ('nodelocal://1/ab.parquet');
-- passes validation, then fails per-row:
-- null value in column "d" violates not-null constraint
```
**Expected behavior**
Auto-mapped imports should treat file-absent columns exactly like explicit-target-list imports (and CSV): apply the column default, falling back to NULL only for nullable, default-less columns.
**Environment:**
- Reproduced on current master (26.3 dev); behavior shipped with parquet IMPORT in v26.2.0.
**Additional context**
This NULLing behavior is also the mechanism behind the mixed-version LIST data loss in #172329 (a 26.2 node treats a LIST column as "missing" and NULLs it).
Jira issue: CRDB-65564
Epic CRDB-66113
Contributor guide
Research direction
Start in pkg/sql/importer/read_import_parquet.go at the auto-mapping path described by “Set any nil datums to DNull,” then compare it with the explicit-target-list default handling. Add regression coverage for missing columns with defaults and verify that auto-mapped Parquet imports apply defaults while nullable, default-less columns still become NULL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100