cockroachdb / cockroachdb/cockroach

sql/importer: STRUCT and bare-repeated parquet columns bypass nested-type rejection and silently corrupt data

Open
#172,333 0 comments 0 reactions 0 assignees View on GitHub
A-import branch-release-26.2 branch-release-26.3 C-bug O-agent O-qa T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

IMPORT claims nested parquet structures are unsupported (#162543) and has a guard that rejects them — but the guard only fires on `maxDefLevel > 1` ([read_import_parquet.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/importer/read_import_parquet.go), nested-structure check). Two legal parquet shapes slip past it and import silently wrong data instead of being rejected:

**1. STRUCT columns are silently flattened; multi-leaf structs are last-leaf-wins.** Leaves under `optional group s { required int64 x }` have maxDefLevel = 1 (fully-required nesting has 0), so struct files bypass the guard, and column metadata labels every leaf with the root group's name. Empirically verified with pyarrow structs:
- single-leaf struct `s{x}`: imports `s.x`'s values under column `s`, no error;
- two-leaf struct `s{x, y}`: imports **y's values** into `s`, silently dropping `x` entirely.

**2. Bare repeated primitives (1-level list encoding) are misread as flat columns, misaligning rows and dropping values.** `repeated int64 nums;` directly under the root is legal parquet (written by parquet-protobuf/parquet-thrift). `detectListColumn` ([read_import_parquet_list.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/importer/read_import_parquet_list.go)) only recognizes the 3-level group shape, so the column is classified as flat; the guard passes it (maxDef = 1); the flat batch reader ignores repetition levels, so each list *element* is consumed as a *row*. Empirically verified with a file containing rows `[10, 20]` and `[30]`: importing into a scalar INT column produced `(1, 10), (2, 20)` — row 2 received element 2 of row 1's list — and the value `30` was silently dropped. Job `succeeded`. Meanwhile the *correct* mapping (an `INT8[]` target) is **rejected** (`int64 type can only be converted to INT or DECIMAL, got ArrayFamily`), so the only accepted path for 1-level-list files is the silently-corrupting one.

**To Reproduce**

Structs (pyarrow):

```python
import pyarrow as pa, pyarrow.parquet as pq
s = pa.array([{"x": 1, "y": 100}, {"x": 2, "y": 200}])
pq.write_table(pa.table({"id": [1, 2], "s": s}), "struct.parquet")
```

```sql
CREATE TABLE t (id INT PRIMARY KEY, s INT);
IMPORT INTO t PARQUET DATA ('nodelocal://1/struct.parquet'); -- succeeded
SELECT * FROM t; -- (1, 100), (2, 200): y's values, x silently dropped
```

Bare repeated (pyarrow can't write this; Go generator, `go mod init repro && go get github.com/apache/arrow/go/v11 && go run .`):

```go
package main

import (
"os"

"github.com/apache/arrow/go/v11/parquet"
"github.com/apache/arrow/go/v11/parquet/file"
"github.com/apache/arrow/go/v11/parquet/schema"
)

func main() {
idNode, _ := schema.NewPrimitiveNode("id", parquet.Repetitions.Optional,
parquet.Types.Int64, -1, -1)
// Bare repeated primitive directly under the root: legal parquet
// (1-level list encoding, as written by parquet-protobuf/thrift).
numsNode, _ := schema.NewPrimitiveNode("nums", parquet.Repetitions.Repeated,
parquet.Types.Int64, -1, -1)
root, _ := schema.NewGroupNode("schema", parquet.Repetitions.Required,
schema.FieldList{idNode, numsNode}, -1)

f, _ := os.Create("repeated.parquet")
w := file.NewParquetWriter(f, root)
rgw := w.AppendRowGroup()

cw, _ := rgw.NextColumn()
cw.(*file.Int64ColumnChunkWriter).WriteBatch([]int64{1, 2}, []int16{1, 1}, nil)
cw.Close()

cw, _ = rgw.NextColumn()
// Row 1: [10, 20] (repLevels 0,1); Row 2: [30] (repLevel 0).
cw.(*file.Int64ColumnChunkWriter).WriteBatch(
[]int64{10, 20, 30}, []int16{1, 1, 1}, []int16{0, 1, 0})
cw.Close()
rgw.Close()
w.Close()
}
```

```sql
CREATE TABLE r (id INT PRIMARY KEY, nums INT);
IMPORT INTO r PARQUET DATA ('nodelocal://1/repeated.parquet'); -- succeeded
SELECT * FROM r; -- (1, 10), (2, 20): misaligned; 30 silently dropped
```

**Expected behavior**

Both shapes should be rejected with the same clean `0A000` unsupported-nested-structure error that LIST-of-LIST and MAP columns already get (or properly supported: the repeated case could be decoded as a 1-level LIST). One guard covers both: reject any leaf whose column root is a group that isn't a recognized LIST, and reject `MaxRepetitionLevel() > 0` on columns not classified as LIST — independent of definition levels.

**Environment:**

- Reproduced on current master (26.3 dev). The guard predates 26.3; 26.2 is likely affected for the struct case (not separately verified).

**Additional context**

Both cases are silent corruption with a `succeeded` job: struct files drop entire subfields, repeated files shift values across rows and drop data.

Jira issue: CRDB-65560

Epic CRDB-66113

Contributor guide

Open the contributing guide

Research direction

Read pkg/sql/importer/read_import_parquet.go and read_import_parquet_list.go, starting at the nested-structure guard and detectListColumn. Run the supplied pyarrow struct and Go bare-repeated reproductions against IMPORT. Done means both shapes receive the existing unsupported-nested-structure error without a succeeded job, while recognized lists retain current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.