cockroachdb / cockroachdb/cockroach
sql/importer: vendored arrow-go v11 rejects modern parquet encodings (BYTE_STREAM_SPLIT, LZ4_RAW); decompress option silently ignored
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The parquet reader uses the vendored `apache/arrow/go/v11` library (released 2022), which predates two things modern writers produce, and the surrounding option handling has a gap:
**1. BYTE_STREAM_SPLIT encoding fails with a nonsense error.**
`reading batch from column "f" (index 2): expected 100 rows, got 0` — the v11 library has no BSS decode support and silently returns 0 values; the batch reader turns that into a row-count mismatch instead of "unsupported encoding". BSS is the recommended float encoding in modern writers (Arrow and DuckDB ≥ 1.2 use it when asked), so this will be hit in the wild.
```python
import pyarrow as pa, pyarrow.parquet as pq
t = pa.table({"i": list(range(100)), "s": [str(i) for i in range(100)],
"f": [i * 1.5 for i in range(100)]})
pq.write_table(t, "bss.parquet", use_dictionary=False,
column_encoding={"f": "BYTE_STREAM_SPLIT"})
```
**2. LZ4_RAW codec unsupported:** `parquet: unable to initialize page reader: compression for LZ4_RAW unimplemented`. At least this error is comprehensible. (`compression="lz4"` in pyarrow writes LZ4_RAW.)
**3. `WITH decompress` is accepted but silently ignored for parquet.** Parquet bypasses the stream-decompression wrapper entirely (it needs seekable, uncompressed access — see `read_import_base.go`). So `WITH decompress = 'gzip'` on a plain parquet file "succeeds" while doing nothing, and a gzip-compressed parquet file (`t.parquet.gz`) cannot be imported at all — with or without the option it fails with the confusing `failed to open Parquet file: parquet: file is smaller than indicated metadata size`.
**Expected behavior**
1–2. Upgrade the vendored arrow-go (v11 → current) to gain BSS decode and LZ4_RAW; failing that, detect the encoding/codec at file open and emit a clear "unsupported encoding: BYTE_STREAM_SPLIT" error instead of a row-count mismatch.
3. Reject `decompress` with a non-none value for PARQUET (the way CSV-only options are rejected), and ideally sniff gzip magic bytes at open to say "parquet files must not be externally compressed".
**Environment:**
- Current master (26.3 dev). All other encoding/codec variants tested fine: dictionary on/off, DELTA_BINARY_PACKED, DELTA_BYTE_ARRAY, data page v1/v2, format versions 1.0/2.6, snappy/gzip/zstd/brotli/uncompressed, per-column mixed codecs.
Jira issue: CRDB-65565
Epic CRDB-66113
Contributor guide
Research direction
Start with the Parquet importer and read_import_base.go, then reproduce the BYTE_STREAM_SPLIT, LZ4_RAW, and decompress cases using the pyarrow examples in the issue. Check how the vendored apache/arrow/go/v11 reader reports encodings and codecs and how CSV-only options are rejected. Done means supported files import correctly or unsupported cases produce clear errors, and non-none Parquet decompression is rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100