cockroachdb / cockroachdb/cockroach

sql/importer: vendored arrow-go v11 rejects modern parquet encodings (BYTE_STREAM_SPLIT, LZ4_RAW); decompress option silently ignored

Open
#172,338 1 comment 0 reactions 0 assignees View on GitHub
A-import C-bug O-agent O-qa T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The parquet reader uses the vendored `apache/arrow/go/v11` library (released 2022), which predates two things modern writers produce, and the surrounding option handling has a gap:

**1. BYTE_STREAM_SPLIT encoding fails with a nonsense error.**
`reading batch from column "f" (index 2): expected 100 rows, got 0` — the v11 library has no BSS decode support and silently returns 0 values; the batch reader turns that into a row-count mismatch instead of "unsupported encoding". BSS is the recommended float encoding in modern writers (Arrow and DuckDB ≥ 1.2 use it when asked), so this will be hit in the wild.

```python
import pyarrow as pa, pyarrow.parquet as pq
t = pa.table({"i": list(range(100)), "s": [str(i) for i in range(100)],
"f": [i * 1.5 for i in range(100)]})
pq.write_table(t, "bss.parquet", use_dictionary=False,
column_encoding={"f": "BYTE_STREAM_SPLIT"})
```

**2. LZ4_RAW codec unsupported:** `parquet: unable to initialize page reader: compression for LZ4_RAW unimplemented`. At least this error is comprehensible. (`compression="lz4"` in pyarrow writes LZ4_RAW.)

**3. `WITH decompress` is accepted but silently ignored for parquet.** Parquet bypasses the stream-decompression wrapper entirely (it needs seekable, uncompressed access — see `read_import_base.go`). So `WITH decompress = 'gzip'` on a plain parquet file "succeeds" while doing nothing, and a gzip-compressed parquet file (`t.parquet.gz`) cannot be imported at all — with or without the option it fails with the confusing `failed to open Parquet file: parquet: file is smaller than indicated metadata size`.

**Expected behavior**

1–2. Upgrade the vendored arrow-go (v11 → current) to gain BSS decode and LZ4_RAW; failing that, detect the encoding/codec at file open and emit a clear "unsupported encoding: BYTE_STREAM_SPLIT" error instead of a row-count mismatch.
3. Reject `decompress` with a non-none value for PARQUET (the way CSV-only options are rejected), and ideally sniff gzip magic bytes at open to say "parquet files must not be externally compressed".

**Environment:**

- Current master (26.3 dev). All other encoding/codec variants tested fine: dictionary on/off, DELTA_BINARY_PACKED, DELTA_BYTE_ARRAY, data page v1/v2, format versions 1.0/2.6, snappy/gzip/zstd/brotli/uncompressed, per-column mixed codecs.

Jira issue: CRDB-65565

Epic CRDB-66113

Contributor guide

Open the contributing guide

Research direction

Start with the Parquet importer and read_import_base.go, then reproduce the BYTE_STREAM_SPLIT, LZ4_RAW, and decompress cases using the pyarrow examples in the issue. Check how the vendored apache/arrow/go/v11 reader reports encodings and codecs and how CSV-only options are rejected. Done means supported files import correctly or unsupported cases produce clear errors, and non-none Parquet decompression is rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.