cockroachdb / cockroachdb/cockroach

sql/importer: parquet DECIMAL support gaps — FLBA-backed rejected, int-backed corrupts INT targets, BYTE_ARRAY-backed misparsed

Open
#172,335 0 comments 0 reactions 0 assignees View on GitHub
A-import branch-release-26.2 branch-release-26.3 C-bug O-agent O-qa T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Parquet DECIMAL columns can be physically backed by INT32/INT64, FIXED_LEN_BYTE_ARRAY, or BYTE_ARRAY. IMPORT only handles the int-backed forms correctly (into DECIMAL targets), and each of the other paths misbehaves:

**1. FLBA-backed DECIMAL is rejected — and it's the ecosystem default.** Import fails with
`column "f": fixed-length byte array type cannot be converted to FloatFamily` (or DecimalFamily).
FLBA is pyarrow's default decimal encoding at *any* precision, and the only encoding for precision > 18 in pyarrow/Spark/DuckDB — e.g. DuckDB types `range * 1.5` as `DECIMAL(21,1)` and writes `fixed_len_byte_array(16)`. Net effect: decimal columns from the most common writers cannot be imported at all. Verified workarounds: cast to DOUBLE before writing, or pyarrow `store_decimal_as_integer=True` (precision ≤ 18 only).

**2. Int-backed DECIMAL with scale > 0 into an INT target silently imports garbage.** Validation admits INT targets for any decimal logical type, but the converter only produces a `DInt` when scale == 0 — otherwise it emits a `DDecimal`, whose value encoding is then decoded as INT. Empirically verified: `DECIMAL(9,2)` values `123.45` and `6.78` **both** imported into an `INT8` column as `2`. Job `succeeded`, post-import INSPECT passed.

```python
import pyarrow as pa, pyarrow.parquet as pq, decimal
t = pa.table({"d": pa.array([decimal.Decimal("123.45"), decimal.Decimal("6.78")],
pa.decimal128(9, 2))})
pq.write_table(t, "dec.parquet", store_decimal_as_integer=True)
```
```sql
CREATE TABLE t (d INT8);
IMPORT INTO t PARQUET DATA ('nodelocal://1/dec.parquet'); -- succeeded
SELECT * FROM t; -- 2, 2
```

**3. BYTE_ARRAY-backed DECIMAL is parsed as decimal *text* (code-read, not reproduced).** The ByteArray validation path never checks `DecimalLogicalType`; conversion falls through to `tree.ParseDDecimal(string(v))` on what is actually big-endian two's-complement unscaled binary. Most values fail as row errors, but all-ASCII-digit payloads parse to a wrong number (e.g. unscaled 12336 scale 2 = `123.36` encodes as bytes `"00"` → imports as `0`). Writer: older Spark/parquet-mr configurations.

**Expected behavior**

1. FLBA- and BYTE_ARRAY-backed decimals decode per spec (big-endian unscaled integer + scale) into DECIMAL targets — this is table stakes for real-world parquet.
2. INT targets for decimal columns require scale == 0 at validation (or the converter errors) — never a silently wrong value.

**Environment:**

- Items 1–2 verified on current master (26.3 dev); the code shipped in v26.2.0, so 26.2 is likely affected as well. Item 3 is from code inspection.

**Additional context**

Item 1 is the biggest practical adoption blocker found in this QA pass (most ecosystem decimal files simply can't be imported); item 2 is silent corruption.

Jira issue: CRDB-65562

Epic CRDB-66113

Contributor guide

Open the contributing guide

Research direction

Start at the Parquet IMPORT decimal validation and conversion paths, including the DecimalLogicalType and BYTE_ARRAY branches, and reproduce the supplied pyarrow DECIMAL(9,2) import into an INT8 target. Done means FLBA- and BYTE_ARRAY-backed decimals import correctly into DECIMAL targets, while scaled decimals targeting INT are rejected rather than silently corrupted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.