cockroachdb / cockroachdb/cockroach
sql/importer: parquet DECIMAL support gaps — FLBA-backed rejected, int-backed corrupts INT targets, BYTE_ARRAY-backed misparsed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Parquet DECIMAL columns can be physically backed by INT32/INT64, FIXED_LEN_BYTE_ARRAY, or BYTE_ARRAY. IMPORT only handles the int-backed forms correctly (into DECIMAL targets), and each of the other paths misbehaves:
**1. FLBA-backed DECIMAL is rejected — and it's the ecosystem default.** Import fails with
`column "f": fixed-length byte array type cannot be converted to FloatFamily` (or DecimalFamily).
FLBA is pyarrow's default decimal encoding at *any* precision, and the only encoding for precision > 18 in pyarrow/Spark/DuckDB — e.g. DuckDB types `range * 1.5` as `DECIMAL(21,1)` and writes `fixed_len_byte_array(16)`. Net effect: decimal columns from the most common writers cannot be imported at all. Verified workarounds: cast to DOUBLE before writing, or pyarrow `store_decimal_as_integer=True` (precision ≤ 18 only).
**2. Int-backed DECIMAL with scale > 0 into an INT target silently imports garbage.** Validation admits INT targets for any decimal logical type, but the converter only produces a `DInt` when scale == 0 — otherwise it emits a `DDecimal`, whose value encoding is then decoded as INT. Empirically verified: `DECIMAL(9,2)` values `123.45` and `6.78` **both** imported into an `INT8` column as `2`. Job `succeeded`, post-import INSPECT passed.
```python
import pyarrow as pa, pyarrow.parquet as pq, decimal
t = pa.table({"d": pa.array([decimal.Decimal("123.45"), decimal.Decimal("6.78")],
pa.decimal128(9, 2))})
pq.write_table(t, "dec.parquet", store_decimal_as_integer=True)
```
```sql
CREATE TABLE t (d INT8);
IMPORT INTO t PARQUET DATA ('nodelocal://1/dec.parquet'); -- succeeded
SELECT * FROM t; -- 2, 2
```
**3. BYTE_ARRAY-backed DECIMAL is parsed as decimal *text* (code-read, not reproduced).** The ByteArray validation path never checks `DecimalLogicalType`; conversion falls through to `tree.ParseDDecimal(string(v))` on what is actually big-endian two's-complement unscaled binary. Most values fail as row errors, but all-ASCII-digit payloads parse to a wrong number (e.g. unscaled 12336 scale 2 = `123.36` encodes as bytes `"00"` → imports as `0`). Writer: older Spark/parquet-mr configurations.
**Expected behavior**
1. FLBA- and BYTE_ARRAY-backed decimals decode per spec (big-endian unscaled integer + scale) into DECIMAL targets — this is table stakes for real-world parquet.
2. INT targets for decimal columns require scale == 0 at validation (or the converter errors) — never a silently wrong value.
**Environment:**
- Items 1–2 verified on current master (26.3 dev); the code shipped in v26.2.0, so 26.2 is likely affected as well. Item 3 is from code inspection.
**Additional context**
Item 1 is the biggest practical adoption blocker found in this QA pass (most ecosystem decimal files simply can't be imported); item 2 is silent corruption.
Jira issue: CRDB-65562
Epic CRDB-66113
Contributor guide
Research direction
Start at the Parquet IMPORT decimal validation and conversion paths, including the DecimalLogicalType and BYTE_ARRAY branches, and reproduce the supplied pyarrow DECIMAL(9,2) import into an INT8 target. Done means FLBA- and BYTE_ARRAY-backed decimals import correctly into DECIMAL targets, while scaled decimals targeting INT are rejected rather than silently corrupted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100