support legacy 2-level Parquet LIST<float> encoding for vector import
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Problem
TiDB Lightning's Parquet importer does not support `LIST` columns. Parquet files that store embeddings/vectors as lists of floats cannot be imported today, which blocks vector workloads sourced from common data pipelines (Hive, Impala, Spark, Avro-derived writers, Arrow-based tools, etc.).
Parquet historically allows two on-wire encodings for `LIST`, both still widely used in the wild per the [backward-compatibility rules](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists):
- Modern 3-level encoding (recommended since parquet-format 2.4):
```
optional group (LIST) {
repeated group list {
optional|required float element;
}
}
```
- Legacy 2-level encoding (Hive, Impala, older Spark, several Avro-derived pipelines):
```
optional group (LIST) {
repeated float element;
}
```
## Proposal
Add `LIST` support to the Parquet reader in `pkg/lightning/mydump` and decode it into `VectorFloat32`. A single code path handles both encodings:
1. Schema validation examines the repeated child of the LIST group (primitive = 2-level, group with a single primitive = 3-level) and accepts leaf `maxDefinitionLevel` ∈ {2, 3}.
2. Level decoding is driven by the leaf `maxDefinitionLevel` instead of hard-coded constants: `def == 0` → null list, `def == 1` → empty list, `def == maxDef` → element present, any intermediate value → null inner element (rejected).
## Scope / non-goals
- Float-only (`VectorFloat32`); double/integer list elements are out of scope.
- Null elements inside a list are rejected (representable only in the 3-level optional-element form).
- No API or user-visible configuration changes.
## Testing
- `go test -run TestParquetVectorList ./pkg/lightning/mydump/` covers all three accepted shapes (3-level optional, 3-level required, 2-level).
- `go test -run TestParquetNullElementInList ./pkg/lightning/mydump/` rejects null elements.
Contributor guide
Assessment
This issue has not been assessed yet.