lance-format / lance-format/lance
bug: Substrait expression transport cannot encode predicates on FixedSizeList columns
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Problem
Lance's Substrait expression transport cannot encode predicates referencing FixedSizeList columns. This affects ordinary predicates such as vector IS NULL as well as expressions that inspect vector values.
prune_schema_for_substrait removes these columns because the DataFusion Substrait producer does not support their type. Pruning is safe only when the expression does not reference the removed fields. Passing an expression that still references them to encode_substrait instead fails with SchemaError(FieldNotFound), even though the dataset contains the column.
Code inspected: Lance commit 271a155ba505fd8e1094c095d4ce356707e93866, rust/lance-datafusion/src/substrait.rs.
Minimal scenario
Given a schema with id: int64 and vector: fixed_size_list<float32, 4>:
- Build an expression referencing
vector, for examplecol("vector").is_null(). - Prune the schema with
prune_schema_for_substrait. - Call
encode_substraitwith that expression and the pruned schema.
The column is absent from the encoding schema. Retaining the original schema alone does not address the underlying producer type-support gap. This is a code-derived scenario, not a standalone executed reproduction.
Expected behavior
Support round-tripping expressions that reference fixed-size-list columns through Lance's Substrait transport, preserving expression semantics and field mappings. This may require changes in DataFusion's Substrait producer/consumer. Until supported, callers need an explicit capability boundary so they can retain these predicates on another execution path rather than treating a valid dataset column as missing.
Coverage should include null/non-null vectors, columns before and after the vector field, and compound predicates. A downstream distributed query path currently works around this by pruning its encoding schema; it must avoid choosing this transport for unsupported predicates in the meantime.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance-datafusion/src/substrait.rs and trace prune_schema_for_substrait and encode_substrait using the fixed-size-list predicate scenario. Inspect the DataFusion Substrait producer and consumer support, then verify that null, non-null, positional, and compound predicates either round-trip with correct field mappings or are explicitly rejected so callers can use another execution path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100