Substrait literal conversion should preserve nested nullability
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
DataFusion’s Substrait consumer currently normalizes some nested literal fields to nullable when converting Substrait literals into ScalarValues.
This is intentional today because `VirtualTable` / `Values` consumption converts the schema and literal rows separately. DataFusion later materializes the values into Arrow arrays and builds a `RecordBatch` using the schema. Arrow requires nested field types to match exactly, including nullability, so a mismatch like this can fail even
when the value itself is valid:
```
schema: List(Field { name: "item", data_type: Int32, nullable: false })
literal: List(Field { name: "item", data_type: Int32, nullable: true })
```
### Describe the solution you'd like
Add an expected-type-aware Substrait literal conversion path.
For `VirtualTable` / `Values`, the consumer has access to the `NamedStruct` schema. When converting each literal row, pass the expected top-level Field into literal conversion, and recursively use the expected nested fields for Struct, List, and Map literals.
The existing schema-less literal conversion path should remain available for ordinary expression literals where no expected field/type is available.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start at the Substrait consumer’s literal conversion and the VirtualTable/Values path, tracing how the NamedStruct schema reaches each literal row. Verify that expected nested fields preserve nullability for Struct, List, and Map literals while the schema-less expression-literal path remains available, then check RecordBatch construction against the mismatch described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100