apache / apache/datafusion

Substrait literal conversion should preserve nested nullability

Open
#22,095 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion’s Substrait consumer currently normalizes some nested literal fields to nullable when converting Substrait literals into ScalarValues.

This is intentional today because `VirtualTable` / `Values` consumption converts the schema and literal rows separately. DataFusion later materializes the values into Arrow arrays and builds a `RecordBatch` using the schema. Arrow requires nested field types to match exactly, including nullability, so a mismatch like this can fail even
when the value itself is valid:

```
schema: List(Field { name: "item", data_type: Int32, nullable: false })
literal: List(Field { name: "item", data_type: Int32, nullable: true })
```

### Describe the solution you'd like

Add an expected-type-aware Substrait literal conversion path.

For `VirtualTable` / `Values`, the consumer has access to the `NamedStruct` schema. When converting each literal row, pass the expected top-level Field into literal conversion, and recursively use the expected nested fields for Struct, List, and Map literals.

The existing schema-less literal conversion path should remain available for ordinary expression literals where no expected field/type is available.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the Substrait consumer’s literal conversion and the VirtualTable/Values path, tracing how the NamedStruct schema reaches each literal row. Verify that expected nested fields preserve nullability for Struct, List, and Map literals while the schema-less expression-literal path remains available, then check RecordBatch construction against the mismatch described.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.