[substrait]: schema roundtrip fails
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
If a DataFusion schema contains a timestamp column with timezone other than "UTC", the roundtrip fails.
Specifically, converts the schema to substrait named struct and then converts the named struct back to schema, the reconstructed schema will always have a timestamp column with timezone being "UTC", other than the original timezone.
### To Reproduce
The following test fails. Note, there's a timestamp column with non-UTC timezone.
#[test]
fn named_struct_names() -> Result<()> {
let schema = DFSchemaRef::new(DFSchema::try_from(Schema::new(vec![
Field::new("int", DataType::Int32, true),
Field::new(
"struct",
DataType::Struct(Fields::from(vec![Field::new(
"inner",
DataType::List(Arc::new(Field::new_list_field(DataType::Utf8, true))),
true,
)])),
true,
),
Field::new("trailer", DataType::Float64, true),
Field::new("time", DataType::Timestamp(TimeUnit::Second, None), true),
Field::new(
"time_with_tz",
DataType::Timestamp(TimeUnit::Second, Some("+00:00".into())),
true,
),
]))?);
let named_struct = to_substrait_named_struct(&schema)?;
// Struct field names should be flattened DFS style
// List field names should be omitted
assert_eq!(
named_struct.names,
vec!["int", "struct", "inner", "trailer", "time", "time_with_tz"]
);
let roundtrip_schema =
from_substrait_named_struct(&test_consumer(), &named_struct)?;
assert_eq!(schema.as_ref(), &roundtrip_schema);
Ok(())
}
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the named_struct roundtrip test shown in the issue and trace to_substrait_named_struct and from_substrait_named_struct. Reproduce the failure with the non-UTC timestamp field, then verify that converting through the Substrait named struct preserves its timezone and that the roundtrip assertion passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100