apache / apache/datafusion

[substrait]: schema roundtrip fails

Open
#15,210 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

If a DataFusion schema contains a timestamp column with timezone other than "UTC", the roundtrip fails.
Specifically, converts the schema to substrait named struct and then converts the named struct back to schema, the reconstructed schema will always have a timestamp column with timezone being "UTC", other than the original timezone.

### To Reproduce

The following test fails. Note, there's a timestamp column with non-UTC timezone.

#[test]
fn named_struct_names() -> Result<()> {
let schema = DFSchemaRef::new(DFSchema::try_from(Schema::new(vec![
Field::new("int", DataType::Int32, true),
Field::new(
"struct",
DataType::Struct(Fields::from(vec![Field::new(
"inner",
DataType::List(Arc::new(Field::new_list_field(DataType::Utf8, true))),
true,
)])),
true,
),
Field::new("trailer", DataType::Float64, true),
Field::new("time", DataType::Timestamp(TimeUnit::Second, None), true),
Field::new(
"time_with_tz",
DataType::Timestamp(TimeUnit::Second, Some("+00:00".into())),
true,
),
]))?);

let named_struct = to_substrait_named_struct(&schema)?;

// Struct field names should be flattened DFS style
// List field names should be omitted
assert_eq!(
named_struct.names,
vec!["int", "struct", "inner", "trailer", "time", "time_with_tz"]
);

let roundtrip_schema =
from_substrait_named_struct(&test_consumer(), &named_struct)?;
assert_eq!(schema.as_ref(), &roundtrip_schema);
Ok(())
}

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the named_struct roundtrip test shown in the issue and trace to_substrait_named_struct and from_substrait_named_struct. Reproduce the failure with the non-UTC timestamp field, then verify that converting through the Substrait named struct preserves its timezone and that the roundtrip assertion passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.