You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If a DataFusion schema contains a timestamp column with timezone other than "UTC", the roundtrip fails.
Specifically, converts the schema to substrait named struct and then converts the named struct back to schema, the reconstructed schema will always have a timestamp column with timezone being "UTC", other than the original timezone.
To Reproduce
The following test fails. Note, there's a timestamp column with non-UTC timezone.
#[test]
fn named_struct_names() -> Result<()> {
let schema = DFSchemaRef::new(DFSchema::try_from(Schema::new(vec![
Field::new("int", DataType::Int32, true),
Field::new(
"struct",
DataType::Struct(Fields::from(vec![Field::new(
"inner",
DataType::List(Arc::new(Field::new_list_field(DataType::Utf8, true))),
true,
)])),
true,
),
Field::new("trailer", DataType::Float64, true),
Field::new("time", DataType::Timestamp(TimeUnit::Second, None), true),
Field::new(
"time_with_tz",
DataType::Timestamp(TimeUnit::Second, Some("+00:00".into())),
true,
),
]))?);
let named_struct = to_substrait_named_struct(&schema)?;
// Struct field names should be flattened DFS style
// List field names should be omitted
assert_eq!(
named_struct.names,
vec!["int", "struct", "inner", "trailer", "time", "time_with_tz"]
);
let roundtrip_schema =
from_substrait_named_struct(&test_consumer(), &named_struct)?;
assert_eq!(schema.as_ref(), &roundtrip_schema);
Ok(())
}
Expected behavior
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
Describe the bug
If a DataFusion schema contains a timestamp column with timezone other than "UTC", the roundtrip fails.
Specifically, converts the schema to substrait named struct and then converts the named struct back to schema, the reconstructed schema will always have a timestamp column with timezone being "UTC", other than the original timezone.
To Reproduce
The following test fails. Note, there's a timestamp column with non-UTC timezone.
Expected behavior
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: