Mismatch in timestamp format between dataframe and recordbatch when read from a delta table
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
I'm trying to read an pre-existing delta table (using delta-rs as the TableProvider) .
when I do `self.ctx.read_table(table).unwrap();` I get a dataframe where a timestamp field is `data_type: Timestamp(Microsecond, None)`.
Now I collect recordbatches by `df.collect().await;` here when I inspect the schema of a record batch I see that the datatype for the field has changed to `data_type: Timestamp(Nanosecond, None)`
When I use the DeltaTable writer (from delta.rs) its comparing doing a diff of the arrow schema's between the table and the recordbatch and this shows a mistmatch. the only difference was this and the write fails.
### To Reproduce
1. Create an deltatable (not using datafusion)
2. open it using deltafusion and collect the dataframe
3. compare the schema between the delta table and record bactch.
4
### Expected behavior
The recordbatch schema should be `data_type: Timestamp(Microsecond, None)`.
### Additional context
_No response_
Contributor guide
Research direction
Start by reproducing the mismatch with the listed read_table and df.collect() steps, then inspect the schema conversion path between the DataFrame and collected RecordBatch. Done means the collected timestamp remains Timestamp(Microsecond, None) so the DeltaTable writer's schema comparison succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100