apache / apache/datafusion

Mismatch in timestamp format between dataframe and recordbatch when read from a delta table

Open
#7,564 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I'm trying to read an pre-existing delta table (using delta-rs as the TableProvider) .

when I do `self.ctx.read_table(table).unwrap();` I get a dataframe where a timestamp field is `data_type: Timestamp(Microsecond, None)`.

Now I collect recordbatches by `df.collect().await;` here when I inspect the schema of a record batch I see that the datatype for the field has changed to `data_type: Timestamp(Nanosecond, None)`

When I use the DeltaTable writer (from delta.rs) its comparing doing a diff of the arrow schema's between the table and the recordbatch and this shows a mistmatch. the only difference was this and the write fails.

### To Reproduce

1. Create an deltatable (not using datafusion)
2. open it using deltafusion and collect the dataframe
3. compare the schema between the delta table and record bactch.
4

### Expected behavior

The recordbatch schema should be `data_type: Timestamp(Microsecond, None)`.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the mismatch with the listed read_table and df.collect() steps, then inspect the schema conversion path between the DataFrame and collected RecordBatch. Done means the collected timestamp remains Timestamp(Microsecond, None) so the DeltaTable writer's schema comparison succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.