apache / apache/datafusion

ListingTable cannot handle partition evolution

Open
#13,270 12 comments 0 reactions 1 assignee Claimed by @zhuqi-lucas View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

With CSV:

```shell
echo "a,b\n1,2" > data1.csv
mkdir a=2
echo "b\n3" > a=2/data2.csv
datafusion-cli
> SELECT * FROM '**/*.csv';
Arrow error: Csv error: incorrect number of fields for line 1, expected 2 got 1
```

With Parquet:

```python
import os
import polars as pl

pl.DataFrame({'a': [1], 'b': [2]}).write_parquet('data1.parquet')
os.mkdir('a=2')
pl.DataFrame({'b': [3]}).write_parquet('a=2/data2.parquet')
```

```shell
datafusion-cli
> SELECT * FROM '**/*.parquet';
+---+---+
| b | a |
+---+---+
| 2 | 1 |
| 3 | |
+---+---+
2 row(s) fetched.
Elapsed 0.055 seconds.
```

### To Reproduce

_No response_

### Expected behavior

Partition evolution is handled and both cases return

```
+---+---+
| b | a |
+---+---+
| 2 | 1 |
| 3 | 2 |
+---+---+
```

### Additional context

Having played around quite a bit with ParquetExec and the SchemaAdapter machinery I think what should happen is:
- Partition values are on a per-file basis, in particular on each `PartitionedFile` and not on the `FileScanConfig`
- Partition values are passed into the SchemaAdapter machinery and for each file it decides if it needs to add a column generated from partition values or not

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.