Schema inference occurs before filter pushdown when reading partitioned Parquet files
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
Hello DuckDB team,
Below is my issue report:
Issue: Schema inference occurs before filter pushdown when reading partitioned Parquet files with evolving schemas
I have a table whose schema has evolved over time by adding new columns. For instance, in my dataset, the Parquet files for 2023 have a schema like:
```
year | col_A | col_B
---------------------
2023 | ... | ...
```
while the Parquet files for 2024 have an extended schema:
```
year | col_A | col_B | col_2024
--------------------------------
2024 | ... | ... | ...
```
When I run a query such as:
```
SELECT col_2024
FROM read_parquet('tabla/year=*', hive_partitioning=true)
WHERE year = 2024;
```
the filter pushdown correctly selects only the 2024 files for data. However, DuckDB infers the overall schema from the entire dataset (including a 2023 file) and ends up using the 2023 schema that does not include col_2024. This causes the query to fail because the schema used for the query is missing the col_2024 column.
Ideally, the schema should be determined after applying the filters, or the inference process should account for the filter so that only the matching partitions (and their schema) are used. This would prevent mismatches when the table's schema has evolved over time.
I propose that DuckDB should either:
Infer the schema only from the files that match the filter conditions (in this case, only the 2024 files), or
Perform a two-step process where the partitions are filtered first and then the schema is inferred from the remaining files.
This change would greatly help when dealing with tables that have evolved over time.
Thank you for your attention to this issue, and I look forward to your feedback.
Best regards,
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the provided SELECT against partitioned Parquet files with evolving schemas and hive_partitioning enabled. Trace the schema inference and filter pushdown entry points to determine where partition filters are applied; done means the 2024-only query resolves col_2024 without being affected by the 2023 schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100