duckdb / duckdb/duckdb-httpfs

Schema inference occurs before filter pushdown when reading partitioned Parquet files

Open
#22 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

Hello DuckDB team,

Below is my issue report:

Issue: Schema inference occurs before filter pushdown when reading partitioned Parquet files with evolving schemas

I have a table whose schema has evolved over time by adding new columns. For instance, in my dataset, the Parquet files for 2023 have a schema like:

```
year | col_A | col_B
---------------------
2023 | ... | ...
```
while the Parquet files for 2024 have an extended schema:

```
year | col_A | col_B | col_2024
--------------------------------
2024 | ... | ... | ...

```
When I run a query such as:

```

SELECT col_2024
FROM read_parquet('tabla/year=*', hive_partitioning=true)
WHERE year = 2024;
```

the filter pushdown correctly selects only the 2024 files for data. However, DuckDB infers the overall schema from the entire dataset (including a 2023 file) and ends up using the 2023 schema that does not include col_2024. This causes the query to fail because the schema used for the query is missing the col_2024 column.

Ideally, the schema should be determined after applying the filters, or the inference process should account for the filter so that only the matching partitions (and their schema) are used. This would prevent mismatches when the table's schema has evolved over time.

I propose that DuckDB should either:

Infer the schema only from the files that match the filter conditions (in this case, only the 2024 files), or
Perform a two-step process where the partitions are filtered first and then the schema is inferred from the remaining files.
This change would greatly help when dealing with tables that have evolved over time.

Thank you for your attention to this issue, and I look forward to your feedback.

Best regards,

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the provided SELECT against partitioned Parquet files with evolving schemas and hive_partitioning enabled. Trace the schema inference and filter pushdown entry points to determine where partition filters are applied; done means the 2024-only query resolves col_2024 without being affected by the 2023 schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.