apache / apache/datafusion

Do't throw the error when projected columns are not in the table schema

Open
#3,492 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

In the case where the table has projected columns (like in `parquet_multiple_partitions` where we project columns from the hive partition directory structure) then the projected columns are not in the table schema.

If the predicate contains any such projected columns then we ignore it anyway. So I think this change is fine but with the one adjustment that we don't return an error. So maybe

```
if let Ok(field) self.table_schema.field_with_name(name) {
// return the null value corresponding to the data type
let null_value = ScalarValue::try_from(field.data_type())?;
return Ok(Expr::Literal(null_value));
}
```

In this case we just won't rewrite the expression but also `FilterCandidateBuilder.projected_columns` will be `true` so we won't crate a row filter predicate for the `Expr`.

_Originally posted by @thinkharderdev in https://github.com/apache/arrow-datafusion/pull/3470#discussion_r970907156_

Contributor guide

Open the contributing guide

Research direction

Locate the expression-rewriting path using table_schema.field_with_name and inspect FilterCandidateBuilder.projected_columns. Reproduce the parquet_multiple_partitions case, then verify that predicates on projected columns are left unre-written without returning an error and do not produce a row filter predicate.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.