Do't throw the error when projected columns are not in the table schema
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
In the case where the table has projected columns (like in `parquet_multiple_partitions` where we project columns from the hive partition directory structure) then the projected columns are not in the table schema.
If the predicate contains any such projected columns then we ignore it anyway. So I think this change is fine but with the one adjustment that we don't return an error. So maybe
```
if let Ok(field) self.table_schema.field_with_name(name) {
// return the null value corresponding to the data type
let null_value = ScalarValue::try_from(field.data_type())?;
return Ok(Expr::Literal(null_value));
}
```
In this case we just won't rewrite the expression but also `FilterCandidateBuilder.projected_columns` will be `true` so we won't crate a row filter predicate for the `Expr`.
_Originally posted by @thinkharderdev in https://github.com/apache/arrow-datafusion/pull/3470#discussion_r970907156_
Contributor guide
Research direction
Locate the expression-rewriting path using table_schema.field_with_name and inspect FilterCandidateBuilder.projected_columns. Reproduce the parquet_multiple_partitions case, then verify that predicates on projected columns are left unre-written without returning an error and do not produce a row filter predicate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100