`all_schema()` will skip ExtensionPlan's own schema and fetches schemas from children plans
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Describe the bug**
I found a regression when debugging https://github.com/GreptimeTeam/greptimedb/issues/1136. Since https://github.com/apache/arrow-datafusion/pull/5236 the `all_schema()` won't use the schema of `ExtensionPlan`, but its children's schemas.
**To Reproduce**
Not important
**Expected behavior**
It should use the schema from `ExtensionPlan`
**Additional context**
As @alamb and @jackwener discussed in https://github.com/apache/arrow-datafusion/pull/5236#pullrequestreview-1293793794, `all_schema()` seems unnecessary. It looks like all the usages of `all_schema()` are for naming a column (the collected schemas are passed to `normalize_with_schemas()`). But, only a few plans can have multiple inputs (join, union, and maybe extension). I'm wondering if we can take special care of them inside `normalize_with_schemas()` and remove `all_schema()` (`using_columns` in `normalize_with_schemas()` looks like something is doing this)
Contributor guide
Research direction
Start by reading all_schema() and its usages, then compare the behavior introduced by apache/arrow-datafusion/pull/5236. Trace how normalize_with_schemas() and using_columns() consume the collected schemas, including ExtensionPlan, join, and union; done means ExtensionPlan's own schema is respected and the normalization behavior is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100