apache / apache/datafusion

`all_schema()` will skip ExtensionPlan's own schema and fetches schemas from children plans

Open
#5,513 0 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Describe the bug**

I found a regression when debugging https://github.com/GreptimeTeam/greptimedb/issues/1136. Since https://github.com/apache/arrow-datafusion/pull/5236 the `all_schema()` won't use the schema of `ExtensionPlan`, but its children's schemas.

**To Reproduce**

Not important

**Expected behavior**

It should use the schema from `ExtensionPlan`

**Additional context**

As @alamb and @jackwener discussed in https://github.com/apache/arrow-datafusion/pull/5236#pullrequestreview-1293793794, `all_schema()` seems unnecessary. It looks like all the usages of `all_schema()` are for naming a column (the collected schemas are passed to `normalize_with_schemas()`). But, only a few plans can have multiple inputs (join, union, and maybe extension). I'm wondering if we can take special care of them inside `normalize_with_schemas()` and remove `all_schema()` (`using_columns` in `normalize_with_schemas()` looks like something is doing this)

Contributor guide

Open the contributing guide

Research direction

Start by reading all_schema() and its usages, then compare the behavior introduced by apache/arrow-datafusion/pull/5236. Trace how normalize_with_schemas() and using_columns() consume the collected schemas, including ExtensionPlan, join, and union; done means ExtensionPlan's own schema is respected and the normalization behavior is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.