apache / apache/datafusion

The `dict_id` was lost when constructing the logic plan.

Open
#6,784 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

One of the simplest sql statements: `select * from table;` In the construction of the logical plan, `Projection` will use `to_field` at the bottom to construct `DFField`, and then `to_field` ignore the `dict_id`. This will lead to encoding errors when using IPC if there are dictionary columns.
We are glad to contribute to the community and solve this problem.
To solve this problem, it may be necessary to add interfaces to the `ExprSchema`, for example by adding `dict_is_ordered` and `dict_id` interfaces, or by adding a direct `get_dffield` interface. Or there is a better way other than the two mentioned above. Both methods have a certain amount of work and we are not sure which one to use or if there is a better way. We hope community can provide some comments and help.

Here is a draft of one of the methods:
https://github.com/CeresDB/arrow-datafusion/pull/3

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at logical-plan construction for Projection, especially ExprSchema and its to_field path, where the issue says dict_id is dropped. Review the draft approach linked in the issue and reproduce the IPC encoding failure with a dictionary column. Done means the logical plan preserves dict_id through DFField construction and the encoding error is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.