The `dict_id` was lost when constructing the logic plan.
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
One of the simplest sql statements: `select * from table;` In the construction of the logical plan, `Projection` will use `to_field` at the bottom to construct `DFField`, and then `to_field` ignore the `dict_id`. This will lead to encoding errors when using IPC if there are dictionary columns.
We are glad to contribute to the community and solve this problem.
To solve this problem, it may be necessary to add interfaces to the `ExprSchema`, for example by adding `dict_is_ordered` and `dict_id` interfaces, or by adding a direct `get_dffield` interface. Or there is a better way other than the two mentioned above. Both methods have a certain amount of work and we are not sure which one to use or if there is a better way. We hope community can provide some comments and help.
Here is a draft of one of the methods:
https://github.com/CeresDB/arrow-datafusion/pull/3
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start at logical-plan construction for Projection, especially ExprSchema and its to_field path, where the issue says dict_id is dropped. Review the draft approach linked in the issue and reproduce the IPC encoding failure with a dictionary column. Done means the logical plan preserves dict_id through DFField construction and the encoding error is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100