apache / apache/datafusion-sqlparser-rs
Eliminating whitespace from the parser logic
- 主要语言
- Rust
- 星标
- 3.5k
- 派生
- 772
- 平均合并
- 4 天 9 小时
- 30 天内合并 PR
- 17
描述
Hi,
At this time, whitespace tokens are stored in the parser, and are then filtered out in several distinct points in the parser logic, such as:
* https://github.com/apache/datafusion-sqlparser-rs/blob/67684c84d4c2589356c411ea4917dcf1defcd77c/src/parser/mod.rs#L4032-L4049
* https://github.com/apache/datafusion-sqlparser-rs/blob/67684c84d4c2589356c411ea4917dcf1defcd77c/src/parser/mod.rs#L4055-L4069
* https://github.com/apache/datafusion-sqlparser-rs/blob/67684c84d4c2589356c411ea4917dcf1defcd77c/src/parser/mod.rs#L4077-L4094
* https://github.com/apache/datafusion-sqlparser-rs/blob/67684c84d4c2589356c411ea4917dcf1defcd77c/src/parser/mod.rs#L4149-L4160
* https://github.com/apache/datafusion-sqlparser-rs/blob/67684c84d4c2589356c411ea4917dcf1defcd77c/src/parser/mod.rs#L4183-L4202
and many more.
SQL, as far as I know, is not a language that cares about spaces like Python - it should be safe to remove all concepts of whitespaces after [the tokenization process](https://github.com/apache/datafusion-sqlparser-rs/blob/67684c84d4c2589356c411ea4917dcf1defcd77c/src/tokenizer.rs#L937-L942) is complete, and this should:
* Reduce memory requirements, as whitespace tokens would not be stored anymore
* Significantly simplify parser logic by removing all of the whitespace-related logic from the parser
* Move the parser closer to a streaming logic, but that will require many more PRs
Since such a PR would require quite a bit of effort on my part, I would appreciate some feedback on it before moving forward with it.
@iffyio do you happen to have any opinion regarding such a refactoring?
Ciao,
Luca
贡献指南
这个仓库没有索引到贡献指南
调研方向
首先阅读 src/tokenizer.rs 中链接的 tokenizer 部分,以及 src/parser/mod.rs 中处理空白的示例。跟踪空白 token 如何从 tokenization 流向 parser filtering,然后查看相关的 parser 测试(如果找到)。当提议的空白处理重构得到一致定义,并且 parser 行为仍有测试覆盖时,即视为完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- rust, sql
- 领域
- compilers, databases
- Issue 类型
- 重构
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100