apache / apache/datafusion-sqlparser-rs
[EPIC] Complete Span (source location) information / feature
- 主要语言
- Rust
- 星标
- 3.5k
- 派生
- 772
- 平均合并
- 4 天 9 小时
- 30 天内合并 PR
- 17
描述
This ticket tracks the work remaining to complete adding source location information into sqlparser
## Background
- @Nyrox and @iffyio introduced the foundations for storing source location information in the AST nodes in https://github.com/apache/datafusion-sqlparser-rs/pull/1435. This information can be used to provide more specific error messages, and potentially syntax highlighting among other great things.
- In order to 1. minimize the disruption to downstream projects that use `sqlparser-rs` and 2. avoid a single massive PR and 3. work together as a community, we are implementing this feature incrementally over several releases.
Let's use this ticket to organize needed / remaining work. If you find additional features are needed / issues, please leave a comment on this ticket
## Source Span Contributing Guidelines
For contributing source spans improvement in addition to the general
[contribution guidelines], please make sure to pay attention to the
following:
- `Ident` always have correct source spans
- We try to minimize downstream breaking changes
- Consider using [`Span::union`] in favor of storing spans on all nodes
- Any metadata added to compute spans must not change semantics (`Eq`, `Ord`, `Hash`, etc.). See [`AttachedToken`] for more information.
[contribution guidelines]: https://github.com/apache/datafusion-sqlparser-rs/blob/main/README.md#contributing
[`Span::union`]: ast::Span::union
[`AttachedToken`]: ast::helpers::attached_token::AttachedToken
When adding support for source spans on a type, consider the impact to consumers of that type and whether your change would require a consumer to do non-trivial changes to their code.
Example of a trivial change
```rust
match node {
ast::Query {
field1,
field2,
location: _, // add a new line to ignored location
}
If adding source spans to a type would require a significant change like wrapping the type, please open an issue to discuss.
# AST Node Equality and Hashes
When adding tokens to AST nodes, make sure to store them using the [AttachedToken](https://docs.rs/sqlparser/latest/sqlparser/ast/helpers/struct.AttachedToken.html) (TODO UPDATE SOURCE REFERENCE)to ensure that semantically equivalent AST nodes compare as equal and hash to the same value. i.e. `select 5` and `SELECT 5` would compare as different `Select` nodes, if the select token was stored directly. f.e.
```rust
struct Select {
select_token: AttachedToken, // only used for spans
/// remaining fields
field1,
field2,
...
}
```
Some high level work (list from https://github.com/apache/datafusion-sqlparser-rs/pull/1435)
- Store keyword `TokenWithLocation` for expressions that currently don't have them
- Implement spans for the rest of the AST, namely `Statement`s
## Tasks
- [ ] https://github.com/apache/datafusion-sqlparser-rs/issues/1563
- [ ] Look into reducing AST size with smaller offset sizes (e.g. `u32` rather than `usize`)
- [ ] Store spans for ast::value::Value
- [x] https://github.com/apache/datafusion-sqlparser-rs/issues/1858
- [ ] https://github.com/apache/datafusion-sqlparser-rs/issues/1548
贡献指南
这个仓库没有索引到贡献指南
调研方向
首先阅读 PR 1435 和 source-span 贡献指南,尤其是 ast::Span::union 和 ast::helpers::attached_token::AttachedToken。检查尚未勾选的任务以及它们提到的 AST 区域,包括 Statements 和 ast::value::Value。完成一项范围明确的剩余任务,同时保持 AST 语义并尽量减少下游更改,即可视为完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- rust, sql
- 领域
- compilers, databases
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100