apache / apache/datafusion-sqlparser-rs

[EPIC] Complete Span (source location) information / feature

未关闭
#1,548 4 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Rust
星标
3.5k
派生
772
平均合并
4 天 9 小时
30 天内合并 PR
17

描述

This ticket tracks the work remaining to complete adding source location information into sqlparser

## Background
- @Nyrox and @iffyio introduced the foundations for storing source location information in the AST nodes in https://github.com/apache/datafusion-sqlparser-rs/pull/1435. This information can be used to provide more specific error messages, and potentially syntax highlighting among other great things.
- In order to 1. minimize the disruption to downstream projects that use `sqlparser-rs` and 2. avoid a single massive PR and 3. work together as a community, we are implementing this feature incrementally over several releases.

Let's use this ticket to organize needed / remaining work. If you find additional features are needed / issues, please leave a comment on this ticket

## Source Span Contributing Guidelines

For contributing source spans improvement in addition to the general
[contribution guidelines], please make sure to pay attention to the
following:

- `Ident` always have correct source spans

- We try to minimize downstream breaking changes

- Consider using [`Span::union`] in favor of storing spans on all nodes
- Any metadata added to compute spans must not change semantics (`Eq`, `Ord`, `Hash`, etc.). See [`AttachedToken`] for more information.

[contribution guidelines]: https://github.com/apache/datafusion-sqlparser-rs/blob/main/README.md#contributing
[`Span::union`]: ast::Span::union
[`AttachedToken`]: ast::helpers::attached_token::AttachedToken

When adding support for source spans on a type, consider the impact to consumers of that type and whether your change would require a consumer to do non-trivial changes to their code.

Example of a trivial change

```rust
match node {
ast::Query {
field1,
field2,
location: _, // add a new line to ignored location
}

If adding source spans to a type would require a significant change like wrapping the type, please open an issue to discuss.

# AST Node Equality and Hashes

When adding tokens to AST nodes, make sure to store them using the [AttachedToken](https://docs.rs/sqlparser/latest/sqlparser/ast/helpers/struct.AttachedToken.html) (TODO UPDATE SOURCE REFERENCE)to ensure that semantically equivalent AST nodes compare as equal and hash to the same value. i.e. `select 5` and `SELECT 5` would compare as different `Select` nodes, if the select token was stored directly. f.e.

```rust
struct Select {
select_token: AttachedToken, // only used for spans
/// remaining fields
field1,
field2,
...
}
```

Some high level work (list from https://github.com/apache/datafusion-sqlparser-rs/pull/1435)
- Store keyword `TokenWithLocation` for expressions that currently don't have them
- Implement spans for the rest of the AST, namely `Statement`s

## Tasks

- [ ] https://github.com/apache/datafusion-sqlparser-rs/issues/1563
- [ ] Look into reducing AST size with smaller offset sizes (e.g. `u32` rather than `usize`)
- [ ] Store spans for ast::value::Value
- [x] https://github.com/apache/datafusion-sqlparser-rs/issues/1858
- [ ] https://github.com/apache/datafusion-sqlparser-rs/issues/1548

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先阅读 PR 1435 和 source-span 贡献指南,尤其是 ast::Span::union 和 ast::helpers::attached_token::AttachedToken。检查尚未勾选的任务以及它们提到的 AST 区域,包括 Statements 和 ast::value::Value。完成一项范围明确的剩余任务,同时保持 AST 语义并尽量减少下游更改,即可视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
rust, sql
领域
compilers, databases
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。