apache / apache/datafusion-sqlparser-rs
[EPIC] Improve sqlparser performance
- 主要言語
- Rust
- スター
- 3.5k
- フォーク
- 772
- 平均マージ
- 4日 9時間
- マージ済み PR(30日)
- 17
説明
## What problem are you trying to solve?
Normally, in a SQL processing system, parsing SQL is not a major bottleneck compared to actually processing data. That being said, given how many SQL strings are parsed by this crate, I think there is significant benefit to improving the performance of the SQL parser in this crate.
That being said, I also think it is important to minimize the impact on downstream crates as much as possible.
Recently, we started [introducing locations into the parser](https://github.com/apache/datafusion-sqlparser-rs/pull/1435) (thanks again @Nyrox!), which we found slows things down a bit (see https://github.com/apache/datafusion-sqlparser-rs/pull/1435#issuecomment-2500664144).
Thankfully, I think there is significant room for improvement. As as part of the adding location information, I spent some time profiling and I think there are some obvious ways to improve the performance without impacting downstream crates.
Here is the flamegraph for anyone who is interested (you can download it locally to get zoom / etc):
[fixed-flamegraph](https://github.com/user-attachments/assets/7ceae3a0-dec2-462d-9a00-0b49afd9d2bc)

## What would you like to see?
The idea would be
1. Run the benchmarks (instructions in https://github.com/apache/datafusion-sqlparser-rs/pull/1555)
2. Maybe add additional benchmarks so they are more representative
3. Improve the benchmarks
## Ideas to improve performance:
- The most obvious one is to next_token / peek to not clone each `Token` (which involves copying strings)L: https://github.com/apache/datafusion-sqlparser-rs/issues/1558
- https://github.com/apache/datafusion-sqlparser-rs/issues/1381
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
PR #1555 のベンチマーク手順から始めて、既存のベンチマークを実行し、その後、リンクされている flamegraph と、issues #1558 および #1381 のパフォーマンスに関するアイデアを確認します。必要な箇所に代表的なベンチマークを追加し、下流の crates に重大な影響を与えることなく parser のパフォーマンスを改善できれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- rust
- 領域
- databases, performance
- issue の種類
- リファクタリング
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 25/100