apache / apache/datafusion-sqlparser-rs

[EPIC] Improve sqlparser performance

Ouverte
#1,557 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Rust
Étoiles
3.5k
Forks
772
Merge moyen
4 j 9 h
PR mergées (30 j)
17

Description

## What problem are you trying to solve?
Normally, in a SQL processing system, parsing SQL is not a major bottleneck compared to actually processing data. That being said, given how many SQL strings are parsed by this crate, I think there is significant benefit to improving the performance of the SQL parser in this crate.

That being said, I also think it is important to minimize the impact on downstream crates as much as possible.

Recently, we started [introducing locations into the parser](https://github.com/apache/datafusion-sqlparser-rs/pull/1435) (thanks again @Nyrox!), which we found slows things down a bit (see https://github.com/apache/datafusion-sqlparser-rs/pull/1435#issuecomment-2500664144).

Thankfully, I think there is significant room for improvement. As as part of the adding location information, I spent some time profiling and I think there are some obvious ways to improve the performance without impacting downstream crates.

Here is the flamegraph for anyone who is interested (you can download it locally to get zoom / etc):

[fixed-flamegraph](https://github.com/user-attachments/assets/7ceae3a0-dec2-462d-9a00-0b49afd9d2bc)

![fixed-flamegraph](https://github.com/user-attachments/assets/7ceae3a0-dec2-462d-9a00-0b49afd9d2bc)

## What would you like to see?
The idea would be
1. Run the benchmarks (instructions in https://github.com/apache/datafusion-sqlparser-rs/pull/1555)
2. Maybe add additional benchmarks so they are more representative
3. Improve the benchmarks

## Ideas to improve performance:
- The most obvious one is to next_token / peek to not clone each `Token` (which involves copying strings)L: https://github.com/apache/datafusion-sqlparser-rs/issues/1558
- https://github.com/apache/datafusion-sqlparser-rs/issues/1381

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par les instructions des benchmarks dans PR #1555 et exécutez les benchmarks existants, puis examinez le flamegraph associé ainsi que les idées d’amélioration des performances dans les issues #1558 et #1381. La tâche est considérée comme terminée lorsque des benchmarks représentatifs ont été ajoutés là où c’est nécessaire et que les performances du parser ont été améliorées sans affecter sensiblement les crates en aval.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
rust
Domaine
databases, performance
Type d'issue
Refactorisation
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.