apache / apache/datafusion-sqlparser-rs

[EPIC] Improve sqlparser performance

Aperta
#1,557 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Rust
Stelle
3.5k
Fork
772
Merge medio
4g 9h
PR unite (30g)
17

Descrizione

## What problem are you trying to solve?
Normally, in a SQL processing system, parsing SQL is not a major bottleneck compared to actually processing data. That being said, given how many SQL strings are parsed by this crate, I think there is significant benefit to improving the performance of the SQL parser in this crate.

That being said, I also think it is important to minimize the impact on downstream crates as much as possible.

Recently, we started [introducing locations into the parser](https://github.com/apache/datafusion-sqlparser-rs/pull/1435) (thanks again @Nyrox!), which we found slows things down a bit (see https://github.com/apache/datafusion-sqlparser-rs/pull/1435#issuecomment-2500664144).

Thankfully, I think there is significant room for improvement. As as part of the adding location information, I spent some time profiling and I think there are some obvious ways to improve the performance without impacting downstream crates.

Here is the flamegraph for anyone who is interested (you can download it locally to get zoom / etc):

[fixed-flamegraph](https://github.com/user-attachments/assets/7ceae3a0-dec2-462d-9a00-0b49afd9d2bc)

![fixed-flamegraph](https://github.com/user-attachments/assets/7ceae3a0-dec2-462d-9a00-0b49afd9d2bc)

## What would you like to see?
The idea would be
1. Run the benchmarks (instructions in https://github.com/apache/datafusion-sqlparser-rs/pull/1555)
2. Maybe add additional benchmarks so they are more representative
3. Improve the benchmarks

## Ideas to improve performance:
- The most obvious one is to next_token / peek to not clone each `Token` (which involves copying strings)L: https://github.com/apache/datafusion-sqlparser-rs/issues/1558
- https://github.com/apache/datafusion-sqlparser-rs/issues/1381

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia con le istruzioni sui benchmark in PR #1555 ed esegui i benchmark esistenti, quindi esamina il flamegraph collegato e le idee sulle prestazioni nelle issue #1558 e #1381. Il lavoro è completato quando vengono aggiunti benchmark rappresentativi dove necessario e vengono migliorate le prestazioni del parser senza influire in modo sostanziale sui crate downstream.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
databases, performance
Tipo di issue
Refactoring
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.