apache / apache/datafusion-sqlparser-rs

Optimize `Token::make_word`

Offen
#1,588 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Rust
Sterne
3.5k
Forks
772
Ø Merge
4 T. 9 Std.
Gemergte PRs (30 T.)
17

Beschreibung

While working on #1587 I noticed that Instruments is showing `Token::make_word` as the second hottest single function, right after `alloc::raw_vec::finish_grow`.

Looking into the implementation I saw that its just doing a binary search across all keywords to find if its a known keyword or not. This is a fairly classical case where we have a known set of strings and want to check if a given string is in that list. There are a bunch of ways that we could speed this up. This issue is to figure out a good compromise between those possible speedups and other project constraints like maintaining a `no_std` ability.

My [first approach](https://github.com/apache/datafusion-sqlparser-rs/commit/4551933dc0a9e892e412be5ca0022a124859dad0) at speeding this up was to create a table for the first byte in every keyword to reduce the number of entries that need to be searched. This small optimization managed to shave off about 400ms of time (of the 1.4ish seconds total).

However, there are other approaches that could speed this up even more. Either by generating parsing/lookup tables or using something like [phf](https://crates.io/crates/phf) to do the heavy lifting for us.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with Token::make_word and the linked first approach commit to understand the current keyword lookup and its performance impact. Compare possible lookup-table or phf-based approaches while preserving the project's no_std constraint, then use the reported Instruments timing to verify that the chosen design improves performance without changing keyword recognition.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
rust
Bereich
compilers
Issue-Typ
Refactoring
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
30/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.