apache / apache/arrow-java

[Java] Improve VectorSchemaRoot.getVector(String name) lookup performance

Offen
#1,148 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Type: enhancement
Vorherrschende Sprache
Java
Sterne
94
Forks
152
Ø Merge
3 T. 16 Std.
Gemergte PRs (30 T.)
11

Beschreibung

### Describe the enhancement requested

The Java `VectorSchemaRoot.getVector(String name)` can be quite slow if a schema has many vectors. The code currently loops over all the field vectors until it finds the string match. [Link to code](https://github.com/apache/arrow-java/blob/3bc34b041761081ac32a7cd3b167f9ab8b628677/vector/src/main/java/org/apache/arrow/vector/VectorSchemaRoot.java#L175).

Could we stroe a hash map of the `FieldVector` String name to the actual `FieldVector`? Basically the same thing as the `fieldVectorsMap` that maps the `Field` to `FieldVector`. For really wide datasets the trade off of a bit more memory for faster vector lookups would be really nice to have!

Happy to open up a PR with the improvement, thanks!

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start in vector/src/main/java/org/apache/arrow/vector/VectorSchemaRoot.java at getVector(String name), then inspect the existing fieldVectorsMap and the construction paths for field vectors. Confirm the name-based lookup preserves current behavior and compare it on a schema with many vectors; done means repeated name lookups no longer scan every field vector.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
data, performance
Issue-Typ
Feature
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
72/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.