apache / apache/arrow-java

[Java] Add API for getBufferSizeFor() with density to BaseVariableWidthVector

Aperta
#389 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Type: enhancement
Lingua principale
Java
Stelle
94
Fork
152
Merge medio
3g 16h
PR unite (30g)
11

Descrizione

Following the discussion on https://github.com/apache/arrow/pull/9187.

Proposed API in BaseVariableWidthVector.java:

```java

/**
* Get the potential buffer size for a particular number of records and density.
* @param valueCount desired number of elements in the vector
* @param density average number of bytes per variable width element
* @return estimated size of underlying buffers if the vector holds
* a given number of elements
*/
public int getBufferSizeFor(final int valueCount, double density)
```

The current `getBufferSizeFor(int valueCount)` for BaseVariableWidthVector requires that validity and offset vectors have already been allocated for at least the given `valueCount`. If the aim of this method is to estimate memory usage for a value count, it's not very useful because it can only give sizes for less than or equal value counts in the currently allocated vector.

A better approach for approximating memory usage is to include a density argument, along with value count. Then the buffer estimate does not require the validity and offset vector to have any allocation. This also is inline with `setInitialCapacity(int valueCount, double density)`

NOTE: this API should also be added to BaseLargeVariableWidthVector and possibly BaseRepeatedValueVector(Large) as well.

**Reporter**: [Bryan Cutler](https://issues.apache.org/jira/browse/ARROW-11739) / @BryanCutler

**Note**: *This issue was originally created as [ARROW-11739](https://issues.apache.org/jira/browse/ARROW-11739). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con BaseVariableWidthVector.java e confronta il getBufferSizeFor(int valueCount) esistente con setInitialCapacity(int valueCount, double density). Esamina la discussione della pull request collegata, quindi verifica se la stessa API si applica a BaseLargeVariableWidthVector e BaseRepeatedValueVector(Large); il lavoro è completato quando l’API basata sulla densità richiesta è definita in modo coerente in tutte le classi di vettori applicabili.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
backend-api-design, data
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.