apache / apache/parquet-java

Avoid per-write virtual dispatch in `DictionaryValuesWriter.shouldFallBack()` by caching the size-exceeded check

Aperta
#3,501 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Type: enhancement
Lingua principale
Java
Stelle
3.1k
Fork
1.6k
Merge medio
3g 12h
PR unite (30g)
33

Descrizione

### Describe the enhancement requested

`DictionaryValuesWriter.shouldFallBack()` is called by `FallbackValuesWriter.checkFallback()` after every single value write. The current implementation dispatches a virtual call to `getDictionarySize()` on every invocation:
```java
public boolean shouldFallBack() {
return dictionaryByteSize > maxDictionaryByteSize || getDictionarySize() > MAX_DICTIONARY_ENTRIES;
}
```
`getDictionarySize()` is an abstract method overridden in each typed subclass (Binary, Long, Double, Integer, Float) to return the backing map's `.size()`. Since `shouldFallBack()` is polled after every write, including writes of duplicate values that do not grow the dictionary, the virtual dispatch and map-size query are redundant work for the common case where most values are already in the dictionary.
Both `dictionaryByteSize` and the dictionary entry count can only increase when a new entry is added (inside the `if (id == -1)` branch of each subclass's write method). Therefore the size-exceeded condition can only transition from `false` to `true` at that exact point.

### Proposal
Replace the per-write check with a cached boolean `dictionarySizeExceeded` flag. Introduce a `checkDictionarySizeLimit(int newDictionarySize)` method that subclass write methods call only when a new dictionary entry is actually added. `shouldFallBack()` then returns the cached flag directly, a simple field read with no virtual dispatch.

### Component(s)

Core

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start with DictionaryValuesWriter.shouldFallBack() and FallbackValuesWriter.checkFallback(), then inspect the Binary, Long, Double, Integer, and Float typed subclass write methods, especially their id == -1 branches. Confirm that the size-exceeded state is updated only when a new dictionary entry is added and that fallback behavior remains unchanged for duplicate writes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
data-engineering
Tipo di issue
Refactoring
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
76/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.