apache / apache/parquet-java

`RequiresFallback.isCompressionSatisfying` is too aggressive with current default page size

Aperta
#3,479 1 commento 1 reazione 0 assegnatari Vedi su GitHub
Type: enhancement
Lingua principale
Java
Stelle
3.1k
Fork
1.6k
Merge medio
3g 12h
PR unite (30g)
33

Descrizione

### Describe the enhancement requested

An issue recently was brought up in arrow-rs (https://github.com/apache/arrow-rs/pull/9700) which brought to my attention the existence of `isCompressionSatisfying` in the `RequiresFallback` interface. In short, after accumulating a page worth of data, `isCompressionSatisfying` is called to see if dictionary encoding is actually compressing the data at all, and if not, then the encoder falls back immediately to the fallback encoder. As far as I could determine, this behavior was introduced very early on, before the advent of the page indexes, so IIRC the page size would have been significantly larger. With page indexes, however, this function is now called after only 20000 rows have been processed. A column with a moderate cardinality might not yet have produced enough repeating values to lead this function to conclude it's best to continue using a dictionary.

For example, a dataframe with an int64 column consisting of one million values mod'd with 32768 will end up ditching dictionary encoding completely, and produce a column chunk of 8.4MB. If the page row count is bumped up to 128k, then dictionary encoding is used throughout and the resultant column chunk is only 2.2MB.

Sadly, it does not appear that this behavior is configurable, so short of increasing the page row count, its behavior cannot be modified.

I can see the need for this type of heuristic, but I think it needs to be modified in light of the current defaults resulting in far too few samples with which to determine if dictionary encoding is beneficial or not. If collecting more samples before falling back is not practical, there should at least be a configuration setting to disable this check.

### Component(s)

Core

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia dal punto di ingresso RequiresFallback.isCompressionSatisfying e traccia come interagiscono gli indici di pagina, il valore predefinito di 20,000 righe e il fallback del dizionario. Confronta i risultati riportati per 20,000 righe e 128,000 righe, quindi determina se la modifica concordata è un’euristica di campionamento rivista o un’opzione di configurazione; il lavoro è completato quando il comportamento è coperto per dati a cardinalità moderata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
data-engineering
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.