apache / apache/parquet-java

Off heap memory leaks with large binary fields using Snappy

Aperta
#2,114 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Component: Java Component: Parquet Priority: Major Type: bug
Lingua principale
Java
Stelle
3.1k
Fork
1.6k
Merge medio
3g 12h
PR unite (30g)
33

Descrizione

When I write a large pages (~100MB) that contains large binary fields (~1MB), the java application uses an unexpected amount of off-heap memory (1.2GB)

This problem was identified when using the `AvroParquetWriter` but its source lies in the parquet-hadoop submodule.

Diving a little bit deeper shows the following:
- writing fields into the ParquetWriter creates a SequenceBytesIn which is actually just a list of `BytesInput` for each field. When calling `bytes.writeAllTo(cos)` in the `CodecFactory`, it actually writes one `ByteInput` (which contains a single field) at a time.
- the `SnappyCompressor` receives the data in `setInput` one large field at a time. This calls `ByteBuffer.allocateDirect` each time with a growing size. But as the memory is actually allocated off-heap, this does not trigger the garbage collector which only sees small objects on the heap. The actual memory associated with the object is the size of all the fields added to the page until then, so off-heap the memory is growing quadratically.

I did not attach a pull request to this issue because I see multiple mitigation to the issue but I'm not really delighted by any of them:
- merge all the fields into one byte array before pushing them down to the `SnappyCompressor`. For instance we could replace the previous statement in the `CodecFactory` with `BytesInput.from(bytes.toByteArray()).writeAllTo(cos)`. But this generates an extra on-heap allocation the size of the whole page.
- force the `DirectBuffer` to be cleaned up with something like `((DirectBuffer)inputBuffer).cleaner().clean()` after having copied it to the new bigger buffer. The issue here would be that `DirectBuffer` is part of the internal API and is likely to be moved. Using reflexion could make the solution more resilient but is even "hackier" IMHO.

**Reporter**: [Remi Dettai](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=remi.dettai)

**Note**: *This issue was originally created as [PARQUET-1188](https://issues.apache.org/jira/browse/PARQUET-1188). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia nel sottomodulo parquet-hadoop, seguendo ParquetWriter e SequenceBytesIn fino a CodecFactory e SnappyCompressor. Riproduci il problema con pagine di circa 100MB contenenti campi binari da 1MB, quindi valuta i percorsi di mitigazione proposti. Il lavoro è completato quando le scritture di grandi dimensioni non causano più una crescita quadratica della memoria off-heap, preservando il comportamento del writer.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
performance
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.