apache / apache/parquet-java

Off heap memory leaks with large binary fields using Snappy

Offen
#2,114 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Component: Java Component: Parquet Priority: Major Type: bug
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.6k
Ø Merge
3 T. 12 Std.
Gemergte PRs (30 T.)
33

Beschreibung

When I write a large pages (~100MB) that contains large binary fields (~1MB), the java application uses an unexpected amount of off-heap memory (1.2GB)

This problem was identified when using the `AvroParquetWriter` but its source lies in the parquet-hadoop submodule.

Diving a little bit deeper shows the following:
- writing fields into the ParquetWriter creates a SequenceBytesIn which is actually just a list of `BytesInput` for each field. When calling `bytes.writeAllTo(cos)` in the `CodecFactory`, it actually writes one `ByteInput` (which contains a single field) at a time.
- the `SnappyCompressor` receives the data in `setInput` one large field at a time. This calls `ByteBuffer.allocateDirect` each time with a growing size. But as the memory is actually allocated off-heap, this does not trigger the garbage collector which only sees small objects on the heap. The actual memory associated with the object is the size of all the fields added to the page until then, so off-heap the memory is growing quadratically.

I did not attach a pull request to this issue because I see multiple mitigation to the issue but I'm not really delighted by any of them:
- merge all the fields into one byte array before pushing them down to the `SnappyCompressor`. For instance we could replace the previous statement in the `CodecFactory` with `BytesInput.from(bytes.toByteArray()).writeAllTo(cos)`. But this generates an extra on-heap allocation the size of the whole page.
- force the `DirectBuffer` to be cleaned up with something like `((DirectBuffer)inputBuffer).cleaner().clean()` after having copied it to the new bigger buffer. The issue here would be that `DirectBuffer` is part of the internal API and is likely to be moved. Using reflexion could make the solution more resilient but is even "hackier" IMHO.

**Reporter**: [Remi Dettai](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=remi.dettai)

**Note**: *This issue was originally created as [PARQUET-1188](https://issues.apache.org/jira/browse/PARQUET-1188). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne im Submodul parquet-hadoop und verfolge ParquetWriter und SequenceBytesIn bis zu CodecFactory und SnappyCompressor. Reproduziere das Problem mit ungefähr 100MB großen Seiten, die binäre Felder von 1MB enthalten, und bewerte anschließend die vorgeschlagenen Lösungswege. Als erledigt gilt die Aufgabe, wenn große Schreibvorgänge kein quadratisches Wachstum des Off-Heap-Speichers mehr verursachen und das Verhalten des Writers erhalten bleibt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
performance
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.