AbsaOSS / AbsaOSS/ABRiS

Memory leak on Spark 4 / Avro 1.12: AvroSchemaUtils.parse re-parses schema every micro-batch, never caches

Aperta
#379 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Scala
Stelle
242
Fork
84
Merge medio
10h 12m
PR unite (30g)
2

Descrizione

I've had my own spark4 version of ABRiS running in test for a while, and noticed a constant increase in memory use. After a bit of testing and debugging I ended up with the following findings - which appears to be very much present in 7.0.0-RC1.

Avro 1.12 quietly flipped a default — fast-reader is now on unless explicitly disabled:

```
// Avro 1.11.2 (Spark 3.5.x pulls this) — fast reader OFF by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP));

// Avro 1.12.x (Spark 4.x pulls this) — fast reader ON by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP, "true"));
```
That routes every read through GenericData's singleton FastReaderBuilder, whose cache has a pre-existing bug ([AVRO-3524](https://issues.apache.org/jira/browse/AVRO-3524)): entries are weak-keyed by Schema, but the cached value holds a strong ref back to that same key, so nothing is ever collectible.

AvroSchemaUtils.parse calls new Schema.Parser().parse(schema) fresh every time with no caching. On Spark 3/Avro 1.11 this was harmless — fast-reader was off, cache unused. On Spark 4/Avro 1.12 it means every Structured Streaming micro-batch (which re-plans AvroDataToCatalyst, re-triggering the parse) feeds a brand-new, never-evictable Schema into that cache — unbounded leak, OOM after ~12h on a long-running job in my setup.

Fix — cache by schema string so repeated parses of the same text return the same instance:

```
private val schemaCache = new ConcurrentHashMap[String, Schema]()

def parse(schema: String): Schema =
schemaCache.computeIfAbsent(schema, s => new Schema.Parser().parse(s))
```

Verified: flat memory over 4+ days with this patch vs. unbounded growth (confirmed still present in 7.0.0-RC1) without it.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.