AbsaOSS / AbsaOSS/ABRiS

Memory leak on Spark 4 / Avro 1.12: AvroSchemaUtils.parse re-parses schema every micro-batch, never caches

Ouverte
#379 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Scala
Étoiles
242
Forks
84
Merge moyen
10 h 12 min
PR mergées (30 j)
2

Description

I've had my own spark4 version of ABRiS running in test for a while, and noticed a constant increase in memory use. After a bit of testing and debugging I ended up with the following findings - which appears to be very much present in 7.0.0-RC1.

Avro 1.12 quietly flipped a default — fast-reader is now on unless explicitly disabled:

```
// Avro 1.11.2 (Spark 3.5.x pulls this) — fast reader OFF by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP));

// Avro 1.12.x (Spark 4.x pulls this) — fast reader ON by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP, "true"));
```
That routes every read through GenericData's singleton FastReaderBuilder, whose cache has a pre-existing bug ([AVRO-3524](https://issues.apache.org/jira/browse/AVRO-3524)): entries are weak-keyed by Schema, but the cached value holds a strong ref back to that same key, so nothing is ever collectible.

AvroSchemaUtils.parse calls new Schema.Parser().parse(schema) fresh every time with no caching. On Spark 3/Avro 1.11 this was harmless — fast-reader was off, cache unused. On Spark 4/Avro 1.12 it means every Structured Streaming micro-batch (which re-plans AvroDataToCatalyst, re-triggering the parse) feeds a brand-new, never-evictable Schema into that cache — unbounded leak, OOM after ~12h on a long-running job in my setup.

Fix — cache by schema string so repeated parses of the same text return the same instance:

```
private val schemaCache = new ConcurrentHashMap[String, Schema]()

def parse(schema: String): Schema =
schemaCache.computeIfAbsent(schema, s => new Schema.Parser().parse(s))
```

Verified: flat memory over 4+ days with this patch vs. unbounded growth (confirmed still present in 7.0.0-RC1) without it.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.