AbsaOSS / AbsaOSS/ABRiS

Memory leak on Spark 4 / Avro 1.12: AvroSchemaUtils.parse re-parses schema every micro-batch, never caches

未關閉
#379 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Scala
星號
242
分支
84
平均合併
10 小時 12 分鐘
30 天內合併 PR
2

描述

I've had my own spark4 version of ABRiS running in test for a while, and noticed a constant increase in memory use. After a bit of testing and debugging I ended up with the following findings - which appears to be very much present in 7.0.0-RC1.

Avro 1.12 quietly flipped a default — fast-reader is now on unless explicitly disabled:

```
// Avro 1.11.2 (Spark 3.5.x pulls this) — fast reader OFF by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP));

// Avro 1.12.x (Spark 4.x pulls this) — fast reader ON by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP, "true"));
```
That routes every read through GenericData's singleton FastReaderBuilder, whose cache has a pre-existing bug ([AVRO-3524](https://issues.apache.org/jira/browse/AVRO-3524)): entries are weak-keyed by Schema, but the cached value holds a strong ref back to that same key, so nothing is ever collectible.

AvroSchemaUtils.parse calls new Schema.Parser().parse(schema) fresh every time with no caching. On Spark 3/Avro 1.11 this was harmless — fast-reader was off, cache unused. On Spark 4/Avro 1.12 it means every Structured Streaming micro-batch (which re-plans AvroDataToCatalyst, re-triggering the parse) feeds a brand-new, never-evictable Schema into that cache — unbounded leak, OOM after ~12h on a long-running job in my setup.

Fix — cache by schema string so repeated parses of the same text return the same instance:

```
private val schemaCache = new ConcurrentHashMap[String, Schema]()

def parse(schema: String): Schema =
schemaCache.computeIfAbsent(schema, s => new Schema.Parser().parse(s))
```

Verified: flat memory over 4+ days with this patch vs. unbounded growth (confirmed still present in 7.0.0-RC1) without it.

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。