AbsaOSS / AbsaOSS/ABRiS

Memory leak on Spark 4 / Avro 1.12: AvroSchemaUtils.parse re-parses schema every micro-batch, never caches

未关闭
#379 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Scala
星标
242
派生
84
平均合并
10 小时 12 分钟
30 天内合并 PR
2

描述

I've had my own spark4 version of ABRiS running in test for a while, and noticed a constant increase in memory use. After a bit of testing and debugging I ended up with the following findings - which appears to be very much present in 7.0.0-RC1.

Avro 1.12 quietly flipped a default — fast-reader is now on unless explicitly disabled:

```
// Avro 1.11.2 (Spark 3.5.x pulls this) — fast reader OFF by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP));

// Avro 1.12.x (Spark 4.x pulls this) — fast reader ON by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP, "true"));
```
That routes every read through GenericData's singleton FastReaderBuilder, whose cache has a pre-existing bug ([AVRO-3524](https://issues.apache.org/jira/browse/AVRO-3524)): entries are weak-keyed by Schema, but the cached value holds a strong ref back to that same key, so nothing is ever collectible.

AvroSchemaUtils.parse calls new Schema.Parser().parse(schema) fresh every time with no caching. On Spark 3/Avro 1.11 this was harmless — fast-reader was off, cache unused. On Spark 4/Avro 1.12 it means every Structured Streaming micro-batch (which re-plans AvroDataToCatalyst, re-triggering the parse) feeds a brand-new, never-evictable Schema into that cache — unbounded leak, OOM after ~12h on a long-running job in my setup.

Fix — cache by schema string so repeated parses of the same text return the same instance:

```
private val schemaCache = new ConcurrentHashMap[String, Schema]()

def parse(schema: String): Schema =
schemaCache.computeIfAbsent(schema, s => new Schema.Parser().parse(s))
```

Verified: flat memory over 4+ days with this patch vs. unbounded growth (confirmed still present in 7.0.0-RC1) without it.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。