AbsaOSS / AbsaOSS/ABRiS

Memory leak on Spark 4 / Avro 1.12: AvroSchemaUtils.parse re-parses schema every micro-batch, never caches

Đang mở
#379 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Scala
Star
242
Fork
84
Merge trung bình
10 giờ 12 phút
Pull request đã merge (30 ngày)
2

Mô tả

I've had my own spark4 version of ABRiS running in test for a while, and noticed a constant increase in memory use. After a bit of testing and debugging I ended up with the following findings - which appears to be very much present in 7.0.0-RC1.

Avro 1.12 quietly flipped a default — fast-reader is now on unless explicitly disabled:

```
// Avro 1.11.2 (Spark 3.5.x pulls this) — fast reader OFF by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP));

// Avro 1.12.x (Spark 4.x pulls this) — fast reader ON by default:
private boolean fastReaderEnabled = "true".equalsIgnoreCase(System.getProperty(FAST_READER_PROP, "true"));
```
That routes every read through GenericData's singleton FastReaderBuilder, whose cache has a pre-existing bug ([AVRO-3524](https://issues.apache.org/jira/browse/AVRO-3524)): entries are weak-keyed by Schema, but the cached value holds a strong ref back to that same key, so nothing is ever collectible.

AvroSchemaUtils.parse calls new Schema.Parser().parse(schema) fresh every time with no caching. On Spark 3/Avro 1.11 this was harmless — fast-reader was off, cache unused. On Spark 4/Avro 1.12 it means every Structured Streaming micro-batch (which re-plans AvroDataToCatalyst, re-triggering the parse) feeds a brand-new, never-evictable Schema into that cache — unbounded leak, OOM after ~12h on a long-running job in my setup.

Fix — cache by schema string so repeated parses of the same text return the same instance:

```
private val schemaCache = new ConcurrentHashMap[String, Schema]()

def parse(schema: String): Schema =
schemaCache.computeIfAbsent(schema, s => new Schema.Parser().parse(s))
```

Verified: flat memory over 4+ days with this patch vs. unbounded growth (confirmed still present in 7.0.0-RC1) without it.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.