microsoft / microsoft/snmalloc

Persistent Memory Growth / Failure to Deallocate under high Object Store Load (DataFusion + delta-rs)

Open
#846 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
2k
Forks
138
Avg merge
11h 19m
Merged PRs (30d)
5

Description

Description:

I am experiencing a memory "leak" (unbounded growth) when using snmalloc as the global allocator in a Rust-based query server. The application uses DataFusion and delta-rs to query object storage.

The Issue:

Memory is not being released back to the OS (or reused effectively) at the end of a REST request lifecycle. This leads to a steady climb in RSS until the process is OOM killed.

With jemalloc: Memory is reclaimed/recycled correctly.
With snmalloc: Memory usage climbs linearly and crashes.

Environment:

OS: Linux
snmalloc-rs version: 0.3.8
Relevant Crates: DataFusion, delta-rs (which relies heavily on arrow and FFI).
Runtime: [Tokio 1.48]

Observations:

Interestingly, in the snmalloc trace, the OOM occurs well before the "Limits" (32GB) defined in our monitoring. It seems the allocator is struggling with the specific allocation patterns of DataFusion's execution plan (large buffers for Arrow record batches).

Request:

Are there specific configurations for snmalloc or known issues with the way it interacts with large, short-lived Arrow buffers that might prevent timely deallocation?

Refrence links

https://github.com/delta-io/delta-rs/issues/4241#issuecomment-4398471570
https://github.com/SchrodingerZhu/snmalloc-rs/issues/210 (Opened same issue in rust binding repo)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the reported REST-request workload under jemalloc and snmalloc, using the Linux environment and the DataFusion/delta-rs setup described. Review the linked delta-rs and snmalloc-rs discussions; done means documenting whether a snmalloc configuration or interaction is responsible and identifying a reproducible next step.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.