[New feature] Support exporting memory snapshots compatible with PyTorch's memory viz format
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 28.5k
- Forks
- 2.3k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 62
Description
Hi, I'm currently implementing support for dumping memory snapshots that will be compatible with PyTorch's memory viz [web app][how-to tutorial], to visualize how the memory allocations/deallocations look like without having to recreate the UI.
The usage will be similar to PyTorch's:
- the user will be able to turn memory recording on and off (something like
mx.record_memory_events(enabled=True|False)) - the user will be able to take a snapshot and dump it to a pickle file (
mx.get_memory_snapshot()andmx.dump_memory_snapshot()) - The output pickle file generated from
mx.dump_memory_snapshot()will be loadable directly in the web app
I've been studying how PyTorch implements it, and my current design (naming not finalized + Metal backend-only for now) is roughly:
-
To turn memory recording on (and off):
- a virtual function to turn on/off recording (including traceback capture) and configuration setup is added in the
Allocatorclass (each backend-allocator, e.g.,MetalAllocatorwill implement it) - a private bool member
record_enabledis added to theMetalAllocatorclass; this is the flag to inform whether to create and store a new memory event
- a virtual function to turn on/off recording (including traceback capture) and configuration setup is added in the
-
To record memory events:
- a private function
maybe_record()is added toMetalAllocator, will be called in malloc/free-related functions to create a memory event and add it into an event buffer. The event creation + addition to the buffer only occur ifrecord_enabledis true. - to obtain the context of each event (e.g., primitive name, traceback stack index) , a scope object is created right before
gpu::eval(arr)/cpu::eval(arr)is run (mlx/transforms.cpp) that sets athread_localcurrent-op context.maybe_record()reads this context and adds it onto each memory event. - to capture a traceback (currently only Python stacks),
array::ArrayDesc::init()(if capture is enabled) calls a hook that walks the current call stack (PyEval_GetFrameandPyFrame_GetBack), builds frames{code: PyObject*, line: int}, interns them, and returns an integerstack_idxto be stored on theArrayDesc. During eval, thestack_idxwill be copied into thethread_localcurrent-op context I mentioned earlier
- a private function
-
To take a snapshot, a function will be called to symbolize the frames and add them into the memory events (and pass them to Python to be exported as a pickle file)
Writing this issue to check:
- Would you be interested in this feature landing in MLX? (I'll open a PR once I'm done, if so.)
- Would love thoughts on the design (e.g., things I should be aware of, recommendations on where it should live, particular naming conventions to follow).
Thank you so much in advance!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The main entry points mentioned are the Allocator and MetalAllocator classes, mlx/transforms.cpp, and array::ArrayDesc::init(). Start by tracing allocator malloc/free paths and the eval scope context, then compare the proposed APIs with PyTorch's memory-viz snapshot format. Done means recording can be toggled, snapshots can be exported, and the resulting pickle loads in the linked web app.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python, pytorch
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100