michaelfeil / michaelfeil/infinity

Memory leak: bge-m3 RSS grows from 2 GiB → 28 GiB over hours on AMD W6800 (gfx1030, ROCm 6.4)

Open
#661 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

## Summary
The `michaelf34/infinity:latest-rocm` image running `BAAI/bge-m3` shows progressive GPU VRAM growth on AMD Radeon PRO W6800 (gfx1030, RDNA2, 32 GiB) until it consumes most of the card.

## Environment
- Image: `michaelf34/infinity:latest-rocm` (exact digest available on request)
- Host OS: Ubuntu 22.04 (host running ROCm 6.4.0-47)
- GPU: AMD Radeon PRO W6800 (gfx1030, RDNA2, 32 GiB)
- Model: `BAAI/bge-m3` (sentence embedding, 1024 dimensions)
- Container env: `PYTORCH_HIP_ALLOC_CONF=expandable_segments:True`, `HIP_VISIBLE_DEVICES` unset (single-tenant per host process)
- Container args: `--model-id BAAI/bge-m3 --batch-size 64 --url-prefix /v1 --device cuda` (Infinity CLI flags)

## Observed behavior
- Container starts: GPU memory ~2-4 GiB (bge-m3 weights + activations as expected)
- After several hours of operation handling normal embed traffic: GPU memory climbs to 28+ GiB
- Throughput remains nominal; HTTP /embeddings POST returns 200 OK throughout
- `docker restart` immediately clears the memory back to ~2-4 GiB

## Reproducibility
Easily reproducible. Container survives normal traffic but VRAM grows unboundedly until restart.

## Comparison with gfx1100
- Same image (`latest-rocm`) running on AMD RX 7900 XT (gfx1100, RDNA3, 20 GiB) does NOT exhibit this leak
- gfx1100 instance uses `PYTORCH_HIP_ALLOC_CONF=max_split_size_mb:512,garbage_collection_threshold:0.8` (different allocator config)
- Suggests the leak may be specific to `expandable_segments:True` on RDNA2, possibly an interaction with PyTorch's HIP caching allocator

## Suspected cause
Possibly the same class of allocator-pre-allocation issue documented for vLLM on gfx1100 ([github.com/shawnq-msft/rx9070-qwen-rocm](https://github.com/shawnq-msft/rx9070-qwen-rocm)). However, this presents as gradual growth rather than upfront over-allocation.

## Requested investigation
- Is bge-m3 + `expandable_segments:True` known-problematic on RDNA2?
- Recommended allocator config for production gfx1030 deployments?
- Any planned upstream fix?

## Workaround in our deployment
Scheduled container restart (manual / via watcher self-heal) — not ideal long-term.

## Willing to provide
- More detailed VRAM growth timeline / metrics
- Logs from the leaking instance
- Stable comparison logs from gfx1100

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the michaelf34/infinity:latest-rocm image, BAAI/bge-m3, the /v1 embeddings endpoint, and the reported gfx1030 environment while recording VRAM over time. Compare the expandable_segments:True configuration with the reported gfx1100 setup; done means identifying the leak scope and documenting a production allocator configuration or confirmed upstream fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, pytorch
Domain
api, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.