michaelfeil / michaelfeil/infinity
Memory leak: bge-m3 RSS grows from 2 GiB → 28 GiB over hours on AMD W6800 (gfx1030, ROCm 6.4)
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
The `michaelf34/infinity:latest-rocm` image running `BAAI/bge-m3` shows progressive GPU VRAM growth on AMD Radeon PRO W6800 (gfx1030, RDNA2, 32 GiB) until it consumes most of the card.
## Environment
- Image: `michaelf34/infinity:latest-rocm` (exact digest available on request)
- Host OS: Ubuntu 22.04 (host running ROCm 6.4.0-47)
- GPU: AMD Radeon PRO W6800 (gfx1030, RDNA2, 32 GiB)
- Model: `BAAI/bge-m3` (sentence embedding, 1024 dimensions)
- Container env: `PYTORCH_HIP_ALLOC_CONF=expandable_segments:True`, `HIP_VISIBLE_DEVICES` unset (single-tenant per host process)
- Container args: `--model-id BAAI/bge-m3 --batch-size 64 --url-prefix /v1 --device cuda` (Infinity CLI flags)
## Observed behavior
- Container starts: GPU memory ~2-4 GiB (bge-m3 weights + activations as expected)
- After several hours of operation handling normal embed traffic: GPU memory climbs to 28+ GiB
- Throughput remains nominal; HTTP /embeddings POST returns 200 OK throughout
- `docker restart` immediately clears the memory back to ~2-4 GiB
## Reproducibility
Easily reproducible. Container survives normal traffic but VRAM grows unboundedly until restart.
## Comparison with gfx1100
- Same image (`latest-rocm`) running on AMD RX 7900 XT (gfx1100, RDNA3, 20 GiB) does NOT exhibit this leak
- gfx1100 instance uses `PYTORCH_HIP_ALLOC_CONF=max_split_size_mb:512,garbage_collection_threshold:0.8` (different allocator config)
- Suggests the leak may be specific to `expandable_segments:True` on RDNA2, possibly an interaction with PyTorch's HIP caching allocator
## Suspected cause
Possibly the same class of allocator-pre-allocation issue documented for vLLM on gfx1100 ([github.com/shawnq-msft/rx9070-qwen-rocm](https://github.com/shawnq-msft/rx9070-qwen-rocm)). However, this presents as gradual growth rather than upfront over-allocation.
## Requested investigation
- Is bge-m3 + `expandable_segments:True` known-problematic on RDNA2?
- Recommended allocator config for production gfx1030 deployments?
- Any planned upstream fix?
## Workaround in our deployment
Scheduled container restart (manual / via watcher self-heal) — not ideal long-term.
## Willing to provide
- More detailed VRAM growth timeline / metrics
- Logs from the leaking instance
- Stable comparison logs from gfx1100
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with the michaelf34/infinity:latest-rocm image, BAAI/bge-m3, the /v1 embeddings endpoint, and the reported gfx1030 environment while recording VRAM over time. Compare the expandable_segments:True configuration with the reported gfx1100 setup; done means identifying the leak scope and documenting a production allocator configuration or confirmed upstream fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, pytorch
- Domain
- api, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100