Azure / Azure/AKS

[Feature] GPU Memory Profiling

Open
#5,770 3 comments 0 reactions 1 assignee Assigned to @mayasingh17 View on GitHub
feature-request GPU
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
As a platform engineer the # 1 pain point of dealing with llm inference in general, undoubtedly, is OOM prevention. Without paging support for GPU memory, the entire GPU memory management needs to be static, e.g., we decide the maximum KV cache size before start up the service and pray the transient memory burst from pytorch (due to fragmentation as far as we can see) will not kill the service.

As a DevOps Engineer responsible for AKS infrastructure that is running AI/ML workloads, I need to be able to easily monitor memory usage of the applications and easily identify memory leaks so I can share anomalies with development teams for optimization/fixes.

**Describe the solution you'd like**
New GPU memory profiling unlocks new insights for AI workloads
Monitor where GPU memory is used - down to pods, code paths, and allocations
Profile GPU memory in production without invasive tooling or restarts
Fix out-of-memory (OOM) errors faster, reclaim wasted GPU capacity, and ship workloads with confidence

**Describe alternatives you've considered**
Before GPU Profiling, AKS customers needed to rely on DCGM exporter metrics and guesswork to detect and root cause GPU memory issues, delaying remediation and wasting scarce GPU capacity. Memory acts as a bottleneck for GPUs due to bandwidth and latency limitations - GPUs rely on memory to process massive amounts of parallel data. This reliance on memory underscores the necessity to ensure memory is being allocated appropriately

**Additional context**

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.