huggingface / huggingface/candle

Support for Nvidia unified memory?

Open
#1,618 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

All,

I saw this morning that Tim Dettmers bitsandbytes python lib uses Nvidia's [Unified Memory](https://developer.nvidia.com/blog/unified-memory-cuda-beginners/) by [default](https://x.com/stasbekman/status/1749968490155696612), see (`csrc/pythonInterface.c:377`).

It doesn't look like candle, via cudarc supports this.

I'm interested in supporting this feature in Rust because my current GPU has a decent compute score, but only 4GB of VRAM (T1000).

Is there any plan to support this in candle?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the bitsandbytes reference at csrc/pythonInterface.c:377 and tracing how candle currently reaches cudarc. Determine whether cudarc can expose Nvidia Unified Memory and what candle integration would require; done means a decided implementation path and verified support for the reported low-VRAM use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.