huggingface / huggingface/candle

Candle leaves ~ model size of memory unused on Spark

Open
#3,530 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Candle seems unable to reason about availability of memory in a UMA system very well:
```sh
$ free -g
total used free shared buff/cache available
Mem: 119 81 38 0 0 37
Swap: 15 0 15
```
Looks offhand like that's about the size of an 80B NVPF4 model which makes me wonder if its being mapped to memory and then copied to the same memory resulting in counting its size twice for bounds calculation.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.