dotnet / dotnet/TorchSharp

Feature Request - implement nmap style model loading

Open
#1,120 2 comments 0 reactions 0 assignees View on GitHub
Missing Feature
Dominant language
C#
Stars
1.9k
Forks
228
PR merge metrics
No merged PRs in 30d

Description

I didn't see a template for this...

> I saw that llama.cpp (github project) has a PR (at the time I saw it) that made use of the nmap? c++ command that allows the model files to be mapped as memory space (like a dedicated swap file) and I was wondering if TorchSharp has something like that or could work with something created in c# to do that?

Having it map the model file(s) memory space to the files on the drive, allows them to be loaded near instantaneously. Also, it decreases the required memory by gigabytes.

I'm not 100% sure about all of the finer details, but I do know that the difference between running the standard llama.cpp and the nmap llama.cpp with the same model files was like night and day. It made getting the program up and running take less than a minute, and didn't slow down the model in a noticeable way.

So, I was wondering if something like this could be implemented, as that would be awesome (and would work cross-platform too.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.