macaroni-os / macaroni-os/mark-issues

[ai-kit] Review ollama ebuild to better support NVIDIA GPU

Open
#636 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ai-kit MARK mark-iii mark-unstable mark-xl
Dominant language
No language data
Stars
5
Forks
1
PR merge metrics
No merged PRs in 30d

Description

In the last releases of ollama the CUDA driver uses the new dev-util/nvidia-cuda-toolkit package with different releases:

  • CUDA 13
  • CUDA 12
  • CUDA 11 (to test)

Every release supplies support for different devices:

if(CUDA_VERSION VERSION_GREATER_EQUAL "13.0")
    # CUDA 13: 
    set(CMAKE_CUDA_ARCHITECTURES "75;80;86;89;90;100;120")
elseif(CUDA_VERSION VERSION_GREATER_EQUAL "12.0")
    # CUDA 12: 
    set(CMAKE_CUDA_ARCHITECTURES "61;70;75;80;86;89;90")
else()
    # CUDA 11: Max Ampere (86), include Maxwell (50)
    set(CMAKE_CUDA_ARCHITECTURES "50;52;61;70;75;80;86")
endif()

What is saw is that the native detection of CUDA GPU currently used in the ollama ebuild doesn't work correctly every time.
Only forcing CMAKE_CUDA_ARCHITECTURES with value 75 permit to have my GeForce RTX 2060 Mobile working correctly.

So what we need to do is:
a) permit to customize CUDA_ARCHITECTURES from users based on their GPUs. Keep them the control is better.
b) try to use the reported values on CMAKE_CUDA_ARCHITECTURES based on the available nvidia-cuda-toolkit package or just keep just point a

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Inspect the ollama ebuild and its current CUDA GPU detection, then compare the requested CUDA 11, 12, and 13 architecture values with the available nvidia-cuda-toolkit releases. The work is done when users can customize CUDA_ARCHITECTURES and the ebuild handles the supported toolkit versions without relying on unreliable native detection.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, ollama
Domain
ai, build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.