[Feature Request] Ability to disable mmproj via an argument and set max context checkpoints

Open
#626 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
cpp
Domain
backend

Research direction

Start by reviewing the server launch arguments and --help output for model-loading options, then trace how mmproj weights and context checkpoints are handled. Confirm whether disabling vision/audio weights and limiting checkpoints are supported concepts, and define completion as documented, usable arguments that produce the intended memory behavior.

Written by the indexing model from the issue text.

Description

Currently (correct me if I'm wrong, please - but I can't find a way in docs, or via --help) there is no way to disable loading vision or audio weights (in case of Gemma 4 models) via parameter when launching the server. Since for Gemma 4 E4B those add up to nearly 2 GB and they aren't always used, it would be nice if we could just easily disable them.

I'm not quite sure about the second point since generally there is a very limited number of server params (so I'm not actually sure if the concept translates from llama.cpp) but I wanted to bring it up in case it's right:

Another potential saving, on llama.cpp you can set a maximum number of context checkpoints - I don't know how FLM handles this, but I would believe that there are a few equivalent checkpoints because the memory occupied by the model seems to keep growing over time.

Since I feel like NPU models are best used when loaded alongside another model via a different backend, those memory savings could be really useful. For me, currently I'm hovering around 6GB free memory when loading models for my usecase, and I could stay in that footprint when using two llama.cpp servers but when I use flm to serve the second model the memory use grows by 2 GB or so after use.

Dominant language
C++
Stars
1.9k
Forks
152
Avg merge
4h 14m
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/FastFlowLM

All issues in ROCm/FastFlowLM

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.