[Issue]: Subpar Gemma 4 E4B output quality on FLM
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 152
- Avg merge
- 4h 14m
- Merged PRs (30d)
- 11
Description
Required system details:
PS C:\WINDOWS\system32> (Get-WmiObject Win32_OperatingSystem).Version
10.0.26200
PS C:\WINDOWS\system32> (Get-WmiObject win32_Processor).Name
AMD Ryzen AI 9 365 w/ Radeon 880M
PS C:\WINDOWS\system32> (Get-WmiObject win32_VideoController).Name
AMD Radeon(TM) 880M Graphics
Hi, I wanted to report that Gemma 4 output quality with the E4B model is pretty clearly subpar compared to other quants of similar size (in this case Unsloth's gemma-4-E4B-it-qat-UD-Q4_K_XL). This may or may not be due to the outdated template or something to do with quantization, but in any ase it's a bit of a shame since normally it would be one of the preferred models to use for small tool-use applications in my case. Since it's a bit hard to just showcase what I'm talking about in one go, I made a small repository that can be used to easily reproduce the issue:
https://github.com/BoneHorror/flm_g4e4b_quality_test
But the tl;dr of what one could expect is:
Scenario:
The model, with previous conversation history, is asked to find balance changes made to "Marines". It does not have earlier knowledge of Marines or other specifics, but Gemma 4 E4B can typically infer most of what's needed to complete this query.
Answers -
When FLM is generating: (G4 E4B)
<tool>Model is listing docs in: balance_changes
I've listed the files in the `balance_changes` directory. It seems to be organized by race, with files for Terran (`terran.txt`), UED (`ued.txt`), Zerg (`zerg.txt`), and Protoss (`protoss.txt`).
To find the specific balance changes made to Marines, I would need you to tell me which race group (Terran, UED, Zerg, or Protoss) Marines belong to in this documentation structure so I can read the correct file for you!
When llama-server is generating with G4 E4B Q4_K_XL:
<tool>Model is listing docs in: balance_changes
<tool>Model is reading: balance_changes/terran.txt
Reading /home/user/Repositories/discord_bot/docs/balance_changes/terran.txt
Based on the `terran.txt` documentation, here are the balance changes made to Marines:
* **Cost:** Decreased to **40** (from 50).
* **Train Time:** Decreased to **320 frames** (from 360).
* **Note:** Terran units no longer include Medics; these roles were moved to the UED faction.
This is a typical scenario, you can find more examples in the repo linked above. You can see that via FLM the model stopped short of completing the query properly, even though as seen in the other case it could absolutely find the information.
I have previously tried to report it on the project's Discord but ended up not having enough time to write it up properly. I hope someone can take a look here, as well.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked flm_g4e4b_quality_test repository and reproduce the Marines query on the reported Windows and AMD Ryzen AI 9 365 setup, comparing FLM's output with llama-server's. Investigate whether the discrepancy relates to the model template or quantization; done means identifying a concrete cause or documenting a confirmed limitation with reproducible results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai-infra-agents
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100