Flm with gemma 4 has very poor ability to call tools
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 152
- Avg merge
- 4h 14m
- Merged PRs (30d)
- 11
Description
I am using flm backend for lemonade server on Linux with Gemma 4 e4b.
And it's support for tools is very poor.
AMD Ryzen AI 9 HX PRO 375 w/ Radeon 890M × 12
Linux Mint 22.3 - Cinnamon 64-bit
7.0.0-31-generic
lemonade version 11.9.0
flm npu v1.0.3
gemma4-it-e4b-FLM 9.10GB
User
Please write a python script that writes json with "name" and "description" keys into out.txt file. Please save this python script into scrypt.py
Assistant (Coder · gemma4-it-e4b-FLM (local) · 11.9s)
Tool: write
Input:
{
"content": "\"\"\"import json\n\ndata = {\n \"name\": \"Example Item\",\n \"description\": \"This is a sample description.\"\n}\n\nwith open('out.txt', 'w') as f:\n json.dump(data, f)\n\"\"",
"filePath": "\"scrypt.py\""
}
Output:
Wrote file successfully.
It wrote file with ["scrypt.py"] name and following invalid content:
"""import json
data = {
"name": "Example Item",
"description": "This is a sample description."
}
with open('out.txt', 'w') as f:
json.dump(data, f)
"
And later just stuck with tool calls:
Tool: write
Input:
{
"content": "import json\n\ndata = {\n \"name\": \"Example Item",
"description": "This is a sample description."
}
Error:
The write tool was called with invalid arguments: SchemaError(Missing key
at ["filePath"]).
Please rewrite the input so it satisfies the expected schema.
Assistant (Coder · gemma4-it-e4b-FLM (local) · 13.5s)
Tool: write
Input:
{
"content": "import json\n\ndata = {\n \"name\": \"Example Item",
"description": "This is a sample description."
}
Error:
The write tool was called with invalid arguments: SchemaError(Missing key
at ["filePath"]).
Please rewrite the input so it satisfies the expected schema.
response:write{value:<|"|>Wrote file successfully.<|"|>}<tool_call|>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported prompt with the flm backend, Gemma 4 e4b-FLM, lemonade 11.9.0, and flm npu v1.0.3 on Linux. Start by tracing the tool-call and write-tool request handling; done means the generated call preserves valid content and the required filePath, completes without schema errors, and creates the requested script.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100