Layr-Labs / Layr-Labs/d-inference

[bug] mlx-community/gemma-4-26b-a4b-it-8bit chat_template is not portable to swift-jinja

Open
#242 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
522
Forks
113
Avg merge
17h 26m
Merged PRs (30d)
111

Description

### Component

provider-swift (Swift CLI)

### What happened?

When tool definitions are included in calls to /v1/chat/completions to providers using the swift-jinja template interpreter, `500 upper filter requires string` is always thrown. This impacts all agent harnesses that send tools. This occurs because the chat_template.jinja for this particular model is defective and only succeeds for other services due to undocumented permissiveness in the CPython jinja2 used by oMLX and other backends.

Defect specifics:
- CPython jinja2 is permissive about `Undefined | upper` and propagates `Undefined / ""` through the filter without raising

Patch description:
- throughout chat_template.jinja, replace `X | upper` with `(X | default('')) | upper` to properly treat `missing/None` as the empty string, failing all comparisons without crashing

Darkbloom actions:
- must patch chat_template.jinja
- must re-vend the patched chat_template.jinja from a new R2 prefix
- must bump `aggregate_sha256` in the coordinator catalog
- should consider upstreaming the patch to mlx-community/gemma-4-26b-a4b-it-8bit
- should expand test coverage to include cases that send tool definitions
- should include defensive try/catch in provider-swift's `MultiModelBatchSchedulerEngine` to downgrade the cascading `model load failed` into a clean 4xx error

### Expected behavior

Chat completions for mlx-community/gemma-4-26b-a4b-it-8bit from harnesses that send tool definitions should succeed for Darkbloom as they succeed when this model is loaded locally in oMLX

### Steps to reproduce

set Darkbloom as a provider in any harness that sends tool definitions

### Environment

computers

### Logs

```shell

```

### Confirmations

- [x] I searched existing issues and this isn't a duplicate

Contributor guide

Open the contributing guide

Research direction

Start with the model's chat_template.jinja and provider-swift's MultiModelBatchSchedulerEngine, then inspect the coordinator catalog and existing tests. Reproduce a chat completion that includes tool definitions, verify the patched template is re-vended from a new R2 prefix and the catalog aggregate_sha256 is updated, and add coverage for tool calls; a clean 4xx for model-load failures is also suggested.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
ai, api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.