microsoft / microsoft/foundry-local

[Bug] Foundry Local v2.0.1 uses max_length config instead of model context_length

Open
#1,067 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the issue

The Foundry Local model `gemma-4-e2b-it-generic-gpu:2` advertises a context length of 131,072 tokens, but Foundry Local SDK/runtime v2.0.1 rejects requests using more than 4,096 total tokens.

The downloaded model package contains conflicting values in `genai_config.json`:

- `model.context_length`: `131072`
- `search.max_length`: `4096`

At runtime, request validation uses the 4,096-token value. A text-only request with approximately 127,826 input tokens and 256 requested output tokens (128,082 total) is below the advertised 131,072-token context but is rejected as exceeding the model's maximum context length of 4,096 tokens.

Short text prompts and image inference work with the same model and WebGPU backend, so this appears to be a context-length/configuration mismatch rather than a general WebGPU execution failure.

This is similar to #969, but affects the Gemma 4 WebGPU model and limits it to 4,096 tokens.

### To reproduce

1. Install Foundry Local SDK/runtime v2.0.1.
2. Resolve and download `gemma-4-e2b-it-generic-gpu:2` using the C++ API.
3. Load the model with the WebGPU provider.
4. Submit a text-only request containing approximately 127,826 input tokens and request up to 256 output tokens.
5. Observe that the request is rejected against a 4,096-token maximum.
6. Inspect the downloaded `genai_config.json` and observe that `model.context_length` is 131,072 while `search.max_length` is 4,096.

### Expected behavior

The runtime should support the advertised 131,072-token context window, with input and requested output counted together. If the model variant intentionally supports only 4,096 tokens, the catalog metadata and `model.context_length` should report that effective limit consistently.

### Platform and architecture

macOS Apple Silicon

### OS Version

macOS

### Installation type

Released package/binary

### Foundry Local version

Foundry Local SDK/runtime 2.0.1

### API or surface area

C++ API

### Hardware acceleration/backend

WebGPU

### Model

`gemma-4-e2b-it-generic-gpu:2`

Contributor guide

Open the contributing guide

Research direction

Start with the downloaded genai_config.json for gemma-4-e2b-it-generic-gpu:2 and trace the C++ API request-validation path that reads search.max_length. Compare that value with model.context_length and the catalog metadata, then run the reported large text-only request with the WebGPU provider. Done means the effective context limit is consistent and input plus requested output are validated against it.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.