microsoft / microsoft/foundry-local

[Bug] gpt-oss-20b WebGPU advertises 131072 context but is limited to 8192 tokens at runtime

Open
#969 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the issue

The Foundry Local model `gpt-oss-20b-generic-gpu:1` advertises a context length of 131072 tokens, but the actual WebGPU runtime is limited to 8192 total tokens.

This is reproducible and the model package itself contains conflicting settings:

- `config.json`
- `max_position_embeddings`: 131072
- `genai_config.json`
- `context_length`: 131072
- `max_length`: 8192

At runtime, a request with 9196 input tokens and 2048 requested output tokens fails with:

This request requires 11244 total tokens (9196 input + 2048 output), which exceeds the model's maximum context length of 8192 tokens.

The exception originates from:

Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator.ValidateMaxTokens

GPU execution itself works correctly for short prompts using the same model and WebGPU backend, so this appears to be specifically a context-length/configuration issue rather than a general GPU execution failure.

For our use case, we need to process full meeting transcripts in one pass. An 8192-token limit is insufficient even for a 12,671-character Japanese meeting transcript.

Please clarify whether the 8192-token limit is intentional for the WebGPU variant or whether this is a packaging/runtime configuration issue.

If the 8192-token limit is intentional, the catalog metadata should reflect the actual supported context length.

If it is not intentional, we would like the GPU variant to support a substantially longer context window, ideally consistent with the advertised 131072-token context length.

### To reproduce

1. Install Foundry Local 0.10.2 and `foundry-local-sdk-winml` 1.2.4.

2. Register the available execution providers using:

`download_and_register_eps()`

3. Confirm that the following execution providers are registered:

- WebGpuExecutionProvider
- OpenVINOExecutionProvider

4. Refresh the Foundry Local model catalog.

5. Resolve the model:

`gpt-oss-20b-generic-gpu:1`

6. Confirm that the SDK reports:

- Device: GPU
- Execution Provider: WebGpuExecutionProvider
- context_length: 131072

7. Download and load the model.

8. Inspect the downloaded model package:

`~/.gpt_oss_minutes_webgpu/cache/models/Microsoft/gpt-oss-20b-generic-gpu-1/v1/`

9. Relevant model configuration values are:

`config.json`
- max_position_embeddings: 131072

`genai_config.json`
- context_length: 131072
- max_length: 8192

10. Submit a request containing a 12,671-character Japanese meeting transcript with `max_tokens=2048`.

11. Foundry Local calculates:

- Input tokens: 9196
- Requested output tokens: 2048
- Total: 11244

12. The request fails with:

`This request requires 11244 total tokens (9196 input + 2048 output), which exceeds the model's maximum context length of 8192 tokens.`

13. As a control test, a short prompt with the same GPU model succeeds:

Prompt:
`Reply with exactly: GPU_OK`

Result:
`GPU_OK`

Inference time:
approximately 3.7 seconds.

### Urgency

High.

This issue blocks evaluation of the GPT-OSS GPU variant for long-context workloads.

The current 8192-token runtime limit prevents processing documents that should fit within the advertised 131072-token context window.

A fix or clarification is important because the observed runtime behavior does not match the published model metadata.

### Platform and architecture

Windows X64

### OS Version

Windows 11, 10.0.26200

### Installation type

Released package/binary

### Foundry Local version

CLI 0.10.2 foundry-local-sdk-winml 1.2.4

### API or surface area

Python SDK

### Hardware acceleration/backend

WebGPU/CUDA

### Backend/runtime version

WebGpuExecutionProvider Intel Graphics driver 32.0.101.8860 foundry-local-sdk-winml 1.2.4 onnxruntime-core 1.26.0 onnxruntime-genai-core 0.14.1; CPU: Intel Core Ultra 7 356H GPU: Intel Graphics GPU driver: 32.0.101.8860 RAM: 64 GB

Contributor guide

Open the contributing guide

Research direction

Start with the downloaded model package's config.json and genai_config.json, then trace the request validation at Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator.ValidateMaxTokens. Determine whether max_length=8192 is intentional or conflicts with the advertised 131072-token context, and verify the result with the 9196-token input plus 2048-token output reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai-infra-agents, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.