microsoft / microsoft/foundry-local
[Bug] gpt-oss-20b WebGPU advertises 131072 context but is limited to 8192 tokens at runtime
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
### Describe the issue
The Foundry Local model `gpt-oss-20b-generic-gpu:1` advertises a context length of 131072 tokens, but the actual WebGPU runtime is limited to 8192 total tokens.
This is reproducible and the model package itself contains conflicting settings:
- `config.json`
- `max_position_embeddings`: 131072
- `genai_config.json`
- `context_length`: 131072
- `max_length`: 8192
At runtime, a request with 9196 input tokens and 2048 requested output tokens fails with:
This request requires 11244 total tokens (9196 input + 2048 output), which exceeds the model's maximum context length of 8192 tokens.
The exception originates from:
Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator.ValidateMaxTokens
GPU execution itself works correctly for short prompts using the same model and WebGPU backend, so this appears to be specifically a context-length/configuration issue rather than a general GPU execution failure.
For our use case, we need to process full meeting transcripts in one pass. An 8192-token limit is insufficient even for a 12,671-character Japanese meeting transcript.
Please clarify whether the 8192-token limit is intentional for the WebGPU variant or whether this is a packaging/runtime configuration issue.
If the 8192-token limit is intentional, the catalog metadata should reflect the actual supported context length.
If it is not intentional, we would like the GPU variant to support a substantially longer context window, ideally consistent with the advertised 131072-token context length.
### To reproduce
1. Install Foundry Local 0.10.2 and `foundry-local-sdk-winml` 1.2.4.
2. Register the available execution providers using:
`download_and_register_eps()`
3. Confirm that the following execution providers are registered:
- WebGpuExecutionProvider
- OpenVINOExecutionProvider
4. Refresh the Foundry Local model catalog.
5. Resolve the model:
`gpt-oss-20b-generic-gpu:1`
6. Confirm that the SDK reports:
- Device: GPU
- Execution Provider: WebGpuExecutionProvider
- context_length: 131072
7. Download and load the model.
8. Inspect the downloaded model package:
`~/.gpt_oss_minutes_webgpu/cache/models/Microsoft/gpt-oss-20b-generic-gpu-1/v1/`
9. Relevant model configuration values are:
`config.json`
- max_position_embeddings: 131072
`genai_config.json`
- context_length: 131072
- max_length: 8192
10. Submit a request containing a 12,671-character Japanese meeting transcript with `max_tokens=2048`.
11. Foundry Local calculates:
- Input tokens: 9196
- Requested output tokens: 2048
- Total: 11244
12. The request fails with:
`This request requires 11244 total tokens (9196 input + 2048 output), which exceeds the model's maximum context length of 8192 tokens.`
13. As a control test, a short prompt with the same GPU model succeeds:
Prompt:
`Reply with exactly: GPU_OK`
Result:
`GPU_OK`
Inference time:
approximately 3.7 seconds.
### Urgency
High.
This issue blocks evaluation of the GPT-OSS GPU variant for long-context workloads.
The current 8192-token runtime limit prevents processing documents that should fit within the advertised 131072-token context window.
A fix or clarification is important because the observed runtime behavior does not match the published model metadata.
### Platform and architecture
Windows X64
### OS Version
Windows 11, 10.0.26200
### Installation type
Released package/binary
### Foundry Local version
CLI 0.10.2 foundry-local-sdk-winml 1.2.4
### API or surface area
Python SDK
### Hardware acceleration/backend
WebGPU/CUDA
### Backend/runtime version
WebGpuExecutionProvider Intel Graphics driver 32.0.101.8860 foundry-local-sdk-winml 1.2.4 onnxruntime-core 1.26.0 onnxruntime-genai-core 0.14.1; CPU: Intel Core Ultra 7 356H GPU: Intel Graphics GPU driver: 32.0.101.8860 RAM: 64 GB
Contributor guide
Research direction
Start with the downloaded model package's config.json and genai_config.json, then trace the request validation at Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator.ValidateMaxTokens. Determine whether max_length=8192 is intentional or conflicts with the advertised 131072-token context, and verify the result with the 9196-token input plus 2048-token output reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai-infra-agents, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100