microsoft / microsoft/onnxruntime
ONNX Runtime GenAI phi-2 tutorial broken: set_search_options() API changed
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
## Description
The phi-2 example code in the ONNX Runtime GenAI repository no longer works with the current version of the library due to an API change in `set_search_options()`.
## Expected Behavior
The following code from the tutorial should work:
```python
params = og.GeneratorParams(model)
params.set_search_options({"max_length": 200})
params.input_ids = tokens
```
## Actual Behavior
The code raises a TypeError:
```
TypeError: set_search_options(): incompatible function arguments. The following argument types are supported:
1. (self: onnxruntime_genai.onnxruntime_genai.GeneratorParams, **kwargs) -> None
Invoked with: , {'max_length': 200}
```
## Steps to Reproduce
1. Create a test script (example code):
```python
import onnxruntime_genai as og
prompt = '''def print_prime(n):
"""
Print all primes between 1 and n
"""'''
model=og.Model(f'example-models/phi2-int4-cpu')
tokenizer = og.Tokenizer(model)
tokens = tokenizer.encode(prompt)
params=og.GeneratorParams(model)
params.set_search_options({"max_length":200}) # This line fails
params.input_ids = tokens
output_tokens=model.generate(params)[0]
text = tokenizer.decode(output_tokens)
print(text)
```
2. Run the script with onnxruntime-genai 0.11.0.dev0
## Environment
- **ONNX Runtime GenAI Version**: 0.11.0.dev0
- **ONNX Runtime Version**: 1.23.2 (GPU)
- **Python Version**: 3.13.9
- **OS**: Ubuntu 24.04.2 LTS (Noble Numbat)
- **Kernel**: Linux 6.16.12-200.fc42.x86_64
## Root Cause
The API has changed from accepting a dictionary to using keyword arguments. The correct syntax is now:
```python
params.set_search_options(max_length=200)
```
Or as shown in the Phi-3 example in the README:
```python
search_options = {}
search_options['max_length'] = 2048
params.set_search_options(**search_options)
```
## Suggested Fix
Update any phi-2 tutorial/example code to use the new `set_search_options()` API with keyword arguments instead of passing a dictionary directly.
Contributor guide
Research direction
Search the repository for the phi-2 tutorial or example and its use of set_search_options(), then compare it with the Phi-3 README example mentioned in the issue. Update the outdated call to use keyword arguments and verify that the example no longer passes a dictionary directly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100