microsoft / microsoft/onnxruntime

ONNX Runtime GenAI phi-2 tutorial broken: set_search_options() API changed

Open
#26,481 2 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

## Description

The phi-2 example code in the ONNX Runtime GenAI repository no longer works with the current version of the library due to an API change in `set_search_options()`.

## Expected Behavior

The following code from the tutorial should work:
```python
params = og.GeneratorParams(model)
params.set_search_options({"max_length": 200})
params.input_ids = tokens
```

## Actual Behavior

The code raises a TypeError:
```
TypeError: set_search_options(): incompatible function arguments. The following argument types are supported:
1. (self: onnxruntime_genai.onnxruntime_genai.GeneratorParams, **kwargs) -> None

Invoked with: , {'max_length': 200}
```

## Steps to Reproduce

1. Create a test script (example code):
```python
import onnxruntime_genai as og

prompt = '''def print_prime(n):
"""
Print all primes between 1 and n
"""'''

model=og.Model(f'example-models/phi2-int4-cpu')
tokenizer = og.Tokenizer(model)
tokens = tokenizer.encode(prompt)

params=og.GeneratorParams(model)
params.set_search_options({"max_length":200}) # This line fails
params.input_ids = tokens

output_tokens=model.generate(params)[0]
text = tokenizer.decode(output_tokens)
print(text)
```

2. Run the script with onnxruntime-genai 0.11.0.dev0

## Environment

- **ONNX Runtime GenAI Version**: 0.11.0.dev0
- **ONNX Runtime Version**: 1.23.2 (GPU)
- **Python Version**: 3.13.9
- **OS**: Ubuntu 24.04.2 LTS (Noble Numbat)
- **Kernel**: Linux 6.16.12-200.fc42.x86_64

## Root Cause

The API has changed from accepting a dictionary to using keyword arguments. The correct syntax is now:
```python
params.set_search_options(max_length=200)
```

Or as shown in the Phi-3 example in the README:
```python
search_options = {}
search_options['max_length'] = 2048
params.set_search_options(**search_options)
```

## Suggested Fix

Update any phi-2 tutorial/example code to use the new `set_search_options()` API with keyword arguments instead of passing a dictionary directly.

Contributor guide

Open the contributing guide

Research direction

Search the repository for the phi-2 tutorial or example and its use of set_search_options(), then compare it with the Phi-3 README example mentioned in the issue. Update the outdated call to use keyword arguments and verify that the example no longer passes a dictionary directly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.