allenai / allenai/olmo-eval

Upgrade openai-agents: native unknown-tool handling, openai client constraint

Ouverte
#332 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
70
Forks
21
Merge moyen
2 j 11 h
PR mergées (30 j)
41

Description

Follow-up requested during the review of the unknown-tool fallback fix: track upgrading `openai-agents` so the SDK's own unknown-tool handling can eventually take over.

Findings from scoping the upgrade:

- The SDK gained `RunConfig.tool_not_found_behavior` in 0.17.4, but it **defaults to `raise_error`** — upgrading alone changes nothing; it must be set explicitly.
- The native message to the model is just `Tool 'X' not found.`, with no listing of valid tools. Our fallback returns the available tool names, which is what makes weak models actually recover — a naive swap could reduce recovery unless `tool_error_formatter` is used to restore the listing.
- As of 0.22.0 the native path still does not cover custom/freeform tools, so the fallback should remain as a backstop even after upgrading.
- Recommended target range: `>=0.18.2,<0.21`. 0.21+ requires `openai` 3.x and an httpx→httpx2 migration — a separate, larger step.
- The reason we are pinned today: `openai` 2.45.0 made `InputTokensDetails.cache_write_tokens` required, which breaks `openai-agents` 0.7.0's `Usage` defaults. Any SDK upgrade needs the `openai` constraint resolved together (current workaround pins `openai~=2.21` in the litellm extra).

Suggested shape: one PR that moves the SDK into the target range, resolves the `openai` constraint, opts into `tool_not_found_behavior` with a formatter that lists valid tools, keeps the fallback tool as backstop, and runs a paired A/B against the current pin before any scored sweep adopts it (SDK loop mechanics can shift generation behavior).

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.