allenai / allenai/olmo-eval

Upgrade openai-agents: native unknown-tool handling, openai client constraint

未關閉
#332 2 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
70
分支
21
平均合併
2 天 11 小時
30 天內合併 PR
41

描述

Follow-up requested during the review of the unknown-tool fallback fix: track upgrading `openai-agents` so the SDK's own unknown-tool handling can eventually take over.

Findings from scoping the upgrade:

- The SDK gained `RunConfig.tool_not_found_behavior` in 0.17.4, but it **defaults to `raise_error`** — upgrading alone changes nothing; it must be set explicitly.
- The native message to the model is just `Tool 'X' not found.`, with no listing of valid tools. Our fallback returns the available tool names, which is what makes weak models actually recover — a naive swap could reduce recovery unless `tool_error_formatter` is used to restore the listing.
- As of 0.22.0 the native path still does not cover custom/freeform tools, so the fallback should remain as a backstop even after upgrading.
- Recommended target range: `>=0.18.2,<0.21`. 0.21+ requires `openai` 3.x and an httpx→httpx2 migration — a separate, larger step.
- The reason we are pinned today: `openai` 2.45.0 made `InputTokensDetails.cache_write_tokens` required, which breaks `openai-agents` 0.7.0's `Usage` defaults. Any SDK upgrade needs the `openai` constraint resolved together (current workaround pins `openai~=2.21` in the litellm extra).

Suggested shape: one PR that moves the SDK into the target range, resolves the `openai` constraint, opts into `tool_not_found_behavior` with a formatter that lists valid tools, keeps the fallback tool as backstop, and runs a paired A/B against the current pin before any scored sweep adopts it (SDK loop mechanics can shift generation behavior).

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

Start by locating the current unknown-tool fallback and the litellm extra's openai~=2.21 constraint, then review the dependency configuration for the compatible openai-agents range. Confirm the native behavior can list valid tools while retaining the fallback for custom tools; done means the constraint is resolved and a paired A/B comparison against the current pin is complete before any scored sweep uses it.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
ai
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
活躍
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。