aaif-goose / aaif-goose/goose

Support cached-input pricing for custom providers

未關閉
#11,509 4 則留言 0 個 reaction 已指派 1 人 已指派給 @DOsinga 在 GitHub 檢視
主要語言
Rust
星號
54.2k
分支
6.2k
平均合併
3 天 2 小時
30 天內合併 PR
262

描述

## Problem

Custom provider model metadata currently supports `input_token_cost` and `output_token_cost`, but it cannot specify a separate price for cached input tokens.

Some OpenAI-compatible providers return cached-token usage and bill those tokens at a substantially discounted rate. For example, Cloudflare Workers AI enables prefix caching for supported models, reports cached token counts in the response `usage` object, and applies a different cached-input price.

Because custom providers cannot configure that price separately, Goose's estimated and accumulated session cost can materially overestimate actual provider billing during long agent sessions with high cache-hit rates.

## Who this affects

Users of custom OpenAI-compatible providers that:

- report cached input tokens;
- charge a distinct cached-input rate;
- use long, repeated prompt prefixes, particularly coding-agent sessions.

Cloudflare Workers AI is one example, but the capability would be useful for any compatible provider with discounted cached input.

## Current behavior

A custom model can define:

- `input_token_cost`
- `output_token_cost`

There is no model metadata field for cached-input cost. Goose already has usage fields for cache-read and cache-write tokens, but custom-provider pricing cannot use a separate cached-input rate.

## Expected behavior

Custom providers should be able to optionally specify a cached-input token price.

When both cached-token usage and cached pricing are available, Goose should use them when calculating session cost. Providers that do not configure cached pricing should retain the current behavior.

The capability should work consistently in session totals and any CLI or Desktop cost display.

## Verification

A suitable verification plan would cover:

- no cached tokens;
- partially cached input;
- fully cached input;
- cached-token usage without cached pricing;
- cached pricing without cached-token usage;
- backward compatibility with existing custom-provider JSON files;
- accumulated cost across multiple requests.

## Example provider documentation

Cloudflare Workers AI prompt caching:

https://developers.cloudflare.com/workers-ai/features/prompt-caching/

Cloudflare Workers AI pricing:

https://developers.cloudflare.com/workers-ai/platform/pricing/

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。