aaif-goose / aaif-goose/goose

Support cached-input pricing for custom providers

Abierto
#11,509 4 comentarios 0 reacciones 1 asignado Asignado a @DOsinga Ver en GitHub
Lenguaje dominante
Rust
Estrellas
54.2k
Forks
6.2k
Merge medio
3 d 4 h
PR fusionados (30 d)
240

Descripción

## Problem

Custom provider model metadata currently supports `input_token_cost` and `output_token_cost`, but it cannot specify a separate price for cached input tokens.

Some OpenAI-compatible providers return cached-token usage and bill those tokens at a substantially discounted rate. For example, Cloudflare Workers AI enables prefix caching for supported models, reports cached token counts in the response `usage` object, and applies a different cached-input price.

Because custom providers cannot configure that price separately, Goose's estimated and accumulated session cost can materially overestimate actual provider billing during long agent sessions with high cache-hit rates.

## Who this affects

Users of custom OpenAI-compatible providers that:

- report cached input tokens;
- charge a distinct cached-input rate;
- use long, repeated prompt prefixes, particularly coding-agent sessions.

Cloudflare Workers AI is one example, but the capability would be useful for any compatible provider with discounted cached input.

## Current behavior

A custom model can define:

- `input_token_cost`
- `output_token_cost`

There is no model metadata field for cached-input cost. Goose already has usage fields for cache-read and cache-write tokens, but custom-provider pricing cannot use a separate cached-input rate.

## Expected behavior

Custom providers should be able to optionally specify a cached-input token price.

When both cached-token usage and cached pricing are available, Goose should use them when calculating session cost. Providers that do not configure cached pricing should retain the current behavior.

The capability should work consistently in session totals and any CLI or Desktop cost display.

## Verification

A suitable verification plan would cover:

- no cached tokens;
- partially cached input;
- fully cached input;
- cached-token usage without cached pricing;
- cached pricing without cached-token usage;
- backward compatibility with existing custom-provider JSON files;
- accumulated cost across multiple requests.

## Example provider documentation

Cloudflare Workers AI prompt caching:

https://developers.cloudflare.com/workers-ai/features/prompt-caching/

Cloudflare Workers AI pricing:

https://developers.cloudflare.com/workers-ai/platform/pricing/

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Look at the custom provider model metadata structure, likely in a file like `src/provider/custom.rs` or `src/model/metadata.rs`. Find where `input_token_cost` and `output_token_cost` are defined. Add a new field for cached input cost. Update the cost calculation logic to use this field when cached token usage is reported. Check the session cost accumulation and any CLI/Desktop display code. Run existing tests to ensure backward compatibility.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
rust
Área
backend-api-design
Tipo de issue
Nueva funcionalidad
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Activo
Claridad
Bien especificado
Aptitud para principiantes
65/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.