cloudwego / cloudwego/eino

feat(schema): track LLM cache-creation (write) tokens in PromptTokenDetails

Open
#1,150 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
13k
Forks
1.1k
Avg merge
4h 6m
Merged PRs (30d)
41

Description

**Is your feature request related to a problem? Please describe.**

`schema.PromptTokenDetails` models only CachedTokens (cache read). There's no field for cache write (creation) tokens. Multiple providers bill cache writes at a premium and report a distinct count - Anthropic cache_creation_input_tokens (1.25x/2x), OpenAI cache_write_tokens (newer manual-cache models, 1.25x), AWS Bedrock cacheWriteInputTokens - but eino has nowhere to put it, so bindings fold it into `PromptTokens`. The total is preserved, but the distinct write count isn't surfaced:

```go
promptTokens := int(usage.InputTokens + usage.CacheReadInputTokens + usage.CacheCreationInputTokens)
PromptTokenDetails{ CachedTokens: int(usage.CacheReadInputTokens) } // write count not surfaced
```

Result: write tokens are counted, but unattributable - they land in the input bucket and get priced at 1x instead of the provider's write rate, so cost tracking under-reports the write premium and can't break out write volume.

**Describe the solution you'd like**

```go
type PromptTokenDetails struct {
CachedTokens int `json:"cached_tokens"` // cache read
CacheCreationTokens int `json:"cache_creation_tokens"` // cache write
}
```

**Describe alternatives you've considered**

- Compute downstream - impossible; the write count is already summed into `PromptTokens` at the binding and can't be separated back out.
- Carry via ResponseMeta.Extra - works but unshaped and per-consumer; a typed field is the right home.

**Additional context**

Every other provider-agnostic framework already models write separately and provider-neutrally: LangChain / LangChain.js (input_token_details.cache_creation), Vercel AI SDK (inputTokens.cacheWrite), LlamaIndex (cache_creation_input_tokens). Gemini is the one provider with no per-request write count (write billed in a separate caches.create call) — a neutral field simply stays zero there.

Contributor guide

Open the contributing guide

Research direction

Start at the Go definition of schema.PromptTokenDetails and trace the provider bindings that construct it from cache usage. Add the distinct cache-creation token field described in the issue, preserving existing cached-token behavior, and verify that provider-reported write counts are surfaced without changing total prompt-token accounting.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.