finos / finos/fluxnova-plugins
[Feature] Add LLM token usage monitoring
- Dominant language
- Java
- Stars
- 3
- Forks
- 7
- Avg merge
- 5d 4h
- Merged PRs (30d)
- 3
Description
### Description of Problem
When I run LLM workflows, I want to see how many tokens each request uses, so I can control costs and catch problems early.
Right now there is no way to see token usage. The only option is the provider's billing page, which is delayed and only shows account totals. That means I can't:
- Tell which feature, user, or workflow is spending the money
- Catch a runaway loop before it runs up a big bill
- Predict costs when scaling up or switching models
### Potential Solutions
**Proposed: built-in token tracking**
- Record input/output tokens for each request (use the provider's response data, or estimate if not available)
- Group totals by model, session, and tags (user, feature, etc.)
- Show the data in logs and a metrics endpoint
- Add optional alerts when usage passes a set limit
*Benefits:* real-time cost visibility, spend tied to specific workflows, early warning on runaway usage.
*Drawbacks:* small overhead per request; estimates may be off for some providers; more config to maintain.
**Alternatives considered**
- **External proxy/gateway:** no code changes, but adds infrastructure and latency, and loses in-app context.
- **Manual logging:** works, but everyone builds it differently and there's no shared standard.
- **Provider billing dashboards:** free, but delayed and account-level only.
Contributor guide
Research direction
The issue names no files, tests, providers, or metrics entry points. Start by locating where provider responses are handled and how logs, metrics, tags, and configuration are exposed; then clarify which providers and alerting behavior are in scope. Done should include per-request input/output usage, grouped totals, logs, a metrics endpoint, and optional limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ai, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100