spring-projects / spring-projects/spring-ai
Prompt and completion tokens (including corresponding prometheus metrics) not tracking when streaming chat client's response
Open
Nobody has claimed this yet.
status: waiting-for-triage
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
The below returns the prompt and completion tokens
ChatClient.CallResponseSpec responseSpec = _chatClient.prompt("Tell me a dad joke about dogs")
.call();
ChatResponse chatResponse = responseSpec.chatResponse();
ChatResponseMetadata chatResponseMetadata = chatResponse.getMetadata();
int promptTokens = chatResponseMetadata.getUsage().getPromptTokens();
int completionTokens = chatResponseMetadata.getUsage().getCompletionTokens();
int totalTokens = chatResponseMetadata.getUsage().getTotalTokens();
_LOGGER.debug("Prompt tokens: {}", promptTokens);
_LOGGER.debug("Completion tokens: {}", completionTokens);
_LOGGER.debug("Total tokens: {}", totalTokens);
But streaming the chat client's response doesn't
ChatClient.StreamResponseSpec streamResponseSpec = _chatClient.prompt("Tell me a dad joke about dogs")
.stream();
Flux<ChatResponse> chatResponseFlux = streamResponseSpec.chatResponse();
chatResponseFlux.map((r) -> {
ChatResponseMetadata chatResponseMetadata = r.getMetadata();
int promptTokens = chatResponseMetadata.getUsage().getPromptTokens();
int completionTokens = chatResponseMetadata.getUsage().getCompletionTokens();
int totalTokens = chatResponseMetadata.getUsage().getTotalTokens();
_LOGGER.debug("Prompt tokens: {}", promptTokens);
_LOGGER.debug("Completion tokens: {}", completionTokens);
_LOGGER.debug("Total tokens: {}", totalTokens);
});
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ChatClient.CallResponseSpec and StreamResponseSpec entry points shown in the issue, comparing how chatResponse() exposes metadata for non-streaming and streaming calls. Trace the streaming response handling and corresponding Prometheus metrics; done means prompt, completion, and total token usage are available and tracked during streaming, with regression coverage for that path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, prometheus
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100