spring-projects / spring-projects/spring-ai

Prompt and completion tokens (including corresponding prometheus metrics) not tracking when streaming chat client's response

Open
#2,671 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

The below returns the prompt and completion tokens

ChatClient.CallResponseSpec responseSpec = _chatClient.prompt("Tell me a dad joke about dogs")
            .call();
        ChatResponse chatResponse = responseSpec.chatResponse();
        ChatResponseMetadata chatResponseMetadata = chatResponse.getMetadata();

        int promptTokens = chatResponseMetadata.getUsage().getPromptTokens();
        int completionTokens = chatResponseMetadata.getUsage().getCompletionTokens();
        int totalTokens = chatResponseMetadata.getUsage().getTotalTokens();

        _LOGGER.debug("Prompt tokens: {}", promptTokens);
        _LOGGER.debug("Completion tokens: {}", completionTokens);
        _LOGGER.debug("Total tokens: {}", totalTokens);

But streaming the chat client's response doesn't

ChatClient.StreamResponseSpec streamResponseSpec = _chatClient.prompt("Tell me a dad joke about dogs")
            .stream();

        Flux<ChatResponse> chatResponseFlux = streamResponseSpec.chatResponse();
        chatResponseFlux.map((r) -> {
            ChatResponseMetadata chatResponseMetadata = r.getMetadata();
            int promptTokens = chatResponseMetadata.getUsage().getPromptTokens();
            int completionTokens = chatResponseMetadata.getUsage().getCompletionTokens();
            int totalTokens = chatResponseMetadata.getUsage().getTotalTokens();

            _LOGGER.debug("Prompt tokens: {}", promptTokens);
            _LOGGER.debug("Completion tokens: {}", completionTokens);
            _LOGGER.debug("Total tokens: {}", totalTokens);
        });

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ChatClient.CallResponseSpec and StreamResponseSpec entry points shown in the issue, comparing how chatResponse() exposes metadata for non-streaming and streaming calls. Trace the streaming response handling and corresponding Prometheus metrics; done means prompt, completion, and total token usage are available and tracked during streaming, with regression coverage for that path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, prometheus
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.