JSON Truncation in Spring AI Streaming Response with Large Sets

Bug description

When using Spring AI's streaming client with larger sets of keywords (~1000), the response is truncated mid-JSON object, resulting in a JsonMappingException. The same code works perfectly with smaller sets (~100 keywords). There's no clear indication if this is due to token limits or other constraints, making it difficult to handle appropriately in production code.

Environment

• Spring AI version: 1.0.0-M5
• Spring Boot version: 3.4.2
• Java version: 23
• Model: GPT_4_O_MINI

Steps to reproduce

1. Create a test with ~100 keywords (passes) and another with ~1000 keywords (fails)
2. Use Spring AI's streaming client (required to avoid timeout issues as per discussion #2195)
3. Attempt to process the response

Expected behavior

Either:

1. Successfully process all keywords, or
2. Receive a clear error indicating why the processing failed (e.g., token limit exceeded)

Minimal Complete Reproducible example

Here are two tests - one that passes with ~100 keywords and one that fails with larger sets:

Test that passes with ~100 keywords

@Test
    void givenLongResponseTime_whenProcessingWithStreamingChatClient_thenResponseIsSuccessful() {
        ChatClient chatClient = ChatClient.create(chatModel);
        var converter = new BeanOutputConverter<>(new ParameterizedTypeReference<AiResponseKeywordsAligned>() {
        });
        String systemPrompt = "You are a keyword research assistant specialized in SEO and Google Keyword Planner, helping users discover high-quality, long-tail keywords for niche businesses. {format}";
        String userMessage = """
                ## How to rate industry alignment
                Using the following scale, rate how closely each keyword aligns with the industry. The specific values represent key reference points or milestones, but your rating should lie on the continuous scale between 0 and 100 in proximity to those milestones.
                100 = Perfect alignment: The keyword is highly relevant to the food and beverage, cafe culture, coffee shops industry.
                50  = Partial alignment: The keyword is somewhat related or adjacent to this industry.
                0   = Misalignment: The keyword is not related to this industry at all. In addition, ensure that keywords related to  are give a low rating.
                
                ## List of keywords to analyze
                150 ml to ounces, 150ml to ounces, 20 oz in pounds, 5 ounces to g, 5 ounces to gr, 5 ounces to grams, 6'2 in cm, 80 ml to ounces, a gallon of water is how many water bottles, bahama mama recipe, benefits of central business district, best afternoon tea london, best cafe near me, best espresso martini recipe, best local coffee shops, blue in spanish, cafe in meiner nähe, cafes to study, caffeine in diet coke, caffeine in espresso shot, chai tea benefits, chocolate cafe, chocolate house, chocolate rain, chocolate rain song, chocolate world, cinnamon tea benefits, coffee loophole recipe, coffee shops with wifi near me, cranberry in spanish, daiquiri recipe, dark chocolate benefits, definition for wholesome, est to ist, exit to eden, factory tea bar, french martini recipe, fruits start with i, gentleman in moscow book, gin and tonic recipe, ginger tea benefits, green bean casserole with fresh green beans, health food store, hershey chocolate world, hershey's chocolate world, hibiscus tea benefits, homemade hot chocolate, horchata recipe, hot chocolate near me, hot chocolate recipe, hot chocolate run, how many bottle of waters is a gallon, how many bottle waters are in a gallon, how many bottled waters are in a gallon, how many grams of sugar in a tablespoon of sugar, how many grams of sugar in a teaspoon, how many grams of sugar in a teaspoon of sugar, how many ounces of water to drink a day, how many water bottle are in a gallon, how many water bottles are in a gallon, how many water bottles are in a gallon of water, how many water bottles is a gallon of water, how many water bottles is in a gallon, how much caffeine in matcha, how much sugar can of coke, how much sugar i n can of coke, how much sugar in a can of coca cola, how much sugar in a can of coke, how much sugar in can of coke, how much sugar in can of soda, how much sugar is in a can of coke, how much sugar is in a can of pop, how much sugar is in a soda can, how much sugar is in coke can, how much sugars in a can of coke, how to descale a keurig, how to make hot chocolate, how to make hummingbird nectar, hugo spritz recipe, ingredients in coke, is carbonated water bad for you, is coconut water good for you, is coffee good for you, is coke zero bad for you, is dark chocolate good for you, is diet coke bad for you, is fluoride bad for you, is matcha good for you, is oat milk good for you, is sparkling water good for you, is vitamin water good for you, juice bar, kakawa chocolate house, lemon drop recipe, loco moco recipe, long island iced tea recipe, lychee martini recipe, lyrics to escape the pina colada song, mai tai recipe, mango smoothie recipe, martini recipe, matcha benefits, mg of caffeine in coffee, movita juice bar, nekter juice bar, oolong tea benefits, oz in grams, peppermint tea benefits, pina colada recipe, pomegranate benefits, raspberry leaf tea benefits, raw milk benefits, tea bar, time for tokyo, time in shan, toothsome chocolate emporium menu, virgin pina colada recipe, water in spanish, what's in spanish, whiskey drink recipes, whole foods coffee bar
                
                ## Output format to return
                Set of keywordsWithThemeAndConfidence objects. Each object must include:
                - The keyword
                - A confidence rating (0–100) indicating how well it aligns with the specified industry.""";
        log.info("User message: {}", userMessage);

        Flux<String> response = chatClient.prompt()
                .system(sp -> sp.text(systemPrompt).param("format", converter.getFormat()))
                .user(userMessage)
                .stream()
                .content();

        assertThat(response)
                .as("Check that the response is not null")
                .isNotNull();

        List<String> chunks = response.collectList().block();
        assertThat(chunks)
                .as("Check that we received multiple chunks")
                .hasSizeGreaterThan(1);

        String content = String.join("", chunks);

        AiResponseKeywordsAligned result = assertDoesNotThrow(() ->
                        converter.convert(content),
                "Response should be convertible to AiResponseKeywordsAligned");
        log.info("Result: {}", result);

        assertThat(result)
                .as("Check that the entity is not null")
                .isNotNull();
        assertThat(result.keywordsWithThemeAndConfidence())
                .as("Check if AI returned any keywords with confidence")
                .isNotNull()
                .isNotEmpty();
    }

Test that fails with ~1000 keywords

@Test
    void given1000Keywords_whenProcessingWithStreamingChatClient_thenResponseIsSuccessful() {
        ChatClient chatClient = ChatClient.create(chatModel);
        var converter = new BeanOutputConverter<>(new ParameterizedTypeReference<AiResponseKeywordsAligned>() {});
        String systemPrompt = "You are a keyword research assistant specialized in SEO and Google Keyword Planner, helping users discover high-quality, long-tail keywords for niche businesses. {format}";

        String keywordsToAnalyze = create1000KeywordsToAnalyze();
        @NotNull Set<String> keywords = Arrays.stream(keywordsToAnalyze.split(","))
                .map(String::trim)
                .collect(Collectors.toSet());
        assertThat(keywords)
                .as("Verify we have exactly 1000 keywords to process")
                .hasSize(1000);

        String userMessage = """
                ## How to rate industry alignment
                Using the following scale, rate how closely each keyword aligns with the industry. The specific values represent key reference points or milestones, but your rating should lie on the continuous scale between 0 and 100 in proximity to those milestones.
                100 = Perfect alignment: The keyword is highly relevant to the food and beverage, cafe culture, coffee shops industry.
                50  = Partial alignment: The keyword is somewhat related or adjacent to this industry.
                0   = Misalignment: The keyword is not related to this industry at all. In addition, ensure that keywords related to  are give a low rating.
                
                ## List of keywords to analyze
                %s
                
                ## Output format to return
                Set of keywordsWithThemeAndConfidence objects. Each object must include:
                - The keyword
                - A confidence rating (0–100) indicating how well it aligns with the specified industry."""
                .formatted(keywordsToAnalyze);
        log.info("User message: {}", userMessage);

        Flux<String> response = chatClient.prompt()
                .system(sp -> sp.text(systemPrompt).param("format", converter.getFormat()))
                .user(userMessage)
                .stream()
                .content();

        assertThat(response)
                .as("Check that the response is not null")
                .isNotNull();

        List<String> chunks = response.collectList().block();
        assertThat(chunks)
                .as("Check that we received multiple chunks")
                .hasSizeGreaterThan(1);

        String content = String.join("", chunks);

        AiResponseKeywordsAligned result = assertDoesNotThrow(() ->
                        converter.convert(content),
                "Response should be convertible to AiResponseKeywordsAligned");
        log.info("Result: {}", result);

        assertThat(result)
                .as("Check that the entity is not null")
                .isNotNull();
        assertThat(result.keywordsWithThemeAndConfidence())
                .as("Check if AI returned any keywords with confidence")
                .isNotNull()
                .isNotEmpty();
    }

Config

@SpringBootConfiguration
public class OpenAiTestConfiguration {

    @Bean
    public OpenAiApi openAiApi() {
        return new OpenAiApi(getApiKey());
    }

    @Bean
    public OpenAiChatModel openAiChatModel(OpenAiApi api) {
        return new OpenAiChatModel(api, OpenAiChatOptions.builder()
                .model(ChatModel.GPT_4_O_MINI)
                .temperature(0.2)
                .build());
    }

    private String getApiKey() {
        String apiKey = System.getenv("SPRING_AI_OPENAI_API_KEY");
        if (!StringUtils.hasText(apiKey)) {
            throw new IllegalArgumentException("You must provide an API key via OPENAI_API_KEY.");
        }
        return apiKey;
    }

}