[vertexai] Inconsistent behavior when using responseSchema with Gemini Flash models, malformed or repetitive JSON output unless schema is removed

We’ve encountered a reproducible issue when using the Java Vertex AI client (com.google.cloud:google-cloud-vertexai:1.18.0) with Gemini Flash models for structured text generation.

When a responseSchema is attached to the GenerationConfig, the model intermittently produces malformed or repetitive JSON outputs, often looping text fragments or inserting stray newline escape sequences until the max output token limit is reached.

Removing the schema entirely eliminates the issue, and the same prompt setup works correctly in the Python Vertex AI SDK, suggesting this may be SDK-specific or related to how the Java client serializes the schema.

Environment details

Key | Value
-- | --
API | Vertex AI Generative AI (Java)
Library | com.google.cloud:google-cloud-vertexai:1.18.0
Java version | 21
OS | Windows 11
Models tested | Gemini 2.0 Flash, Gemini 2.5 Flash Lite, Gemini 2.5 Flash
Behavior | Issue occurs with 2.0 Flash and 2.5 Flash Lite; 2.5 Flash mitigates it partially

Steps to reproduce

1. 
   

   Configure a GenerativeModel with deterministic decoding:

   
   
   
   
   • 
     

     temperature = 0.0f, topP = 0.0f, topK = 1, candidateCount = 1, seed = 42

     
     
   
   
   • 
     

     responseMimeType = "application/json"

     
     
   
   
   
   


2. 
   

   Attach a complex responseSchema describing nested arrays and objects (see example below).

   
   


3. 
   

   Send a document-extraction prompt requesting structured JSON per the schema.

   
   


4. 
   

   Observe that:

   
   
   
   
   • 
     

     The model often ignores the schema’s structure.

     
     
   
   
   • 
     

     Output becomes recursive or repetitive ("Company Company Company...").

     
     
   
   
   • 
     

     Output terminates abruptly at token limit with unclosed quotes or brackets.

     
     
   
   
   
   


5. 
   

   Remove the schema (keep all other settings identical).

   
   


6. 
   

   Observe that the output is now clean and well-formed JSON.

   
   

Code snippet (simplified)

GenerationConfig cfg = GenerationConfig.newBuilder()

    .setTemperature(0.0f)

    .setTopP(0.0f)

    .setTopK(1)

    .setCandidateCount(1)

    .setSeed(42)

    .setResponseMimeType("application/json")

    .setResponseSchema(ResponseSchemaFactory.getExtractionSchema()) // When set, issue occurs

    .build();


GenerativeModel model = baseModel

    .withSystemInstruction(ContentMaker.fromString(systemPrompt))

    .withGenerationConfig(cfg);

GenerateContentResponse response = model.generateContent(promptText);

String jsonOutput = response.getText(); // often malformed

Example schema shape:

Schema workExperience = Schema.newBuilder()

  .setType(Type.OBJECT)

  .putProperties("company", Schema.newBuilder().setType(Type.STRING).build())

  .putProperties("tenure", Schema.newBuilder().setType(Type.STRING).build())

  .putProperties("skills", Schema.newBuilder()

      .setType(Type.ARRAY)

      .setItems(Schema.newBuilder().setType(Type.STRING).build())

      .build())

  .build();

Observed output (excerpt, simulated)

{

  "experience": [

    {

      "company": "TechCorp TechCorp TechCorp TechCorp TechCorp ...",

      "tenure": "2 yrs",

      "skills": ["Java", "Spring Boot"]

    }

  ],

  "summary": "\n\n {\n.\n.\n\\n\\n\\n\\n\\n\\n\\n\\n\n"

}

Occasionally, the output fails JSON parsing due to missing closing quotes or brackets:

com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input:

 was expecting closing quote for a string value

 at [Source: (String)"{ "experience": [ { "company": "ABC

 "tenure": "3 yrs"...]; line: 1, column: 4211]

Expected behavior

When responseSchema is provided, the model should consistently honor the schema and produce syntactically valid JSON following the defined structure.

Additional context

• 
  

  Removing the schema entirely fixes the problem.

  
  


• 
  

  Using identical prompts and schema definitions in Python Vertex AI SDK does not reproduce the issue.

  
  


• 
  

  Switching to Gemini 2.5 Flash improves output stability, possibly due to increased reasoning or token budget.

  
  


• 
  

  This suggests the issue may lie in schema serialization or how the Java SDK encodes the request payload.

  
  

Would appreciate guidance on whether this is:

• 
  

  A known limitation or bug in the Java Vertex AI client,

  
  


• 
  

  A misalignment between the Java SDK’s schema format and backend expectations,

  
  


• 
  

  Or a potential model-side behavior that needs handling guidance.

  
  

从 Java Vertex AI 客户端的 GenerationConfig 和 responseSchema 入口开始，使用提供的 schema 和确定性生成设置来复现格式错误的输出。将移除 schema 时的请求行为与使用 Python Vertex AI SDK 时的行为进行比较，然后检查 Java 的 schema 序列化是否不同于预期的后端 payload。完成标准是：要么确认客户端修复能够生成符合 schema 约束的有效 JSON，要么记录证据证明该行为发生在模型侧。

由索引模型根据 Issue 内容生成。