`response` field has incorrect type when using JSON Mode

Current Issue

See https://github.com/cloudflare/cf-platform-issues/issues/18#issuecomment-5319666039

Original Issue

What versions & operating system are you using?

  System:
    OS: macOS 15.7.1
    CPU: (12) arm64 Apple M2 Pro
    Memory: 149.84 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 24.11.0 - /Users/gamer/.nvm/versions/node/v24.11.0/bin/node
    npm: 11.6.1 - /Users/gamer/.nvm/versions/node/v24.11.0/bin/npm
    pnpm: 10.28.2 - /opt/homebrew/bin/pnpm
    bun: 1.3.7 - /Users/gamer/.bun/bin/bun
    Deno: 2.1.4 - /Users/gamer/.deno/bin/deno
  npmPackages:
    @cloudflare/workers-types: ^4.20260203.0 => 4.20260203.0 
    wrangler: ^4.62.0 => 4.62.0 

Please provide a link to a minimal reproduction

https://github.com/earth-app/cloud

Describe the Bug

There seems to be issues with the json_schema response type and trying to use the huggingface model @hf/nousresearch/hermes-2-pro-mistral-7b.

Snippet from earth-app/cloud:

const quizModel = '@hf/nousresearch/hermes-2-pro-mistral-7b';

export type ArticleQuizQuestion = {
	question: string;
} & (
	| {
			type: 'multiple_choice';
			options: string[];
			correct_answer: string;
			correct_answer_index: number;
	  }
	| {
			type: 'true_false';
			options: ('true' | 'false')[];
			correct_answer: 'true' | 'false';
			correct_answer_index: number;
			is_true: boolean;
			is_false: boolean;
	  }
);

const articleQuizAiSchema = {
	type: 'object',
	properties: {
		questions: {
			type: 'array',
			minItems: 2,
			maxItems: 5,
			items: {
				type: 'object',
				properties: {
					question: {
						type: 'string',
						maxLength: 100
					},
					type: { type: 'string', enum: ['multiple_choice', 'true_false'] },
					options: {
						type: 'array',
						maxItems: 4,
						items: {
							type: 'string',
							maxLength: 60
						}
					},
					correct_answer: { type: 'string' },
					correct_answer_index: { type: 'number' },
					is_true: { type: 'boolean' },
					is_false: { type: 'boolean' }
				},
				required: ['question', 'type', 'options', 'correct_answer', 'correct_answer_index']
			}
		}
	},
	required: ['questions']
};

const QUIZ_CUTOFF = 300;

export async function createArticleQuiz(
	article: Pick<Article, 'title' | 'content' | 'ocean' | 'tags'>,
	ai: Ai
): Promise<ArticleQuizQuestion[]> {
	try {
		const content = article.ocean.content || article.ocean.abstract || '';
		const firstPart = content.substring(0, QUIZ_CUTOFF);
		const lastPart = content.substring(content.length - QUIZ_CUTOFF);
		const quizResult = await ai.run(quizModel, {
			messages: [
				{ role: 'system', content: prompts.articleQuizSystemMessage.trim() },
				{
					role: 'user',
					content:
						content.length > QUIZ_CUTOFF * 2
							? firstPart + '... (truncated) ...' + lastPart
							: content
				},
				{ role: 'user', content: prompts.articleQuizPrompt.trim() }
			],
			max_tokens: 512,
			temperature: 0.3,
			response_format: {
				type: 'json_schema',
				json_schema: articleQuizAiSchema
			}
		});

		const parsedResult = JSON.parse(quizResult?.response || '{"questions":[]}');
		const quizData = (parsedResult.questions || []) as ArticleQuizQuestion[];
		return quizData;
	} catch (error) {
		console.error('Quiz generation failed, continuing without quiz:', error);
		return []; // Return empty quiz rather than failing article creation
	}
}

Changing max_tokens does not matter, nor does trying to downsize the schema with maxLength / maxItems or article content provided with QUIZ_CUTOFF. I still get the same 1024 error, and large cutoffs only produce smaller movements toward the 1024 cutoff.

It seems that max_tokens should be passed to max_new_tokens, but for some reason it is being ignored. This is especially true since the docs specify that the default is 256, but it always 1024.

Please provide any relevant error logs

Quiz generation failed, continuing without quiz: InferenceUpstreamError [AiError]: 3025: error with TGI API: failed to parse response (invalid type: map, expected a sequence at line 1 column 0): {"error":"Input validation error: `max_new_tokens` must be <= 1024. Given: 1321","error_type":"validation"}

      at Ai._parseError (cloudflare-internal:ai-api:208:24)
      at async Ai.run (cloudflare-internal:ai-api:186:19)
      at async createArticleQuiz
  (file:///Users/gamer/gmitch215/earth-app/cloud/src/content/boat.ts:523:22)
      at null.<anonymous> (async
  file:///Users/gamer/gmitch215/earth-app/cloud/.wrangler/tmp/dev-Ftlejr/index.js:105986:22)
      at [object Object]
      at async Object.scheduled
  (file:///Users/gamer/gmitch215/earth-app/cloud/src/scheduled.ts:36:3)
      at async scheduled
  (file:///Users/gamer/gmitch215/earth-app/cloud/node_modules/wrangler/templates/middleware/middleware-scheduled.ts:8:3)
      at async drainBody
  (file:///Users/gamer/gmitch215/earth-app/cloud/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10)

La reproduction appelle Ai.run depuis src/content/boat.ts:523, et la planification l’atteint depuis src/scheduled.ts:36 ; commencez par suivre la manière dont response_format et max_tokens sont traduits pour le modèle Hermes. Reproduisez l’erreur de validation 1321 max_new_tokens, puis vérifiez que la requête respecte la limite de tokens de l’appelant et renvoie la réponse JSON attendue.