web-infra-dev / web-infra-dev/midscene

[Bug]: Using version 1.10.5, the issue frequently occurs where the JSON format returned by the LLM is incorrect, causing midscene to report errors and fail.

Open
#2,831 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
14.9k
Forks
1.2k
Avg merge
1d 14h
Merged PRs (30d)
96

Description

Version
System:
    OS: Windows 11 10.0.26200
    CPU: (8) x64 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
    Memory: 2.10 GB / 15.71 GB
  Browsers:
    Chrome: 150.0.7871.116
    Edge: Chromium (150.0.4078.65)
    Internet Explorer: 11.0.26100.8115
  npmPackages:
    @midscene/cli: ^1.10.5 => 1.10.5
    @midscene/web: ^1.10.5 => 1.10.5
Details
Image 使用1.10.5版本,频繁出现LLM返回的json格式不对导致midscene报错失败的问题,之前的版本没有出现过,是否是新版本出现的问题? json的{} 变成了[],LLM格式问题是否应该加校验或格式清洗手段 Image
Reproduce link

Reproduce Steps

LLM信息
MIDSCENE_MODEL_NAME=qwen3.7-plus
MIDSCENE_MODEL_FAMILY=qwen3
发送的是点击事件
ai-call.log
[2026-07-20T19:21:57.585+08:00] adapter chat completion params: {"config":{"temperature":0,"enable_thinking":false}}
[2026-07-20T19:21:57.585+08:00] sending request to qwen3.7-plus
[2026-07-20T19:22:00.080+08:00] response reasoning content:
[2026-07-20T19:22:00.080+08:00] response content: ```json
[
"bbox": [345, 529, 391, 565],
"errors": []
}

[2026-07-20T19:22:00.081+08:00] adapter chat completion params: {"config":{"temperature":0,"enable_thinking":false}}
[2026-07-20T19:22:00.082+08:00] sending request to qwen3.7-plus
[2026-07-20T19:22:00.245+08:00] response reasoning content: 
[2026-07-20T19:22:00.245+08:00] response content: ```json
[
  "bbox": [345, 529, 391, 565],
  "errors": []
}

ServiceError: failed to locate element:
failed to parse LLM response into JSON. Error - Error: Unexpected character "`" at position 59. Response -

[
 "bbox": [345, 529, 391, 565],
 "errors": []
}
at Service.locate (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\service\index.ts:181:13)
at processTicksAndRejections (node:internal/process/task_queues:104:5)
at Object.executor (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\agent\task-builder.ts:551:28)
at TaskRunner.flush (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\task-runner.ts:335:25)
at TaskExecutor.runPlans (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\agent\tasks.ts:349:20)
at PuppeteerPageAgent.callActionInActionSpace (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\agent\agent.ts:697:24)
at D:\project_code\midscene-monitor\node_modules\@midscene\core\src\agent\agent.ts:719:7
at PuppeteerPageAgent.aiTap (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\agent\agent.ts:718:5)
at ScriptPlayer.playFlowItem (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\yaml\player.ts:585:7)
at ScriptPlayer.playTask (D:\project_code\midscene-monitor\node_modules\@midscene\core\src\yaml\player.ts:308:9)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/service/index.ts:181, where the stack trace reports the LLM response parsing failure, and inspect the response handling used by the locate flow. Reproduce with the logged fenced response if possible, then determine the expected object shape and how malformed JSON should be handled. Done means the reported response no longer causes an unhandled parsing failure and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.