OpenHands / OpenHands/software-agent-sdk

goal judge: split judge prompt into system + user messages

Open Beginner friendly
#5,145 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

invariants
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Problem

judge_goal() in openhands-sdk/openhands/sdk/conversation/goal/judge.py builds its LLM request as a single user message with no system message before it:

response = judge_llm.completion(
    messages=[Message(role="user", content=[TextContent(text=prompt)])]
)

Placed at lines ~65–66. The prompt embeds JUDGE_PROMPT (steering/format instructions) together with the objective + transcript.

This violates the repo convention that every LLM request starts with a system message (see parent tracking issue #5144). It also means the judge's verdict format instructions live in the user slot rather than the system/instructions slot, which matters for the Responses API path where a lone user message ends up entirely in input.

Proposed change

Split the judge request into a canonical system + user pair:

  • system message carrying JUDGE_PROMPT (the steering / verdict-format instructions).
  • user message carrying the objective + transcript payload.

Alternatively, prepend a small system message with the shared judge role definition. Either way the request must begin with a system role.

Parent issue: #5144

Note: the judge is an independent, single-turn, side-channel call (judge_llm.completion(messages=[...]), forced stream=False); it builds its message list from scratch and does not touch the agent's main conversation history, so adding a system lead-in is safe and isolated.


This issue was created by an AI agent (OpenHands) on behalf of juanmichelini.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open openhands-sdk/openhands/sdk/conversation/goal/judge.py and inspect judge_goal() around lines 65–66, along with the request conventions tracked by issue #5144. Verify how JUDGE_PROMPT and the objective/transcript are assembled, then run the relevant judge or SDK tests. Done means the independent request starts with a system message and preserves the existing verdict behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.