OpenHands / OpenHands/software-agent-sdk
goal judge: split judge prompt into system + user messages
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Problem
judge_goal() in openhands-sdk/openhands/sdk/conversation/goal/judge.py builds its LLM request as a single user message with no system message before it:
response = judge_llm.completion(
messages=[Message(role="user", content=[TextContent(text=prompt)])]
)
Placed at lines ~65–66. The prompt embeds JUDGE_PROMPT (steering/format instructions) together with the objective + transcript.
This violates the repo convention that every LLM request starts with a system message (see parent tracking issue #5144). It also means the judge's verdict format instructions live in the user slot rather than the system/instructions slot, which matters for the Responses API path where a lone user message ends up entirely in input.
Proposed change
Split the judge request into a canonical system + user pair:
systemmessage carryingJUDGE_PROMPT(the steering / verdict-format instructions).usermessage carrying the objective + transcript payload.
Alternatively, prepend a small system message with the shared judge role definition. Either way the request must begin with a system role.
Parent issue: #5144
Note: the judge is an independent, single-turn, side-channel call (judge_llm.completion(messages=[...]), forced stream=False); it builds its message list from scratch and does not touch the agent's main conversation history, so adding a system lead-in is safe and isolated.
This issue was created by an AI agent (OpenHands) on behalf of juanmichelini.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Open openhands-sdk/openhands/sdk/conversation/goal/judge.py and inspect judge_goal() around lines 65–66, along with the request conventions tracked by issue #5144. Verify how JUDGE_PROMPT and the objective/transcript are assembled, then run the relevant judge or SDK tests. Done means the independent request starts with a system message and preserves the existing verdict behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend-api-design
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100