OpenHands / OpenHands/software-agent-sdk
[Enhancement] Create CLI-specific prompts that remove browser action space
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
What problem or use case are you trying to solve?
When using OpenHands CLI, the agent attempts to use browser actions to parse images and interact with web content, but the CLI runtime (CLIRuntime) doesn't support browser functionality. This leads to several issues:
- Failed image parsing: The agent tries to use browser actions to view/parse images but fails because browser functionality is not available in CLI mode
- Confusing error messages: Users get errors about browser functionality not being implemented in CLIRuntime
- Suboptimal agent behavior: The agent may waste time attempting browser-based solutions when CLI-appropriate alternatives exist
- Poor user experience: Users are confused when the agent suggests browser-based solutions that cannot work in CLI mode
Root Cause
While the CLI runtime automatically disables browsing in the agent configuration (enable_browsing = False), the agent's system prompts still contain references to browser actions. The agent may still attempt to use browser functionality because:
- The system prompt doesn't change based on runtime type
- The agent may have learned patterns that involve browser usage from training
- The current prompt doesn't explicitly guide the agent away from browser-based solutions
Describe the UX or technical implementation you have in mind
Proposed Solution
Create a separate set of CLI-specific prompts that:
- Remove all browser action references: No mention of browser tools, web browsing, or browser-based image viewing
- Emphasize CLI-appropriate alternatives: Focus on file system operations, command-line tools, and local file processing
- Provide clear guidance: Explicitly state that browser functionality is not available and suggest alternatives
Implementation Plan
- Create CLI-specific prompt templates: openhands/agenthub/codeact_agent/prompts/cli/system_prompt.j2
- Detect when running in CLI mode (check runtime == 'cli' or enable_browsing == False), and set
agent_config.enable_browsing=falsewhen that's true.
Focus on available tools: bash, file editing, Python execution (if enabled)
Additional context
If you find this feature request or enhancement useful, make sure to add a 👍 to the issue
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with openhands/agenthub/codeact_agent/prompts/cli/system_prompt.j2 and inspect how the existing system prompt and agent_config.enable_browsing are selected. Trace the CLI runtime path and check how runtime or browsing configuration is passed into the agent. Done means CLI prompts avoid browser actions, emphasize bash, file editing, and optional Python execution, and browsing is disabled for CLI mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100