OpenHands / OpenHands/software-agent-sdk

[Enhancement] Create CLI-specific prompts that remove browser action space

Open
#4,237 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement roadmap
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

What problem or use case are you trying to solve?

When using OpenHands CLI, the agent attempts to use browser actions to parse images and interact with web content, but the CLI runtime (CLIRuntime) doesn't support browser functionality. This leads to several issues:

  • Failed image parsing: The agent tries to use browser actions to view/parse images but fails because browser functionality is not available in CLI mode
  • Confusing error messages: Users get errors about browser functionality not being implemented in CLIRuntime
  • Suboptimal agent behavior: The agent may waste time attempting browser-based solutions when CLI-appropriate alternatives exist
  • Poor user experience: Users are confused when the agent suggests browser-based solutions that cannot work in CLI mode

Root Cause

While the CLI runtime automatically disables browsing in the agent configuration (enable_browsing = False), the agent's system prompts still contain references to browser actions. The agent may still attempt to use browser functionality because:

  • The system prompt doesn't change based on runtime type
  • The agent may have learned patterns that involve browser usage from training
  • The current prompt doesn't explicitly guide the agent away from browser-based solutions

Describe the UX or technical implementation you have in mind

Proposed Solution

Create a separate set of CLI-specific prompts that:

  • Remove all browser action references: No mention of browser tools, web browsing, or browser-based image viewing
  • Emphasize CLI-appropriate alternatives: Focus on file system operations, command-line tools, and local file processing
  • Provide clear guidance: Explicitly state that browser functionality is not available and suggest alternatives

Implementation Plan

  • Create CLI-specific prompt templates: openhands/agenthub/codeact_agent/prompts/cli/system_prompt.j2
  • Detect when running in CLI mode (check runtime == 'cli' or enable_browsing == False), and set agent_config.enable_browsing=false when that's true.

Focus on available tools: bash, file editing, Python execution (if enabled)

Additional context

If you find this feature request or enhancement useful, make sure to add a 👍 to the issue

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with openhands/agenthub/codeact_agent/prompts/cli/system_prompt.j2 and inspect how the existing system prompt and agent_config.enable_browsing are selected. Trace the CLI runtime path and check how runtime or browsing configuration is passed into the agent. Done means CLI prompts avoid browser actions, emphasize bash, file editing, and optional Python execution, and browsing is disabled for CLI mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.