Proposal: Replace regex-based shell command analysis with tree-sitter-bash

Problem

The security module (openhands-sdk/openhands/sdk/security/defense_in_depth/) relies on regex pattern matching against flattened shell command text. The terminal module (openhands-tools/openhands/tools/terminal/) uses bashlex for command splitting and escaping. Both approaches have fundamental limitations that a proper parser would solve.

Regex cannot understand shell syntax

The patterns in pattern.py (lines 70–107) and policy_rails.py (lines 78–94) are well-designed and hardened with normalization, but regex fundamentally cannot understand quoting, escaping, operators, or nesting. This creates bypass classes that no amount of regex refinement can close:

Bypass	Why it works	Reference
r"m" -rf /	Quoted segment breaks \brm\b word boundary	pattern.py:73
/bin/rm -rf /	Path-qualified command breaks \brm\s+ anchor	pattern.py:73
dd with of=/dev/ beyond 100 chars	.{0,100} window is too small	pattern.py:80, policy_rails.py:105
curl ... | node	node not in hardcoded interpreter list (sh|bash|python|perl|ruby)	policy_rails.py:82, documented xfail in test_adversarial.py:294–314
curl x|bash	Escaped pipe is literal text, but regex sees | as pipe operator	pattern.py:92
bash -c 'rm -rf /'	Nested command inside -c argument is opaque to regex	pattern.py:73
Payload past 30KB	_EXTRACT_HARD_CAP truncates to prevent ReDoS	utils.py:41, documented xfail in test_adversarial.py:241–259
rm --no-preserve-root -rf /	Intervening flag breaks the strict ordering regex expects	pattern.py:73, policy_rails.py:87–94 (subject of PR #2718)

bashlex is unmaintained and crashes on valid input

openhands-tools depends on bashlex>=0.18 (pyproject.toml:9). It's used in two functions in terminal/utils/command.py:

• split_bash_commands() (lines 14–67): parses multi-command strings to enforce single-command execution. Catches four exception types (ParsingError, NotImplementedError, TypeError, AttributeError) because bashlex crashes on various valid shell constructs. On any failure, it silently returns the raw string — bypassing multi-command validation entirely.

• escape_bash_special_chars() (lines 70–151): walks the bashlex AST to escape unquoted operators. Uses regex on word text (lines 109–113) to detect quoted strings and command substitutions ("...", '...', $(...), `...`), because bashlex doesn't expose these as distinct node types. Same 4-exception catch-all with silent fallback.

bashlex was last meaningfully updated around 2020, has a single maintainer, and has no error recovery — any malformed input (common from LLM output) produces a ParsingError.

PR #2718 is heading in the right direction but adds complexity

PR #2718 improved rm -rf detection by replacing the regex with shlex tokenization + a hand-rolled shell segment splitter. This correctly fixes the flag-ordering bypass, but adds ~170 lines of new parsing code to utils.py: a 60-line char-by-char parser for ;/&&/||/|, a shlex.split() wrapper with fallback, regex-based redirection detection, and token-based flag inspection.

shlex is not a shell parser — it doesn't understand pipes, redirections, command substitution, or compound commands. Every new detection pattern will require more hand-rolled parsing on top of it.

The codebase parses bash three different ways

Package	Module	Approach	Limitations
openhands-sdk	security/defense_in_depth/pattern.py	Compiled regex patterns	Can't understand syntax
openhands-sdk	security/defense_in_depth/policy_rails.py	Composed regex conditions	Same
openhands-sdk	security/defense_in_depth/utils.py (PR #2718)	shlex + hand-rolled splitter	Not a shell parser
openhands-tools	terminal/utils/command.py	bashlex	Crashes on valid input, unmaintained, no error recovery
openhands-sdk	context/skills/execute.py	Regex for !`cmd` extraction	No security check before subprocess.run(shell=True)

---

Proposal

Replace all shell parsing with tree-sitter + tree-sitter-bash.

Why tree-sitter

• Full parse tree: commands, arguments, flags, operators, redirections, quoting — all understood structurally
• Error recovery: malformed shell (common from LLM output) still produces a partial AST with error nodes instead of raising exceptions. This directly solves the bashlex crash problem
• O(n) parsing: eliminates the 30KB hard cap (utils.py:41) currently needed to prevent regex ReDoS (documented xfail in test_adversarial.py:241–259)
• Actively maintained: used by GitHub (syntax highlighting for every repo), Neovim, Zed; the bash grammar is comprehensive and regularly tested
• Small footprint: tree-sitter (~2 MB) + tree-sitter-bash grammar

---

What changes

1. PatternSecurityAnalyzer (pattern.py)

Replace the HIGH-risk regex patterns (lines 70–101) with tree-sitter queries. Example for rm -rf:

import tree_sitter_bash as tsbash
from tree_sitter import Language, Parser

BASH = Language(tsbash.language())
parser = Parser(BASH)

def _has_rm_recursive_force(command: str) -> bool:
    tree = parser.parse(command.encode())
    for node in _walk_commands(tree.root_node):
        words = [c.text.decode() for c in node.children if c.type == "word"]
        if not words or not (words[0] == "rm" or words[0].endswith("/rm")):
            continue
        flags = set()
        for w in words[1:]:
            if w == "--":
                break
            if w == "--recursive":
                flags.add("r")
            elif w == "--force":
                flags.add("f")
            elif w.startswith("-") and w != "-":
                flags.update(w[1:])
        if {"r", "f"} <= flags or {"R", "f"} <= flags:
            return True
    return False

This immediately fixes:

• /bin/rm -rf / — words[0].endswith("/rm") handles path-qualified commands
• rm --no-preserve-root -rf / — flags in any position
• rm / -rf — flags after positional args

Similarly for dd of=/dev/ (extract of= from word nodes, no char-count window), curl | sh (query pipeline nodes, check RHS command name — not a hardcoded list), and mkfs.

Patterns that are not shell-specific stay as regex:

• Injection patterns (inject.override, inject.mode_switch, inject.identity) — NLP, not syntax
• Python code patterns (eval(), os.system(), subprocess.*()) — not bash

2. PolicyRailSecurityAnalyzer (policy_rails.py)

• Fetch-to-exec rail (lines 78–102): query the parse tree for pipeline nodes. Check if LHS is curl/wget and RHS is any command — eliminates the hardcoded interpreter list. This fixes the curl | node xfail (test_adversarial.py:294–314)
• Raw-disk-op rail (lines 104–112): parse dd command, extract of= value from word nodes, check if it starts with /dev/. No fixed-width window
• Catastrophic-delete rail (lines 114–128): reuse _has_rm_recursive_force from above, then check target arguments against the critical path list

3. Utils cleanup (utils.py)

If PR #2718 merges first, delete the hand-rolled parsing it adds:

• _split_shell_segments() — replaced by tree-sitter's understanding of ;, &&, ||, |
• _tokenize_shell_segment() — replaced by tree-sitter node children
• _is_redirection_token() — tree-sitter has a redirect node type
• _segment_has_rm_recursive_force() and _has_rm_recursive_force() — replaced by tree-sitter version

The normalization pipeline (_normalize, lines 310–338) stays — invisible char stripping and NFKC normalization are still valuable as pre-processing before tree-sitter parsing.

The extraction logic (_extract_exec_segments, _extract_content, etc.) stays — it controls which fields are scanned.

4. openhands-tools: replace bashlex (terminal/utils/command.py)

split_bash_commands() (lines 14–67):

• tree-sitter parses malformed input gracefully (partial AST with error nodes) instead of raising 4 exception types
• The catch-all fallback that silently bypasses multi-command validation disappears
• Multi-command detection becomes: walk the tree for top-level command nodes

escape_bash_special_chars() (lines 70–151):

• tree-sitter provides explicit node types for string (double-quoted), raw_string (single-quoted), command_substitution, heredoc_body — no need to detect these via regex on string content (lines 109–113)
• The recursive visit_node() + regex approach becomes a tree walk

Dependency change in openhands-tools/pyproject.toml:

- "bashlex>=0.18",
+ "tree-sitter>=0.24",
+ "tree-sitter-bash>=0.23",

The public API of both functions stays the same — only the internal implementation changes.

5. Skill execution (context/skills/execute.py)

Currently, commands extracted via the !`command` regex (line 44–48) are executed directly via subprocess.run(shell=True) (line 66) with no security analysis. With tree-sitter available in openhands-sdk, extracted commands should be run through the security analyzer before execution.

6. Shared parser across packages

The key benefit: both packages share one parser. Today openhands-sdk (security) uses regex + shlex and openhands-tools (terminal) uses bashlex. With tree-sitter-bash, both share one dependency and one parsing strategy. Security checks can run on the same AST the terminal uses for command splitting.

---

What stays the same

• Normalization pipeline (utils.py:310–338) — invisible char stripping, NFKC, whitespace collapse remain as pre-processing
• Extraction architecture (utils.py:75–157) — whitelisted field extraction, per-segment evaluation, hard cap (can be raised or removed with O(n) parsing)
• Injection patterns (pattern.py:112–132) — NLP patterns, not shell syntax; regex is appropriate
• Python code patterns (pattern.py:82–89) — eval(), os.system(), subprocess.*() are not bash
• Per-segment evaluation (policy_rails.py:68–130) — correct architecture, each segment is now parsed instead of regex-scanned
• Terminal escape filtering (openhands-tools/.../escape_filter.py) — byte-level ANSI patterns, not bash syntax

Out of scope

• Cyrillic homoglyph bypasses — charset problem, requires Unicode TR39 confusable tables (documented xfail in test_adversarial.py:201–219)
• Combining mark bypasses — requires diacritic stripping (documented xfail in test_adversarial.py:221–239)
• Variable indirection (x=rm; $x -rf /) — requires runtime evaluation
• Alias/function expansion — requires shell state

Test plan

• All existing tests in test_pattern.py, test_policy_rails.py, test_adversarial.py must pass
• The hard-cap xfail (test_adversarial.py:241–259) can be converted to a passing test if the cap is raised/removed
• The curl | node xfail (test_adversarial.py:294–314) can be converted to a passing test with pipeline-based detection
• Add regression tests for bypasses tree-sitter fixes:
  • /bin/rm -rf / (path-qualified command)
  • bash -c 'rm -rf /' (nested command)
  • dd if=/dev/zero of=/dev/sda with of= beyond 100 chars
• All existing openhands-tools tests for split_bash_commands and escape_bash_special_chars must pass

(cc. @csmith49 @Fieldnote-Echo )

Start with openhands-sdk/openhands/sdk/security/defense_in_depth/pattern.py, policy_rails.py, and utils.py, then inspect openhands-tools/openhands/tools/terminal/utils/command.py and both packages' pyproject.toml files. Run the existing test_pattern.py, test_policy_rails.py, test_adversarial.py, and terminal tests to establish current behavior. Done means the shared tree-sitter-bash approach preserves public APIs, covers the listed regressions, and passes the existing suites.