OpenHands / OpenHands/software-agent-sdk

[Bug]: UTF-8 Chinese text misclassified as binary when sampling splits a multibyte character

Open
#5,038 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug priority:medium release-note-required tools
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Is there an existing issue for the same bug?
  • Searched SDK issues for binaryornot and binary UTF-8; found no issue covering this sample-truncation problem.
Bug Description

FileEditor.validate_file() uses binaryornot.is_binary() to detect binary files. Its 1,024-byte sample can split a multibyte UTF-8 character, causing an otherwise valid Chinese Markdown file to be classified as binary and preventing it from being viewed or edited.

Expected Behavior

Valid UTF-8 text without binary control characters should be viewable and editable through the file editor.

Actual Behavior

Raises FileValidationError: File appears to be binary and this file type cannot be read or edited by this tool.

Steps to Reproduce / Minimal Code Sample

Run from the SDK source directory after installing dependencies:

uv sync --frozen --dev
uv run python - <<'PY'
from pathlib import Path
from tempfile import TemporaryDirectory
from openhands.tools.file_editor.editor import FileEditor

with TemporaryDirectory() as directory:
    path = Path(directory) / "report.md"
    # Chinese text encoded as escapes to keep this description in English.
    path.write_text("\u66f4\u6df1\u4e00\u5c42\u7684\u5f52\u5c5e\u611f\u3002" * 200, encoding="utf-8")
    result = FileEditor()(command="view", path=str(path))
    assert not result.is_error
PY

For this content, the first 1,023 or 1,026 bytes decode successfully as UTF-8, and is_binary_string() returns False. At 1,024 or 1,025 bytes, the sample ends in a truncated character and the function returns True. The complete file decodes successfully using strict UTF-8 decoding.

Acceptance Criteria
  • Valid Chinese text can be viewed and replaced at different sample alignments, with correct full UTF-8 contents after replacement.
  • A valid UTF-8 prefix must not allow a file containing subsequent NUL bytes, binary control characters, or invalid UTF-8 bytes to bypass validation.
  • Preserve existing file-size limits, image handling, and other file-editor behavior.
Installation Method / SDK Version

Installed from source using uv sync --frozen --dev; reproduced on upstream main b5c8ab950 (SDK 1.47.0).

Python Version / Operating System / Model

Python 3.13.7, Linux; no model calls required.

Additional Context

Related to, but distinct from, PR #4784: that PR changes encoding selection in EncodingManager. This issue occurs in the binary rejection check in validate_file() and requires a separate fix.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at FileEditor.validate_file() in openhands.tools.file_editor.editor and run the provided UTF-8 reproduction after installing dependencies with uv sync --frozen --dev. Add focused coverage for different sample alignments, then verify that valid UTF-8 remains editable while later NUL bytes, binary controls, or invalid UTF-8 still fail validation and existing size and image behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.