OpenHands / OpenHands/software-agent-sdk
[Bug]: UTF-8 Chinese text misclassified as binary when sampling splits a multibyte character
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Is there an existing issue for the same bug?
- Searched SDK issues for
binaryornotandbinary UTF-8; found no issue covering this sample-truncation problem.
Bug Description
FileEditor.validate_file() uses binaryornot.is_binary() to detect binary files. Its 1,024-byte sample can split a multibyte UTF-8 character, causing an otherwise valid Chinese Markdown file to be classified as binary and preventing it from being viewed or edited.
Expected Behavior
Valid UTF-8 text without binary control characters should be viewable and editable through the file editor.
Actual Behavior
Raises FileValidationError: File appears to be binary and this file type cannot be read or edited by this tool.
Steps to Reproduce / Minimal Code Sample
Run from the SDK source directory after installing dependencies:
uv sync --frozen --dev
uv run python - <<'PY'
from pathlib import Path
from tempfile import TemporaryDirectory
from openhands.tools.file_editor.editor import FileEditor
with TemporaryDirectory() as directory:
path = Path(directory) / "report.md"
# Chinese text encoded as escapes to keep this description in English.
path.write_text("\u66f4\u6df1\u4e00\u5c42\u7684\u5f52\u5c5e\u611f\u3002" * 200, encoding="utf-8")
result = FileEditor()(command="view", path=str(path))
assert not result.is_error
PY
For this content, the first 1,023 or 1,026 bytes decode successfully as UTF-8, and is_binary_string() returns False. At 1,024 or 1,025 bytes, the sample ends in a truncated character and the function returns True. The complete file decodes successfully using strict UTF-8 decoding.
Acceptance Criteria
- Valid Chinese text can be viewed and replaced at different sample alignments, with correct full UTF-8 contents after replacement.
- A valid UTF-8 prefix must not allow a file containing subsequent NUL bytes, binary control characters, or invalid UTF-8 bytes to bypass validation.
- Preserve existing file-size limits, image handling, and other file-editor behavior.
Installation Method / SDK Version
Installed from source using uv sync --frozen --dev; reproduced on upstream main b5c8ab950 (SDK 1.47.0).
Python Version / Operating System / Model
Python 3.13.7, Linux; no model calls required.
Additional Context
Related to, but distinct from, PR #4784: that PR changes encoding selection in EncodingManager. This issue occurs in the binary rejection check in validate_file() and requires a separate fix.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at FileEditor.validate_file() in openhands.tools.file_editor.editor and run the provided UTF-8 reproduction after installing dependencies with uv sync --frozen --dev. Add focused coverage for different sample alignments, then verify that valid UTF-8 remains editable while later NUL bytes, binary controls, or invalid UTF-8 still fail validation and existing size and image behavior is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100