mpfaffenberger / mpfaffenberger/code_puppy
Additional surrogate-sanitizing file read outside `code_puppy/tools`.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 814
- Forks
- 278
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 76
Description
Follow up on the ongoing issue #463 and PR #756
Currently, inside #756, code_puppy/tools/common.py::_sanitize_string is implemented like this:
def _sanitize_string(text: str) -> str:
"""Sanitize a string to remove invalid Unicode surrogates.
This handles encoding issues common on Windows with copy-paste operations.
"""
if not text:
return text
try:
# Try encoding — if it works, string is clean.
text.encode("utf-8")
return text
except UnicodeEncodeError:
pass
try:
# Encode allowing surrogates, then decode replacing them.
return text.encode("utf-8", errors="surrogatepass").decode(
"utf-8", errors="replace"
)
except (UnicodeEncodeError, UnicodeDecodeError):
# Last resort: filter out surrogate characters.
return "".join(
char if ord(char) < 0xD800 or ord(char) > 0xDFFF else "\ufffd"
for char in text
)
Additional duplications
I ran rg "surrogatepass" on StarsExpress:fix-read-text-sanitized, which is the branch supporting #756, and found these:
1. code_puppy/command_line/completers.py::_sanitize_for_encoding
Appears to be dead code in production; its only reference is tests/command_line/test_remaining_coverage.py::test_sanitize_for_encoding_unicode_error, which exists solely for test coverage.
Candidates for removal are this function + its dedicated test.
2. code_puppy/agents/_runtime.py::_sanitize_prompt
Actively used with duplicated surrogate-stripping logic as code_puppy/tools/common.py::_sanitize_string.
Consolidation candidate here — call _sanitize_string via a thin wrapper. Like this:
from code_puppy.tools.common import _sanitize_string
def _sanitize_prompt(prompt: str) -> str:
"""Strip lone UTF-16 surrogates (common on Windows copy-paste)."""
return _sanitize_string(prompt)
3. code_puppy/config.py::normalize_command_history
Consolidation candidate here — call _sanitize_string.
4. code_puppy/config.py::save_command_to_history
Consolidation candidate here — call _sanitize_string.
Plan
All these will be covered inside a brand new PR once #756 merges into main.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading PR #756 and the existing implementations in code_puppy/tools/common.py, code_puppy/agents/_runtime.py, and code_puppy/config.py. Inspect the dedicated test in tests/command_line/test_remaining_coverage.py and run the relevant tests before changing the duplicated sanitization paths. Done means the active callers use the shared sanitizer, dead code and its dedicated test are removed if confirmed unused, and surrogate handling remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100