openai / openai/codex

asciicheck.py flags CRLF line endings as invalid characters (false positive on Windows checkouts)

Open Beginner friendly
#41,025 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug windows-os
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

Description

scripts/asciicheck.py (run in CI via .github/workflows/repo-checks.yml as ./scripts/asciicheck.py README.md) reads a file, decodes it as UTF-8, and flags every character outside 0x20-0x7E (plus a small allowlist) as an error:

for lineno, line in enumerate(text.splitlines(keepends=True), 1):
    for colno, char in enumerate(line, 1):
        codepoint = ord(char)
        if char == "\n":
            continue
        if (
            not (0x20 <= codepoint <= 0x7E)
            and codepoint not in allowed_unicode_codepoints
        ):
            errors.append((lineno, colno, char, codepoint))

Only \n is exempted — \r (U+000D) is not. If the file being checked has CRLF line endings (e.g. checked out on Windows with core.autocrlf=true, a common Git-for-Windows default), every single line is reported as containing an "invalid character". --fix doesn't help either: \r isn't in the substitutions map, so the rewritten file still contains \r and the script still exits non-zero.

The repo has no .gitattributes line forcing eol=lf for the files this script checks, so a Windows contributor who checks out the repo with the common core.autocrlf=true setting and runs this script locally against README.md gets a wall of false-positive errors, even though the intent of the script (per its own docstring) is to catch non-breaking spaces/smart quotes/dashes, not legitimate line endings.

Steps to reproduce
from pathlib import Path
p = Path("crlf_test.md")
p.write_bytes(b"hello world\r\nsecond line\r\n")
$ python scripts/asciicheck.py crlf_test.md
Invalid character at line 1, column 12: U+000D (\r)
Invalid character at line 2, column 12: U+000D (\r)
$ echo $?
1
Expected behavior

CRLF line endings should not be flagged as invalid characters.

Actual behavior

Every line of a CRLF file produces a spurious "Invalid character ... U+000D (\r)" error, and --fix cannot resolve it since \r isn't in the substitution table.

Environment
  • Commit: 7c37479 (main, 2026-08-27), Python 3.14, Windows 11
  • File: scripts/asciicheck.py:93-94
  • No existing test file covers this script.
Suggested fix
-            if char == "\n":
+            if char in ("\n", "\r"):
                 continue

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/asciicheck.py at the character-checking loop described in the issue, and review its invocation in .github/workflows/repo-checks.yml. Reproduce the behavior with a CRLF file and run the checker against it. Done means CRLF line endings are accepted without errors while the existing invalid-character checks still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.