ruff does not honor declaration of character coding
- Dominant language
- Rust
- Stars
- 49.7k
- Forks
- 2.4k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 458
Description
According to PEP263, a character encoding can be declared in a Python program file. This is done with a specially formatted comment placed in the first or second line of the program:
```
#!/usr/bin/python
# -*- coding: latin-1 -*-
```
It seems that Ruff (0.0.285) does not honor the coding declaration. Ruff seems to assume that input files are always encoded with utf8. The following Python program demonstrates the problem. It first generates three short Python program files with different encodings and than runs ruff and python3 on them.
```
#!/usr/bin/env python3
# -*- coding: us-ascii -*-
import subprocess
prog = """# -*- coding: {} -*-
print(\"\u00D8resund og Sj\u00E6lland\")
"""
## create artefacts with differing encodings
filenames = []
filenames.append("prog-utf8.py")
print(f"writing file '{filenames[-1]}'")
with open(filenames[-1], "wb") as outstream:
outstream.write(prog.format("utf8").encode("utf8"))
filenames.append("prog-usascii.py")
print(f"writing file '{filenames[-1]}'")
with open(filenames[-1], "wb") as outstream:
# declared encoding differs from the true encoding
outstream.write(prog.format("us-ascii").encode("utf8"))
filenames.append("prog-latin1.py")
print(f"writing file '{filenames[-1]}'")
with open(filenames[-1], "wb") as outstream:
outstream.write(prog.format("latin-1").encode("latin1"))
## re-check encodings of the artefacts
print("\nTrue encodings")
for filename in filenames:
subprocess.call(["file", "-i", filename,])
## run python3 and ruff on the artefacts
for filename in filenames:
cmd = ["python3", filename,]
print("---\n" + " ".join(cmd))
subprocess.call(cmd)
cmd = ["ruff", filename,]
print("\n" + " ".join(cmd))
subprocess.call(cmd)
```
The first file with utf8 encoding runs flawlessly with ruff and with python3. This is the expected behaviour.
The second file comprises characters in utf8 encoding, but wrongly declares us-ascii encoding. This file throws an error when run with python3, but successfully passes ruff. I would expect that ruff complains on this file.
The third file comprises characters in latin1 encoding and correctly declares its encoding. This program runs successfully with python3, but throws an error when checked with ruff. I would expect that ruff does not complain on this file.
Contributor guide
Research direction
Start by running the supplied reproducer, which creates prog-utf8.py, prog-usascii.py, and prog-latin1.py, then invokes python3 and ruff on each file. Trace Ruff's input decoding from the command-line entry point and compare it with Python's PEP263 behavior. Done means Ruff rejects the incorrectly declared us-ascii file and accepts the correctly declared latin-1 file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100