_FileSystemSpecBackend.open() ignores the utf-8 contract stated in Backend.open()
- Dominant language
- Python
- Stars
- 259
- Forks
- 34
- Avg merge
- 57m
- Merged PRs (30d)
- 1
Description
`Backend.open()` documents its own contract at `backend.py:47`:
```python
"""`open`. Encoding should be utf-8."""
```
`_OsPathBackend.open()` honours it, passing `encoding='utf-8'` for text mode (`backend.py:124-133`). `_FileSystemSpecBackend.open()` does not:
```python
def open(self, path: PathLike, mode: str) -> typing.IO[Union[str, bytes]]:
return self.fs(path).open(path, mode=mode)
```
With no `encoding=`, fsspec falls back to `locale.getpreferredencoding()`. On a Windows machine with a non-UTF-8 locale that is `cp1252`, and any non-latin1 text fails to write:
```python
# Windows 11, CPython 3.13.13, locale.getpreferredencoding() == 'cp1252'
fs = fsspec.filesystem('file')
with fs.open(p, mode='w') as f:
f.write('abc统一码def')
# UnicodeEncodeError: 'charmap' codec can't encode characters in position 3-5
open(p, 'rb').read()
# b'' - the file is created and left empty
```
The same write through `epath.Path` on the os backend produces the expected UTF-8 bytes, so the two backends disagree about a contract the abstract base class states explicitly.
The failure is not Windows-specific in principle. It reproduces on any platform whose preferred encoding is not UTF-8; `ubuntu-latest` defaults to UTF-8, which is why `pytest_and_autopublish.yml` has never surfaced it.
`test_backend[_test_open]` reports `SUBFAILED[fsspec]` with this `UnicodeEncodeError`, but the module has never run on Windows because it fails collection there on `import grp`.
I have no Linux or macOS machine, so the claim about non-UTF-8 POSIX locales is reasoned from `locale.getpreferredencoding()` semantics rather than measured.
Contributor guide
Research direction
Start in backend.py, comparing _FileSystemSpecBackend.open() with the documented Backend.open() contract and _OsPathBackend.open(). Run the test_backend[_test_open] case, including its fsspec variant, to reproduce the encoding failure. Done means both backends honor the stated UTF-8 behavior and the relevant test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100