google / google/etils

_FileSystemSpecBackend.open() ignores the utf-8 contract stated in Backend.open()

Open Beginner friendly
#817 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
259
Forks
34
Avg merge
57m
Merged PRs (30d)
1

Description

`Backend.open()` documents its own contract at `backend.py:47`:

```python
"""`open`. Encoding should be utf-8."""
```

`_OsPathBackend.open()` honours it, passing `encoding='utf-8'` for text mode (`backend.py:124-133`). `_FileSystemSpecBackend.open()` does not:

```python
def open(self, path: PathLike, mode: str) -> typing.IO[Union[str, bytes]]:
return self.fs(path).open(path, mode=mode)
```

With no `encoding=`, fsspec falls back to `locale.getpreferredencoding()`. On a Windows machine with a non-UTF-8 locale that is `cp1252`, and any non-latin1 text fails to write:

```python
# Windows 11, CPython 3.13.13, locale.getpreferredencoding() == 'cp1252'
fs = fsspec.filesystem('file')
with fs.open(p, mode='w') as f:
f.write('abc统一码def')
# UnicodeEncodeError: 'charmap' codec can't encode characters in position 3-5

open(p, 'rb').read()
# b'' - the file is created and left empty
```

The same write through `epath.Path` on the os backend produces the expected UTF-8 bytes, so the two backends disagree about a contract the abstract base class states explicitly.

The failure is not Windows-specific in principle. It reproduces on any platform whose preferred encoding is not UTF-8; `ubuntu-latest` defaults to UTF-8, which is why `pytest_and_autopublish.yml` has never surfaced it.

`test_backend[_test_open]` reports `SUBFAILED[fsspec]` with this `UnicodeEncodeError`, but the module has never run on Windows because it fails collection there on `import grp`.

I have no Linux or macOS machine, so the claim about non-UTF-8 POSIX locales is reasoned from `locale.getpreferredencoding()` semantics rather than measured.

Contributor guide

Open the contributing guide

Research direction

Start in backend.py, comparing _FileSystemSpecBackend.open() with the documented Backend.open() contract and _OsPathBackend.open(). Run the test_backend[_test_open] case, including its fsspec variant, to reproduce the encoding failure. Done means both backends honor the stated UTF-8 behavior and the relevant test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.