fsspec / fsspec/filesystem_spec

fs.open: newline=None translates to os.linesep on write

Open
#1,066 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Since I introduced read_text/write_text with newline= parameters, I was trying to implement the async versions of it.
Since asyncfs don't have an async version of open, for ease, I was thinking of using cat_file/pipe_file and building on it.
For the record, the implementation is going to be something like follows:

async def _read_text(self, path, encoding=None, errors=None, newline=None):
    encoding = encoding or locale.getpreferredencoding(False)
    errors = errors or "strict"
    assert newline in (None, "", "\n", "\r", "\r\n")
    contents = await self._cat_file(path)
    text = contents.decode(encoding, errors)
    if newline is None:
        # needs an optimization when there may be no `\r` after 1st replacement
        text = text.replace(b"\r\n", b"\n").replace(b"\r", "\n")
    return text


async def _write_text(self, path, value: str, encoding=None, errors=None, newline=None):
    encoding = encoding or locale.getpreferredencoding(False)
    errors = errors or "strict"
    assert newline in (None, "", "\n", "\r", "\r\n")
    if newline is None:
        newline = os.linesep
    if newline not in ("", "\n") and "\n" in value:
        value = value.replace("\n", newline)
    contents = value.encode(encoding, errors)
    await self._pipe_file(path, contents)

While implementing this, I was going through the docs and noticed this:

When writing output to the stream, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep.

I always knew that it converts to os.linesep on write, but it forced me to think about whether this makes sense in the context of fsspec.

The universal newlines when reading does make sense to me, but I am not sure about write. Of course, this can be fixed by passing newline=''. It does seem like a minor issue, just wanted to see what others think, especially in the context of fs.open/fs.read_text/fs.write_text.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the fs.open, read_text, and write_text entry points described in the issue, and compare their newline handling with the proposed async _read_text and _write_text paths using cat_file and pipe_file. The issue does not define a preferred write convention; clarify that behavior first, then document or implement the chosen semantics consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.