fsspec / fsspec/filesystem_spec
fs.open: newline=None translates to os.linesep on write
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Since I introduced read_text/write_text with newline= parameters, I was trying to implement the async versions of it.
Since asyncfs don't have an async version of open, for ease, I was thinking of using cat_file/pipe_file and building on it.
For the record, the implementation is going to be something like follows:
async def _read_text(self, path, encoding=None, errors=None, newline=None):
encoding = encoding or locale.getpreferredencoding(False)
errors = errors or "strict"
assert newline in (None, "", "\n", "\r", "\r\n")
contents = await self._cat_file(path)
text = contents.decode(encoding, errors)
if newline is None:
# needs an optimization when there may be no `\r` after 1st replacement
text = text.replace(b"\r\n", b"\n").replace(b"\r", "\n")
return text
async def _write_text(self, path, value: str, encoding=None, errors=None, newline=None):
encoding = encoding or locale.getpreferredencoding(False)
errors = errors or "strict"
assert newline in (None, "", "\n", "\r", "\r\n")
if newline is None:
newline = os.linesep
if newline not in ("", "\n") and "\n" in value:
value = value.replace("\n", newline)
contents = value.encode(encoding, errors)
await self._pipe_file(path, contents)
While implementing this, I was going through the docs and noticed this:
When writing output to the stream, if newline is None, any '\n' characters written are translated to the system default line separator,
os.linesep.
I always knew that it converts to os.linesep on write, but it forced me to think about whether this makes sense in the context of fsspec.
The universal newlines when reading does make sense to me, but I am not sure about write. Of course, this can be fixed by passing newline=''. It does seem like a minor issue, just wanted to see what others think, especially in the context of fs.open/fs.read_text/fs.write_text.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the fs.open, read_text, and write_text entry points described in the issue, and compare their newline handling with the proposed async _read_text and _write_text paths using cat_file and pipe_file. The issue does not define a preferred write convention; clarify that behavior first, then document or implement the chosen semantics consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100