psf / psf/requests

iter_lines is still broken?

Open
#5,540 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

I don't understand.

I can't iterate "\r\n" because "\r" and "\n" can be read into different chunks.

But I can't even get solve this problem by explicitly setting the final delimiter "\n"

  for line in (_.strip() for _ in resp.iter_lines(delimiter=b"\n")):
      print(line)

because if chunk end with "\r\n", then lines = chunk.split(delimiter) in iter_lines append empty extra line (but chunk.splitlines() not)

the same problem and just with "\n" - if chunk end with delimiter, then we get an extra empty line. Which forces you to set a large buffer size if line length is unknown

So if the split(...) adds an extra line, why not remove it?

    if delimiter:
        lines = chunk.split(delimiter)
        if lines and not lines[-1]:
            lines.pop(-1)
    else:
        lines = chunk.splitlines()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the iter_lines entry point and trace how chunks are split when a delimiter ends a chunk or when \r\n spans chunks. Reproduce both cases and add or update coverage for the observed behavior; done means iteration does not emit an unintended empty line while preserving delimiters that belong to the next chunk.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.