python / python/cpython

_io.TextIOWrapper._CHUNK_SIZE ignores underlying buffer size, defaults 8192

Open
#153,656 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules performance stdlib topic-IO type-feature
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

If I try to read a line from a file containing a very long line using open(path, buffering=n).readline(), it reads in chunks of 8192 regardless of the value of n. I would like it to respect n so that I can reduce the number of syscalls.

I can manually work around this by setting the undocumented attribute _CHUNK_SIZE, but I think this should be done automatically. The number 8192 is hard-coded in _io_TextIOWrapper___init___impl; I propose automatically setting it to the buffer size of the underlying _io.BufferedReader / _io.BufferedWriter / _io.BufferedRandom (if the underlying object is one of those).

docker run --rm -it python:3.14.6 bash
apt-get update && apt-get install -y strace
python3 -c 'print("x"*100000)' > f
# f.readline ignores the buffer size
strace -e trace=read python3 -c 'open("f", buffering=100000).readline()' # calls `read(3, ..., 8192)`
# workaround: set f._CHUNK_SIZE
strace -e trace=read python3 -c 'f = open("f", buffering=100000); f._CHUNK_SIZE = 1000000; f.readline()' # calls `read(3, ..., 1000000)`
# aside: f.buffer.readline respects the buffer size
strace -e trace=read python3 -c 'f = open("f", buffering=100000); f.buffer.readline()' # calls `read(3, ..., 100000)`
CPython versions tested on:

3.14

Operating systems tested on:

Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Modules/_io/textio.c at _io_TextIOWrapper___init___impl, where the issue identifies the hard-coded 8192 chunk size. Reproduce the behavior with the provided open(), readline(), and strace commands, then trace how the underlying buffered object's size is exposed. Done means readline() uses that buffer size for the relevant buffered object types without requiring _CHUNK_SIZE.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.