nodejs / nodejs/node

stream.readable._read() description is unclear

Open
#42,291 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

doc stale stream
Dominant language
JavaScript
Stars
122k
Forks
37.3k
Avg merge
4d 2h
Merged PRs (30d)
283

Description

Affected URL(s)

https://nodejs.org/api/stream.html#readable_readsize

Description of the problem

I'm implementing a readable stream, and found the following part of the documentation to be unclear:

When readable._read() is called, if data is available from the resource, the implementation should begin pushing that data into the read queue using the this.push(dataChunk) method. _read() will be called again after each call to this.push(dataChunk) once the stream is ready to accept more data. _read() may continue reading from the resource and pushing data until readable.push() returns false. Only when _read() is called again after it has stopped should it resume pushing additional data into the queue.

Once the readable._read() method has been called, it will not be called again until more data is pushed through the readable.push() method. Empty data such as empty buffers and strings will not cause readable._read() to be called.

Specifically, these three quotes seem to conflict one another, and the relationship/behavior of _read() and push() are not very obvious to me:

  1. _read() will be called again after each call to this.push(dataChunk) once the stream is ready to accept more data
  2. Once the readable._read() method has been called, it will not be called again until more data is pushed through the readable.push()
  3. data. _read() may continue reading from the resource and pushing data until readable.push() returns false

The first seems to claim that ._read() gets called after every call of .push(), but then also claims that its conditional on the stream consumer's readiness to accept more data. The second seems to claim that after the first _read(), it won't get called again until there is at least one .push(). The third seems to claim that ._read()'s implementation can call .push() multiple times, so presumably, the first can't be true or you'd end up with an infinite loop of sorts.


The examples under stream.readable._construct() and stream.readable.push() are also confusing in that they suggest two very different sorts of operations of ._read() and .push().

The former suggests that initiating stream consumption calls the first ._read(), which in the example calls a single .push(), which then triggers the stream to call ._read() once again, rinse and repeat.

The latter seems to suggest that initiating stream consumption calls the first ._read(), which triggers continuous and asynchronous calls to .push() which only stop when it returns false. Another subsequent ._read() call would then restart the .push()es

I'm guessing that in this second example, ._read() is actually called multiple times, but this._source.readStart(); is assumed to be idempotent and can be called multiple times with no effect. If that's the case, I think it would be helpful to clarify this example, and also be a little more explicit in the ._read() description.


Maybe something like (assuming my understanding is correct):

When readable._read() is called, if data is available from the resource, the implementation should begin pushing that data into the read queue using the this.push(dataChunk) method. The stream implementation may continue reading from the resource and pushing data until readable.push() returns false.

After the first call, readable._read() will only be called again after data is pushed through the readable.push() method, and will be called with every push(). Empty data such as empty buffers and strings will not cause readable._read() to be called again.

When readable.push() returns false, the stream consumer is not ready to accept more data, so the stream implementation should stop calling .push(). Only after _read() is called again should the stream resume pushing additional data into the queue.

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the readable._read(size) and readable.push(chunk, encoding) sections at the affected stream.html URLs, then compare the readable._construct() and readable.push() examples. Clarify the timing and interaction of _read() and push(), including the false-return stopping condition, so the prose and examples describe one consistent behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.