nodejs / nodejs/node

stream: Writable.toWeb()/Duplex.toWeb() settles write() before a mutable chunk is consumed

オープン
#64,549 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

主要言語
JavaScript
スター
122k
フォーク
37.3k
平均マージ
4日 2時間
マージ済み PR(30日)
283

説明

Version

v24.15.0

Platform
Microsoft Windows NT 10.0.26200.0 x64
Subsystem

stream, webstreams

What steps will reproduce the bug?

When fs.WriteStream is converted using Writable.toWeb(), the promise returned
by writer.write() can settle before the file stream has consumed the supplied
Uint8Array.

This makes it unsafe to reuse a buffer after awaiting writer.write().

const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { Writable } = require('node:stream');

(async () => {
  const file = path.join(
    os.tmpdir(),
    `node-writable-to-web-${process.pid}.bin`,
  );

  const fileStream = fs.createWriteStream(file);
  const writer = Writable.toWeb(fileStream).getWriter();

  // Reusable scratch buffer.
  const storage = new Uint8Array(4);

  storage.set([1, 2, 3, 4]);
  await writer.write(storage.subarray());

  // The previous write promise has settled, so reuse the buffer.
  storage.set([5, 6, 7, 8]);
  await writer.write(storage.subarray());

  await writer.close();

  const content = fs.readFileSync(file);
  console.log([...content]);

  fs.unlinkSync(file);
})().catch(console.error);

Run:

$ node repro.js
[
  5, 6, 7, 8,
  5, 6, 7, 8
]
How often does it reproduce? Is there a required condition?

It reproduces consistently when:

  1. A mutable Uint8Array or subarray() is written.
  2. The native writable.write() call returns true.
  3. The backing buffer is reused after the Web Streams write() promise settles.
  4. The native writable has not yet consumed the original bytes.

The problem can depend on chunk size. When the native write() returns false,
the adapter waits for backpressure, so larger chunks may appear to work.

The writable side returned by Duplex.toWeb() is affected by the same adapter
behavior.

What is the expected behavior? Why is that the expected behavior?

The file should contain the values supplied by the two completed writes:

[
  1, 2, 3, 4,
  5, 6, 7, 8
]

The producer mutates the buffer only after the first writer.write() promise
has settled.

The Web Streams specification advises producers not to mutate a mutable chunk
until the promise returned by write() settles:

https://streams.spec.whatwg.org/#default-writer-write

Following that guidance should ensure that the underlying sink processes the
same bytes that were passed to write().

What do you see instead?

The file contains the second value twice:

[
  5, 6, 7, 8,
  5, 6, 7, 8
]

The first writer.write() promise settles while fs.WriteStream still retains
a reference to the first subarray(). Reusing the backing buffer therefore also
changes the bytes of the pending file write.

Changing subarray() to slice() avoids the corruption because slice() copies
the bytes, but it requires an allocation and copy for every write.

What do you see instead?

The boolean returned by native writable.write() only represents backpressure.
It does not indicate that the specific chunk has finished being processed.

Node's native writable API provides a per-write callback:

writable.write(chunk, callback);

https://nodejs.org/api/stream.html#writablewritechunk-encoding-callback

A possible fix would be for the Web Streams adapter to:

  • Settle the Web write() promise from the native per-write callback.
  • If native write() returns false, also honor the drain event.
  • Reject the Web write promise if the callback reports an error.

This would allow callers to safely reuse mutable buffers after awaiting
writer.write() without requiring slice() copies.

Additional information

No response

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

調査の方向性

Writable.toWeb() と Duplex.toWeb() のエントリポイントから開始し、native writable.write() が Web Streams の書き込み promise をどのように確定させるかを追跡します。提供されている fs.WriteStream スクリプトで問題を再現し、可変 Uint8Array の再利用に対するリグレッションテストを追加します。各完了済みの書き込みが元のバイトを保持し、backpressure または callback のエラーが引き続き処理されれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
javascript, node.js
領域
backend
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
おおむね明確
初心者へのやさしさ
48/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。