gleam-lang / gleam-lang/stdlib

Feature request: Add `bytes_tree.uncons` or similar

Open
#932 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Gleam
Stars
710
Forks
225
Avg merge
5h 52m
Merged PRs (30d)
3

Description

On the JavaScript target, the `BytesTree` type is fully opaque. The only function that allows me to access the data in the `BytesTree` is to use the `to_bit_array` function, which has to allocate and copy the entire tree into one continuous buffer in memory. However, the main use-case for this function is to avoid this copy, for example when the output is streamed directly to a network socket.

In JavaScript, this would be modelled as a `ReadableStream`, which can be implemented in 2 flavours:

- With `push` streams, you would call `controller.enqueue` on individual chunks, filling up the internal queue. When the desired size drops below `0`, you are supposed to pause queueing new chunks, and resume once the stream is consumed.
- With `pull` streams, the callback of the same name will be called whenever the stream needs more data. This makes it easier to have back-pressure in the system.
de's special flavour of Streams, which work similarly but are their own distinct things.

In both cases, you'd want to have access to the underlying ByteTree chunks, while also coordinating with JavaScript, interleaving event handlers and promises.

I propose adding an `uncons` (`shift_chunk`? I honestly don't have a good name) function, which allows me to consume chunks of a ByteTree in order:

```gleam
pub fn uncons(tree: BytesTree) -> Result(#(BitArray, BytesTree), Nil) {
case tree {
Bytes(chunk) -> Ok(#(chunk, new())
Text(string) -> Ok(#(string_tree.to_bit_array(string), new())
Many([]) -> Error(Nil)
Many([chunk, ..rest]) ->
case uncons(chunk) {
Ok(#(chunk, remaining)) -> Ok(#(chunk, Many([remaining, ..rest])))
Error(Nil) -> uncons(Many(rest))
}
}
}
```

(Erlang would need a custom implementation since `iolists` are way more flexible)

Alternatively, i considered if a `try_fold` or similar would work, but it looks like to me that this would not be usable with `pull` streams, so it would mostly just be used as a `flatten`-style function that converts the tree to a depth-first array first.

~ 💜

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the JavaScript-target BytesTree implementation and its existing to_bit_array function. Compare the proposed uncons behavior with the Bytes, Text, and Many cases described in the issue, and consider the Erlang target separately. Done means a chunk-consuming public operation that avoids copying the whole tree and supports the stated streaming use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.