gleam-lang / gleam-lang/stdlib
Feature request: Add `bytes_tree.uncons` or similar
- Dominant language
- Gleam
- Stars
- 710
- Forks
- 225
- Avg merge
- 5h 52m
- Merged PRs (30d)
- 3
Description
On the JavaScript target, the `BytesTree` type is fully opaque. The only function that allows me to access the data in the `BytesTree` is to use the `to_bit_array` function, which has to allocate and copy the entire tree into one continuous buffer in memory. However, the main use-case for this function is to avoid this copy, for example when the output is streamed directly to a network socket.
In JavaScript, this would be modelled as a `ReadableStream`, which can be implemented in 2 flavours:
- With `push` streams, you would call `controller.enqueue` on individual chunks, filling up the internal queue. When the desired size drops below `0`, you are supposed to pause queueing new chunks, and resume once the stream is consumed.
- With `pull` streams, the callback of the same name will be called whenever the stream needs more data. This makes it easier to have back-pressure in the system.
de's special flavour of Streams, which work similarly but are their own distinct things.
In both cases, you'd want to have access to the underlying ByteTree chunks, while also coordinating with JavaScript, interleaving event handlers and promises.
I propose adding an `uncons` (`shift_chunk`? I honestly don't have a good name) function, which allows me to consume chunks of a ByteTree in order:
```gleam
pub fn uncons(tree: BytesTree) -> Result(#(BitArray, BytesTree), Nil) {
case tree {
Bytes(chunk) -> Ok(#(chunk, new())
Text(string) -> Ok(#(string_tree.to_bit_array(string), new())
Many([]) -> Error(Nil)
Many([chunk, ..rest]) ->
case uncons(chunk) {
Ok(#(chunk, remaining)) -> Ok(#(chunk, Many([remaining, ..rest])))
Error(Nil) -> uncons(Many(rest))
}
}
}
```
(Erlang would need a custom implementation since `iolists` are way more flexible)
Alternatively, i considered if a `try_fold` or similar would work, but it looks like to me that this would not be usable with `pull` streams, so it would mostly just be used as a `flatten`-style function that converts the tree to a depth-first array first.
~ 💜
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the JavaScript-target BytesTree implementation and its existing to_bit_array function. Compare the proposed uncons behavior with the Bytes, Text, and Many cases described in the issue, and consider the Erlang target separately. Done means a chunk-consuming public operation that avoids copying the whole tree and supports the stated streaming use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- api
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100