WebAssembly / WebAssembly/component-model
Provide a synchronous constructor for non-empty streams
Nobody has claimed this yet.
- Dominant language
- WebAssembly
- Stars
- 1.4k
- Forks
- 130
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 15
Description
TL;DR: There is not currently a way to construct a non-empty stream in a synchronous context, making it impossible for a guest component to virtualize several WASI APIs. While planned features like cooperative threads will eventually make it possible to populate a stream by spawning background tasks, I believe that having a straightforward stream constructor that takes a list would be generally useful and hopefully (?) much simpler to implement.
(Apologies if this is not the best repo for this issue, please feel free to move it as appropriate.)
WASI-Virt currently supports WASI versions v0.2.1 and v0.2.3. Both of these are quite old relative the ecosystem; cargo build --target wasm32-wasip2 produces components that depend on WASI v0.2.9. The latest 2.x release is 0.2.12, and the first 0.3.x release landed last month. WASI-Virt also uses exact-match dependency resolution, so while a v0.2.12 export could satisfy a v0.2.3 import in theory, it gets rejected in practice. Functionally, this means that WASI-Virt cannot be used with recent toolchains.
I attempted to enable semver-compatible dependency resolution, but learned that it wouldn't be possible so long as WASI-Virt was using WASI p2. P2's use of resources to represent IO streams always lead to transitive import/export conflicts:
4: world exports
wasi:io/streams@0.2.3but it's also transitively used by an import which means that this is not valid
These conflicts should be obviated by P3's use of stream instead of resources. I set out to test this by making a simple component that would virtualize wasi:filesystem (and nothing else). However, I quickly ran into an issue when trying to implement descriptor.read-via-stream (see #wasi > Creating a `stream` in a sync context).
In short: read-via-stream is synchronous function. A guest implementation can construct a stream reader/writer pair via wit_stream::new(), but the guest cannot (meaningfully) spawn a background task to populate a stream via the writer. Cooperative threads should make this possible when they arrive early next year.
The WASI-Virt use case does not require a sophisticated writer implementation. The bytes that represent a file are known at virtualization time; the file descriptors are backed by a simple Vec<u8>. However, so far as I can tell, there is no way to construct a stream from a list<u8> in a synchronous context.
I know there's a larger design effort happening around slicing and splicing stream; I wonder if there's room for a simple synchronous constructor. (Regrettably, the lack of user generics does mean that you couldn't define a func from-list(list<T>) -> stream<T> in the near term.)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the WASI stream discussion linked from the issue, especially wit_stream::new() and descriptor.read-via-stream, then review the referenced work on stream slicing and splicing. The goal is to determine and implement a synchronous constructor that can populate a stream from a known list<u8> without a background task, and verify that it supports the WASI-Virt virtualization use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- wasm
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100