Repository navigation
Provide a synchronous constructor for non-empty streams #677
Description
TL;DR: There is not currently a way to construct a non-empty stream in a synchronous context, making it impossible for a guest component to virtualize several WASI APIs. While planned features like cooperative threads will eventually make it possible to populate a stream by spawning background tasks, I believe that having a straightforward stream constructor that takes a list would be generally useful and hopefully (?) much simpler to implement.
(Apologies if this is not the best repo for this issue, please feel free to move it as appropriate.)
WASI-Virt currently supports WASI versions v0.2.1 and v0.2.3. Both of these are quite old relative the ecosystem; cargo build --target wasm32-wasip2 produces components that depend on WASI v0.2.9. The latest 2.x release is 0.2.12, and the first 0.3.x release landed last month. WASI-Virt also uses exact-match dependency resolution, so while a v0.2.12 export could satisfy a v0.2.3 import in theory, it gets rejected in practice. Functionally, this means that WASI-Virt cannot be used with recent toolchains.
I attempted to enable semver-compatible dependency resolution, but learned that it wouldn't be possible so long as WASI-Virt was using WASI p2. P2's use of resources to represent IO streams always lead to transitive import/export conflicts:
4: world exports
wasi:io/streams@0.2.3but it's also transitively used by an import which means that this is not valid
These conflicts should be obviated by P3's use of stream instead of resources. I set out to test this by making a simple component that would virtualize wasi:filesystem (and nothing else). However, I quickly ran into an issue when trying to implement descriptor.read-via-stream (see #wasi > Creating a `stream` in a sync context).
In short: read-via-stream is synchronous function. A guest implementation can construct a stream reader/writer pair via wit_stream::new(), but the guest cannot (meaningfully) spawn a background task to populate a stream via the writer. Cooperative threads should make this possible when they arrive early next year.
The WASI-Virt use case does not require a sophisticated writer implementation. The bytes that represent a file are known at virtualization time; the file descriptors are backed by a simple Vec<u8>. However, so far as I can tell, there is no way to construct a stream from a list<u8> in a synchronous context.
I know there's a larger design effort happening around slicing and splicing stream; I wonder if there's room for a simple synchronous constructor. (Regrettably, the lack of user generics does mean that you couldn't define a func from-list(list<T>) -> stream<T> in the near term.)
Activity
lukewagner commentedon Jul 16, 2026
Thanks for working on WASI-Virt, doing all this exploration and writing up a nice clear issue!
So I believe that we can hopefully solve your original virtualization issue in a wasip2 context as commented here. I also think there's a path for WASI-Virt to virtualize
read-via-streamin a 0.3.0 context as commented here. So hopefully those can address the immediate WASI-Virt use cases without having to add anything.Considering just the design question of whether we should add a new built-in to create a
streamfrom an array of bytes: I think this would end up requiringstreams to maintain an internal memory buffer which we've otherwise tried to avoid: currently streams are always unbuffered and just synchronize a direct copy from astream.writer-supplied buffer to astream.reader-supplied buffer. So my impression is that we shouldn't add a built-in to do this.However, one old-and-still-great idea that I think would directly address your WASI-Virt use case is #138, which proposes a built-in to (synchronously) create a (readable end of a) stream directly from a passive data segment, which would avoid the intermediate copy through linear memory entirely and simply work in a synchronous
read-via-streamcontext. In particular, the key advantage of creating a stream from a data segment is that data segments are immutable and thus the stream wouldn't need to make an eager copy (as it would if creating the stream from a buffer in linear memory). If anyone wants to work on #138, I'd be happy to help write it up as a spec proposal.