wasi:filesystem alternative - force guests to do all path resolution #933

Activity

1. alexcrichton

   changed the title [-]wasi:filesytem alternative - force guests to do all path resolution[/-] [+]wasi:filesystem alternative - force guests to do all path resolution[/+] on Jun 20, 2026
2. alexcrichton commented on Jun 20, 2026

   

   alexcrichton

   on Jun 20, 2026

   Last edited by alexcrichton

   ContributorAuthor

   More actions

   To elaborate a bit on differences from today's WITs, here's how I would envision behavior/API changes for today's functions on descriptor and such:

   • All path or *-path arguments are required to be a "segment". notably cannot contain / or cannot contain ../. as path elements. For example foo.txt is allowed, but ./foo.txt is not. Nor is /foo.txt. The one exception to this is the old-path argument to the symlink-at API. As the content of the symlink this is not validated at all.
   • All path-flags arguments are removed and hardcoded as nofollow-always-enabled.

   If it's possible to get away with it, I wouldn't propose adding a sort of descriptor.parent() -> option<descriptor> API to find the parent directory (returning none for preopens). I say this because I don't know how it would be implemented on the host, but the lack of this API forces the guest to maintain a stack of descriptors for all open directories so they can resolve to .. appropriately.

   For preopens I think something needs to change about the structure of the names. If a guest gets preopens for . and / that's sort of nonsensical from a guest trying to view preopens as "just a single filesystem" as there's no relation to . and /. Without having thought about this too-too-hard, what I'd propose is something like:

   interface preopens {
       use types.{descriptor, error-code};
   
       root: func() -> option<preopen>;
   
       resource preopen {
           descriptor: func() -> option<descriptor>;
   
           // here `path` must be a "path segment"
           child: func(path: string) -> result<option<preopen>, error-code>;
           children: func() -> list<tuple<string, preopen>>;
       }
   }

   Notably this (a) imposes a tree structure, (b) forces guests to think of "one root fs", (c) enables overlap as is allowed today, and (d) keeps path conventions purely in the guest by only operating on segments. Hosts would have to do a little work to modify the "list of unstructured name plus dir" that they manage today, but modeling it as a something like this I would hope wouldn't be too bad.
3. lukewagner commented on Jun 22, 2026

   

   lukewagner

   on Jun 22, 2026

   Member

   More actions

   Moving path-resolution logic into the guest makes a bunch of sense, especially for these cases of assembling a VFS from multiple native directories. For this, what you sketched above sgtm!

   To your performance-concern bullet though: I am also somewhat worried that, for cases where the host does just want to expose the whole filesystem or just one chroot-style directory (which both seem like valid and not-uncommon use cases) unconditionally doing all the path resolution in the guest with host-calls (and, I assume, syscalls) for each path segment might be a pretty significant performance penalty that in these cases doesn't buy the host or guest much value in exchange for the overhead.

   One assumption I believe we all have is that most guest source code only wants to think about a single rooted filesystem; I think it's only in rather advanced/specialized virtualization scenarios where the guest code wants to "see below the hood" and think about preopens, etc. Thus, even if we want the path resolution to happen in the guest for the above reasons, it seems like the host's choice of how it wants to assemble this single rooted (possibly-virtual) filesystem.

   Based on this, I wonder if we can reconcile these goals by defining two interfaces in wasi:filesystem:

   • preopens as you've sketched above
   • root (or some TBD better name), which also contains use types.{descriptor, error-code} but has a open: async func(path: string) -> result<descriptor, error-code> that does do path resolution

   Then we say that the wasi:cli/command world imports only root, not preopens, so that, by default our producer tools are spitting out components that assume path resolution is handled "on the other side" of the import call. But we also define a new world wasi:cli/preopen-vfs (or some TBD better name) that exports root and imports preopens (and the other usual CLI stuff). And then WASI or wasi-sdk would publish a canonical component .wasm that targets wasi:cli/preopen-vfs, implementing root's path resolution in terms of preopens in guest code.

   So then if the host wants to expose the OS filesystem in a basic way, it provides a native wasi:filesystem/root impl, or if the host wants to do something more advanced, it natively implements wasi:filesystem/preopens and automatically links in the canonical preopen-vfs implementation to virtualization wasi:filesystem/root. If we're doing host-/sys-call-per-segment anyways, the additional overhead of the preopen-vfs component would hopefully be amortized.

   A few nice consequences fall out of this hybrid approach:

   • It helps us isolate the impact of VFS sandboxing in perf testing (and in any external benchmarks folks do)
   • In non-VFS scenarios, it reduces the baseline size and overhead of wasi-libc
   • It opens the door to more advanced VFS scenarios that go beyond preopens, such as, e.g., "mounting" various future WASI stores as directories (analogous to what Emscripten's VFS does in the browser). I expect trying to address all these use cases in wasi-libc would feel like a bridge too far (b/c it adds overhead to all wasi-libc users), whereas if it's in a separate opt-in component (of which there can be several, targeting different worlds), it seems fair game.

   (I mentioned this here b/c it could help address the perf downside mentioned in the OP, but if it's too much of a tangent, happy to move discussion to a separate issue.)
4. badeend commented on Jun 22, 2026

   

   badeend

   on Jun 22, 2026

   Last edited by badeend

   Member

   More actions

   I'm slowly starting to like this idea. Though I have two main concerns:

   Correctness

   Path resolution logic is non-trivial. It needs to:

   • handle . & .. segments (logically, not syntactically),
   • maintain a stack of parent directories in order to know what .. actually refers to,
   • recurse into symlinks,
   • detect loops in symlinks,
   • handle symlinks in the middle of paths differently than symlinks at the end of the path.
   • be resistant to TOCTOU vulnerabilities,
   • and probably more.. For reference, see wasmtime's implementation here: https://github.com/bytecodealliance/cap-std/blob/main/cap-primitives/src/fs/manually/open.rs

   On the one hand, you could argue that this complexity is exactly why moving it into the guest is a good idea.
   On the other hand, all guests need to implement this logic near-identically in order to stay interoperable with each other. Otherwise we can end up in a situation where symlinks created by guest A are unresolvable by guest B or vice-versa, despite running on the same host.

   Whichever route we take, we need to make sure that the path resolution logic is fully specified as part of the WASI itself.

   Performance

   Naively resolving paths in user space takes at least (N-1)*2 + 1 syscalls, where N is the total number of path segments:

   • An open+close pair for each intermediate segment, and
   • a final open for the basename.

   Example: resolving a relatively modest path /a/b/c/d/e/f takes 11 syscalls. If symlinks are involved, that number could easily multiply.

   ---

   Requiring the resolution logic to live inside the guest strips away all possibilities for the host to accelerate this logic. It forces the worst-possible performance characteristics onto the wasm application, even when the host doesn't care about the additional security.
   A valid use case (IMHO) for not needing the additional sandboxing is: running the WASI workload inside a Docker container, where the filesystem virtualization is already handled "one level up".

   Split interface

   I think we can solve these problems by splitting the interface into two aspects:

   • the core interface that exposes the descriptor-based primitives.
   • a "utility" interface, which is semantically fully specified in terms of the lower level core interface. This can be implemented in the guest as e.g. a reusable wasm component. Or it can be implemented directly by the host for extra performance.

   Abbreviated example:

   interface core {
       get-root: func() -> descriptor;
   
       resource descriptor {
           /// Opens exactly *one* segment. Never follows symbolic links. Never traverses `..` etc.
           open: async func(name: string, ..) -> result<descriptor, error-code>;
   
           /// Performs the action on the open file descriptor itself.
           stat: async func() -> result<descriptor-stat, error-code>;
   
           // (...)
       }
   }
   
   interface paths {
       /// Opens an arbitrary path, starting at a base directory. Also traversing `..` and symlinks (if requested)
       /// This is fully implementable in terms of the `core` interface, but will be more efficient when implemented natively.
       open-at: async func(base-dir: descriptor, path: string, ..) -> result<descriptor, error-code>;
       
       /// Semantically equivalent to `open-at(base-dir, path, ..)?.stat()`, but potentially more efficient.
       stat-at: async func(base-dir: descriptor, path: string, ..) -> result<descriptor-stat, error-code>;
   
       // (...)
   }

   Notice the layering:

   • paths::open-at is defined in terms of core::descriptor::open.
     • Reducing the number of syscalls from N to 1.
   • paths::stat-at is in turn defined in terms of paths::open-at and core::descriptor::stat.
     • Reducing the number of syscalls from 3 (open+fstat+close) to 1 (fstatat).

   The POSIX stat family boils down to various incantations of the single WASI stat method:

   • fstat(fd, ..) is fd.stat()
   • fstatat(fd, path, ..) is paths::stat-at(fd, path) (and in turn paths::open-at(fd, path)?.stat())
   • stat(path, ..) is paths::stat-at(core::get-root(), path)

   Single root preopen

   The example above also exposes the root preopen as a regular descriptor. Now that the descriptor API has been drastically simplified and they are truly are "just" simple nodes in a tree, I don't think it adds significant implementation complexity for hosts to implement it this way. Realistically speaking, the guest just wants a single tree and doesn't care how it came to be there. This also resolves the ambiguity of having overlapping preopens.
5. alexcrichton commented on Jun 24, 2026

   

   alexcrichton

   on Jun 24, 2026

   ContributorAuthor

   More actions

   You both bring up excellent points! I want to write down a few thoughts of mine on perf/correctness before responding to the dual-interface ideas as well.

   Performance

   One thing I can mention is that for performance the big-O number of syscalls I think is how everything works today in Wasmtime when RESOLVE_BENEATH isn't available. I believe cap-std (which Wasmtime uses) already does the whole resolve-symlinks-manually algorithm which @badeend linked. In that sense the per-segment interpretation is just simply required on macOS, Windows, other non-Linux OS-es, and Linux when RESOLVE_BENEATH isn't available.

   With this information, a per-segment interface performs worse in three situations AFAIK:

   1. On Linux when RESOLVE_BENEATH is available (as the kernel can "do the sandboxing").
   2. On Unix when combined with chroot.
   3. On any OS when the entire filesystem is exposed.

   Subjectively I'd say that these situations might be rare enough that it might not be worth bending over backwards for them. For (1) IMO we still need to move away from list-of-preopens to one-preopen-thingie meaning that to use RESOLVE_BENEATH requires the full filesystem defined on the host, aka no VFS. For (2) that clashes with multi-tenancy because chroot is a process-wide change and additionally apparently requires super-user privledges according to man (although not on Linux I think). For (3) I'm not sure how much a completely un-sandboxed view will come up, but I'll note that we cannot provide an un-sandboxed view of Windows filesystems because we don't currently have any way to map all the drives (e.g. C:/, D:/, ...) into the guest.

   One possibly-saving-grace is that performance of actual operations on files (e.g. reads/writes/etc) are unaffected by this proposal. It's just path traversal that'd be affected, and even the naive benchmark of "process an entire directory recursively" wouldn't be affected too much since everything's already operating on a segment-at-a-time anyway. I don't mean to say performance does matter, but there might be a plausible case that this performance becomes a bit niche over time. I could be totally wrong on this though.

   Correctness

   I've no doubt implementing path resolution in a guest is going to be non-trivial, but I'll go back to my point (1) in the OP and point out that many hosts do not even attempt sandboxing today due to the complexity. To me that seems like one of the highest-priority motivations for a proposal like this -- enabling all hosts to reasonably implement a sandboxed filesystem.

   Subjectively we also have a lot more code sharing in guests right now with wasi-libc than we do with hosts (which AFAIK there's pretty much 0 sharing of code between many pairwise hosts). The end state might be more hosts in the world than guest languages/libraries implementing wasi:filesystem, but this is again an un-founded assertion that could be entirely wrong.

   ---

   For multiple interfaces, I like the idea of enabling the host to accelerate things where possible to ensure that we haven't hamstrung ourselves with this permanent performance penalty. I also like the idea of this being a separate interface to sort of "get the best of both worlds". That being said precisely how this interface is expected to be integrated and used is something that I'd want to clarify.

   @lukewagner with your approach I'd be worried that this wouldn't actually move the needle much on guest/host complexity. Wasmtime, for example, can't provide the root interface you'd like to be the default most of the time. Any non-Linux platform with more-than-one-preopen means that the OS can't be relied on for path resolution. Additionally wasi-libc, as an example, is such a core dependency that if it hardcodes usage of root as opposed to preopen-vfs I don't think there's going to be any reasonable way to change that. Basically as such a low-level implementation detail I'd be dubious of the viability of being able to swap out one implementation vs the other.

   I really like @badeend's idea though of a core / paths split (naming TBD with a bikeshed) where paths is exclusively defined in terms of core. This still has the same question, though, of what does a core library like wasi-libc actually use? This also in theory has the same downsides of trying to swap one-vs-the-other with @lukewagner your idea.

   One way to thread that needle though would be optional imports, IMO. For example wasi-libc would unconditionally import core and assume it's only thing that can be assumed. Or, well, ok maybe this is optional too but that's for hosts that provide no filesystem whatsoever. For wasi-libc, though, the paths interface (or "thing that does path resolution externally") would be an optional import. Operations in wasi-libc would then conditionally use the paths interface if applicable, and otherwise it would fall back to core. This only works for POSIX guests where paths would be defined to have POSIX semantics, but that covers all guests today anyway really.

   What would y'all think of that? Wasmtime, for example, would pretty rarely provide that paths interface (if ever?) but it could readily provide the core interface. Other hosts could (hopefully easily) provide the core interface and never provide paths, and guests would work seamlessly (in theory) across all these hosts. Now for this to be viable I do think we need something to provide paths such that it can actually be tested, and my best idea there would be a chroot-ing Wasmtime which we'd have to build out -- but basically something that's only applicable in some specific situations as opposed to trying to move all historical usage to this.

   ---

   Orthogonally from all of this, in terms of actual API design of this proposal (the core interface @badeend has dubbed), there's a few other clarifications/modifications I'd want to make:

   • As @badeend mentioned, I agree that my resource preopen idea is unnecessary. The core.descriptor type is so simple it is reasonable to expect hosts to implement that for VFS-style dirs. Given that I think this issue is actually still in concert with Drop explicit preopens #742 ("just have a single preopen") and it's mostly all about the path resolution side of things.
   • Another change is that get-initial-cwd I think will want to return an option<descriptor>, not a path. With core trying to defer all path resolution to the guest this would close another hole where an externally-specified "full path" makes its way to the guest.

   I also haven't though too too hard about Windows here in terms of open-handles-prevent-deletion. I don't know how difficult it will be to ensure that the guest can close all relevant handles when trying to delete a directory or file. My hunch is not the hardest thing in the world, but if things like get-initial-cwd return a descriptor and the host tries to delete the cwd, things'll probably get a bit weird. (e.g. a handle on the host preventing the guest from deleting it)
6. programmerjake commented on Jun 24, 2026

   

   programmerjake

   on Jun 24, 2026

   Contributor

   More actions

   one option to reduce the number of syscalls necessary is to have a userspace cache of recently accessed files/dirs so, ignoring stuff that gets dropped from the cache, you only need the syscalls the first time you resolve any particular path segment and can use the cached result the rest of the time. you can either assume your wasm engine has exclusive access to the files or use something like inotify to track creation/deletion/renames to know when the cache is out-of-date and/or update it.
7. programmerjake commented on Jun 24, 2026

   

   programmerjake

   on Jun 24, 2026

   Contributor

   More actions

   • Another change is that get-initial-cwd I think will want to return an option<descriptor>, not a path. With core trying to defer all path resolution to the guest this would close another hole where an externally-specified "full path" makes its way to the guest.

   how would libc implement getcwd with that API? does descriptor have a method to get its absolute path?
8. sbc100 commented on Jun 24, 2026

   

   sbc100

   on Jun 24, 2026

   Last edited by sbc100

   Member

   More actions

   you can either assume your wasm engine has exclusive access to the files or use something like inotify to track creation/deletion/renames to know when the cache is out-of-date and/or update it.

   But we cannot make that assumption in the general case.. and there is no inotify mechanism just yet is there in WASI?
9. programmerjake commented on Jun 24, 2026

   

   programmerjake

   on Jun 24, 2026

   Contributor

   More actions

   you can either assume your wasm engine has exclusive access to the files or use something like inotify to track creation/deletion/renames to know when the cache is out-of-date and/or update it.

   But we cannot make that assumption in the general case.. and there is no inotify mechanism just yet is there in WASI?

   I was referring to host-side caching.

   👍React with 👍1Reacted by Sam Clegg
10. sbc100 commented on Jun 24, 2026

    

    sbc100

    on Jun 24, 2026

    Member

    More actions

    you can either assume your wasm engine has exclusive access to the files or use something like inotify to track creation/deletion/renames to know when the cache is out-of-date and/or update it.

    But we cannot make that assumption in the general case.. and there is no inotify mechanism just yet is there in WASI?

    I was referring to host-side caching.

    Ah yes, I assumed that that in above discussion of "userspace" we were talking about the guest wasm code and "syscalls" meant wasi syscalls.

    👍React with 👍1Reacted by Jacob Lifshay
11. Ericson2314 commented on Jun 25, 2026

    

    Ericson2314

    on Jun 25, 2026

    Last edited by Ericson2314

    More actions

    I don't do much with WASI (yet!) but I wanted to chime in that the basic idea here is how our VFS layer in Nix works.

    We (independently) decided a few years ago that trying to allow the Host OS implementation to "accelerate" symlink following just wasn't worth the attendant complexity. and doing it ourselves with a little bit of caching and opentat2 (on Linux) just isn't so bad. We do however do support multi-segment operations for RESOLVE_BENEATH acceleration.

    Good luck!

    CC @xokdvium
12. lukewagner commented on Jun 28, 2026

    

    lukewagner

    on Jun 28, 2026

    Member

    More actions

    The core/paths split idea does seem like an interesting alternative worth considering. If paths is made optional though and guests have to always implement their own path resolution anyways, it does reopen some of the concerns @badeend mentioned above.

    I do think that perhaps I didn't clearly explain what I was suggesting with the root/preopens split though and it could help reconcile these tensions. In particular, in this part of @alexcrichton's concern:

    Additionally wasi-libc, as an example, is such a core dependency that if it hardcodes usage of root as opposed to preopen-vfs I don't think there's going to be any reasonable way to change that.

    In what I was suggesting, preopen-vfs isn't an interface that wasi-libc would have to know about, it's a component that implements the root interface. Thus, wasi-libc would only ever call root and the ability to transparently plug in a preopen-vfs component (that implements root by performing VFS path resolution in terms of the lower-level preopens interface) is just component virtualizability as usual. Thus, regarding:

    I'd be worried that this wouldn't actually move the needle much on guest/host complexity.

    I think, with the above clarification (preopen-vfs is a component, not an interface), the net host complexity of the approach I'm suggesting is equivalent to the core-with-optional-paths approach (in particular, a host would natively implement root in the same scenarios that it would natively implement paths). The main difference is that wasi-libc (and other filesystem abstractions) wouldn't ever need to do their own VFS path resolution; they could simply use the root's open function. Given that we have N hosts and M wasi-libc-like-guest-libraries, the preopen-vfs component basically gets to sit in the middle, avoiding duplication in either. There are still tradeoffs to discuss (like the complexity of introducing an extra component), but I think it's an interesting candidate.
13. alexcrichton commented on Jun 29, 2026

    

    alexcrichton

    on Jun 29, 2026

    ContributorAuthor

    More actions

    Oh sorry about that I definitely conflated yours/@badeend's comments together. I agree that's definitely a candidate for how to make this more viable, and it's nice to be able to have a single canonical component do "hard stuff" as opposed to trying to replicate that. Runtimes supporting --dir-like flags (e.g. Wasmtime) would need to bake in this canonical component by default to handle --dir transparently, where the CLI would compose-on-the-fly according to --dir, which would be kind of complicated, but also not the end of the world.

    A consequence though of this approach is:

    • Libraries like wasi-libc will require external path-resolution by default.
    • Folks generally expect to be able to run the raw output of a compiler in runtimes, so wasi-libc raw components will be fed to runtimes.
    • Thus, runtimes will be expected to implement path-resolution behavior by default.

    The consequence here being that implementing sandboxed path-resolution on the host in a cross-platform manner is extremely difficult, and most runtimes don't even try today. While this scheme would simplify guests like wasi-libc a bit it'd still leave hosts in pretty much the same complicated spot they're in today (minus the VFS-related bits hosts are expected to implement).

    Personally I think I'd still favor the split @badeend mentioned where wasi-libc optional-imports path resolution and then otherwise requires segment-only APIs with internal logic for path resolution. Effectively I'd personally rank the "make it more reasonable for hosts to implement the sandbox" as a higher priority.
14. alexcrichton commented on Jul 13, 2026

    

    alexcrichton

    on Jul 13, 2026

    ContributorAuthor

    More actions

    Time has passed, I've thought about this some more, I've discussed this with some folks in meetings, and I want to provide some more thoughts on this.

    Overall it seems clear to me that we're not in a position to pursue a design such as this. The open questions are sort of hard blockers for something like this and no one's really in a position to evaluate them. For example it's not clear what the performance impact of this proposal is, and the default otherwise thinking is "probably bad" as it'd do more syscalls. It's also not clear how difficult it would be to implement this inside of wasi-libc, for example. The difficulty in implementing this wasi-libc would be a key driver in answering the question of "what should wasi-libc import by default," but without this it's hard to answer that. Effectively while (to me at least) it seems laudable to try to make host implementations easier the general community is just not staffed and/or populated with folks with enough interest to carry the burden to push a design like this forward.

    If a strategy like this cannot be done, then my thinking is that #742 is the way forward. Right now hosts and guests are all performing incorrect path resolution. Guests think they can pick a preopen and resolve a path within that, but they can't. Hosts think they can provide directories to a guest and the guest will do the path resolution correctly, but they don't. With #742 we'll at least move to a world where guests can't be wrong, they always resolve relative to the root. In this situation the onus is on hosts to implement path resolution correctly.

    This issue, and my conclusion at this time, is that hosts will continue to not implement path resolution correctly, however. WIth #742 most hosts will probably continue to not even attempt to provide sandboxing. Additionally hosts like Wasmtime which provide a VFS-lookalike (e.g. multiple --dir options) will also not implement path resolution correctly, much for the same reasons that pursuing this issue is not viable (no one is interested in pursing this and/or has a need for it).

    In my opinion, however, that's at least still a little bit better than it is today. A host can in theory do correct path resolution and sandboxing, even if none currently do. This to me is demonstrably better than today's situation which is effectively un-salvagable -- doing path resolution correctly requires doing the slow thing we don't want to do (aka per-segment file/dir resolution) in guests. Perhaps one day in the future someone can write a component that imports this issue's proposed API (per-segment resolution + root) and provides the API of #742 and today (root + guest imports ability to resolve within this root). That could in theory be used to determine performance/etc, and maybe at that point it'll be much easier to distribute such a component. For example Wasmtime might be able to bundle this component and then automatically compose it in when the host has multiple directory mounts for a VFS or something like that.

    ---

    Basically, tl;dr, I think the only viable option at this time is to close this issue and pursue #742. Notably deprecate preopens as-is with a new interface that just provides a single root descriptor. Hosts will be left to fend for themselves to retain preexisting functionality, and that's the tradeoff to make.
15. ricochet

    added

    P-filesystemProposal: wasi-filesystemProposal: wasi-filesystem

    on Aug 20, 2026