WASI testsuite

It would be nice to have a standard testsuite for the WASI API

Is there an official testsuit currently available? Even very simple.

There's a testsuite for the wasi-sdk, which is perhaps the closest we have right now.

I'm not sure it'd make sense to do something more directly for WASI itself (as opposed to the libc abstraction on top of it, which wasi-sdk uses) right now: once WASI is defined in terms of Interface Types[1], I think the tests would probably have to look somewhat different.

[1]: see the last paragraph in the design principle doc's section on security.

I'm not sure it'd make sense to do something more directly for WASI itself (as opposed to the libc abstraction on top of it, which wasi-sdk uses)

I didn't realise these are separate; but yeah, the testsuite of wasi-sdk looks pretty comprehensive. Do you think it's something that could be extracted into a separate submodule that other implementations could rely on / contribute to independently from wasi-sdk implementation?

@devsnek

Maybe I rushed with "comprehensive" bit; looking through the suite, it doesn't seem to test most of filesystem operations, for example, or randomness, or timers.

https://github.com/devsnek/node-wasi/tree/master/test/c by @devsnek, referenced from one of related issues above, seems better in this regard.

We've got more tests now: https://github.com/nodejs/node/tree/master/test/wasi/c

This is exactly what I want to express, there are many wasm vm implementations in the world, and they all have their own wasi test cases, which seems inconsistent.

I think we can write a unified testsuit, these testcases are not so platform-dependent, and which can ensure that the data structure, function signature and function logic are correct.

Some existing test cases are too complicated, I mean "not so much like unit tests". For example, when I want to testing fd_readdir, the case imports too many other wasi functions like:

  (import "wasi_unstable" "proc_exit" (func $__wasi_proc_exit (type 2)))
  (import "wasi_unstable" "fd_readdir" (func $__wasi_fd_readdir (type 3)))
  (import "wasi_unstable" "fd_close" (func $__wasi_fd_close (type 4)))
  (import "wasi_unstable" "fd_fdstat_get" (func $__wasi_fd_fdstat_get (type 5)))
  (import "wasi_unstable" "path_open" (func $__wasi_path_open (type 6)))
  (import "wasi_unstable" "args_sizes_get" (func $__wasi_args_sizes_get (type 5)))
  (import "wasi_unstable" "args_get" (func $__wasi_args_get (type 5)))
  (import "wasi_unstable" "fd_prestat_get" (func $__wasi_fd_prestat_get (type 5)))
  (import "wasi_unstable" "fd_prestat_dir_name" (func $__wasi_fd_prestat_dir_name (type 0)))
  (import "wasi_unstable" "fd_seek" (func $__wasi_fd_seek (type 7)))
  (import "wasi_unstable" "fd_write" (func $__wasi_fd_write (type 8)))

@devsnek

I have some tests for Deno's std/wasi implementation here and there are some legacy tests for libc in the pre-merge repository I haven't brought over; overall fairly similar coverage to @devsnek's tests, looks like I go a bit overboard on explicitly testing stdio 🤔 .

Ultimately I'd like to have a bunch of tests in the WebAssembly text format (for sharing between toolchains) that test the snapshots directly rather than relying fully on integration tests.

Tests written in wat are great because they easily work in many engines and don't require users to have various compilers for various languages installed in order to run them. So if people have tests of this form and/or are interesting in writing such tests, it'd be great to collect them.

That said though, tests written in wat directly, and targeting WASI APIs directly, are often very verbose and difficult to maintain, so it'd be good to collect compiled tests as well. Ideally we'd have the original Rust/C/etc. source files in the repository, so that people with the appropriate toolchains installed can edit and debug, and also the compiled output in the repository, in either wat or wasm format, so that it's possible for others to run the tests without having the toolchains installed.

@sunfishcode

I agree with @sunfishcode .. I don't think we should require that tests be authored in wat. Writing them in C or rust and providing supplying the compiled output in wat/wasm format should be fine.

@devsnek

Sounds like merging Node & Deno tests and precompiling to WAT alongside existing C files would be a good start then. @devsnek & @caspervonb do you want to uh talk to each other? :)

That said though, tests written in wat directly, and targeting WASI APIs directly, are often very verbose and difficult to maintain, so it'd be good to collect compiled tests as well.

Another alternative could be to generate wat files from another language, JS/Python. We've done that for a few proposals now, see https://github.com/WebAssembly/bulk-memory-operations/tree/master/test/meta and https://github.com/WebAssembly/simd/tree/master/test/core/simd/meta as examples.

I was working on a test generator (written in JS) to that effect, and I came to the conclusion that it's simpler to just write the tests in rust/c/wat/etc and keep the compiled versions in the source tree.

@devsnek

@devsnek that makes sense. It's a bit sad because those examples go from ~300 byte C files to ~30k wasm files. But I can definitely understand that the burden of writing in wat with or without a generator may outweigh this slight debugging cost.

@sunfishcode

So a little update, the test suite has moved here for the time being so it's not on my personal account and contributors can be added (hint hint).

We've merged the prebuilt test-suite into Deno which is working out great and I just published a little compatibility table over at https://khronosproject.github.io/wasi-compat-table.

I saw @sunfishcode mentioned this to the agenda of yesterday's meeting as-well. Anything I should be in the loop about?

@caspervonb

@caspervonb Is the plan still to have .wasm files committed to the repo? I'm not sure everyone would want to rebuild the .rs files.

@RReverser

Yep @RReverser!

Prebuilt is at https://github.com/khronosproject/wasi-test-suite which is what Deno pulls in as a submodule.

Ah great, thanks! To double-check - does this testsuite already include Node.js tests as well or only yours? So far I've been using the ones copied from Node.js in my WASI implementation, I wonder if I can just switch to this new repo instead, or need to combine both.

To double-check - does this testsuite already include Node.js tests as well or only yours?

Node's C tests are not included at the moment

I did have my initial batch of C tests for a while but diverged in favor of doing direct tests against wasi syscalls which is still a work in progress.

For libc i've been thinking its better to just bring in, or adapt libc-test than rolling our ad-hoc set of tests for it.
If it's good enough for musl it should be good enough for us.

TLDR; for a little while it may be beneficial to combine both; because deno/wasi is a sandbox within deno's sandbox conditions that would result in ENOTCAPABLE tests in particular are a bit lacking.

Ah, right, that makes sense. Perhaps we could at least make sure that equivalent syscalls / arguments are covered in the new repo, then the Node's tests could migrate fairly easily?

(Catching up here, appologies for being slow).

I have an experimental adaptation of musl's libc-test that I've tested WASI libc with here, in case it's useful: https://github.com/CraneStation/wasi-libc-test. I wrote my own harness instead of using the provided Makefile, which may or may not be the best approach, but it made it easier to customize things. It hasn't moved into the WebAssembly organization yet because libc-test contains code with a variety of licenses, which I expect would require approval.

Perhaps we could at least make sure that equivalent syscalls / arguments are covered in the new repo, then the Node's tests could migrate fairly easily?

Went through Node's tests and currently wasi-test-suite is lacking equivalents of these tests:

cant_dotdot.c
freopen.c
poll.c
escape_symlink.c

With the exception of poll they more or less equate to one assertion so I'll try to get those taken care of today; if not today, really soon (Deno release tomorrow and I have things pending merge 😉).

I have an experimental adaptation of musl's libc-test that I've tested WASI libc with here, in case it's useful: https://github.com/CraneStation/wasi-libc-test. I wrote my own harness instead of using the provided Makefile, which may or may not be the best approach, but it made it easier to customize things. It hasn't moved into the WebAssembly organization yet because libc-test contains code with a variety of licenses, which I expect would require approval.

Good to know; I'll take a proper look once I've got full coverage for direct wasi syscall tests 👍

The way I'm envisioning this is that straight ABI tests should be the source of truth; where-as libc, libc++ and libstd (as in rust libstd) tests are nice to have optional smoke tests.

I started work integrating https://github.com/khronosproject/wasi-test-suite into emscripten: emscripten-core/emscripten#12704

Once most tests are passing (not yet) I will update https://github.com/khronosproject/wasi-test 's compat.py to include an emscripten test function!

@sunfishcode

A little update:

I've been meaning to split the tests into sub repositories for a while and finally got around to it a couple of days ago, then I wasn't happy with the harness so I did it again.

The wasi-test repository is going the way of the dodo, it was trying to be a crate but it just made it convoluted to work with.

There are currently three source repositories:

These get built and merged into (https://github.com/caspervonb/wasi-testsuite); work in progress so coverage there is actually a bit less.

I've dropped the json preludes from the source files and instead gone with a simple file matching approach very similar to what is in wasi-sdk; meaning that each binary may have any of the following auxiliary files next to it:

<basename>.arg
<basename>.env
<basename>.stdin
<basename>.stdout
<basename>.stderr
<basename>.status

Intend to expand it with a couple of more auxiliary files and directories as I rewrite the old tests including but not limited to:

.dir
.invoke

A preliminary test runner used in the source repositories can be found at https://github.com/caspervonb/wasi-test-tools.

Heads up, it's bare bones, just enough to run the tests in CI.

At this point I'm pretty much accidentally converging with what we discussed earlier so going forward so I don't see much point in continuing this work outside of the webassembly/wasi-* repositories as I'm just starting on the rewrite so I'll start topping up @sunfishcode's review queue.

However, it seems however that the restructure still is not complete but I suppose that doesn't really interfere with this.

Change license to (Apache-2.0 WITH LLVM-exception) OR Apache-2.0 OR MIT

Port the wasi-clocks API to the new wit format.

Merge pull request WebAssembly#9 from sunfishcode/sunfishcode/isatty

Navigation Menu

Repository navigation

WASI testsuite #9

28 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

WASI testsuite #9

Description

Activity

mohanson commented on Jun 22, 2020

tschneidereit commented on Jun 22, 2020

RReverser commented on Jun 29, 2020

RReverser commented on Jun 29, 2020

devsnek commented on Jun 29, 2020

mohanson commented on Jun 29, 2020

caspervonb commented on Jun 29, 2020

sunfishcode commented on Jun 29, 2020

sbc100 commented on Jun 29, 2020

RReverser commented on Jun 29, 2020

binji commented on Jun 29, 2020

devsnek commented on Jun 29, 2020

binji commented on Jun 29, 2020

28 remaining items

caspervonb commented on Aug 28, 2020

RReverser commented on Sep 9, 2020

caspervonb commented on Sep 9, 2020

RReverser commented on Sep 9, 2020

caspervonb commented on Sep 9, 2020

RReverser commented on Sep 9, 2020

sunfishcode commented on Sep 10, 2020

caspervonb commented on Sep 10, 2020

caspervonb commented on Sep 10, 2020

sbc100 commented on Nov 5, 2020

caspervonb commented on Jan 14, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions