Sketch: allow a variable number of imports/exports of fixed type in Wit

@dicej

Motivation

Let's say I'd like to build a component that consumes 3 configuration values a, b and c (which in a 12-factor app I'd take as 3 environment variables). I could define a component with type:

(component
  (import "get-config" (func (param "key" string) (result (option string))))
  ...
)

However, this loses the fact that my component specifically wants a, b and c. If I'm using or deploying this component, I have to learn about these values from the docs or observe the behavior of get-config at runtime. If I make a mistake, the error will only show up at runtime.

If instead I give my component this type:

(component
  (import "config" (instance
    (export "a" (func (result string)))
    (export "b" (func (result string)))
    (export "c" (func (result string)))
  ))
  ...
)

then there's a lot more declarative information that a host can leverage to provide a better developer experience. E.g., at deployment time, the host can check that there are indeed configuration values a, b and c available and give a deployment-time error if not. There are also new optimizations made available at runtime: a, b and c can be held in a dense array accessed by internally-computed static index, thereby avoiding the hash-table lookups at runtime (which can really matter when the target instantiation-time is in microseconds).

This example is in the domain of configuration, but you can also find analogous examples in:

metrics (with one (inlineable) (func (param "increment" u32)) import per metric)
logging (supporting multiple logging backends)
HTTP upstreams (with one import of wasi:http/outgoing-handler per upstream.
KV/Blob stores (with one import per distinct storage backend)

In all these cases, the workaround for supporting multiple instances of the same interface inevitably involves using a dynamic string parameter which loses the otherwise-declarative dependency information. (To be clear, some use cases do really need a runtime-dynamic string name; we're talking here about the cases where the string would otherwise be a magic constant in the code.)

So given that we'd like to write components with the above type, how do we capture all these varying types in Wit? Of course we can write the Wit for any single component; for example, a Wit world that supports the above component is:

world my-world {
  import config: interface {
    a: func() -> string
    b: func() -> string
    c: func() -> string
  }
  ...
}

The challenge is writing a single world that captures this component and all the other components, each with their own varying set of configuration values.

The proposal is to allow Wit to express not just a single interface or world, but a family of interfaces/worlds produced by substituting various parameters. Concretely, for the above I'd like to write (roughly; the exact syntax here is open for debate, of course):

world my-world {
  import config: interface {
   *: func() -> string
  }
  ...
}

Using this, it would be natural for WASI standards to use * in standardized interfaces such as:

default interface "wasi:config/values" {
  *: func() -> string
}

which would then allow my-world to be rewritten as:

world my-world {
  import config: "wasi:config/values"
  ...
}

and leverage standard implementations of wasi:config/values.

Sketch

While in general I think we'll want to allow putting parameters everywhere in Wit (types, names, parameters, results, fields, cases, imports, exports, ...), as a first step, I'd like to keep things scoped to just the case I showed above where a * can show up in the name of the lone field of an interface. While this won't be easy (there are a number of interesting producer/consumer design questions to work out), I think it'll be much easier than parameters in types, and so a good starting point. For terminology, I've been calling the general feature "Wit templates", and I think this first milestone could be called "variable imports and exports", but open to hearing alternatives.

From a Wit grammar perspective, the addition is fairly tiny, defining a:

variable-id ::= id | '*'

and then using variable-id in func-item and typedef-item. As an additional validation-time constraint, * would only be allowed inside an interface block as the only item. (Or we could capture this constraint in the grammar; but I think we'll want to loosen this constraint over time.)

To allow encoding Wit documents as .wasm binaries, we'll also need to extend Binary.md to support * in names. One thing to be clear about here is that a concrete component won't be allowed to have any *s in its imports/exports; only Wit documents. In a far-post-MVP future, one could imagine generalizing components with a staged-compilation model that did allow components to talk about * names, so while we don't need to design how that all would work, it would be good to pick an encoding of * that could be retconned into this far-post-MVP future if needed.

Significant design work will be needed per-language-toolchain around how to allow the developer to fill in the *s in the world they are targeting. Riffing on an idea from @dicej and @fibonacci1729, this could come from buildconfig, with each line adding a field. E.g., in the context of cargo component, I could write:

[package.metadata.component.target]
path = "my-world.wit"

[package.metadata.component.parameters]
config.a = "string"
config.b = "string"
config.c = "string"

which would declare 3 imported configuration values a, b and c, looking ahead to a future milestone where non-string types could be allowed. More design work is probably necessary here to think through how to express all the not-so-simple cases. But the idea is that, from this language-specific buildconfig, tooling could derive a language-agnostic substitution which would then be applied to the target world to produce a "monomorphized" world with no *s that is fed into bindings generation, keeping the rest of the build pipeline working like normal.

This is just a sketch, and more work is needed to flesh out the design, but I thought it'd be useful to put up this much now since the interface design questions above are coming up in a number of places at the moment. @dicej and @fibonacci1729 have also done a lot of thinking about this, so I'd invite them to drop in whatever they're thinking too.

Nice write-up! The cargo component example really makes sense and makes it easy for developers to express configs. I have a few questions regarding the later part of the write-up in this proposal:

Is the scope of star imports restricted to functions without any parameters and single string return type, like *: func() -> string? If not, I'd be interested to understand what *: func(param1, param2, ...) -> (ret1, ret2) would mean for a configuration.
The example in cargo component is really nice. I'd be interested to see how other languages would express this in their buildconfig, especially some dyanmic languages like Python and JavaScript.
Following up on question 2, I am not entirely sure about what a buildconfig even is. What is a buildconfig for Python or JavaScript? Could a buildconfig be implemented as a language-independent configuration file using YAML. Could it be a ConfigMap in Kubernetes?

@lukewagner

This looks great @lukewagner , thanks for capturing all of this!

+1 to not encoding the rule * would only be allowed inside an interface block as the only item directly into the grammar. Plenty of languages allow more expressiveness per their grammar while enforcing particular constraints during parse.

I might not be making the connection here, but how is the RHS string type of the parameter in Cargo.toml used during the substitution, for instance when substituting in the above *: func()->string?

@fibonacci1729

Ah, I think my Cargo.toml example was confusing. Because wasi:config/values as-written has a fixed string result type, there is no right-hand-side necessary. Since TOML doesn't allow empty right-hand-side, maybe I should have written this instead:

[package.metadata.component.target]
path = "my-world.wit"

[package.metadata.component.parameters]
config = [ "a", "b", "c" ]

What I wrote would make sense once type parameters were possible and wasi:config/values was generalized interface { *: func() -> _ }), so I sortof jumped straight there in my example, where I'm substituting string for the _.

I think that answers @fibonacci1729 and partially @Mossaka's question 1. Answering the rest:

The * should be usable with any function type (the "string" in the Cargo.toml was a red herring).
For JS, I think you'd write the analogous thing but in package.json. I can't speak to the Python though.
Yes, great point! I think we could definitely have a language-agnostic TOML (or JSON or ...) file that was fed into wit-bindgen. Maybe this is even our starting point, so it sortof sets up a "reference" format, and then Cargo.toml and package.json are just developer conveniences that avoid one extra config file.

@fibonacci1729

FYI, @fibonacci1729 and I have this pretty much implemented; we'll demo it at tomorrow's Component Model meeting, then clean things up and start opening PRs. I'm sure @alexcrichton will have opinions :)

bytecodealliance/wasm-tools@main...fermyon:wasm-tools:wit-templates
bytecodealliance/wit-bindgen@main...dicej:wit-templates
bytecodealliance/wasmtime@main...dicej:wasmtime:wit-templates

I unfortunately won't be able to make the meeting tomorrow, but I look forward to digging into these bits on Monday!

support worlds which import and/or export "wildcard" interfaces

First draft PR for feedback: bytecodealliance/wasmtime#5925
I anticipate that one will generate the most discussion, since there are a variety of ways host bindings could be generated, and I picked the one that seemed most flexible to me.
@Kylebrown9 FYI

support worlds which import and/or export "wildcard" interfaces

Per WebAssembly/component-model#172, this implements "part 1" of WIT

implement "part 1" of WIT Templates

@lukewagner

@lukewagner Brian and I ran into what seems to be a pretty fundamental issue while implementing this.

Consider this world:

interface foo {
    *: func() -> u32
}

default world wildcards {
    import imports1: self.foo
    import imports2: self.foo
}

Now imagine we want to build a component that targets that world, using ["a", "b", "c"] to expand imports1's wildcard and ["x", "y", "z"] to expand imports2's wildcard. AFAICT there's currently no way to represent that component type, whether as WIT or in binary -- we can say foo has functions a, b, and c or we can say it has functions x, y, and z, but we can't say both at the same time.

Of course, we could split foo into two separate interfaces, but how would we name them, and how would you do a subtype comparison between the original WIT template and the concretized WIT?

I just realized my example above is rejected by wasm-tools anyway since you can't import the same interface more than once (I'd be curious to know the reason for that, BTW; EDIT: Brian pointed me to bytecodealliance/wit-bindgen#529 for context). It does accept importing and exporting the same interface, though, so it's still a problem in that scenario.

Also, perhaps I was incorrect in claiming there's no way to represent the component type of the "concretized" world, given that wasm-tools component wit -t wildcards.wit currently duplicates interface types when generating a component type, so you could imagine a substitutions-aware version of wasm-tools doing something like this:

input wildcards.wit:

interface foo {
    *: func() -> u32
}

default world wildcards {
    import imports: self.foo
    export exports: self.foo
}

input substitutions.toml:

[wildcards]
imports = ["a", "b", "c"]
exports = ["x", "y", "z"]

output concrete.wat:

(component
  (type (;0;)
    (component
      (type (;0;)
        (instance
          (type (;0;) (func (result u32)))
          ;; TODO: this part won't validate -- still need proper representation for wildcards:
          ;; (export (;0;) "*" (func (type 0))) 
        )
      )
      (export (;0;) "foo" "pkg:/wildcards/foo" (instance (type 0)))
      (type (;1;)
        (component
          (type (;0;)
            (instance
              (type (;0;) (func (result u32)))
              (export (;0;) "a" (func (type 0)))
              (export (;0;) "b" (func (type 0)))
              (export (;0;) "c" (func (type 0)))
            )
          )
          (import "imports" "pkg:/wildcards/foo" (instance (type 0)))
          (type (;1;)
            (instance
              (type (;0;) (func (result u32)))
              (export (;0;) "x" (func (type 0)))
              (export (;0;) "y" (func (type 0)))
              (export (;0;) "z" (func (type 0)))
            )
          )
          (export (;0;) "exports" "pkg:/wildcards/foo" (instance (type 1)))
        )
      )
      (export (;0;) "wildcards" "pkg:/wildcards/wildcards" (component (type 1)))
    )
  )
  (export (;1;) "wildcards" "pkg:/wildcards" (type 0))
)

The above component validates just fine, but there's no way we could convert it back into WIT, since we've given an inconsistent definition of pkg:/wildcards/foo. So I guess you could argue this is just a WIT limitation, but even though the above component validates, I'd still consider it malformed given the inconsistency. In the end, I think we'd want to change both the component model and WIT if we want to support instantiating the same interface more than once in a given world.

So where does that leave us? I see two ways forward:

Punt for now and just disallow instantiating the same wildcard-containing interface more than once in a given world.
Formally define what multiple instantiation of a wildcard-containing interface means and how it is represented in both WIT and the component model in a consistent, unambiguous way

Assuming we do want to modify WIT and the CM to enable multiple instantiation of wildcard interfaces, I think it might be helpful to compare it to the existing "genericity" in WIT/CM: list, result, option, and tuple. Note that none of those things are types -- they're type constructors, i.e. they are "functions" take a type (or more than one in the cases of result and tuple) and produce a type. In Haskell parlance, list has kind * -> *, while u8 has kind *.

Analogously, I would suggest that an interface containing a wildcard is not an interface at all -- it's an interface constructor, i.e. a "function" that takes a list of names and produces an interface. And just as you can't say type foo = list, I'd contend that you shouldn't be able to say import imports: self.foo if foo has a wildcard because you can't instantiate an interface constructor -- you need to monomorphize it first. And I think the key to allowing multiple instantiation of wildcard interfaces is to make an explicit distinction between interfaces and interface constructors in both WIT and the CM.

We can bikeshed the syntax, but for the moment let's imagine we used a similar syntax for interface constructors (and world constructors!) to what we use for type constructors:

// Polymorphic interface constructor:
interface foo<W...> {
    <W...>: func() -> u32 // expand W into zero or more functions of this type
}

// Monomorphic world: needs no substitutions
world monomorphic {
    import imports: self.foo<a, b, c>
    export exports: self.foo<x, y, z>
}

// Polymorphic world constructor; substitutions needed to monomophize:
world polymorphic<W1..., W2...> {
    import imports: self.foo<W1>
    import imports: self.foo<W2>
}

The main virtue of this syntax is that you can see at a glance which things are worlds or interfaces (monomorphic) vs. which ones are world or interface constructors (polymorphic). And crucially for our purposes, we can invoke a given interface or world constructor multiple times without ambiguity.

To be clear, I'm not necessarily advocating for that specific syntax -- just that we make the distinction between interfaces/worlds and interface/world constructors clear and unambiguous in both WIT and the CM, so we can round-trip between them losslessly -- both before and after monomorphization. This will be even more important when we introduce type holes (which are really just type parameters to interface/world constructors).

implement "part 1" of WIT Templates

feat: preliminary support for WIT templates

implement "part 1" of WIT Templates

feat: preliminary support for WIT templates

implement "part 1" of WIT Templates

@lukewagner

I went ahead and created draft PRs containing a minimum viable implementation:
bytecodealliance/wasm-tools#964
bytecodealliance/wit-bindgen#541
bytecodealliance/wasmtime#5934

I like @lukewagner 's idea for specifying ids for otherwise-anonymous interfaces. For now, though, our implementation only clones the wildcard interface as an anonymous interface prior to expanding the wildcard, which is enough to get an end-to-end test working.

feat: preliminary support for WIT templates

implement "part 1" of WIT Templates

Does this mean reading some information from an additional toml file? I find this inconvenient.

Is there a solution that is compatible with inline packages(#313), where one file contains all the necessary information?

No, the toml file was just one option for specific use cases where it lines up with what you want to say. But there are lots of other producer toolchain options or WIT features that could be used to fill in the *'s (and we didn't get far enough along to begin to explore them all). But as a base case, you can always write a concrete (non-templated) world in WIT by hand that matches (is a subtype of) a templated world, filling in the *'s manually -- everything else would just be sugar for synthesizing this concrete world from something more intuitive (as other examples: the specifier strings of JS import statements or a Rust proc-macro).

Navigation Menu

Repository navigation

Sketch: allow a variable number of imports/exports of fixed type in Wit #172

Motivation

Sketch

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sketch: allow a variable number of imports/exports of fixed type in Wit #172

Description

Motivation

Sketch

Activity

Mossaka commented on Feb 28, 2023

fibonacci1729 commented on Feb 28, 2023

lukewagner commented on Feb 28, 2023

dicej commented on Mar 3, 2023

alexcrichton commented on Mar 3, 2023

dicej commented on Mar 3, 2023

dicej commented on Mar 8, 2023

dicej commented on Mar 9, 2023

dicej commented on Mar 9, 2023

8 remaining items

dicej commented on Mar 20, 2023

oovm commented on May 23, 2024

lukewagner commented on May 23, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions