cmd/compile: allow the use of go:wasmimport globally #59149

Background

#38248 defined a new compiler directive, go:wasmimport, for declaring a dependency on a function provided by the runtime environment. It also defined a restriction on the use of this compiler directive, such that it cannot be used in code outside of the runtime and syscall/js standard library packages. #58141 relaxed this restriction to allow the use of the directive in the syscall package to allow accessing WASI defined syscall APIs necessary to support WASI.

The go:wasmimport directive is defined as follows:

//go:wasmimport importmodule importname

Where importmodule is the Wasm module of the function and importname is the name of the function within the module. The effect of this compiler directive is that any Go function calls of the function will be replaced with a call instruction to the function identified by the module and function name during compilation. Parameters and return values to the Go function are translated to Wasm according to the following table:

Any other parameter types are disallowed by the compiler.

For more information see the ssagen compiler package.

Proposal

We propose that the package restriction on the use of the go:wasmimport directive be removed entirely.

Rationale

Much of the discussion around the WASI proposal (#58141) revolved around the question of how and when the WASI target should move between syscall APIs (i.e. from depending on the wasm_snapshot_preview1 API to the wasm_snapshot_preview2 API and beyond). A major reason for this anxiety is because of the conflict between the desire for stability and the desire for access to newer runtime features afforded by newer syscall APIs.

One way to help avoid this conflict would be to provide access to newer APIs through purpose built third-party packages (e.g. golang.org/x/wasi/preview2/net) implementing various syscall level APIs through the use of go:wasmimport and exposing Go native types implementing interfaces such as net.Listener and net.Conn. Users who want to opt-in to new functionality can import one of these packages with the understanding that it may be unstable. Users who prefer to wait to use this functionality until it’s in the standard library can stay on the older API version. We’re exploring adding functionality to wit-bindgen to generate Go files with go:wasmimport directives and function signatures.

In addition to this, it would also allow users or vendors to create their own packages around vendor specific runtime extensions (e.g. WasmEdge defines its own sockets API, Fermyon defines a custom HTTP handler API) and expose them as easily consumable Go modules. This would help make Go more flexible and useful in different Wasm runtime environments.

Discussion

Rationale for the existing restriction

The existing restriction was added as a last minute addition to the original proposal. The primary rationale for this seems to have been an anxiety around exposing this widely without having had any experience of using it. If changes had to be made, it could break existing users.

We believe this concern should be smaller now that we have experience of using the directive while implementing go:wasmimport and working on implementing the WASI target.

Regardless, we do want to reserve the right to make breaking changes to this directive until such a point as the wasm architecture itself is declared stable. This is in line with the reservations made in #38248.

Effect on js/wasm

The js/wasm target has similar functionality exposed via the use of the syscall/js standard library package. The existing standard library interfaces will remain the preferred way of accessing functions defined on the JavaScript host. The primary use case within the js/wasm target would be as a means of accessing functions defined by other Wasm modules loaded into the same environment.

A note on wasm module names

The Wasm Core 1.0 spec defines a name as used in module and function names as a sequence of UTF-8 codepoints. The current definition of the go:wasmimport directive uses a space to separate the module name from the function name, making it impossible to specify a module or function name with a space in it. This proposal does not suggest changing the definition to support this use case.

Other language implementations

TinyGo

TinyGo uses the export pragma both for exporting to and importing methods from the environment. It also allows the user to specify the module to import from using the //go:wasm-module pragma. An example of this in the wild is the Fastly Compute@Edge SDK:

//go:wasm-module fastly_abi
//export init
//go:noescape
func fastlyABIInit(abiVersion prim.U64) FastlyStatus

TinyGo similarly restricts the types supported in parameters though I have struggled to find documentation for this exact limitation.

Rust

Rust uses the extern "C" raw interface pattern to define Wasm host imports. This is similar to our proposal in that it allows the user to specify the module and function name.

#[link(wasm_import_module = "the-wasm-import-module")]
extern "C" {
    // imports the name `foo` from `the-wasm-import-module`
    fn foo();

    // functions can have integer/float arguments/return values
    fn translate(a: i32) -> f32;

    // Note that the ABI of Rust and wasm is somewhat in flux, so while this
    // works, it's recommended to rely on raw integer/float values where
    // possible.
    fn translate_fancy(my_struct: MyStruct) -> u32;

    // you can also explicitly specify the name to import, this imports `bar`
    // instead of `baz` from `the-wasm-import-module`.
    #[link_name = "bar"]
    fn baz();
}

Rust supports using rich types in arguments but does not provide guarantees of the stability and encourages the use of integer/float values only.

Future work

Richer type support for imported functions

The current implementation of go:wasmimport supports only functions with integer, floating point or pointer parameters and a single return value. In the future, we would like to support types such as strings and structs and multiple return values. This would require defining a more elaborate calling convention for the imported functions than the one currently used.

The current mechanism follows the same semantics in use by other Wasm compilers such as Clang, which effectively treat Wasm the same as C, requiring the caller and callee to share the same ABI to be able to decode values in memory.

A 32 bit Wasm port

During the discussion of #59156, it became evident that many of the UX issues with the directive would disappear if the Wasm port was a 32 bit port, since all existing Wasm runtimes use 32 bit pointers, and the mismatch between the memory layout of Go and Wasm would disappear. There are still some concerns around whether the Go struct layout may change in the future, which would prevent us from guaranteeing direct memory mapping.

In the case of a future 32 bit Wasm port, the restriction on types allowed in the go:wasmimport parameters would be relaxed in a way that doesn't risk introducing memory corruption, but provides significant UX improvements to the compiler directive. The current wasm architecture, which is a 64 bit port, will likely retain the parameter limitations for the forseeable future.

64 bit Wasm runtimes

In the future, some runtimes may add support for 64 bit pointers. In such a runtime, the ABI translation rules for the use of the unsafe.Pointer type may change. We would evaluate the effects of this when/if adding support for 64 bit pointers.

Authors

@johanbrandhorst, @Pryz, @evanphx, @achille-roussel