Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Macros] Add support for wasm macros #73031

Open
wants to merge 132 commits into
base: main
Choose a base branch
from

Conversation

kabiroberai
Copy link
Contributor

@kabiroberai kabiroberai commented Apr 15, 2024

This PR introduces support for WebAssembly-based macros via -load-plugin-executable Foo.wasm#Foo.

Wasm macros are blazing fast (no SwiftSyntax build required for consumers!) and secure (fully sandboxed on all OSes)

time $BIN/swiftc Client.swift \
  -load-plugin-executable ExampleRaw.wasm#ExampleRaw
Ninja-RelWithDebInfoAssert/swift-macosx-arm64/bin/swiftc -c 0.06s user 0.03s system 33% cpu 0.244 total

The approach I've taken here is to update swift-plugin-server to add special handling for Wasm macros. These macros are executed using WasmKit, but in the future we can add support for other (faster, JIT) runtimes like JavaScriptCore.

Note also that these wasm macros currently have an ad-hoc ABI as I wanted to avoid a hard dependency on WASI. It's thus theoretically possible to build macros without a dependency on WASI, though I've had trouble doing so because Swift currently builds for wasm as WASI xor Embedded Mode — we'd need a middle ground with support for String etc. The current ABI is based on the one I built for Wacro. We'll have to iron out the execution model though: my current mental model is that plugins are request-response style (i.e. every HostToPluginMessage must be followed by exactly one PluginToHostMessage) and so the macro effectively exports a char *macro_parse(char *request_json) but I don't know if this is a guaranteed contract.

There's currently a few things that still need to be done:

  • Support other platforms with WasmKit. (*)
  • Unit tests

Followups:

  • Figure out the story for building macro targets — the approach I'm using right now is to have the "source" macro depend on Wacro but we'll probably want to upstream some sort of Wasm macro module into swift-syntax. (*)
  • SwiftPM support + evolution proposal? I'm envisioning a .binaryMacro target similar to .binaryTarget, though this will make the edit-compile-run cycle non-trivial as it splits building the macro into a separate phase. (*)

Marking this as a draft until the above items are done but I wanted to make this PR so I could get some input as well as to brainstorm (particularly about the items marked as (*))

Depends on swiftlang/swift-syntax#2623

There's also a corresponding PR in swift-driver, which is blocked by this one: swiftlang/swift-driver#1582

lib/AST/PluginLoader.cpp Outdated Show resolved Hide resolved
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-jit</key>
Copy link
Contributor Author

@kabiroberai kabiroberai Apr 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Necessary in order for JSC to expose globalThis.WebAssembly — I don't think there's any security risk here because /System/Library/Frameworks/JavaScriptCore.framework/Versions/Current/Helpers/jsc already exposes significantly more surface area. Plus we're able to keep the restrictive sandbox from Sandbox::apply intact when running this via swiftc.

@kabiroberai kabiroberai marked this pull request as draft April 15, 2024 18:15
@kabiroberai
Copy link
Contributor Author

paging @DougGregor @MaxDesiatov as stakeholders, would love to get your thoughts on this

@kabiroberai
Copy link
Contributor Author

also wondering how I should add end-to-end tests for this: I would include some wasm blob fixtures but I don't want to be mistaken for the Second Coming of Jia Tan

@kabiroberai
Copy link
Contributor Author

Another question: how much should we generalize this right now? As in, should we have a specialized wasm executor embedded into swift-wasm-plugin-server or would it maybe make sense to ship a general swift-wasm-runtime that can be used for other things (such as Swift Package Plugins) as well? I'm leaning towards the former because I think we shouldn't prematurely abstract but there are pros to the other approach as well.

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

MaxDesiatov added a commit to swiftlang/swift-syntax that referenced this pull request Dec 11, 2024
This method deterministically and synchronously cleans up resources that cannot be dealt with in a deinitializer due to possible errors thrown during the clean up. Usually this includes closure of file handles, sockets, shutting down external processes and IPC resources set up for these processes, etc.

This method is required for `swift-plugin-server` to properly clean up file handles there were opened for communication with Wasm macros, as prototyped in swiftlang/swift#73031.
@MaxDesiatov
Copy link
Contributor

MaxDesiatov added a commit to swiftlang/swift-syntax that referenced this pull request Dec 12, 2024
This method deterministically and synchronously cleans up resources that cannot be dealt with in a deinitializer due to possible errors thrown during the clean up. Usually this includes closure of file handles, sockets, shutting down external processes and IPC resources set up for these processes, etc.

This method is required for `swift-plugin-server` to properly clean up file handles there were opened for communication with Wasm macros, as prototyped in swiftlang/swift#73031.
@MaxDesiatov
Copy link
Contributor

@swift-ci test

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

@MaxDesiatov
Copy link
Contributor

@swift-ci test

@MaxDesiatov
Copy link
Contributor

@swift-ci test

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

@MaxDesiatov
Copy link
Contributor

@swift-ci test

@MaxDesiatov
Copy link
Contributor

@swift-ci test

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants