[libshortfin] Add workflow to build nightly packages.

[ScottTodd]

Progress on #130, follow-up to #230.

Overview

This gets us the general structure of a release pipeline that:

1. Computes some version information metadata based on build id and date
2. Calls shortfin/build_tools/build_linux_package.sh, which runs python -m pip wheel under a manylinux Docker container
3. Uploads wheels to both GitHub artifacts (for the workflow run itself) and GitHub releases (for archival and developer usage)

This follows how stablehlo (https://github.com/openxla/stablehlo/releases/tag/dev-wheels) and torch-mlir (https://github.com/llvm/torch-mlir-release/releases/tag/dev-wheels) both publish to a constantly growing "dev-wheels" release rather than separate releases like IREE does. That makes it harder to directly "promote" a candidate release, but it avoids polluting the release history with nightly builds.

Testing

• Sample run: https://github.com/ScottTodd/SHARK-Platform/actions/runs/11078636913
• Dev wheels release: https://github.com/ScottTodd/SHARK-Platform/releases/tag/dev-wheels
• Install with python -m pip install shortfin -f https://github.com/ScottTodd/SHARK-Platform/releases/expanded_assets/dev-wheels

What's next

In no particular order,

• Verify that the Tracy build enabled by SHORTFIN_ENABLE_TRACING is functional
• Set up dependencies and optional components such that a user can just pip install shortfin and have it bring along a compatible version of packages like iree-runtime and iree-compiler
• Update the dockerfile version from manylinux2014 (that is independent of this workflow)
• Build for Python 3.13 free threaded and test that
• Build for other platforms/architectures (macOS, Windows, arm64, etc.)
• Set up a workflow like IREE's pkgci that installs the packages and runs some tests on them
• Document how to install the developer packages

[marbre]

It makes sense to not set this to 0.9 here but just as a note, we still do this here
https://github.com/nod-ai/SHARK-Platform/blob/6c8272042a39860e13ee548967bb1c1fcd404c60/shortfin/CMakeLists.txt#L18

[ScottTodd]

Yeah more work will be needed to fully plumb versions (including stable, nightly, and dev) through each part of the projects. I tried to focus on the minimal changes needed to get a nightly release going here. In IREE we have some CMake options: https://github.com/iree-org/iree/blob/6c47f631855ced5f48d4897df11784f81ae24386/CMakeLists.txt#L84-L88 that get set through setup.py and other scripts: https://github.com/iree-org/iree/blob/6c47f631855ced5f48d4897df11784f81ae24386/compiler/setup.py#L162-L169

[marbre]

Do we want to pre-build with a non-free-threaded 3.13 at all?

[ScottTodd]

Probably? @stellaraccident , opinions? I think we'll want to publish for both flavors of python 3.13, but I'm not sure how pip handles that.

[ScottTodd]

Reading through https://py-free-threading.github.io/.

• Building with cp313-cp313t (note the t) seems to work, even with manylinux2014.
• Going to figure out how to test next

[ScottTodd]

Tested. Free threaded seems to work fine.

$ sudo apt-get install python3.13-nogil
$ python3.13t -m venv 3.13t.venv
$ source 3.13t.venv/bin/activate

$ python -c "import sys; print(sys._is_gil_enabled())"
False

$ python -m pip install shortfin -f https://github.com/ScottTodd/SHARK-Platform/releases/expanded_assets/dev-wheels

Looking in links: https://github.com/ScottTodd/SHARK-Platform/releases/expanded_assets/dev-wheels
Collecting shortfin
  Downloading https://github.com/ScottTodd/SHARK-Platform/releases/download/dev-wheels/shortfin-20240930.17-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 46.9 MB/s eta 0:00:00
Installing collected packages: shortfin
Successfully installed shortfin-20240930.17

$ python
>>> import shortfin as sf