libshortfin CI and Releases

[stellaraccident]

Bringing up libshortfin CI and releases

Overall Description

libshortfin is a C++ project providing serving oriented APIs atop the IREE runtime. It aims to supercede prior pure-Python systems that used IREE via its low-level synchronous API. From the get-go, libshortfin is:

• Multi-device
• Parallel (designed to scale in a single process)
• Exposing a high level Python API that directly mirrors the C++ API
• Async (integrating with both Python asyncio and C++ coroutines)

Internally, it consists of three primary components:

• libshortfin C++ library (built as both a static and a dynamic library)
• _shortfin native Python module
• shortfin pure Python module (re-exports from the native module and adds
  higher level Python-only features)

Note that we are transitioning to using the containing repository (currently called sharktank) as a monorepo for a variety of related model development and serving projects. As such, plan for some organization to the way that venvs/dev setup is done across the repo (and CI, etc).

Dependencies

Native deps

libshortfin currently depends purely on the IREE runtime C APIs. In the future, it will also expose an interface to the
IREE compiler C API via a delay loaded stub, allowing the compiler to be dynamically loaded (same as some other integrations). The compiler will remain an optional dep and pure-inference or deployed systems can leave it off so long as whatever is built on libshortfin does not need it (i.e. if pipelines are pre-compiled).

libshortfin currently has the following additional native dependencies:

• spdlog
• fmt (dependency of spdlog and used internally)
• xtensor
• gtest (if testing is enabled)

Python deps

shortfin does not have any Python deps for minimal functionality, but it is expected that a number of deps will be useful for various applications of it, and it may provide features based on these deps in an optional fashion that people can use if helpful/available:

• pytest (for testing)
• numpy
• torch
• uvicorn (or other ASGI web server)
• iree-turbine (for torch compilation)
  • Transitive deps on iree-compiler and iree-runtime
• onnx (for onnx compilation)
• etc

Variants

The native _shortfin.lib Python package is provided, similar to iree.runtime, by redirecting at runtime to a concrete native library module based on env flags, allowing the selection of multiple packages runtimes such as "default", "tracy", "assert", etc. As with the iree.runtime, we expect that our normal wheels will bundle the default and tracy versions so that users can always use an instrumented build.

Dependent Projects

It is intended that both C++ and pure/native Python extensions will be built atop libshortfin, providing application level support for various models, etc.

Moving towards release

There are several steps towards a robust release of the project.

Initial Packaging and CI

Initial Build Work

See the current README for developer quality instructions. This just uses find_package for both IREE and other dependencies. A setup.py is added that works for development, but it needs to be completed to build a hermetic/production build.

This will necessitate supporting CMake options to enable bundled/pinned deps vs relying on find_package. The trickiest of these is IREE itself, and for that we should support two modes:

• Bring your own source (SHORTFIN_IREE_SOURCE_DIR set, for example) which will do a add_subdirectory on IREE and include it directly in the build.
• FetchContent a pinned IREE release in bundled mode and build off of that.

These bundling modes can be developed/tested in pure cmake in isolation. Once functional, setup.py can be extended to invoke CMake itself. See IREE runtime/setup.py for inspiration. The result should be that pip install libshortfin/ works on any reasonably modern Linux and Windows system with no further fuss.

There are a few things that should be done at this point:

• Set a soname on libshortfin.so or shortfin-{soname}.dll and let it be configured (needed for coexistence and dependent extension linking).
• Enable LTO build options for IREE, spdlog, and xtensor.
• Set compiler/linker flags to include a linker script that enables symbol versioning.
• Tweak hidden visibility and API re-export settings for deps (they should already be right for libshortfin and are tested).
• Set up libshortfin CMake install/exports
• Bundle development headers and CMake exports with the build Python wheels so that applications can be auto configured without fiddling with native deps.

Initial CI work

CI should be brought up to:

• Build Python packages.
• Run native unit tests via CTest (running on CPU is fine for these).
• Run pytest integration tests on combinations of machines with specific hardware.
• Push built binaries to the GH releases page using some form of pre-release versioning.

Note that the project is self contained enough that it should build just fine on free linux and windows runners.

Versioning and Releasing

Releasing libshortfin properly will necessitate working out the versioning and releasing policies for some of its related deps (IREE, iree-turbine, sharktank, etc). This may be the end of the road for the nightly builds of those, and we may want to take this chance to come up with a real versioning policy and apply it uniformly to all of the projects. It would be great to have normal pre-releases and regular releases in stable places where everything can be installed by adding one index-url (vs the current state where everything is somewhere bespoke with all kinds of -f flags).

We also get requests for conda packages from some in the community, and this would be a good time to enable those.

[ScottTodd]

For the package builds, do we want to use manylinux?

Dockerfile choices:

• nod-ai: https://github.com/nod-ai/base-docker-images/pkgs/container/manylinux_x86_64
  • source: https://github.com/nod-ai/base-docker-images/blob/main/dockerfiles/manylinux_x86_64.Dockerfile)
• iree-org: https://github.com/iree-org/base-docker-images/pkgs/container/manylinux_x86_64
  • source: https://github.com/iree-org/base-docker-images/blob/main/dockerfiles/manylinux_x86_64.Dockerfile)
• upstream pypa: https://quay.io/repository/pypa/manylinux_2_28_x86_64
  • source: https://github.com/pypa/manylinux

[stellaraccident]

You always want to build python packages that will be run on any other os with a many Linux image. It's typically easier to just do that from the get go. I almost always use one that has been forked locally and extended with needed packages.

[ScottTodd]

Can we yank the old shortfin releases (https://pypi.org/project/shortfin/#history) ? That only has version 0.1.dev3, for Python 3 (including pre-3.12).

If I try running python -m pip install shortfin -f https://github.com/ScottTodd/SHARK-Platform/releases/expanded_assets/dev-wheels from Python 3.10, that gets the stale package, since the new packages are only for Python 3.12 and 3.13.

[ScottTodd]

Can we yank the old shortfin releases (https://pypi.org/project/shortfin/#history) ? That only has version 0.1.dev3, for Python 3 (including pre-3.12).

If I try running python -m pip install shortfin -f https://github.com/ScottTodd/SHARK-Platform/releases/expanded_assets/dev-wheels from Python 3.10, that gets the stale package, since the new packages are only for Python 3.12 and 3.13.

Bah, https://peps.python.org/pep-0440/#handling-of-pre-releases

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

That old 0.1.dev3 pre-release package for https://pypi.org/project/shortfin/ (and similarly for https://pypi.org/project/sharktank/) is making testing new packaging tricky:

• The old shortfin package is for any Python version, not just 3.12+
• The old shortfin package depends on sharktank, iree-runtime, etc.
• The sharktank package depends on shark-turbine, which still needs to be migrated to iree-turbine

[ScottTodd]

Brain dump before vacation (could merge these into a single tasklist in the original issue comment):

• Nightly releases of shortfin for Python 3.12, 3.13, and 3.13t are being published to https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels by https://github.com/nod-ai/SHARK-Platform/actions/workflows/build_packages.yml.
• Nightly releases of shortfin can be installed using

  python -m pip install shortfin -f https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels
  

• Instructions for building release packages locally are at https://github.com/nod-ai/SHARK-Platform/tree/main/shortfin#package-python-release-builds
• Draft a shark-platform meta package to organize Python packages. #244 is in review
• The example at shortfin/examples/python/mobilenet_server works on my machine as of 8e074b4 . We'll want more examples and tests for those examples as these packages mature.
• shortfin API docs are hosted at https://shortfin.readthedocs.io/en/latest/ (manage that from https://readthedocs.org/projects/shortfin/, Marius and I are maintainers). We should check that the docs are useful enough to build examples with... I found them a bit lacking while fixing the mobilenet server code.

TODO:

• Build and publish releases of sharktank to the same location.
  • That should be simpler since sharktank is a pure Python package (no C/C++/CMake), but will likely need a few patches to sharktank/setup.py, as well as .github/workflows/build_packages.yml
• Settle on a versioning scheme and roll that out across each setup.py file and the various projects in scope (iree-compiler, iree-runtime, iree-turbine, sharktank, shortfin, etc.)
• Ensure that all projects in scope work across our supported Python versions and operating systems. For IREE, I have Build and publish Python 3.13 and 3.13t wheels iree-org/iree#18652 in progress
• Add CI workflows that test the built packages (new smoketests, existing unit tests, nightly full model tests/workflows)
• Rebase the shortfin build process from manylinux2014 to manylinux_2_28
• Build for other platforms/architectures (macOS, Windows, arm64, etc.)
• Publish releases to pypi, possibly using --pre for prerelease versions. Put less emphasis on installing from source or nightly builds
• Refactor requirements.txt files and README.md files across https://github.com/nod-ai/SHARK-Platform , focusing on what each individual project needs for development and what the aggregate "platform" needs to be used

[ScottTodd]

The main chunk of work here is now done. We could keep this open for some of the smaller items (expanded CI coverage, upgrading to manylinux_2_28, making the CMake build more compatible with downstream projects, etc.)... or we could close this as fixed now.

[marbre]

I think this has still some information / requests we either want to transfer to new issues or we keep this open.

[ScottTodd]

shortfin builds are working on Windows again, so we can start thinking about publishing Windows packages too.

From the repo root, this works for me on Windows:

python -m pip wheel --disable-pip-version-check --no-deps -v -w %CD%/shortfin/wheelhouse %CD%/shortfin

ls shortfin\wheelhouse\
# shortfin-3.1.0.dev0-cp311-cp311-win_amd64.whl

Should be able to add a build_windows_package.sh (or .ps1) next to https://github.com/nod-ai/shark-ai/blob/main/shortfin/build_tools/build_linux_package.sh and hook that up to a workflow pretty easily - no manylinux or Docker needed. Can also reference the script that IREE has been using: https://github.com/iree-org/iree/blob/main/build_tools/python_deploy/build_windows_packages.ps1.