Draft a shark-platform meta package to organize Python packages.

[ScottTodd]

Progress on #130, posting this for discussion.

Current local build instructions:

# Setup venv the project root.
# Note: Until IREE publishes packages for 3.13, stick to 3.12.
#     * See https://github.com/iree-org/iree/issues/18652
cd SHARK-Platform/packaging/shark-platform
python3.12 -m venv 3.12.venv
source 3.12.venv/bin/activate

# Build the meta package.
# Note: The `--find-links` is required until we push a new version to pypi.
#     * https://pypi.org/project/shortfin/ has an old pre-release version that we
#       _don't_ want to fetch, since that depends on a similarly old
#       pre-release version of https://pypi.org/project/sharktank/
export BUILD_DIR=$PWD/wheelhouse
python -m pip wheel \
    --disable-pip-version-check \
    -v -w $PWD/wheelhouse \
    --find-links https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels \
    .

deactivate

Current install instructions:

# Setup a fresh venv to install the meta package into.
mkdir -p /tmp/shark-platform-packaging
cd /tmp/shark-platform-packaging
python3.12 -m venv 3.12.venv
source 3.12.venv/bin/activate

# Install the meta package with some extras
# Note: The `--find-links` is required until we push a new version to pypi.
python -m pip install \
    $BUILD_DIR/shark_platform-0.1.dev3-py3-none-any.whl[onnx] \
    --find-links https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels
# Installing collected packages: shortfin, mpmath, sympy, shark-platform, protobuf, numpy, onnx, iree-runtime, iree-compiler
# Successfully installed iree-compiler-20240828.999 iree-runtime-20240828.999 mpmath-1.3.0 numpy-2.1.1 onnx-1.17.0 protobuf-5.28.2 shark-platform-0.1.dev3 shortfin-20241002.4 sympy-1.13.3

[stellaraccident]

I'm not sure how I feel about there being a top-level Python project and projects nested below it. In my mind, this was a leaf directory but trying to think of a name:

• packaging/
• shark-platform/
• packaging/shark-platform

Basically needs to be a subdirectory somewhere. Python projects have an annoying way of repeating primary directory names... I kind of like having a packaging/ tree where we keep general project packaging infra/support and can also stash the meta package under there where it won't hurt anyone.

[ScottTodd]

Cool, I was considering a new top level folder too.

I also have my eye on the various requirements.txt files scattered around and the setup instructions in READMEs. For example, the root README has sharktank instructions and the base https://github.com/nod-ai/SHARK-Platform/blob/main/requirements.txt file has a bunch of test and serving deps. Should make a pass through those and check that each project tracks its own minimal and organized deps.

[stellaraccident]

Cool, I was considering a new top level folder too.

I also have my eye on the various requirements.txt files scattered around and the setup instructions in READMEs. For example, the root README has sharktank instructions and the base https://github.com/nod-ai/SHARK-Platform/blob/main/requirements.txt file has a bunch of test and serving deps. Should make a pass through those and check that each project tracks its own minimal and organized deps.

Yeah. Would recommend refreshing a view of how others do this and see if there are more modern practices... and simplify as part of having one top level README/scripts for getting a full dev environment setup.

[ScottTodd]

Moved into packaging/shark-platform and started on a packaging/README.md

[marbre]

sharktank has iree-turbine as install required, thus it is pulled in anyway.

[ScottTodd]

Closing in favor of #475