-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bindings: python: add a script to generate armv7l wheels #93
Closed
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,158 @@ | ||
#!/usr/bin/env bash | ||
# SPDX-License-Identifier: GPL-2.0-or-later | ||
# SPDX-FileCopyrightText: 2024 Paulo Ferreira de Castro <[email protected]> | ||
# | ||
# This script makes armv7l gpiod Python wheels for combinations of libc (glibc | ||
# and musl) and CPython versions, complementing the wheels made by cibuildwheel | ||
# (through the 'generate_pypi_artifacts.sh' script). | ||
# | ||
# Modify the TARGET_PYTHON_VERSIONS global variable below in order to change | ||
# the targeted CPython versions. | ||
# | ||
# Dependencies: Docker and binfmt_misc for ARM emulation. | ||
# | ||
# Usage: | ||
# ./generate_armv7l_wheels.sh | ||
# | ||
# The wheels will be placed in the ./dist directory. | ||
# | ||
|
||
TARGET_PYTHON_VERSIONS=(3.9 3.10 3.11 3.12) | ||
TARGET_ARCH='armv7l' | ||
|
||
# Docker image used to run Python commands locally (whichever host CPU). | ||
NATIVE_PYTHON_IMG="python:3.12-alpine" | ||
|
||
quit() { | ||
echo -e "\n${1}" | ||
exit 1 | ||
} | ||
|
||
# Print the gpiod Python binding version from gpiod/version.py. | ||
print_gpiod_version() { | ||
set -x | ||
docker run --rm -iv "${PWD}/gpiod:/gpiod" "${NATIVE_PYTHON_IMG}" python <<-EOF | ||
import sys | ||
sys.path.insert(0, "/gpiod") | ||
from version import __version__ | ||
print(__version__) | ||
EOF | ||
{ local status="$?"; set +x; } 2>/dev/null | ||
return "${status}" | ||
} | ||
|
||
# Make the sdist if it does not already exist in the './dist' directory. | ||
ensure_sdist() { | ||
local gpiod_version | ||
gpiod_version="$(print_gpiod_version)" || \ | ||
quit "Failed to determine the gpiod Python binding version" | ||
|
||
SDIST_TARBALL="gpiod-${gpiod_version}.tar.gz" | ||
if [ -f "dist/${SDIST_TARBALL}" ]; then | ||
return | ||
fi | ||
if [ -z "${LIBGPIOD_VERSION}" ]; then | ||
quit "Please set the LIBGPIOD_VERSION env var to enable making the sdist." | ||
fi | ||
set -x | ||
docker run --rm -v "${PWD}:/py_bindings" -w /py_bindings -e LIBGPIOD_VERSION \ | ||
"${NATIVE_PYTHON_IMG}" python setup.py sdist | ||
{ local status="$?"; set +x; } 2>/dev/null | ||
return "${status}" | ||
} | ||
|
||
# Set the BASE_IMG array with details of the Docker image used to build wheels. | ||
set_base_img_array() { | ||
local libc="$1" # The string 'glibc' or 'musl' | ||
declare -Ag BASE_IMG | ||
case "${libc}" in | ||
glibc) BASE_IMG[name]='python' | ||
BASE_IMG[distro]='bullseye' | ||
BASE_IMG[platform]='linux/arm/v7' | ||
BASE_IMG[wheel_plat]="manylinux_2_17_${TARGET_ARCH}" | ||
BASE_IMG[deps]=" | ||
RUN apt-get update && apt-get install -y autoconf-archive | ||
# auditwheel requires patchelf v0.14+, but Debian Bullseye comes with v0.12. | ||
RUN curl -Ls 'https://github.com/NixOS/patchelf/releases/download/0.18.0/patchelf-0.18.0-${TARGET_ARCH}.tar.gz' \ | ||
| tar -xzC /usr ./bin/patchelf | ||
";; | ||
musl) BASE_IMG[name]='python' | ||
BASE_IMG[distro]='alpine3.20' | ||
BASE_IMG[platform]='linux/arm/v7' | ||
BASE_IMG[wheel_plat]="musllinux_1_2_${TARGET_ARCH}" | ||
BASE_IMG[deps]=" | ||
RUN apk add autoconf autoconf-archive automake bash binutils curl \ | ||
g++ git libtool linux-headers make patchelf pkgconfig | ||
";; | ||
esac | ||
} | ||
|
||
# Make a "wheel builder" docker image used to build gpiod wheels. | ||
# The base image is the official Python image on Docker Hub, in either the | ||
# Debian variant (glibc) or the Alpine variant (musl). Currently using | ||
# Debian 11 Bullseye because it supports building wheels targeting glibc | ||
# v2.17. The newer Debian 12 Bookworm supports higher glibc versions only. | ||
make_wheel_builder_image() { | ||
local builder_img_tag="$1" | ||
local python_version="$2" | ||
set -x | ||
docker build --platform "${BASE_IMG[platform]}" --pull --progress plain \ | ||
--tag "${builder_img_tag}" --file - . <<-EOF | ||
FROM '${BASE_IMG[name]}:${python_version}-${BASE_IMG[distro]}' | ||
${BASE_IMG[deps]} | ||
RUN pip install auditwheel | ||
EOF | ||
{ local status="$?"; set +x; } 2>/dev/null | ||
return "${status}" | ||
} | ||
|
||
# Build a gpiod wheel in a container of the given builder image tag. | ||
build_wheel() { | ||
local builder_img_tag="$1" | ||
set -x | ||
docker run --rm --platform "${BASE_IMG[platform]}" -iv "${PWD}/dist:/dist" \ | ||
-w /tmp "${builder_img_tag}" bash <<-EOF | ||
set -ex | ||
pip wheel --no-deps "/dist/${SDIST_TARBALL}" | ||
auditwheel repair --plat "${BASE_IMG[wheel_plat]}" --wheel-dir /dist ./*.whl | ||
EOF | ||
{ local status="$?"; set +x; } 2>/dev/null | ||
return "${status}" | ||
} | ||
|
||
# Build gpiod wheels for combinations of libc and Python versions. | ||
build_combinations() { | ||
local builder_img_tag= | ||
|
||
for libc in glibc musl; do | ||
set_base_img_array "${libc}" | ||
|
||
for ver in "${TARGET_PYTHON_VERSIONS[@]}"; do | ||
builder_img_tag="gpiod-wheel-builder-cp${ver/./}-${libc}-${TARGET_ARCH}" | ||
|
||
make_wheel_builder_image "${builder_img_tag}" "${ver}" || \ | ||
quit "Failed to build Docker image '${builder_img_tag}'" | ||
|
||
build_wheel "${builder_img_tag}" || \ | ||
quit "Failed to build gpiod wheel with image '${builder_img_tag}'" | ||
|
||
set -x | ||
docker image rm "${builder_img_tag}" | ||
{ set +x; } 2>/dev/null | ||
done | ||
done | ||
} | ||
|
||
main() { | ||
set -e | ||
local script_dir # Directory where this script is located | ||
script_dir="$(cd -- "$(dirname "$0")" >/dev/null 2>&1 || true; pwd -P)" | ||
cd "${script_dir}" || quit "'cd ${script_dir}' failed" | ||
ensure_sdist || quit "Failed to find or make the sdist tarball." | ||
build_combinations | ||
echo | ||
ls -l dist/*"${TARGET_ARCH}"* | ||
echo -e "\nAll done! Hint: Use 'docker system prune' to free disk space." | ||
} | ||
|
||
main |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it still a separate script? Can it not be integrated into the existing one for some reason?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let’s first make a decision on whether or not
armv7l
wheels will be published to PyPI.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not super straight forward unfortunately. cibuildwheel does have hooks for specifying the docker container used to generate a specific set of wheels, however, they didn't include armv7l until recently (and even then, only partially) so it's not as simple as specifying CIBW_{MANY,MUSL}LINUX_ARMV7L_IMAGE= and kicking off cibuildwheel.
pypa/cibuildwheel#2017
pypa/cibuildwheel#1421
I think they, too, are deferring to piwheels to provide those images for glibc environments because RPi OS is pretty much the defacto glibc based distro.
To add this support, a custom docker container needs to be generated with requisite dependencies, such as pip/build/etc and then invoked, preferably following the steps that cibuildwheel does it:
https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/linux.py#L161
So, stage one would generate standard wheels via cibuildwheel then a second stage would have to construct the container to generate armv7l wheels. I don't think there's any reason it couldn't be folded into the current script.
I'm not sure if we want to reinvent the wheel (pun partially intended). It looks like some people have build containers available: https://github.com/bjia56/armv7l-wheel-builder.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh wow, so two weeks ago
cibuildwheel
added support formusllinux armv7l
wheels, but notmanylinux
. I didn’t see that one coming! 🙂Re https://github.com/bjia56/armv7l-wheel-builder, their Dockerfile is building binutils and glibc from source, and targeting
armv6
. It’s much simpler and faster to use the official Python base image on Docker Hub, as I’ve done in this PR.If we are happy in principle to have
armv7l
wheels published to PyPI, I’d be happy to amend the existing script to usecibuildwheel
formusllinux
, and to add a few script functions to cover themanylinux
case, along the lines of the work I have already done in this PR.