Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bindings: python: add a script to generate armv7l wheels #93

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions bindings/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,14 @@ If you still need the deprecated pure-Python bindings, install with:
pip install gpiod==1.5.4
```

Pre-built binary `manylinux` and `musllinux` wheels are published to PyPI.org
for selected platforms, to speed up installation. `armv7l` users: Note that the
pre-built armv7l wheels make use of the VFP instruction set extension (a.k.a.
hard float). If your system does not support VFP, `pip install` will rebuild the
wheel from source on a `manylinux` system, but possibly not on a `musllinux`
system. If you have a musl-libc-based armv7l system without VFP support and you
come across issues using the pre-built gpiod wheel, please get in touch.

## Examples

Check a GPIO chip character device exists:
Expand Down
158 changes: 158 additions & 0 deletions bindings/python/generate_armv7l_wheels.sh
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it still a separate script? Can it not be integrated into the existing one for some reason?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s first make a decision on whether or not armv7l wheels will be published to PyPI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not super straight forward unfortunately. cibuildwheel does have hooks for specifying the docker container used to generate a specific set of wheels, however, they didn't include armv7l until recently (and even then, only partially) so it's not as simple as specifying CIBW_{MANY,MUSL}LINUX_ARMV7L_IMAGE= and kicking off cibuildwheel.

pypa/cibuildwheel#2017

pypa/cibuildwheel#1421

I think they, too, are deferring to piwheels to provide those images for glibc environments because RPi OS is pretty much the defacto glibc based distro.

To add this support, a custom docker container needs to be generated with requisite dependencies, such as pip/build/etc and then invoked, preferably following the steps that cibuildwheel does it:

https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/linux.py#L161

So, stage one would generate standard wheels via cibuildwheel then a second stage would have to construct the container to generate armv7l wheels. I don't think there's any reason it couldn't be folded into the current script.

I'm not sure if we want to reinvent the wheel (pun partially intended). It looks like some people have build containers available: https://github.com/bjia56/armv7l-wheel-builder.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wow, so two weeks ago cibuildwheel added support for musllinux armv7l wheels, but not manylinux. I didn’t see that one coming! 🙂

Re https://github.com/bjia56/armv7l-wheel-builder, their Dockerfile is building binutils and glibc from source, and targeting armv6. It’s much simpler and faster to use the official Python base image on Docker Hub, as I’ve done in this PR.

If we are happy in principle to have armv7l wheels published to PyPI, I’d be happy to amend the existing script to use cibuildwheel for musllinux, and to add a few script functions to cover the manylinux case, along the lines of the work I have already done in this PR.

Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: GPL-2.0-or-later
# SPDX-FileCopyrightText: 2024 Paulo Ferreira de Castro <[email protected]>
#
# This script makes armv7l gpiod Python wheels for combinations of libc (glibc
# and musl) and CPython versions, complementing the wheels made by cibuildwheel
# (through the 'generate_pypi_artifacts.sh' script).
#
# Modify the TARGET_PYTHON_VERSIONS global variable below in order to change
# the targeted CPython versions.
#
# Dependencies: Docker and binfmt_misc for ARM emulation.
#
# Usage:
# ./generate_armv7l_wheels.sh
#
# The wheels will be placed in the ./dist directory.
#

TARGET_PYTHON_VERSIONS=(3.9 3.10 3.11 3.12)
TARGET_ARCH='armv7l'

# Docker image used to run Python commands locally (whichever host CPU).
NATIVE_PYTHON_IMG="python:3.12-alpine"

quit() {
echo -e "\n${1}"
exit 1
}

# Print the gpiod Python binding version from gpiod/version.py.
print_gpiod_version() {
set -x
docker run --rm -iv "${PWD}/gpiod:/gpiod" "${NATIVE_PYTHON_IMG}" python <<-EOF
import sys
sys.path.insert(0, "/gpiod")
from version import __version__
print(__version__)
EOF
{ local status="$?"; set +x; } 2>/dev/null
return "${status}"
}

# Make the sdist if it does not already exist in the './dist' directory.
ensure_sdist() {
local gpiod_version
gpiod_version="$(print_gpiod_version)" || \
quit "Failed to determine the gpiod Python binding version"

SDIST_TARBALL="gpiod-${gpiod_version}.tar.gz"
if [ -f "dist/${SDIST_TARBALL}" ]; then
return
fi
if [ -z "${LIBGPIOD_VERSION}" ]; then
quit "Please set the LIBGPIOD_VERSION env var to enable making the sdist."
fi
set -x
docker run --rm -v "${PWD}:/py_bindings" -w /py_bindings -e LIBGPIOD_VERSION \
"${NATIVE_PYTHON_IMG}" python setup.py sdist
{ local status="$?"; set +x; } 2>/dev/null
return "${status}"
}

# Set the BASE_IMG array with details of the Docker image used to build wheels.
set_base_img_array() {
local libc="$1" # The string 'glibc' or 'musl'
declare -Ag BASE_IMG
case "${libc}" in
glibc) BASE_IMG[name]='python'
BASE_IMG[distro]='bullseye'
BASE_IMG[platform]='linux/arm/v7'
BASE_IMG[wheel_plat]="manylinux_2_17_${TARGET_ARCH}"
BASE_IMG[deps]="
RUN apt-get update && apt-get install -y autoconf-archive
# auditwheel requires patchelf v0.14+, but Debian Bullseye comes with v0.12.
RUN curl -Ls 'https://github.com/NixOS/patchelf/releases/download/0.18.0/patchelf-0.18.0-${TARGET_ARCH}.tar.gz' \
| tar -xzC /usr ./bin/patchelf
";;
musl) BASE_IMG[name]='python'
BASE_IMG[distro]='alpine3.20'
BASE_IMG[platform]='linux/arm/v7'
BASE_IMG[wheel_plat]="musllinux_1_2_${TARGET_ARCH}"
BASE_IMG[deps]="
RUN apk add autoconf autoconf-archive automake bash binutils curl \
g++ git libtool linux-headers make patchelf pkgconfig
";;
esac
}

# Make a "wheel builder" docker image used to build gpiod wheels.
# The base image is the official Python image on Docker Hub, in either the
# Debian variant (glibc) or the Alpine variant (musl). Currently using
# Debian 11 Bullseye because it supports building wheels targeting glibc
# v2.17. The newer Debian 12 Bookworm supports higher glibc versions only.
make_wheel_builder_image() {
local builder_img_tag="$1"
local python_version="$2"
set -x
docker build --platform "${BASE_IMG[platform]}" --pull --progress plain \
--tag "${builder_img_tag}" --file - . <<-EOF
FROM '${BASE_IMG[name]}:${python_version}-${BASE_IMG[distro]}'
${BASE_IMG[deps]}
RUN pip install auditwheel
EOF
{ local status="$?"; set +x; } 2>/dev/null
return "${status}"
}

# Build a gpiod wheel in a container of the given builder image tag.
build_wheel() {
local builder_img_tag="$1"
set -x
docker run --rm --platform "${BASE_IMG[platform]}" -iv "${PWD}/dist:/dist" \
-w /tmp "${builder_img_tag}" bash <<-EOF
set -ex
pip wheel --no-deps "/dist/${SDIST_TARBALL}"
auditwheel repair --plat "${BASE_IMG[wheel_plat]}" --wheel-dir /dist ./*.whl
EOF
{ local status="$?"; set +x; } 2>/dev/null
return "${status}"
}

# Build gpiod wheels for combinations of libc and Python versions.
build_combinations() {
local builder_img_tag=

for libc in glibc musl; do
set_base_img_array "${libc}"

for ver in "${TARGET_PYTHON_VERSIONS[@]}"; do
builder_img_tag="gpiod-wheel-builder-cp${ver/./}-${libc}-${TARGET_ARCH}"

make_wheel_builder_image "${builder_img_tag}" "${ver}" || \
quit "Failed to build Docker image '${builder_img_tag}'"

build_wheel "${builder_img_tag}" || \
quit "Failed to build gpiod wheel with image '${builder_img_tag}'"

set -x
docker image rm "${builder_img_tag}"
{ set +x; } 2>/dev/null
done
done
}

main() {
set -e
local script_dir # Directory where this script is located
script_dir="$(cd -- "$(dirname "$0")" >/dev/null 2>&1 || true; pwd -P)"
cd "${script_dir}" || quit "'cd ${script_dir}' failed"
ensure_sdist || quit "Failed to find or make the sdist tarball."
build_combinations
echo
ls -l dist/*"${TARGET_ARCH}"*
echo -e "\nAll done! Hint: Use 'docker system prune' to free disk space."
}

main