Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pypy3.7 7.3.5 #1103

Merged
merged 1 commit into from
May 23, 2021
Merged

Add pypy3.7 7.3.5 #1103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ Image content

All images currently contain:

- CPython 3.5, 3.6, 3.7, 3.8 and 3.9, installed in
- CPython 3.6, 3.7, 3.8 and 3.9, PyPy 3.7 installed in
``/opt/python/<python tag>-<abi tag>``. The directories are named
after the PEP 425 tags for each environment --
e.g. ``/opt/python/cp37-cp37m`` contains a CPython 3.7 build, and
Expand All @@ -165,6 +165,8 @@ default ``sys.abiflags`` became an empty string: the ``m`` flag for pymalloc
became useless (builds with and without pymalloc are ABI compatible) and so has
been removed. (e.g. ``/opt/python/cp38-cp38``)

Note that PyPy is not available on ppc64le & s390x.

Building Docker images
----------------------

Expand Down
7 changes: 5 additions & 2 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,10 @@ COPY build_scripts/cpython-pubkey-310-311.txt /build_scripts/cpython-pubkeys.txt
RUN manylinux-entrypoint /build_scripts/build-cpython.sh 3.10.0b1


FROM build_cpython AS all_cpython
FROM build_cpython AS all_python
COPY build_scripts/install-pypy.sh /build_scripts/install-pypy.sh
COPY build_scripts/pypy.sha256 /build_scripts/pypy.sha256
RUN manylinux-entrypoint /build_scripts/install-pypy.sh 3.7 7.3.5
COPY --from=build_cpython36 /opt/_internal /opt/_internal/
COPY --from=build_cpython37 /opt/_internal /opt/_internal/
COPY --from=build_cpython38 /opt/_internal /opt/_internal/
Expand All @@ -144,7 +147,7 @@ FROM runtime_base
COPY --from=build_git /manylinux-rootfs /
COPY --from=build_swig /manylinux-rootfs /
COPY --from=build_cpython /manylinux-rootfs /
COPY --from=all_cpython /opt/_internal /opt/_internal/
COPY --from=all_python /opt/_internal /opt/_internal/
COPY build_scripts/finalize.sh build_scripts/update-system-packages.sh build_scripts/python-tag-abi-tag.py build_scripts/requirements.txt build_scripts/requirements-tools.txt /build_scripts/
RUN manylinux-entrypoint /build_scripts/finalize.sh && rm -rf /build_scripts

Expand Down
8 changes: 6 additions & 2 deletions docker/build_scripts/finalize.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ MY_DIR=$(dirname "${BASH_SOURCE[0]}")
source $MY_DIR/build_utils.sh

mkdir /opt/python
for PREFIX in $(find /opt/_internal/ -mindepth 1 -maxdepth 1 -name 'cpython*'); do
for PREFIX in $(find /opt/_internal/ -mindepth 1 -maxdepth 1 \( -name 'cpython*' -o -name 'pypy*' \)); do
# Some python's install as bin/python3. Make them available as
# bin/python.
if [ -e ${PREFIX}/bin/python3 ] && [ ! -e ${PREFIX}/bin/python ]; then
Expand All @@ -28,7 +28,11 @@ for PREFIX in $(find /opt/_internal/ -mindepth 1 -maxdepth 1 -name 'cpython*');
ln -s ${PREFIX} /opt/python/${ABI_TAG}
# Make versioned python commands available directly in environment.
PYVERS=$(${PREFIX}/bin/python -c "import sys; print('.'.join(map(str, sys.version_info[:2])))")
ln -s ${PREFIX}/bin/python /usr/local/bin/python${PYVERS}
if [[ "${PREFIX}" == *"/pypy"* ]]; then
ln -s ${PREFIX}/bin/python /usr/local/bin/pypy${PYVERS}
else
ln -s ${PREFIX}/bin/python /usr/local/bin/python${PYVERS}
fi
done

# Create venv for auditwheel & certifi
Expand Down
70 changes: 70 additions & 0 deletions docker/build_scripts/install-pypy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
#!/bin/bash

# Stop at any error, show all commands
set -exuo pipefail

# Get script directory
MY_DIR=$(dirname "${BASH_SOURCE[0]}")

# Get build utilities
source $MY_DIR/build_utils.sh


PYTHON_VERSION=$1
PYPY_VERSION=$2
PYPY_DOWNLOAD_URL=https://downloads.python.org/pypy


function get_shortdir {
local exe=$1
$exe -c 'import sys; print("pypy%d.%d-%d.%d.%d" % (sys.version_info[:2]+sys.pypy_version_info[:3]))'
}


mkdir -p /tmp
cd /tmp

case ${AUDITWHEEL_ARCH} in
x86_64) PYPY_ARCH=linux64;;
i686) PYPY_ARCH=linux32;;
aarch64) PYPY_ARCH=aarch64;;
*) echo "No PyPy for ${AUDITWHEEL_ARCH}"; exit 0;;
esac

TARBALL=pypy${PYTHON_VERSION}-v${PYPY_VERSION}-${PYPY_ARCH}.tar.bz2
TMPDIR=/tmp/${TARBALL/.tar.bz2//}
PREFIX="/opt/_internal"

mkdir -p ${PREFIX}

fetch_source ${TARBALL} ${PYPY_DOWNLOAD_URL}

# We only want to check the current tarball sha256sum
grep " ${TARBALL}\$" ${MY_DIR}/pypy.sha256 > ${TARBALL}.sha256
# then check sha256 sum
sha256sum -c ${TARBALL}.sha256

tar -xf ${TARBALL}

# the new PyPy 3 distributions don't have pypy symlinks to pypy3
if [ ! -f "${TMPDIR}/bin/pypy" ]; then
ln -s pypy3 ${TMPDIR}/bin/pypy
fi

# rename the directory to something shorter like pypy3.7-7.3.4
PREFIX=${PREFIX}/$(get_shortdir ${TMPDIR}/bin/pypy)
mv ${TMPDIR} ${PREFIX}

# add a generic "python" symlink
if [ ! -f "${PREFIX}/bin/python" ]; then
ln -s pypy ${PREFIX}/bin/python
fi

# remove debug symbols
rm ${PREFIX}/bin/*.debug

# We do not need the Python test suites
find ${PREFIX} -depth \( -type d -a -name test -o -name tests \) | xargs rm -rf

# We do not need precompiled .pyc and .pyo files.
clean_pyc ${PREFIX}
3 changes: 3 additions & 0 deletions docker/build_scripts/pypy.sha256
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
85d83093b3ef5b863f641bc4073d057cc98bb821e16aa9361a5ff4898e70e8ee pypy3.7-v7.3.5-aarch64.tar.bz2
3dd8b565203d372829e53945c599296fa961895130342ea13791b17c84ed06c4 pypy3.7-v7.3.5-linux32.tar.bz2
9000db3e87b54638e55177e68cbeb30a30fe5d17b6be48a9eb43d65b3ebcfc26 pypy3.7-v7.3.5-linux64.tar.bz2
16 changes: 11 additions & 5 deletions tests/run_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,18 @@ for PYTHON in /opt/python/*/bin/python; do
$PYTHON $MY_DIR/manylinux-check.py ${AUDITWHEEL_POLICY} ${AUDITWHEEL_ARCH}
# Make sure that SSL cert checking works
$PYTHON $MY_DIR/ssl-check.py
# Make sure sqlite3 module can be loaded properly and is the manylinux version one
# c.f. https://github.com/pypa/manylinux/issues/1030
$PYTHON -c 'import sqlite3; print(sqlite3.sqlite_version); assert sqlite3.sqlite_version_info[0:2] >= (3, 34)'
# pythonx.y shall be available directly in PATH
IMPLEMENTATION=$(${PYTHON} -c "import sys; print(sys.implementation.name)")
PYVERS=$(${PYTHON} -c "import sys; print('.'.join(map(str, sys.version_info[:2])))")
LINK_VERSION=$(python${PYVERS} -V)
if [ "${IMPLEMENTATION}" == "pypy" ]; then
LINK_PREFIX=pypy
else
LINK_PREFIX=python
# Make sure sqlite3 module can be loaded properly and is the manylinux version one
# c.f. https://github.com/pypa/manylinux/issues/1030
$PYTHON -c 'import sqlite3; print(sqlite3.sqlite_version); assert sqlite3.sqlite_version_info[0:2] >= (3, 34)'
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't these sanity checks be run on PyPy too? If they fail there is a problem with the PyPy build ...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't run the sanity check of the pipx.y symlink in /usr/local/bin since there are no symlink for PyPy pip in there. As mentioned in an earlier comment, going to remove those anyway.

For sqlite3, this is specific to cpython being rebuilt from source with a specific version (well should be specific, this has been an issue before). Given PyPY comes as a full featured binary, this check isn't right (& fails because the embedded sqlite3 is version 3.28 or 3.29 - I don't remember exactly)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the embedded sqlite3 is version 3.28 or 3.29

Hmm. Is it documented somewhere what sqlite3 a specific python version expects? If cpython is statically linking to 3.35 that would suggest PyPy should as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The minimum sqlite3 version required by cpython is fairly low: https://github.com/python/cpython/blob/220dd80a2671f57486055955d5422163cf73ed33/Modules/_sqlite/module.c#L33

The default version in manylinux1 was even older than this requirement though:

It's been updated since then in order not to get issue reported because of CVE on older versions.

Windows/macOS installers of CPython also update sqlite3 regularly as noted in their release notes.

# pythonX.Y / pypyX.Y shall be available directly in PATH
LINK_VERSION=$(${LINK_PREFIX}${PYVERS} -V)
REAL_VERSION=$(${PYTHON} -V)
test "${LINK_VERSION}" = "${REAL_VERSION}"
done
Expand Down