GH-45237: [Python] Raise minimum supported cython to >=3

[raulcd]

Rationale for this change

We do not require to support Cython 2 anymore.

What changes are included in this PR?

Bump required version of Cython

Are these changes tested?

Via CI

Are there any user-facing changes?

Yes, pyarrow cannot be compiled with Cython2 anymore.

• GitHub Issue: [Python] Raise minimum supported cython to >=3  #45237

[github-actions]

⚠️ GitHub issue #45237 has been automatically assigned in GitHub to PR creator.

[raulcd]

@github-actions crossbow submit -g python

[github-actions]

Revision: c6bf8d4

Submitted crossbow builds: ursacomputing/crossbow @ actions-07fd1dc2a0

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[raulcd]

I tried to remove the pytest-cython pin for the docs builds but it fails. I've added an issue to follow that up as this is not related to this issue:

• [Python] Remove pin for pytest-cython 0.2.2 #45239

[raulcd]

@github-actions crossbow submit test-conda-python-3.11-hypothesis

[github-actions]

Revision: 2889c63

Submitted crossbow builds: ursacomputing/crossbow @ actions-6811b48c94

Task	Status
test-conda-python-3.11-hypothesis

[pitrou]

LGTM, but I think we can also remove this line:

arrow/python/CMakeLists.txt

Lines 203 to 205 in 04249b9

	# See https://github.com/cython/cython/issues/2731. Change introduced in
	# Cython 0.29.1 causes "unsafe use of type 'bool' in operation"
	string(APPEND CMAKE_CXX_FLAGS " /wd4804")

[jorisvandenbossche]

There are also some other snippets in the cython code that we keep around only for cython 2 compat, I think, but that probably can be removed later as well

(for example our custom declaration of optional in lib.pxd, the SharedPtrNoGIL has a comment indicating it was also meant to work with both cython 2 and 3, and there are a bunch of Required for cython < 3 comments)

[raulcd]

@github-actions crossbow submit -g python

[raulcd]

@jorisvandenbossche I've removed the unnecessary binding decorator but I've opened:

• [Python] Remove / Update code that might not be required with Cython > 2 #45253

To tackle the std::optional and the SharedPtrNoGIL as this might take me longer and can be done afterwards.
Let me know if that's ok with you.

[github-actions]

Revision: 87bd74f

Submitted crossbow builds: ursacomputing/crossbow @ actions-20576e4f2d

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[raulcd]

@github-actions crossbow submit wheel-windows-cp310-cp310-amd64

[github-actions]

Revision: 87bd74f

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c1d7348b5

Task	Status
wheel-windows-cp310-cp310-amd64

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 8d367d2.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.