Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable kikuchipy #65

Merged
merged 15 commits into from
Dec 12, 2024
Merged

Conversation

hakonanes
Copy link
Contributor

We've just released kikuchipy v0.11.0 on PyPI, compatible with HyperSpy >= 2.2 (https://pypi.org/project/kikuchipy/).

I've therefore re-enabled kikuchipy here.

@ericpre ericpre force-pushed the enable-kikuchipy-again branch from 7f02653 to 4879043 Compare November 11, 2024 20:14
@ericpre
Copy link
Member

ericpre commented Nov 11, 2024

python -m pytest --pyargs kikuchipy
============================= test session starts ==============================
platform linux -- Python 3.11.10, pytest-8.3.3, pluggy-1.5.0
Matplotlib: 3.9.[2](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11785534636/job/32826974375?pr=65#step:26:2)
Freetype: 2.12.1
rootdir: /home/runner/work/hyperspy-extensions-list/hyperspy-extensions-list
plugins: mpl-0.17.0, rerunfailures-14.0, xdist-[3](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11785534636/job/32826974375?pr=65#step:26:3).6.1
collected 0 items

============================ no tests ran in 0.16s =============================

@hakonanes
Copy link
Contributor Author

We don't package tests after changing to src-layout. I can look at including them in the next release.

@hakonanes
Copy link
Contributor Author

pytest --pyargs kikuchipy won't discover any tests even though it is installed from GitHub because tests are not included in site-packages. We need to make a release where we include it in the tarball under kikuchipy/tests. So I'd skip testing it all together for now.

@ericpre
Copy link
Member

ericpre commented Nov 12, 2024

Can you push a fix so that it can be tested when installing from github for now? The alternative would be to install in editable mode.
Anyway, this step is skipped because there is still a dependency issue with quaternion/orix/pyxem.

@hakonanes
Copy link
Contributor Author

Can you push a fix so that it can be tested when installing from github for now?

Not sure how to do this, if it's at all possible? May be that an editable install is the only option.

Anyway, this step is skipped because there is still a dependency issue with quaternion/orix/pyxem.

Oh no, what's the issue?

@ericpre
Copy link
Member

ericpre commented Nov 13, 2024

Not sure how to do this, if it's at all possible? May be that an editable install is the only option.

Update installation configuration to include the tests files? It doesn't need a release because we install from source?

Oh no, what's the issue?

from latest build: https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65

Fatal Python error: Segmentation fault

Thread 0x00007f181430e640 (most recent call first):
  File "/home/runner/miniconda3/envs/test/lib/python3.11/socket.py", line 294 in accept
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/pytest_rerunfailures.py", line 433 in run_server
  File "/home/runner/miniconda3/envs/test/lib/python3.11/threading.py", line 982 in run
  File "/home/runner/miniconda3/envs/test/lib/python3.11/threading.py", line [10](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65#step:28:11)45 in _bootstrap_inner
  File "/home/runner/miniconda3/envs/test/lib/python3.[11](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65#step:28:12)/threading.py", line 1002 in _bootstrap

Current thread 0x00007f1816[12](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65#step:28:13)4440 (most recent call first):
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/numba/np/ufunc/gufunc.py", line 279 in __call__
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/quaternion.py", line 1282 in qu_multiply
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/quaternion.py", line 207 in __mul__
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/quaternion.py", line 1085 in outer
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/rotation.py", line 385 in outer
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/symmetry.py", line 396 in from_generators
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/symmetry.py", line 566 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1[14](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65#step:28:15)7 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/orientation_region.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1[17](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65#step:28:18)6 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/misorientation.py", line 32 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/orientation.py", line 31 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/quaternion/__init__.py", line 25 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1126 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/crystal_map/phase_list.py", line 33 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/crystal_map/crystal_map.py", line 25 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/orix/crystal_map/__init__.py", line 26 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/pyxem/utils/indexation_utils.py", line 29 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/pyxem/utils/calibration.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/pyxem/utils/__init__.py", line 39 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1126 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/pyxem/signals/common_diffraction.py", line 33 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/pyxem/signals/__init__.py", line 23 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/home/runner/miniconda3/envs/test/lib/python3.11/site-packages/pyxem/__init__.py", line 29 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1204 in _gcd_import
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  ...

Extension modules: markupsafe._speedups, numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg, PIL._imaging, kiwisolver._cext, gmpy2.gmpy2, traits.ctraits, yaml._yaml, cytoolz.utils, cytoolz.itertoolz, cytoolz.functoolz, cytoolz.dicttoolz, cytoolz.recipes, psutil._psutil_linux, psutil._psutil_posix, scipy._lib._ccallback_c, numba.core.typeconv._typeconv, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt[19](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65#step:28:20)937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.fftpack.convolve, box.exceptions, work._ruamel_yaml, _ruamel_yaml, msgpack._cmsgpack, box.converters, box.box, box.box_list, box.config_box, box.from_file, box.shorthand_box, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, numba.mviewbuf, numba.types.itertools, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, skimage._shared.geometry, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.signal._sosfilt, scipy.signal._spectral, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.signal._peak_finding_utils, sklearn.__check_build._check_build, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.utils._fast_dict, sklearn.cluster._hierarchical_fast, sklearn.cluster._k_means_common, sklearn.cluster._k_means_elkan, sklearn.cluster._k_means_lloyd, sklearn.cluster._k_means_minibatch, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.utils.arrayfuncs, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.cluster._dbscan_inner, sklearn.cluster._hdbscan._tree, sklearn.cluster._hdbscan._linkage, sklearn.cluster._hdbscan._reachability, sklearn._isotonic, sklearn.tree._utils, sklearn.tree._tree, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.neighbors._quad_tree, sklearn.manifold._barnes_hut_tsne, sklearn.manifold._utils, _cffi_backend (total: 194)
/home/runner/work/_temp/e0aebf4c-9008-4693-a0be-71eeb1a5e55f.sh: line 1:  48[35](https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/11801796768/job/32875961663?pr=65#step:28:36) Segmentation fault      (core dumped) python -m pytest --pyargs pyxem

@CSSFrancis

@ericpre
Copy link
Member

ericpre commented Nov 15, 2024

Sorry, my previous statement was most likely incorrect: the issue is with numpy 2.0 - possibly not orix/quaternion but with pyfai dependencies in pyxem?

@hakonanes
Copy link
Contributor Author

Update installation configuration to include the tests files? It doesn't need a release because we install from source?

With pip install? I don't know how I can tell pip to include the tests/ directory from GitHub within site-packages/kikuchipy/tests for pytest to discover them.

@ericpre
Copy link
Member

ericpre commented Nov 18, 2024

The build backend should be taking take of that.

@hakonanes
Copy link
Contributor Author

Yes, and to do that I need to make a new post release. But you said I didn't need to, or did I misunderstand?

@hakonanes
Copy link
Contributor Author

Yes, and to do that I need to make a new post release. But you said I didn't need to, or did I misunderstand?

Just push a fix to GitHub, you meant... 🦥

…y-extensions-list into enable-kikuchipy-again

Signed-off-by: Håkon Wiik Ånes <[email protected]>
@hakonanes
Copy link
Contributor Author

hakonanes commented Nov 30, 2024

The most recent kikuchipy 0.11.1.post0 wheel from PyPI has tests included.

Edit: the conda-forge release also has the tests packaged.

@hakonanes
Copy link
Contributor Author

The new conda-forge release 0.11.1.post0 also includes PyVista. This was done so that all optional dependencies now are installed from conda-forge ("batteries included").

@hakonanes hakonanes force-pushed the enable-kikuchipy-again branch from 18b0094 to 90e43cc Compare December 1, 2024 09:57
@hakonanes
Copy link
Contributor Author

The kikuchipy test suite now passes when

  • installing the latest release from conda-forge
  • installing develop branch from GH with HyperSpy release next patch

All other runs fail the tests with a seg fault before running any tests. It's not clear to me why.

@ericpre
Copy link
Member

ericpre commented Dec 3, 2024

Segmentation fault at loading modules is usually related to a binary incompatibility, but there is nothing that looks wrong with the libraries installed.

@hakonanes
Copy link
Contributor Author

So you want to wait merging this until all checks pass?

Copy link
Member

@ericpre ericpre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After clarification on the comment on skipping test, I am happy to merge as it is currently. Hopefully, pyxem supporting numpy 2 (not sure what is the situation right now) will make easier to sort out the binary incompatibility.

if: ${{ always() }}
# run the tests headlessly
run: |
xvfb-run python -m pytest --pyargs kikuchipy -k "not test_spherical_pyvista and not test_not_allow_download_raises"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it necessary to skip these tests? Please add a note to justify.
Usually we skip test when:

  • temporally: an issue has been identified and it is fixed upstream and skipping test manually (for example here) is expected to be removed at some point
  • workaround: there is a failure on CI which is not understood, for example it can be related to different blas implementation, etc. most likely relevant to a subtle bug or ill defined test.

I think that this is good practise to keep skipping test as minimal as possible. I hope that it makes sense!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope that it makes sense!

Makes perfect sense!

Unskipped the PyVista test since we now have a virtual frame buffer.

The remaining skip is reported in pyxem/kikuchipy#707 with a plan to fix. This can be unskipped when a fix is pushed.

@ericpre
Copy link
Member

ericpre commented Dec 12, 2024

The new failures seem unrelated.

@ericpre ericpre merged commit 54122af into hyperspy:main Dec 12, 2024
0 of 9 checks passed
@ericpre
Copy link
Member

ericpre commented Dec 12, 2024

Thank you @hakonanes!

@hakonanes
Copy link
Contributor Author

No, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants