Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python311Packages.jaxlib: fix bazel hash #321559

Merged
merged 1 commit into from
Jun 23, 2024
Merged

Conversation

GaetanLepage
Copy link
Contributor

@GaetanLepage GaetanLepage commented Jun 21, 2024

Description of changes

Should fix #296737...

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@GaetanLepage
Copy link
Contributor Author

Result of nixpkgs-review pr 321559 run on aarch64-darwin 1

@GaetanLepage
Copy link
Contributor Author

Result of nixpkgs-review pr 321559 run on x86_64-darwin 1

@ofborg ofborg bot requested a review from ndl June 21, 2024 17:04
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 11-100 labels Jun 21, 2024
@GaetanLepage
Copy link
Contributor Author

Result of nixpkgs-review pr 321559 run on aarch64-linux 1

-> The pymc failure is addressed in #321568.

@GaetanLepage
Copy link
Contributor Author

Result of nixpkgs-review pr 321559 run on aarch64-linux 1

36 packages marked as broken and skipped:
  • open-webui
  • open-webui.dist
  • python311Packages.aeppl
  • python311Packages.aeppl.dist
  • python311Packages.aesara
  • python311Packages.aesara.dist
  • python311Packages.bambi
  • python311Packages.bambi.dist
  • python311Packages.dm-sonnet
  • python311Packages.dm-sonnet.dist
  • python311Packages.tensorflow-datasets
  • python311Packages.tensorflow-datasets.dist
  • python311Packages.torchrl
  • python311Packages.torchrl.dist
  • python311Packages.treeo
  • python311Packages.treeo.dist
  • python312Packages.argos-translate-files
  • python312Packages.argos-translate-files.dist
  • python312Packages.argostranslate
  • python312Packages.argostranslate.dist
  • python312Packages.ctranslate2
  • python312Packages.ctranslate2.dist
  • python312Packages.faster-whisper
  • python312Packages.faster-whisper.dist
  • python312Packages.pettingzoo
  • python312Packages.pettingzoo.dist
  • python312Packages.tensorflow-bin
  • python312Packages.tensorflow-bin.dist
  • python312Packages.torchrl
  • python312Packages.torchrl.dist
  • python312Packages.translatehtml
  • python312Packages.translatehtml.dist
  • python312Packages.treeo
  • python312Packages.treeo.dist
  • tts
  • tts.dist
118 packages built:
  • libretranslate (python311Packages.libretranslate)
  • libretranslate.dist (python311Packages.libretranslate.dist)
  • minari (python311Packages.minari)
  • minari.dist (python311Packages.minari.dist)
  • python311Packages.ale-py
  • python311Packages.ale-py.dist
  • python311Packages.argos-translate-files
  • python311Packages.argos-translate-files.dist
  • python311Packages.argostranslate
  • python311Packages.argostranslate.dist
  • python311Packages.arviz
  • python311Packages.arviz.dist
  • python311Packages.augmax
  • python311Packages.augmax.dist
  • python311Packages.awkward
  • python311Packages.awkward.dist
  • python311Packages.blackjax
  • python311Packages.blackjax.dist
  • python311Packages.chex
  • python311Packages.chex.dist
  • python311Packages.coffea
  • python311Packages.coffea.dist
  • python311Packages.correctionlib
  • python311Packages.correctionlib.dist
  • python311Packages.ctranslate2
  • python311Packages.ctranslate2.dist
  • python311Packages.dask-awkward
  • python311Packages.dask-awkward.dist
  • python311Packages.equinox
  • python311Packages.equinox.dist
  • python311Packages.faster-whisper
  • python311Packages.faster-whisper.dist
  • python311Packages.gymnasium
  • python311Packages.gymnasium.dist
  • python311Packages.islpy
  • python311Packages.islpy.dist
  • python311Packages.jax
  • python311Packages.jax.dist
  • python311Packages.jaxlib (python311Packages.jaxlib-build ,python311Packages.jaxlibWithoutCuda)
  • python311Packages.jaxlib.dist (python311Packages.jaxlib-build.dist ,python311Packages.jaxlibWithoutCuda.dist)
  • python311Packages.jaxopt
  • python311Packages.jaxopt.dist
  • python311Packages.jmp
  • python311Packages.jmp.dist
  • python311Packages.loopy
  • python311Packages.loopy.dist
  • python311Packages.mplhep
  • python311Packages.mplhep.dist
  • python311Packages.nanobind
  • python311Packages.nanobind.dist
  • python311Packages.numpyro
  • python311Packages.numpyro.dist
  • python311Packages.objax
  • python311Packages.objax.dist
  • python311Packages.optax
  • python311Packages.optax.dist
  • python311Packages.optax.testsout
  • python311Packages.pettingzoo
  • python311Packages.pettingzoo.dist
  • python311Packages.pymc
  • python311Packages.pymc.dist
  • python311Packages.pytensor
  • python311Packages.pytensor.dist
  • python311Packages.shimmy
  • python311Packages.shimmy.dist
  • python311Packages.skrl
  • python311Packages.skrl.dist
  • python311Packages.tensorflow-bin
  • python311Packages.tensorflow-bin.dist
  • python311Packages.translatehtml
  • python311Packages.translatehtml.dist
  • python311Packages.uproot
  • python311Packages.uproot.dist
  • python311Packages.vector
  • python311Packages.vector.dist
  • python312Packages.ale-py
  • python312Packages.ale-py.dist
  • python312Packages.awkward
  • python312Packages.awkward.dist
  • python312Packages.blackjax
  • python312Packages.blackjax.dist
  • python312Packages.chex
  • python312Packages.chex.dist
  • python312Packages.coffea
  • python312Packages.coffea.dist
  • python312Packages.correctionlib
  • python312Packages.correctionlib.dist
  • python312Packages.dask-awkward
  • python312Packages.dask-awkward.dist
  • python312Packages.equinox
  • python312Packages.equinox.dist
  • python312Packages.gymnasium
  • python312Packages.gymnasium.dist
  • python312Packages.jax
  • python312Packages.jax.dist
  • python312Packages.jaxlib (python312Packages.jaxlib-build ,python312Packages.jaxlibWithoutCuda)
  • python312Packages.jaxlib.dist (python312Packages.jaxlib-build.dist ,python312Packages.jaxlibWithoutCuda.dist)
  • python312Packages.jaxopt
  • python312Packages.jaxopt.dist
  • python312Packages.jmp
  • python312Packages.jmp.dist
  • python312Packages.minari
  • python312Packages.minari.dist
  • python312Packages.mplhep
  • python312Packages.mplhep.dist
  • python312Packages.numpyro
  • python312Packages.numpyro.dist
  • python312Packages.optax
  • python312Packages.optax.dist
  • python312Packages.optax.testsout
  • python312Packages.uproot
  • python312Packages.uproot.dist
  • python312Packages.vector
  • python312Packages.vector.dist
  • whisper-ctranslate2
  • whisper-ctranslate2.dist
  • wyoming-faster-whisper
  • wyoming-faster-whisper.dist

Copy link
Contributor

@oddlama oddlama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@GaetanLepage
Copy link
Contributor Author

Result of nixpkgs-review pr 321559 run on x86_64-linux 1

36 packages marked as broken and skipped:
  • python311Packages.bambi
  • python311Packages.bambi.dist
  • python311Packages.distrax
  • python311Packages.distrax.dist
  • python311Packages.dm-sonnet
  • python311Packages.dm-sonnet.dist
  • python311Packages.elegy
  • python311Packages.elegy.dist
  • python311Packages.rlax
  • python311Packages.rlax.dist
  • python311Packages.tensorflow-datasets
  • python311Packages.tensorflow-datasets.dist
  • python311Packages.treeo
  • python311Packages.treeo.dist
  • python311Packages.treex
  • python311Packages.treex.dist
  • python312Packages.argos-translate-files
  • python312Packages.argos-translate-files.dist
  • python312Packages.argostranslate
  • python312Packages.argostranslate.dist
  • python312Packages.ctranslate2
  • python312Packages.ctranslate2.dist
  • python312Packages.faster-whisper
  • python312Packages.faster-whisper.dist
  • python312Packages.pettingzoo
  • python312Packages.pettingzoo.dist
  • python312Packages.tensorflow-bin
  • python312Packages.tensorflow-bin.dist
  • python312Packages.torchrl
  • python312Packages.torchrl.dist
  • python312Packages.translatehtml
  • python312Packages.translatehtml.dist
  • python312Packages.treeo
  • python312Packages.treeo.dist
  • tts
  • tts.dist
142 packages built:
  • libretranslate (python311Packages.libretranslate)
  • libretranslate.dist (python311Packages.libretranslate.dist)
  • minari (python311Packages.minari)
  • minari.dist (python311Packages.minari.dist)
  • open-webui
  • open-webui.dist
  • python311Packages.aeppl
  • python311Packages.aeppl.dist
  • python311Packages.aesara
  • python311Packages.aesara.dist
  • python311Packages.ale-py
  • python311Packages.ale-py.dist
  • python311Packages.argos-translate-files
  • python311Packages.argos-translate-files.dist
  • python311Packages.argostranslate
  • python311Packages.argostranslate.dist
  • python311Packages.arviz
  • python311Packages.arviz.dist
  • python311Packages.augmax
  • python311Packages.augmax.dist
  • python311Packages.awkward
  • python311Packages.awkward.dist
  • python311Packages.blackjax
  • python311Packages.blackjax.dist
  • python311Packages.chex
  • python311Packages.chex.dist
  • python311Packages.coffea
  • python311Packages.coffea.dist
  • python311Packages.correctionlib
  • python311Packages.correctionlib.dist
  • python311Packages.ctranslate2
  • python311Packages.ctranslate2.dist
  • python311Packages.dalle-mini
  • python311Packages.dalle-mini.dist
  • python311Packages.dask-awkward
  • python311Packages.dask-awkward.dist
  • python311Packages.dm-haiku
  • python311Packages.dm-haiku.dist
  • python311Packages.equinox
  • python311Packages.equinox.dist
  • python311Packages.faster-whisper
  • python311Packages.faster-whisper.dist
  • python311Packages.flax
  • python311Packages.flax.dist
  • python311Packages.gymnasium
  • python311Packages.gymnasium.dist
  • python311Packages.islpy
  • python311Packages.islpy.dist
  • python311Packages.jax
  • python311Packages.jax.dist
  • python311Packages.jaxlib (python311Packages.jaxlib-build ,python311Packages.jaxlibWithoutCuda)
  • python311Packages.jaxlib.dist (python311Packages.jaxlib-build.dist ,python311Packages.jaxlibWithoutCuda.dist)
  • python311Packages.jaxlibWithCuda
  • python311Packages.jaxlibWithCuda.dist
  • python311Packages.jaxopt
  • python311Packages.jaxopt.dist
  • python311Packages.jmp
  • python311Packages.jmp.dist
  • python311Packages.loopy
  • python311Packages.loopy.dist
  • python311Packages.mplhep
  • python311Packages.mplhep.dist
  • python311Packages.nanobind
  • python311Packages.nanobind.dist
  • python311Packages.numpyro
  • python311Packages.numpyro.dist
  • python311Packages.objax
  • python311Packages.objax.dist
  • python311Packages.optax
  • python311Packages.optax.dist
  • python311Packages.optax.testsout
  • python311Packages.orbax-checkpoint
  • python311Packages.orbax-checkpoint.dist
  • python311Packages.pettingzoo
  • python311Packages.pettingzoo.dist
  • python311Packages.pymc
  • python311Packages.pymc.dist
  • python311Packages.pytensor
  • python311Packages.pytensor.dist
  • python311Packages.shimmy
  • python311Packages.shimmy.dist
  • python311Packages.skrl
  • python311Packages.skrl.dist
  • python311Packages.tensorflow-bin
  • python311Packages.tensorflow-bin.dist
  • python311Packages.torchrl
  • python311Packages.torchrl.dist
  • python311Packages.translatehtml
  • python311Packages.translatehtml.dist
  • python311Packages.uproot
  • python311Packages.uproot.dist
  • python311Packages.vector
  • python311Packages.vector.dist
  • python311Packages.vqgan-jax
  • python311Packages.vqgan-jax.dist
  • python312Packages.ale-py
  • python312Packages.ale-py.dist
  • python312Packages.awkward
  • python312Packages.awkward.dist
  • python312Packages.blackjax
  • python312Packages.blackjax.dist
  • python312Packages.chex
  • python312Packages.chex.dist
  • python312Packages.coffea
  • python312Packages.coffea.dist
  • python312Packages.correctionlib
  • python312Packages.correctionlib.dist
  • python312Packages.dask-awkward
  • python312Packages.dask-awkward.dist
  • python312Packages.equinox
  • python312Packages.equinox.dist
  • python312Packages.gymnasium
  • python312Packages.gymnasium.dist
  • python312Packages.jax
  • python312Packages.jax.dist
  • python312Packages.jaxlib (python312Packages.jaxlib-build ,python312Packages.jaxlibWithoutCuda)
  • python312Packages.jaxlib.dist (python312Packages.jaxlib-build.dist ,python312Packages.jaxlibWithoutCuda.dist)
  • python312Packages.jaxlibWithCuda
  • python312Packages.jaxlibWithCuda.dist
  • python312Packages.jaxopt
  • python312Packages.jaxopt.dist
  • python312Packages.jmp
  • python312Packages.jmp.dist
  • python312Packages.minari
  • python312Packages.minari.dist
  • python312Packages.mplhep
  • python312Packages.mplhep.dist
  • python312Packages.numpyro
  • python312Packages.numpyro.dist
  • python312Packages.optax
  • python312Packages.optax.dist
  • python312Packages.optax.testsout
  • python312Packages.orbax-checkpoint
  • python312Packages.orbax-checkpoint.dist
  • python312Packages.uproot
  • python312Packages.uproot.dist
  • python312Packages.vector
  • python312Packages.vector.dist
  • whisper-ctranslate2
  • whisper-ctranslate2.dist
  • wyoming-faster-whisper
  • wyoming-faster-whisper.dist

@samuela
Copy link
Member

samuela commented Jun 21, 2024

I'm still confused why this was breaking in the first place... Do we have a sense of what's going on with these hashes? It seems that every once in a while we get a report that the hashes are wrong even though they build fine for others.

@GaetanLepage
Copy link
Contributor Author

I'm still confused why this was breaking in the first place... Do we have a sense of what's going on with these hashes? It seems that every once in a while we get a report that the hashes are wrong even though they build fine for others.

No, unfortunately it is pretty unclear.
Maybe there is some non-deterministic aspect in the bazel-build-dependencies building process...

@wegank wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Jun 22, 2024
@SomeoneSerge
Copy link
Contributor

I'm also inclined to assume that hashes in fetchAttrs just aren't reproducible. Not sure how to test this though

@GaetanLepage
Copy link
Contributor Author

Should we merge this fix at least ?

@oddlama
Copy link
Contributor

oddlama commented Jun 23, 2024

I think so, but maybe we can also figure out what exactly causes this somehow

Copy link
Member

@natsukium natsukium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a good idea to create an issue.

@natsukium natsukium merged commit 706c64a into NixOS:master Jun 23, 2024
35 checks passed
@GaetanLepage GaetanLepage deleted the jaxlib branch June 23, 2024 07:23
@drupol
Copy link
Contributor

drupol commented Jun 25, 2024

Looks like it's not over...

faker> tests/providers/test_address.py::TestKaGe::test_city_name PASSED         [ 20%]
bazel-build-jaxlib-0.4.28-deps.tar.gz> installPhase completed in 8 minutes 9 seconds
error: hash mismatch in fixed-output derivation '/nix/store/l1maq7336n7zzpsv8y17kbnczxmdkwhx-bazel-build-jaxlib-0.4.28-deps.tar.gz.drv':
         specified: sha256-R1TIIyyyLlDqAlUkuhJhtyTxZMra2q5S/jX0OCInsEQ=
            got:    sha256-R5Bm+0GYN1zJ1aEUBW76907MxYKAIawHHJoIb1RdsKE=
error: 1 dependencies of derivation '/nix/store/vink8rps5x1bjqgk1x0vwxj170y5m3xa-bazel-build-jaxlib-0.4.28.drv' failed to build

@SomeoneSerge
Copy link
Contributor

@GaetanLepage please say you still have the previous FOD output cached and we can compare it with @drupol's 🙃

@drupol
Copy link
Contributor

drupol commented Jun 25, 2024

Note, I added this custom overlay:

(self: super: {
  openblas = super.openblas.overrideAttrs (oldAttrs: {
    makeFlags = oldAttrs.makeFlags ++ [
      "DEBUG=1"
    ];
  });
})

To try to debug a segfault with Openblas, do you think it might be the cause?

Note2: I have no clue how to verify if that custom overlay could have an impact.

@SomeoneSerge
Copy link
Contributor

@drupol ehhh if the FOD contained references to any other store paths I think we'd have noticed much earlier... but just technically, if you wanted to test for...

Note2: I have no clue how to verify if that custom overlay could have an impact.

...you could just add some thrash into your openblas; just adding another argument beside DEBUG=1 would change the path, and thus the reference; you could even actually modify the binaries in postFixup in case they're being copied; but again, I think this is a purely hypothetical talk

@GaetanLepage
Copy link
Contributor Author

@GaetanLepage please say you still have the previous FOD output cached and we can compare it with @drupol's 🙃

I should still have a working one cached.

@GaetanLepage
Copy link
Contributor Author

gaetan in 🌐 chamaeleon in nixpkgs on  master 
❯ nb python311Packages.jaxlib
Finished at 22:27:38 after 2s
/nix/store/9phzdivi68lrb70ih0c3dfccy68hq67a-python3.11-jaxlib-0.4.28

@SomeoneSerge
Copy link
Contributor

SomeoneSerge commented Jun 25, 2024

@GaetanLepage can you build /nix/store/l1maq7336n7zzpsv8y17kbnczxmdkwhx-bazel-build-jaxlib-0.4.28-deps.tar.gz.drv

EDIT: if the .drv doesn't exist you can try nix build nixpkgs#python3Packages.jaxlib.src
EDIT2: omg it's o stupid we only expose bazel-build as a string in src and not, at the very least, as a derivation/an attrset in passthru

@drupol
Copy link
Contributor

drupol commented Jun 25, 2024

EDIT2: omg it's o stupid we only expose bazel-build as a string in src and not, at the very least, as a derivation/an attrset in passthru

Is it the reason why I struggle finding a way to update and fix the hash mismatch in a custom overlay ?

@SomeoneSerge
Copy link
Contributor

SomeoneSerge commented Jun 25, 2024

 Is it the reason why I struggle finding a way to update and fix the hash mismatch in a custom overlay ?

No, this is just what makes it less convenient to debug. I'm pretty sure the reason starts with the letter "B".

Gaetan just found a cached ...-deps.tar.gz from the previous iteration, he can now compare it with the new one

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/building-openblas-with-debug-1-only-for-open-webui/47667/5

@drupol

This comment has been minimized.

@oddlama
Copy link
Contributor

oddlama commented Jun 28, 2024

Reverting this PR won't fix it, since the hash you get now is a new one again. Something non-reproducible in the bazel build seems to change every few days

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: python 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 11-100 12.approvals: 1 This PR was reviewed and approved by one reputable person
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Build failure: python3Packages.jaxlibWithCuda
8 participants