python312Packages.llama-cpp-python: init at 0.3.1

[kirillrdy]

unlike previous attempt #268712

this uses bundled version of llama-cpp, also tested CUDA support

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[hoh]

Thanks for the work !

Can you explain the motivations behind switching to the bundled version of llama-cpp ?

The package looks much simpler than the previous version that used patches on llama-cpp, but lacks support for other back-ends (OpenCL, ROCm).

[alexvorobiev]

Trying to build with Cuda results in the version error:

llama-cpp-python>   /nix/store/slx40i35cmd7kb3wvdqzckfww8smcy6s-cuda_nvcc-12.2.140/include/crt/host_config.h:143:2:
llama-cpp-python>   error: #error -- unsupported GNU version! gcc versions later than 12 are
llama-cpp-python>   not supported! The nvcc flag '-allow-unsupported-compiler' can be used to
llama-cpp-python>   override this version check; however, using an unsupported host compiler
llama-cpp-python>   may cause compilation failure or incorrect run time execution.  Use at your
llama-cpp-python>   own risk.

The usual workaround

....overridePythonAttrs(attrs: { stdenv = super.gcc12Stdenv; });

works, so it should be possible to add the override to the package.

[alexvorobiev]

Thanks for the work !

Can you explain the motivations behind switching to the bundled version of llama-cpp ?

The package looks much simpler than the previous version that used patches on llama-cpp, but lacks support for other back-ends (OpenCL, ROCm).

I am not the author but the code refers to the llama.cpp's subdirectories which are not included in nixpkg.llama-cpp. For instance https://github.com/abetlen/llama-cpp-python/blob/7403e002b8e033c0a34e93fba2b311e2118487fe/CMakeLists.txt#L110.

[kirillrdy]

@hoh

Can you explain the motivations behind switching to the bundled version of llama-cpp ?

llama-cpp and llama-cpp-python often get out of sync, so using llama-cpp from nixpkgs requires either breaking lama-cpp-python or not updating llama-cpp

@alexvorobiev CUDA support builds without any overrides, do you change default cuda in your overlays ?

[alexvorobiev]

@alexvorobiev CUDA support builds without any overrides, do you change default cuda in your overlays ?

I have to use CUDA 12.2 for now, could that be the issue?

[kirillrdy]

@alexvorobiev CUDA support builds without any overrides, do you change default cuda in your overlays ?

I have to use CUDA 12.2 for now, could that be the issue?

yes, it seems to only build with 12.4 ( which is default in nixpkgs )

[kirillrdy]

@alexvorobiev I've fixed cuda build, tested with 12_3, 12_2, 12_1, 12_0, 11_8 ( stopped here )

[alexvorobiev]

@alexvorobiev I've fixed cuda build, tested with 12_3, 12_2, 12_1, 12_0, 11_8 ( stopped here )

Thank you!

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-development-environment-for-python-ai-llama-3-1/56143/3

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-development-environment-for-python-ai-llama-3-1/56143/4

[MatthewCroughan]

Unsure if there are any outstanding issues, but best to merge instead of leaving the PR for a long time, so have merged and will try to use it and identify any outstanding issues.