Bug (extract-lora): CUSOLVER_STATUS_INVALID_VALUE

[ngxson]

Hi, I got an error with this command:

mergekit-extract-lora huihui-ai/Qwen2.5-72B-Instruct-abliterated Qwen/Qwen2.5-72B-Instruct outputs/tmpa7xncge5 --rank=32 --device=cuda

My hardware: running on HF space with 1xL40S

Log:

(Model already download and all shards loaded successfully)

  0%|          | 0/723 [00:00<?, ?it/s]
  0%|          | 0/723 [00:11<?, ?it/s]
Traceback (most recent call last):
  File "/home/user/.local/bin/mergekit-extract-lora", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/user/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/app/mergekit/mergekit/scripts/extract_lora.py", line 582, in main
    lora_weights, ranks = extract_lora(
  File "/home/user/app/mergekit/mergekit/scripts/extract_lora.py", line 291, in extract_lora
    lora_embedding_A, lora_embedding_B = decompose_delta_weight(
  File "/home/user/app/mergekit/mergekit/scripts/extract_lora.py", line 78, in decompose_delta_weight
    L, R = low_rank_decomposition(delta_weight, max_rank)
  File "/home/user/app/mergekit/mergekit/scripts/extract_lora.py", line 39, in low_rank_decomposition
    U, S, Vh = torch.linalg.svd(weight.float(), full_matrices=False)
torch._C._LinAlgError: cusolver error: CUSOLVER_STATUS_INVALID_VALUE, when calling `cusolverDnSgesvdj_bufferSize(handle, jobz, econ, m, n, A, lda, S, U, ldu, V, ldv, lwork, params)`. This error may appear if the input matrix contains NaN. If you keep seeing this error, you may use `torch.backends.cuda.preferred_linalg_library()` to try linear algebra operators with other supported backends. See https://pytorch.org/docs/stable/backends.html#torch.backends.cuda.preferred_linalg_library