Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Warp CUDA error #408

Open
robotNo201 opened this issue Dec 27, 2024 · 4 comments
Open

[BUG] Warp CUDA error #408

robotNo201 opened this issue Dec 27, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@robotNo201
Copy link

Bug Description

isn't that funny?
Image
Image
Image

System Information

Image Image Image
@robotNo201 robotNo201 added the bug Something isn't working label Dec 27, 2024
@shi-eric
Copy link
Contributor

I'm not sure what's going on and I can't reproduce an issue on my system, but have you tried installing warp-lang from PyPI? From the image you posted, it seems you installed warp_lang-1.5.0+cu11-py3-none-manylinux2014_x86_64.whl, which works on my system with driver 560.35.03. But I guess we might get more information if you run into the same issue with the warp-lang on PyPI.

FYI, Warp actually doesn't use the nvcc in your system path unless you are building Warp from source.

There was a recent commit from @nvlukasz on the main branch that fixed an issue with driver entry points, but it was meant to fix something else: 2df89ed Might be worth building Warp from the main branch on your local system and seeing what happens.

@nvlukasz
Copy link
Contributor

nvlukasz commented Jan 2, 2025

Interesting. Looks like we fail to get the cuGetProcAddress() function from the driver. This function is needed to get all required driver entry points, including cuDriverGetVersion(). Since we fail to get that function, the detected driver version falls back to 0.0.

It's unclear why cuGetProcAddress() is not found, it should be present in the driver.

Based on nvidia-smi, the driver is 561.09, which I believe is a Windows driver. The kernel cache dir looks Linux-like (/home/.../.cache/warp/1.5.0). Is it running on WSL?

@nvlukasz
Copy link
Contributor

nvlukasz commented Jan 2, 2025

FWIW, I tried to reproduce this with WSL but didn't see a problem.

Image

The nvidia-smi and driver versions match yours:
Image

There might be an issue with your driver installation or WSL setup. I suggest to update your driver to latest (try custom installation option with clean install). And like Eric suggested, maybe try the latest PyPI build of Warp:

pip install --upgrade --force-reinstall warp-lang

If that still fails then the issue might be related to WSL somehow.

@nvlukasz
Copy link
Contributor

nvlukasz commented Jan 2, 2025

Just to get another data point, you could try running some Torch CUDA code in that environment.

test_torch.py:

import torch
a = torch.arange(10, device="cuda:0")
print(a)
$ pip install torch
$ python test_torch.py

See if that works correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants