Error finding CUDA devices!

[LewPhan]

My simulations have been running well for around a month (FLAMEGPU1) bringing in some great data! However, a few days ago I got the following message on my Ubuntu terminal:

"GPUassert: unspecified launch failure src/dynamic/simulation.cu 15094"

Following this message, I haven't been able to run any FLAME code and get the following message

"Error finding CUDA devices! Do you have a CUDA-capable GPU installed?"

There hadn't been any change to the system (inc. updates or file changes) and even re-running past simulations which had once worked using the GPU now give this message. Typing nvcc -v gives the correct CUDA version and using lspci -nnk to check the graphics card usage returns that the card is still being detected. Removing and re-inserting the graphics card didn't solve the issue the either.

Yesterday, the same issue effected my personal laptop and this was solved (not sure how) after I booted into windows and tested the GPU using a graphically demanding video game for 2 minuets. I am unsure how this solved the issue on my personal laptop and so don't know how to get FLAME working on my work PC again.

Thanks!

[ptheywood]

When FLAMEGPU 1 returns Error finding CUDA devices! Do you have a CUDA-capable GPU installed? it is only returned when cudaGetDeviceCount fails (source).
Due to the order of CUDA operations in FLAME GPU 1, this should only be the case when CUDA is non-functional on the machine.

Are you about to run nvidia-smi and see see regular output (a couple of tables printed to stdout, including a list of detected GPUs)

Recent Ubuntu's do included unnatended upgrades by default, which may have updated your GPU driver in the background, and could result in a mismatch but that shouldn't be the case if you've rebooted to re-seat the GPU to no avial.

What GPU(s) and Driver do you currently have installed?

We can't really provide much assistance re the original unspecified launch failure, as it is in a dynamically generated source file which is model specific. They can be caused by a number of reasons including out of bounds reads/writes in a kernel. Using cuda-memcheck and other cuda debugging tools may help, but requires a working CUDA system to run.

[LewPhan]

Thanks Peter! When I type nvidia-smi, I get the following output:

"Failed to initialize NVML: Driver/library version mismatch"

I'm using an RTX 3080, yet am running CUDA 10.1 to allow FLAMEGPU 1 to work as I don't have access to an older NVIDIA desktop card. I used the suggestions you gave via email a few years back during setup and have had no issues until last week. My laptop, which had this same issue for a few hours before it self-resolved, uses an RTX 2070 along with CUDA 10.1.

[Robadob]

You most likely have conflicting Nvidia driver installs (e.g. from an unattended GPU driver upgrade as Pete mentioned in OP), there's alot of suggestions for resolving this on StackOverflow (e.g. linked below).

I'd start with Robert Crovella's advice, he works for Nvidia.

https://stackoverflow.com/questions/43022843/nvidia-nvml-driver-library-version-mismatch#comment73133147_43022843

[ptheywood]

As Rob's suggested, anything Rob Crovella has suggested on SO is usually the right thign to do.

I think your 3080 will require driver 455.23 or newer, which shipped with CUDA 11.1, so if you've managed to end up with an older driver this might be the cause of the problem, though I'd lean towards a reboot potentially fixing it if it is due to an unatteneded upgrade.

Personally I've had the most luck using the ubuntu network installer, installing cuda-drivers from the most recent CUDA, and then cuda-toolkit-X-Y for each version I require, changing cuda-toolkit is then a case of the PATH env var etc as described in the cuda installation post-install instructions (though I use lmod module files to manage this, as common on HPC). CUDA 10 is not available for Ubuntu 22.04+ however, so the runfile method would probably be neccesary if that is the case.
Occasionally there is poor packaging too leading to conflicts when installing more recent CUDA versions, but installing these with apt install -f -o DPkg::options::="--force-overwrite" has yet to lead to any noticable issues...

If you only care for linux, there is also a branch in the FLAMEGPU 1 repository you could switch to which compiles with atleast CUDA 11.0 (cuda-11-0-RC). Unfortunately this was never merged in to the master branch as it broke Windows/MSVC builds, and development time was moved to FLAMEGPU 2. This would still require your driver mismatch to be resolved however.

---

To find the cuda driver you are currently running, if installed via apt you can find this using dpkg:

$ sudo dpkg -l | grep ii | grep cuda-drivers
ii  cuda-drivers                                   530.30.02-1                                 amd64        CUDA Driver meta-package, branch-agnostic
ii  cuda-drivers-530                               530.30.02-1                                 amd64        CUDA Driver meta-package, branch-specific

I.e. I have cuda drivers 530.30.02 installed currentlty.

This won't help if you didn't install via the cuda-drivers package or via the runfile though (or potentially if a ubuntu unnattended upgrade removed it / replaced it with a different driver, which would likely be named nvidia-driver-XYZ