mmod_human_face_detector auto fallback from CUDA to CPU

[ha7sh17]

dlib ver : 19.24
server : NVIDIA L4

---

We are successfully using the mmod_human_face_detector in two modes by building dlib with DLIB_USE_CUDA set to ON and OFF for the CUDA and CPU versions, respectively.

In the CUDA version, when GPU resources are stable, the detector always returns consistent results. However, when GPU resources are constrained (e.g., SM usage approaches 100), the returned rect coordinates and confidence values occasionally differ from the usual results. Interestingly, these "different" rect coordinates match exactly with those returned by the CPU version of the mmod_human_face_detector. (In such cases, the confidence values in the CUDA version are slightly different from those in the CPU version.)

Given this behavior, does dlib—even when built with DLIB_USE_CUDA=ON—have an automatic fallback mechanism to use the CPU for calculations during runtime when GPU resources are insufficient?

[arrufat]

Currently, if you compile dlib with DLIB_USE_CUDA=ON, all the dnn parts will use the GPU, there's no way to change the backend at runtime.

[arrufat]

Honestly, I don't know, I've never experienced such a thing... I don't know enough about how the NVIDIA runtime works.

[ha7sh17]

Ok.
Thank you for your kind support.
I will check in my side, more

[arrufat]

Let us know if you find what is it.

[davisking]

It's because changing the order in which you add floating point numbers slightly changes the resulting value. And in cuda there are a ton of threads doing just that. So when something changes the way those threads get scheduled you will sometimes get slightly different results.

[ha7sh17]

@davisking
Thank you for your advice.
We will check CUDA floating point calculation mechanism deeply to avoid getting different results.
If we success, I will share our solution here.

[davisking]

It's not something you can change. Just how the cuda code being used here works.