nvmlDeviceGetMemoryInfo, Insufficient Permissions

[DrAuYueng]

The running container uses the MIG device. Use the API to obtain MemoryInfo in the container. Sample code:

>>>import pynvml
>>> pynvml.nvmlInit()
>>> handle = pynvml.nvmlDeviceGetHandleByIndex(0)
>>> pynvml.nvmlDeviceGetMemoryInfo(handle)

Error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/pynvml.py", line 2934, in nvmlDeviceGetMemoryInfo
    _nvmlCheckReturn(ret)
  File "/usr/local/lib/python3.10/dist-packages/pynvml.py", line 979, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.NVMLError_NoPermission: Insufficient Permissions

Execute nvidia-smi -L in the container, the result is:

How to deal with this situation?

[frosk1]

same issue

[riccardo32]

same issue

Further details: Servers are running Flatcar OS and using the latest release of container-toolkit v1.17.3 and Production branch driver 550.127.08.

Issue is easily reproducible on H100 and A100 MIG devices, but works fine if device is not partitioned.

Please advise if you need any additional information