[Detector Support]: Crash loading YOLONAS model using ROCm detector on 0.15.0 Beta 2

[dodgepong]

Describe the problem you are having

I'm attempting to set up the ROCm detector on the 0.15.0 beta 2 build on my machine with a Ryzen 8845HS. I'm using the -rocm Docker image, I've built the YOLONAS ONNX model according to the instructions, and I've set up my config according to this documentation, but when Frigate starts, I get a crash that seems to be related to loading the model file. Would appreciate any insight as to what I'm doing wrong, or if this is a bug. Thanks!

Version

0.15.0-e76f4e9

Frigate config file

mqtt:
  enabled: false

go2rtc:
  streams:
    front_door: 
      nest: <snip>

record:
  enabled: true
  retain:
    days: 3
    mode: all

snapshots:
  enabled: true

detectors:
  rocm:
    type: rocm

model:
  model_type: yolonas
  width: 320 # <--- should match whatever was set in notebook
  height: 320 # <--- should match whatever was set in notebook
  input_pixel_format: bgr
  path: /config/yolo_nas_s.onnx
  labelmap_path: /labelmap/coco-80.txt

objects:
  track:
    - person
    - cat

cameras:
  front_door:
    enabled: true
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://127.0.0.1:8554/front_door
          input_args: preset-rtsp-restream
          hwaccel_args: preset-vaapi
          roles:
            - detect
    detect:
      enabled: true
      width: 1600
      height: 1200
    zones:
      Front_Area:
        coordinates: 0.77,0.498,0.998,0.585,0.996,0.998,0.267,0.996,0.271,0.456
        loitering_time: 0
version: 0.15-0

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.15.0-beta2-rocm
    shm_size: "128mb"
    devices:
      - /dev/dri:/dev/dri # For ROCm
      - /dev/kfd:/dev/kfd # For ROCm
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/stacks/frigate:/config
      - /mnt/nvr:/media/frigate  # NAS SMB mount
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: ${FRIGATE_RTSP_PASSWORD}

Relevant Frigate log output

2024-12-09 10:25:10.091100626  [2024-12-09 10:25:10] frigate.detectors.plugins.rocm WARNING : AMD/ROCm: detected gfx1103, overriding HSA_OVERRIDE_GFX_VERSION=11.0.0
2024-12-09 10:25:10.121308091  [2024-12-09 10:25:10] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: loaded migraphx module
2024-12-09 10:25:10.121329350  [2024-12-09 10:25:10] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: switching HIP to blocking mode to conserve CPU
2024-12-09 10:25:10.181398189  [2024-12-09 10:25:10] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: loading model from /config/yolo_nas_s.onnx
2024-12-09 10:25:10.194835838  [2024-12-09 10:25:10] frigate.detectors.plugins.rocm INFO    : AMD/ROCm: compiling the model
2024-12-09 10:25:14.537489553  [2024-12-09 10:25:10] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2024-12-09 10:25:14.537514118  [2024-12-09 10:25:10] frigate.api.fastapi_app        INFO    : FastAPI started
2024-12-09 10:25:14.539998220  Process detector:rocm:
2024-12-09 10:25:14.540001406  Traceback (most recent call last):
2024-12-09 10:25:14.540003079    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-12-09 10:25:14.540007758      self.run()
2024-12-09 10:25:14.540009632    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2024-12-09 10:25:14.540010794      return run(*args, **kwargs)
2024-12-09 10:25:14.540035860    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-12-09 10:25:14.540036962      self._target(*self._args, **self._kwargs)
2024-12-09 10:25:14.540038705    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2024-12-09 10:25:14.540039767      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-12-09 10:25:14.540040859    File "/opt/frigate/frigate/object_detection.py", line 68, in __init__
2024-12-09 10:25:14.540041881      self.detect_api = create_detector(detector_config)
2024-12-09 10:25:14.540071787    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-12-09 10:25:14.540072809      return api(detector_config)
2024-12-09 10:25:14.540095180    File "/opt/frigate/frigate/detectors/plugins/rocm.py", line 113, in __init__
2024-12-09 10:25:14.540096703      self.model.compile(
2024-12-09 10:25:14.540099508  RuntimeError: /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/AMDMIGraphX/src/targets/gpu/include/migraphx/gpu/context.hpp:82: create_stream: Failed to allocate stream

Install method

Docker Compose

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

NOTE: These screenshots are using OpenVINO for detection, which works fine. Frigate doesn't start when using the above ROCm settings, so this was the only way to actually get a screenshot.

Any other information that may be helpful

No response

[NickM-27]

looks like there may not be enough GPU RAM available

however, given the performance and speed of using openvino, I see no reason to use rocm

[dodgepong]

Ah, I see. I think this is me being a newbie. I assumed that I needed to use ROCm in order to make use of the 8845HS's XDNA2 NPU, but I think that's just wrong, and that OpenVINO could be making use of it after all? I assumed OpenVINO was using pure CPU, but I suppose a <3ms inference speed ought to be telling here.

[NickM-27]

yeah, there is no way that inference time is just on the CPU