TFLite - Cannot set tensor: Got value of type UINT8 but expected type FLOAT32

[pstephenwille]

Bit of a newbie here but I think I'm in the right place, and that TFLite is the sensor. Please let me know if I'm wrong, and what the sensor actually is. Many thanks in advance.

I have a working Yolov8 model that detects playing cards in groups or individually. It was trained on images and I would like it to work with video clips. I have exported it to a tflite format and it works for images in the original Python project but fails with video clips in Frigate with the error message below. I understand Frigate can use models trained on images but I am wondering if that's true for converted models. I'm now throughtly stumped by the error. I have generated and tried tflite models labeled as float32 as well. Regardless the tflite model I use, the error is the same.

error - Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: inputs_0

Version

SYSTEM 0.13.2-6476F8A

Frigate config file

mqtt:
  enabled: false

model:
  path: /config/models/best-255_saved_model/best-255_int8.tflite
  width: 640
  height: 640
  input_tensor: "nhwc"
  input_pixel_format: "rgb"

cameras:
  cam-1:
    enabled: true
    ffmpeg:
#      global_args: -hide_banner
#      haccel_args: preset-nvidia-h264
#      protection_mode: off
      inputs:
      - path: rtsp://192.168.1.22:554/stream=0
        roles:
        - detect

    zones:
      player_1:
        coordinates: 175,1139,210,0,2001,0,2001,1140
record:
  enabled: true
  retain:
    days: 3
    mode: motion
  events:
    required_zones:
      - player_1
    retain:
      default: 30
      mode: motion

snapshots:
  enabled: true
  timestamp: true
  required_zones:
    - player_1

docker-compose file or Docker CLI command

services:
  mosquitto:
    image: eclipse-mosquitto
    network_mode: host
    volumes:
      - ./conf:/mosquitto/conf
      - ./data:/mosquitto/data
      - ./log:/mosquitto/log

  frigate:
    container_name: frigate
    depends_on:
      - mosquitto
    runtime: nvidia
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    cap_add:
      - CAP_PERFMON
    shm_size: "128mb"
    volumes:
      - ./config:/config
      - ./storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    devices:
      - /dev:/dev
    deploy: # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [ gpu ]
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
#      - "554:554" # RTSP feeds

Relevant log output

frigate      | 2024-07-09 00:13:32.319313697  Process detector:cpu:
frigate      | 2024-07-09 00:13:32.328587484  Traceback (most recent call last):
frigate      | 2024-07-09 00:13:32.328591855    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate      | 2024-07-09 00:13:32.328592757      self.run()
frigate      | 2024-07-09 00:13:32.328593729    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate      | 2024-07-09 00:13:32.328594622      self._target(*self._args, **self._kwargs)
frigate      | 2024-07-09 00:13:32.328596971    File "/opt/frigate/frigate/object_detection.py", line 125, in run_detector
frigate      | 2024-07-09 00:13:32.328600542      detections = object_detector.detect_raw(input_frame)
frigate      | 2024-07-09 00:13:32.328616138    File "/opt/frigate/frigate/object_detection.py", line 75, in detect_raw
frigate      | 2024-07-09 00:13:32.328616863      return self.detect_api.detect_raw(tensor_input=tensor_input)
frigate      | 2024-07-09 00:13:32.328630009    File "/opt/frigate/frigate/detectors/plugins/cpu_tfl.py", line 41, in detect_raw
frigate      | 2024-07-09 00:13:32.328630655      self.interpreter.set_tensor(self.tensor_input_details[0]["index"], tensor_input)
frigate      | 2024-07-09 00:13:32.328631399    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 572, in set_tensor
frigate      | 2024-07-09 00:13:32.328631995      self._interpreter.SetTensor(tensor_index, value)
frigate      | 2024-07-09 00:13:32.328632734  ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: inputs_0

Operating system

Debian

Install method

Docker Compose

Object Detector

Other

Any other information that may be helpful

repo - https://github.com/pstephenwille/frigate-rtsp

[NickM-27]

values are sent as int, but the model was trained on float input. You would either need to make a change to the code or wait for frigate to support different input configurations

[pstephenwille]

Thanks, that's good to know. That gives me something to work with.

[pstephenwille]

found this answer that describes the steps needed to get Int8 data working. It's too much work to be practical so I'll have to wait on Frigate to support Float input.

ultralytics/ultralytics#5250 (comment)

[lraulet]

Hi,
I do not understant why using the "yolo export format=tflite int8=True" does not allow to get a Yolo v11 model that we can use with Frigate.
Any idea with that ?