API timeline anomalies

[blacknell]

I'm trying to process snapshots and finding I need to get various different frames to get good results with ANPR.

Basically, given an event in mqtt I want to get the event timeline and grab the frames from the recording (I know there is an offset)

I'm using http://frigatepi.local:5000/api/timeline?camera=front_door&source_id=1710943479.155059-urn6l5 where this is the id of the event.

It's giving me back

[
   {
      "camera":"front_door",
      "class_type":"visible",
      "data":{
         "attribute":"",
         "box":[
            0.8140625,
            0.3263888888888889,
            0.1546875,
            0.55
         ],
         "label":"car",
         "region":[
            0.225,
            0.0,
            0.775,
            1.3777777777777778
         ]
      },
      "source":"tracked_object",
      "source_id":"1706716117.73791-9d8vqm",
      "timestamp":1706716117.73791
   },

But the event happened yesterday 1710943479.155059 is 20 March 2024 14:04:39.155

But the timestamps in the results are back in January eg 1706716117.73791 is 31 January 2024 15:48:37.737

The timeline API asks for source_id not an event_id and I think this is the confusion.

[NickM-27]

source id is the same thing as event id, I would suggest not providing a camera since the camera is included as part of the event anyway. The UI uses this exact api

[blacknell]

I'm getting pretty poor results from codeproject-alpr.

Even when the plate is a decent size it often comes back with nothing especially if the plate is at an angle or slightly blurred.

As I think @NickM-27 indicated in his script, alpr doesn't seem to do a good job of finding a plate in a large image so I've played with cropping around the box with various borders. Better.

I've tried retrieving timeline snapshots from the recording and seeing if they work but I don't know where the plate is on these. It fails on every image.

I'm wondering if the alpr is using YOLO to find the plate which is not as good as with the Coral.

Am I right in thinking I could let anpr use the Coral too? Is this a simple as giving it access to the device /dev/bus/usb:/dev/bus/usb in docker compose?

[NickM-27]

it's not about code project ai accessing the coral, it is about it running the frigate+ model. Also, only one process can access a coral at a time so this would break detection in frigate if it was used with the same coral.

I have had pretty good results doing cropping, you can always setup codeproject.ai with a different yolo model and see if the license plate results are better.

[blacknell]

I'm trying by moving go2rtc to a separate faster machine and then connecting frigate to those rtsp streams. The latency is far far lower since my go2rtc machine is so quick. I don't have a decent 1280x720 stream from the camera so I'm using this to downscale it from 4k.

Box A. go2rtc

streams:
  front_door_720:
    - ffmpeg:rtsp://[email protected]/cam/realmonitor?channel=1&subtype=0#width=1280#video=h264
  front_door:
    - ffmpeg:rtsp://stream:[email protected]/cam/realmonitor?channel=1&subtype=0#video=h264

Then I'm using this on frigate

Box B. Frigate

ffmpeg:
  inputs:
    - path: rtsp://192.168.1.51:8554/front_door_720
      input_args: preset-rtsp-restream-low-latency
      roles:
        - detect
    - path: rtsp://192.168.1.51:8554/front_door
      input_args: preset-rtsp-restream-low-latency
      roles:
        - record
  hwaccel_args: preset-rpi-64-h264

Since I've specified preset-rtsp-restream-low-latency which is frigate ffmeg process still reporting such high CPU at 20-35%

Both the input streams front_door_720 & front_door have already been converted to h264 and I'd expect the ffmeg on the frigate instance to be doing almost nothing (as per the preset-rtsp-restream-low-latency directive?

[NickM-27]

preset-rtsp-restream-low-latency simply removes the buffer. It is unlikely to affect cpu usage at all.

I notice a few issues here:

1. on box a you are not using hardware so the transcoding is being done on cpu
2. you are not setting fps for the transcode so it is likely still high fps which causes high CPU usage on a low power device like rpi still since it has to decode every frame that is sent

[blacknell]

Ok. The go2rtc docs say Raspberry always uses software. But I added #hardware

• Box A is an RPi 5 8GB no overclocking but it does have NVME drive. So it's pretty quick.
• Box B is an RPi 4 8GB running as it always has.

I find the docs on go2rtc very confusing and written for people who understand ffmeg very well!

For example,

I could not figure out how to write this...

front_door_tr:
  - ffmpeg:rtsp://[email protected]/cam/realmonitor?channel=1&subtype=0#video=h264#raw=-vf crop=1280:720:2560:0

using a more readable structure to something like this which would build the same ffpmeg command into the restream

front_door_tr:
  - rtsp://[email protected]/cam/realmonitor?channel=1&subtype=0
  - ffmpeg: #video=h264
  - ffmpeg: #raw=-vf crop=1280:720:2560:0

[NickM-27]

oh yeah, the rpi 5 doesn't even have an h264 decoder so it definitely has to use software.

it would look like this:

front_door_tr:
  - rtsp://[email protected]/cam/realmonitor?channel=1&subtype=0
  - ffmpeg:front_door_tr#video=h264#raw=-r 5 -vf fps=5,scale=1280:720

[blacknell]

Can you breakdown the ffmpeg directives further eg.

front_door_tr:
  - rtsp://[email protected]/cam/realmonitor?channel=1&subtype=0
  - ffmpeg:front_door_tr#video=h264
  - ffmpeg:front_door_tr#raw=-r 5 -vf scale=1280:720
  - ffmpeg:front_door_tr#raw=-r 5 -vf fps=5

[NickM-27]

no you can't, every line you add is an individual stream

[blacknell]

ok. thanks. this is what I was struggling with last night

eg. this works (my 4k cropped to 1280x720 the top right sector)

front_door_tr:
  - ffmpeg:rtsp://[email protected]/cam/realmonitor?channel=1&subtype=0#video=h264#raw=-vf crop=1280:720:2560:0

but splitting does not work as

front_door_tr:
  - rtsp://[email protected]/cam/realmonitor?channel=1&subtype=0
  - ffmpeg:front_door_tr#video=h264#raw=-vf crop=1280:720:2560:0

[NickM-27]

might need quotes around it

[blacknell]

After posting to the API to register a sub-label, are any message sent by frigate to MQTT? I'm wondering the best way of alerting HA that a plate's been identified. It seems a bit unclean to manage my own new topic.

[NickM-27]

Yes, an update is sent the next time that object is processed internally.

[blacknell]

2nd question, after the events expire (eg i set recording etc to 7 days), am I right in thinking that the distinct list of sub_labels is not cleaned up? I have a lot of stragglers that clearly don't result in any filtered results.

[NickM-27]

The list comes directly from the events DB so if the UI shows a sub label then there is an event in the DB with that sub label

[blacknell]

But will it be processed internally if that was the ‘end’ event? RegardsPaulOn 7 Apr 2024, at 12:36, Nicolas Mowen ***@***.***> wrote: Yes, an update is sent the next time that object is processed internally. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[NickM-27]

No, if the object is no longer detected (end) then no further message will be sent