Ideas for Frigate+ UI improvement

[blacknell]

I think there's a case for making the UI more speedy when tagging photos to send to Frigate+. The Send To Frigate button prompts This is a Person, This is not a Person and Cancel.

When you click Yes or No, the 3rd option changes to Close and I think it would be faster if it just closed the dialog.

I'm going through hopefully dozens of snapshots and it would make it a much more streamlined end user experience.

[blacknell]

Also, In Frigate+ when reviewing images, it would help if you could cycle though images with a left/right arrow.

Currently, when tagging/labelling it brings up the next unverified image when saving. Which is great.

But I've realised I need to run through previously verified images to add more missing labels and this is very tedious as you have to click one, assess it, then click Back and guess at the next image in the array.

With that in mind, would also be very helpful if you could filter the main array screen for labels eg. car, not car, not license_plate etc.

[blakeblackshear]

Thanks for the feedback. Others have suggested similar things too.

[blacknell]

I'm busy (sort of) training in prep for my model. I will subscribe shortly.

I know there are plans to allow choosing a point in time from the clip to upload but at the moment you are stuck with the snapshot. Obviously I want to upload a frame with license plate/face etc.

I am using VLC which has a relatively ok method for saving a frame. But it's tedious for 100's of photos

But does it matter in Frigate+ if the resolution is high? The Frigate uploaded ones are 1280x720 but the VLC ones are 3840 × 2160, Aspect ratio is the same so I'm guessing not.

[NickM-27]

in chrome you can right click and save as image on any video player

submitting higher resolution won't cause problems, the regions are resized for training in any case

[blacknell]

hmm. i did see this once but now I can't find it

[blacknell]

@blakeblackshear - please can you invite me to subscribe (sent an email but wasn't sure that was the route)

[blacknell]

I'm now at 87 images on my front door and 43 on the garden.

But I can't tell how many are tagged with various (and therefore see which is important) labels.

Most important would be a way of quickly cycling through the images in Frigate+ so reviewed for the correct labels and boxes.

Is there any chance this might be added soon?

[blakeblackshear]

Can you describe your workflow? It's currently designed assuming that you label all objects in the image at once and then verify the image. Then you are taken to the next unverified image until no unverified images are left.

[blacknell]

Sure. Yes, that's what I've been following.

But I realised that I had (and continue to) miss some objects out and so the labels were incomplete/wrong. Another is I decided I was being too demanding on face - I added the label to side profiles not face on profiles and I figured this was unrealistic. Likewise with cars that are partially obscured by a hedge.

So I'd like to walk through all my images quickly and edit those I got wrong - before kicking off a model.

[blakeblackshear]

That's helpful, thanks. If you would like, I can mark all your images as unverified so you can go back through them with the expected workflow. I don't expect to get to the other enhancements in the near term.

[blacknell]

Yes, please.

As long as it doesn't remove the actual boxes and labels!

[blacknell]

how important is camera height in the model definition

[NickM-27]

I don't think that is currently used for anything

[blacknell]

hi.
i've got my new model installed and so far so good. cats are being detected as false positives which is better than not at all.

however, now that i'm getting cars and license plates coming through, I've realised that the snapshots are from the detect stream. How do I get the snapshot generated from the record stream. The latter is 4k so by the time it segments/scales out the number plate the next step is in with a chance of being to process it.

i'm aware that the detect stream is much better for detecting (or rather it's a waste of CPU downscaling it for detection). Also aware that there's a time sync between frames on detect and record.

Wondering how to tackle this

[NickM-27]

There's no way currently to do this in real time. There is an api to request a frame from recordings after the recording segment has been saved.

I use alpr with detect stream 1280x720 and it works well

[blacknell]

My detect stream is 704 x 576 (set to 1280 x 720 in the config)
My record stream is 3840 x 2160

I just tried pointing detect and record at the 4k stream but the CPU went through the roof and the lag increased to 10s from about 1s

I can see the snapshot API you mention but I think the segment would have finish first before being saved?

Also, how to you get the timestamp from the detected snapshot - I can't see a snapshot info API

[NickM-27]

timestamp is part of the mqtt payload which also contains information about the location of the license plate / face in the image

[blacknell]

Where's the best place to be asking about Frigate+ tuning false negatives?

This one is a mixture of postive and negative. The bounding box is a false negative but there is a portion of car to the left with the lights. Should I mark is a negative and then draw the correct sub box for the bit that is a car?

[NickM-27]

Discussions is the right place, and yes that's what I'd recommend

[blacknell]

I've noticed in the last day or two that I'm getting a huge number of false positives for cars in my driveway at night. Nothing has changed that I can think of. There's so many it makes it difficult to spot anything for real.

Looking at the mqtt message for one example, it's showing 89% score.

Here's an example below.

I'm very confused about this as the new custom model has been superb in most other regards.

{
   "time":"2024-03-13T18:30:18.271049",
   "topic":"frigate/events",
   "message":{
      "before":{
         "id":"1710354600.703979-mcjj2x",
         "camera":"front_door",
         "frame_time":"2024-03-13T18:30:13.344059",
         "snapshot":{
            "frame_time":1710354601.372689,
            "box":[
               900,
               88,
               1177,
               330
            ],
            "area":67034,
            "region":[
               899,
               31,
               1219,
               351
            ],
            "score":0.921875,
            "attributes":[
               
            ]
         },
         "label":"car",
         "sub_label":null,
         "top_score":0.83203125,
         "false_positive":false,
         "start_time":1710354600.703979,
         "end_time":null,
         "score":0.51953125,
         "box":[
            339,
            34,
            443,
            68
         ],
         "area":3536,
         "ratio":3.0588235294117645,
         "region":[
            233,
            0,
            553,
            320
         ],
         "stationary":false,
         "motionless_count":0,
         "position_changes":1,
         "current_zones":[
            
         ],
         "entered_zones":[
            "driveway"
         ],
         "has_clip":true,
         "has_snapshot":true,
         "attributes":{
            
         },
         "current_attributes":[
            
         ]
      },
      "after":{
         "id":"1710354600.703979-mcjj2x",
         "camera":"front_door",
         "frame_time":1710354613.504625,
         "snapshot":{
            "frame_time":1710354601.372689,
            "box":[
               900,
               88,
               1177,
               330
            ],
            "area":67034,
            "region":[
               899,
               31,
               1219,
               351
            ],
            "score":0.921875,
            "attributes":[
               
            ]
         },
         "label":"car",
         "sub_label":null,
         "top_score":0.83203125,
         "false_positive":false,
         "start_time":1710354600.703979,
         "end_time":1710354617.90293,
         "score":0.89453125,
         "box":[
            324,
            17,
            466,
            76
         ],
         "area":8378,
         "ratio":2.406779661016949,
         "region":[
            204,
            0,
            524,
            320
         ],
         "stationary":false,
         "motionless_count":0,
         "position_changes":1,
         "current_zones":[
            
         ],
         "entered_zones":[
            "driveway"
         ],
         "has_clip":true,
         "has_snapshot":true,
         "attributes":{
            
         },
         "current_attributes":[
            
         ]
      },
      "type":"end"
   }
}

[NickM-27]

that's odd, I've not seen any issues like that. May need to reduce motion detection sensitivity

[blacknell]

seems it was the motion mask for the streamed timestamp. i had tweaked it to only cover the seconds. i've put it back now