Running yolov8 on Google Coral

[lkaino]

I modified the code support the other Google Coral ready made models + yolov8.

Unfortunately the compilation of yolov8 for edgetpu couldn't convert all the operations:

Number of operations that will run on Edge TPU: 233
Number of operations that will run on CPU: 24

yolov8s inference speed on my machine is showing 150 ms, so it is slower than the default edgetpu model.

Here's an image from the default model:

And here's one from the yolov8s (320x320):

Time will show if it helps with false positives.

I can make it into a PR if others are interested in testing it out. Maybe someone can better optimize the yolo model to run on edgetpu.

[lkaino]

While working on the code I noticed that the code might not be doing letterboxing to the image that is sent for inference. It might help to implement it to increase the accuracy.

[NickM-27]

it happens in https://github.com/blakeblackshear/frigate/blob/3c25c578f07a7e79d27d1a99864f4d59ac84ab77/frigate/util/image.py

[lkaino]

So it seems. The method names suggest that they are transforming the color space, not letterboxing, thus I didn't look that deep

[NickM-27]

sorry the link lost the line number, but the comments are describing exactly what is going on

frigate/frigate/util/image.py

Lines 246 to 270 in 3c25c57

	# get the crop box if the region extends beyond the frame
	crop_x1 = max(0, region[0])
	crop_y1 = max(0, region[1])
	# ensure these are a multiple of 4
	crop_x2 = min(width, region[2])
	crop_y2 = min(height, region[3])
	crop_box = (crop_x1, crop_y1, crop_x2, crop_y2)
	y, u1, u2, v1, v2 = get_yuv_crop(frame.shape, crop_box)
	# if the region starts outside the frame, indent the start point in the cropped frame
	y_channel_x_offset = abs(min(0, region[0]))
	y_channel_y_offset = abs(min(0, region[1]))
	uv_channel_x_offset = y_channel_x_offset // 2
	uv_channel_y_offset = y_channel_y_offset // 4
	# create the yuv region frame
	# make sure the size is a multiple of 4
	# TODO: this should be based on the size after resize now
	size = (region[3] - region[1]) // 4 * 4
	yuv_cropped_frame = np.zeros((size + size // 2, size), np.uint8)
	# fill in black
	yuv_cropped_frame[:] = 128
	yuv_cropped_frame[0:size, 0:size] = 16

[lkaino]

Thanks! Would be nice to have a debug mode that would show the cropped images which are sent for prediction from the frames.

[NickM-27]

This already exists with the process_clips.py tool

[Cheerpipe]

I can make it into a PR if others are interested in testing it out. Maybe someone can better optimize the yolo model to run on edgetpu.

Hi.

I am using yolov8 with openvino and would be very happy testing yolov8 on my Coral.

[lkaino]

I can make it into a PR if others are interested in testing it out. Maybe someone can better optimize the yolo model to run on edgetpu.

Hi.

I am using yolov8 with openvino and would be very happy testing yolov8 on my Coral.

Please find it here: https://github.com/lkaino/frigate/tree/edgetpu_yolo_support. It is based on v0.12.1 tag.

Here are the models:
yolov8_models.zip

Configure them into use like this:

model:
  path: "/media/frigate/models/yolov8s_full_integer_quant_edgetpu.tflite"
  width: 320
  height: 320

[lkaino]

Here are yolov5s and yolov5n models if you want to test those:
yolov5.zip

[Cheerpipe]

Hi.

This is just amazing, i am running yolov8n on latest dev with your changes. It is still slower than on intel igpu with openvino but most important is it lowered my cpu usage almost 5% and server power usage about 15W.

I will wait one day to check for false positives and detection rate.

Any chances you export an 288x288 yolov8n?

Thanks for your work.

[lkaino]

Here you go:
yolov8n_288_edgetpu.zip

I added the python script which you can use to generate your own in: https://github.com/lkaino/edgetpu_yolo. See the README for quickstart.

[xavan83]

Do you merge code to dev branch or only used the uploaded yolo models to your config?

[kirsch33]

@lkaino are the NMS/IoU steps necessary? Frigate has its own version of these processes after calling detect_raw I believe, so you are doubling up are you not?

[lkaino]

@lkaino are the NMS/IoU steps necessary? Frigate has its own version of these processes after calling detect_raw I believe, so you are doubling up are you not?

The shared memory for the detector is initialized to hold 20 detections. Without the NMS this limit might be filled. Of course it would be easy to allocate more slots in the SHM.

Nevertheless I implemented it without the NMS here: https://github.com/lkaino/frigate/tree/edgetpu_yolo_support (it is now based on master). There is a variable that can be used to toggle the NMS on/off: https://github.com/lkaino/frigate/blob/8a5f52d70bf758c4cd8b9512f53b99163f7c9e6f/frigate/detectors/plugins/edgetpu_tfl.py#L20C13-L20C13. There is also an info print here if your detection boxes are going over the 20 limit: https://github.com/lkaino/frigate/blob/8a5f52d70bf758c4cd8b9512f53b99163f7c9e6f/frigate/detectors/plugins/edgetpu_tfl.py#L389.

There is no big difference in execution time to using the NMS or not. I measured the NMS taking 2-3ms, where as without it it takes 1-2 ms.

[Cheerpipe]

[lkaino]

That's why the NMS is needed for now. Either the SHM size should be increased or some sort of filtering is needed. The problem with the SHM size is that its hardcoded with magic number in many places.

You can just enable the NMS back on.

[kirsch33]

@lkaino thanks for the reply. i would be interested to test these models on your changes:
https://cs.degirum.com/zoos

These guys have apparently been able to map yolov8 and v5 fully 100% to the edgetpu (no operations on CPU) but the output tensor is not the same as native yolov8 so it would require some code modifications to work with your changes. Here is a post on the ultralytics page that announced it:
ultralytics/ultralytics#1185 (comment)

[kirsch33]

ill be honest its a bit above my head, something about 6 outputs vs a single concatnated output. see slides 11 and 12 here that I ran across when reading last night:

https://www.slideshare.net/UmitYapanel/quantyolomodelspdf-259989222

Also, here is a screenshot of the yolov8s_relu_coco model from them uploaded to Netron, you can see the output tensor is different. I know the 8400 is for 84 coco classes with 100 items per class, but no idea what the 144 is?

[kirsch33]

here is some code on one of their repos that would likely be helpful, but like I said a little over my head:

https://github.com/DeGirum/yolo_inference/blob/master/model.py

[lkaino]

Seems like doable, with some digging in the github fork (https://github.com/DeGirum/ultralytics_degirum/commits/main). Their models' input sizes are huge, would be nice to have smaller ones.

I wonder if they modified the yolo export process with their new magic, so everyone can generate them.

[kirsch33]

i mangled together some code from the link I posted above and from your fork and managed to get their model working, but the inference time is no better. still a little over 100ms on my end.

id be curious if you have any better luck, since I may not have things correct

[lkaino]

I got it working, but as you said the inference time is the same as with my compiled models:

yolov8n_relu6_coco--576x576_quant_tflite_edgetpu_1.tflite
133 ms

yolov8n_full_integer_quant_edgetpu.tflite (320x320)
127 ms

Their model input size is much bigger though. It would be nice to get a model with the same 320x320 input size for comparison. I tried generating new models from https://github.com/DeGirum/ultralytics_degirum, but their scripts are not working. There is a clear typo in https://github.com/DeGirum/ultralytics_degirum/blob/131c0b71c03bf3455d43c6ede5af2813c7dfa64f/dg_quantize.py#L11, tflite.Interpreter should be tf.lite.Interpreter. Even after fixing that it does not work.

Could you @kirsch33 try to ask for a model with smaller input size (320)?

[SuperMaximus1984]

What is the maximum capacity for Coral Edge TPU, YoloV8n?
What about YoloV8X? For me the biggest issue is that there's no YoloV8X for Frigate using TensorRT, I would opt for it definitely.

[kirsch33]

As you can see above you’re looking at inference times of >100ms even for yolo small models so it’s not really practical at all, much less for anything larger. Mobiledet has an inference time of usually less than 10ms so there’s a big disparity

I’ve read somewhere that Yolo, unlike MobileDet, does actually yield a smaller model size when classes are reduced, so if someone could re train a Yolo model with only like person, dog, car for example that might help the situation, but I’m not sure what I’m saying is accurate either

[SuperMaximus1984]

Yes, I'm looking only at person and car/truck/bicycle classes. I focus on accuracy (the location is not a crowded place with multiple objects at once). Will try to test the author's YoloV8S model as standard one (using Coral) or YoloV7-tiny-416 (TensorRT w/Nvidia GPU) leaves much to be desired in terms of false positives.

[d123456temp]

@lkaino first of all, thank you for your contribution. In my situation "default" Frigate models are not working reliably so it would be great to have YOLO as an alternative. What surprises me is that (judging by lack of tracking) this effort is apparently not interesting for @NickM-27 or @blakeblackshear (thank you for your great work guys). It would be great if you guys could provide any feedback on this. Any hopes that we get YOLO support in Frigate? Even with problems described below YOLO managed to detect some objects/situation that were non-detectable by the default model, so it's a real improvement.

That saying, @lkaino have you progressed with this any further? Having YOLO would be great. I'm using your code (with yolov8s on an edge TPU - I simply patch-mounted your edgetpu_tfl.py to v12.0.1 image), but I'm new to the whole Frigate/image-recognition/models so some things are not clear for me:

1. Without NMS I have detection events, but "Too many detections" is triggered very often/always? ( https://github.com/lkaino/frigate/blob/8a5f52d70bf758c4cd8b9512f53b99163f7c9e6f/frigate/detectors/plugins/edgetpu_tfl.py#L389 )
   what's the real impact it has on the whole process? I understand that some detections are dropped on the floor. If I understand this correctly (pardon my ignorrance) one of the reasons for this is that YOLO model has too many object classes defined? If so, having smaller version for person only OR some way to limit model output by providing filter parameters to model itself would help?

2. With NMS Frigate got killed every time during detection as a result of an exception within the NMS procedure being thrown from line https://github.com/lkaino/frigate/blob/8a5f52d70bf758c4cd8b9512f53b99163f7c9e6f/frigate/detectors/plugins/edgetpu_tfl.py#L344 . I added catch-all to visualize this situation (and fallback to NMS), here is output from NMS (offending line got offsetted from 344 to 348 as a result of my catch-all)

2024-01-06 12:17:44.352886141 [2024-01-06 12:17:44] frigate.detectors.plugins.edgetpu_tfl INFO : Got exception from NMS (will fall-back to non-NMS): Traceback (most recent call last):
2024-01-06 12:17:44.352901141 File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 348, in yolo_postprocess
2024-01-06 12:17:44.352904234 detections = non_max_suppression(y, conf_thres=self.min_score)
2024-01-06 12:17:44.352907641 File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 175, in non_max_suppression
2024-01-06 12:17:44.352909808 mask = mask.reshape(box.shape[0], 0)
2024-01-06 12:17:44.352912271 ValueError: cannot reshape array of size 252 into shape (83,0)

NOTE: the last line size and shape messages happens for different values of these parameters:

2024-01-06 12:17:44.352912271 ValueError: cannot reshape array of size 252 into shape (83,0)
2024-01-06 12:37:40.240686190 ValueError: cannot reshape array of size 168 into shape (82,0)
2024-01-06 12:37:43.365554349 ValueError: cannot reshape array of size 504 into shape (86,0)
2024-01-06 12:39:48.052868053 ValueError: cannot reshape array of size 1512 into shape (98,0)
2024-01-06 12:39:48.136465116 ValueError: cannot reshape array of size 336 into shape (84,0)
2024-01-06 12:42:58.801906093 ValueError: cannot reshape array of size 168 into shape (82,0)

NOTE2: NMS doesn't fail always (output from my catch-call):

2024-01-06 12:53:26.068832070 [2024-01-06 12:53:26] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:26.354174207 [2024-01-06 12:53:26] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:26.559972452 [2024-01-06 12:53:26] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:26.711246216 [2024-01-06 12:53:26] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:26.914114779 [2024-01-06 12:53:26] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:27.189416833 [2024-01-06 12:53:27] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:27.383319244 [2024-01-06 12:53:27] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:27.469990111 [2024-01-06 12:53:27] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:27.719938870 [2024-01-06 12:53:27] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:27.937595542 [2024-01-06 12:53:27] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:28.112724991 [2024-01-06 12:53:28] frigate.detectors.plugins.edgetpu_tfl INFO : NMS worked ok
2024-01-06 12:53:28.363416981 [2024-01-06 12:53:28] frigate.detectors.plugins.edgetpu_tfl INFO : Got exception from NMS (will fall-back to non-NMS): Traceback (most recent call last)

catch-all block:

detections = None
        if USE_YOLO_NMS:
            try:
                detections = non_max_suppression(y, conf_thres=self.min_score)
                logger.info(f"NMS worked ok")
            except Exception as e:
                logger.info(f"Got exception from NMS (will fall-back to non-NMS): {traceback.format_exc()}")

        if detections is None:
            detections = get_boxes_from_yolo_output(y, conf_thres=self.min_score)

[blakeblackshear]

YOLO is supported for other detector types. I haven't seen anything that indicates it performs substantially better, so it just hasn't been a priority. I wouldn't object to a PR adding support.

[lkaino]

I haven't continued with this. I'm running frigate on a laptop which has a nvidia card now. I sold my edgetpu 😔

[andrey-skat]

I haven't seen anything that indicates it performs substantially better

Maybe it doesn't perform better, but it more easier to train own yolo model than mobilenet (bundled model mostly gives false positives or false negatives 😔). Also I am (and others) have problems with openvino GPU support (also this #7870 adds some troubles)

[philippedegeer]

I am a very happy Frigate user. My setup is made of two Axis surveillance cameras pointing to an area with several trees and a lot of moving shadows. I am essentially detecting persons.
Adding my 2 cents to the discussion, for this use case, I have found a significant reduction in false positive using yolov8m (openvino) versus anything I had tested on Coral AI before (MobileNet, MobileDet or EfficientDet). I have also noted an improvement with night vision detection. All mentioned models were COCO-trained.
The only thing is that, to improve inference speed and add cameras, I would need another Intel CPU/iGPU and change hardware. So any additional support for yolov8 (edgetpu or Nvidia) would be great. I am not a developper though and would not know where to start. May be other users can report their experience with Yolov8.

[d123456temp]

Since I see some tracking here :), as a follow-up to my messages above, I just want to let you know that I found a quick and dirty way (no experience with CV nor Python so far) that works for me with a great success in recognition quality. What I discovered is that @lkaino 's code doesn't support object type filters from configuration hence have the tendency to hit the recognition count limit. Because I need to discover only persons/people I simply:

• disabled USE_YOLO_NMS - which brought me more troubles than positive outcome
• added filtering for COCO object class and get rid of detections limit

Here's a patch against Ikaino's code to showcase this dirty attempt (still containing some debug logs)

diff patches/edgetpu_tfl.py patches-ikaino-original/edgetpu_tfl.py
5d4
< import traceback
344d342
<         detections = None
346,352c344,345
<             try:
<                 detections = non_max_suppression(y, conf_thres=self.min_score)
<                 logger.info(f"NMS worked ok")
<             except Exception as e:
<                 logger.info(f"Got exception from NMS (will fall-back to non-NMS): {traceback.format_exc()}")
< 
<         if detections is None:
---
>             detections = non_max_suppression(y, conf_thres=self.min_score)
>         else:
391,394d383
<         skipped_detections = list()
<         accepted_detections_count = 0
<         filtered_out_detections = list()
<         accepted_detections = list()
399,400c388,389
<             if class_ids[i] > 0.0:
<                 filtered_out_detections.append(f"{class_ids[i]} : {float(scores[i])}")
---
>             if i == self.max_detections:
>                 logger.info(f"Too many detections ({count})!")
402,422c391,398
<             if accepted_detections_count < self.max_detections:
<                 detections[i] = [
<                     class_ids[i],
<                     float(scores[i]),
<                     boxes[i][0],
<                     boxes[i][1],
<                     boxes[i][2],
<                     boxes[i][3],
<                 ]
<                 accepted_detections_count += 1
<                 accepted_detections.append(f"{class_ids[i]} : {float(scores[i])}")
< 
<             else:
<                 skipped_detections.append(f"{class_ids[i]} -> {float(scores[i])}")
< 
<         if len(accepted_detections) > 0: #or len(filtered_out_detections) > 0:
<             report = f"\ndetections:\nfiltered out: {filtered_out_detections}\ndetections accepted: {accepted_detections}\n"
<             if len(skipped_detections) > 0:
<                 report +=f"\ndetections skipped (count > 20): {skipped_detections}"
< 
<             logger.info(report)
---
>             detections[i] = [
>                 class_ids[i],
>                 float(scores[i]),
>                 boxes[i][0],
>                 boxes[i][1],
>                 boxes[i][2],
>                 boxes[i][3],
>             ]

[d123456temp]

To summarize: adding filter-by-type to Ikaino's code and removing some IMHO unnecessary code related to NMS, this can go straight to production (I've been running this dirty version with yolov8s_full_integer_quant_edgetpu.tflite for 2 weeks constantly now without any problems and recognition rate is really great).

[xavan83]

What is your inference time with that code/model?

[d123456temp]

will provide graph when return home, but c.a. 100-150ms on a Rpi4 with TPU.

[canadarob]

I'm running yolov8s with 384×384 resolution. (As above that less tasks are converted to the tpu). With a pentium 8505 mini pc im running 38ms inference. Running two corals now as well

[SuperMaximus1984]

Can you let us test yolov8s_full_integer_quant_edgetpu.tflite ?

[canadarob]

you can use ikaino's patch file. then also d123456temp patch ontop of that. Ive been using it a week in my frigate 13rc1 beta instance.

[SuperMaximus1984]

Could you please provide the exact links for patches? I wasn't able to find them here. Do you run YoloV8 on Coral either?

[canadarob]

Check out ikainos github page for his fork of frigate 12.1 patched for edgetpu(top of this thread). Within that fork the only change was to one file. You can use that edgetpu_tlf.py then apply d123456temp patch he supplied a few comments ago upon that file. Then swap that file into your frigate installation.

[d123456temp]

I'm AFK now. Since this has tracking, in 1-2 days I will provide output from my tests - graphs showcasing that in my case yolov8s turned out to be way better than other two models I tested (EfficientDet-Lite3 and SSD MobileNet V2 from https://coral.ai/models/object-detection/) - person detections rate faaaar better with inference times way better than EfficientDet-Lite3 and worse than SSD MobileNet V2.

[philippedegeer]

same experience here....

[philippedegeer]

as an additional info, I am getting around 55-60ms inference time on a Celeron 5105 with Openvino, using the iGPU

[d123456temp]

As promised, here's my 2 weeks worth of statistics gathering for people-only detection on a Rpi4 2GB with a Coral USB TPU. YoloV8s clearly stands out (in my empirical/subjective observations as well) - my tests were performed in a pretty demanding environment (long distance to observed sector for anonymization purposes - people were of 1/10th height of the video frame with detection camera input stream 1920x1080) but with pretty decent quality camera (Dahua IPC-HDW5442T-ZE-2712). Night time shots have some street lamps interfering but camera had WDR enabled to battle this. Cannot share photos/videos due to anonymization concerns.
Regarding charts timeline - my previous tests shows that rythm of people flow for this sector is pretty stable week-by-week with only minor fluctuations, but of course you should take this comparison with a grain of salt - this is not an ideal a/a a/b test of course.

edit: above, the rightmost/detailed timeline red annotation lines are offset by -1 day by mistake

[d123456temp]

If someone with more Frigrate coding experience could extend my patch with support for normal object filtering instead of my crude person filter (if class_ids[i] > 0.0:) we could have a non-person-detection-only support - but this detection filter configuration has to be kept limited to not cause too much detections. There's also this original @lkaino 's idea about cutting down the yolov model to some objects subset.

[NickM-27]

the detections should already be returned sorted from highest to lowest score, so I don't really see a problem personally with just letting it hit the limit. I see very few cases where there are actually more than 20 true positives in a single region

[harakas]

I've been working on AMD/ROCm GPU with yolov8 but I also have a coral. So I added support for it into my frigate branch: harakas@80ae30e Slightly based on what @lkaino did but mostly just condensed/minimal. It uses some utilities I wrote for yolov8 postprocessing.

[RobertLukan]

It works. Fantastic, I used Human face and detection works very nicely. The inference time is about 36ms, still fine with me. I have only two cameras at 5fps. No need to retrain the model. Thank you for your help.

[d123456temp]

I've been working on AMD/ROCm GPU with yolov8 but I also have a coral. So I added support for it into my frigate branch: harakas@80ae30e Slightly based on what @lkaino did but mostly just condensed/minimal. It uses some utilities I wrote for yolov8 postprocessing.

@harakas @blakeblackshear @NickM-27 what's the status of this coral + yolov8 integration provided by @harakas ? We were cooperating on the subject here, then harakas code was merged but has it been removed afterwards? I can see other yolov8 integrations in docs but not this one (I saw it added to docs by a PR merge so has it been removed recently?). I understand there was an issue with yoloX licensing, but it looks like it is not a problem in other cases. Can we keep/restore this useful integration (coral + yolov8) in Frigate? It works like a charm and I'm limited to RPI+coral platform...

[NickM-27]

All yolov8 was removed due to licensing. I'm working on adding support with yolo NAS

[d123456temp]

Thank you very much for your answer, this is great news. Will this work on raspberry/arm64 + coral?

[NickM-27]

Sorry I misunderstood what thread this was, I was referring to AMD ROCm

[xavan83]

What the difference between yolov8n-oiv7_320x320_edgetpu.tflite and yolov8n_320x320_edgetpu.tflite

[harakas]

Reading code there is no difference I think. Each detector entry (whatever type) is run in its own process. But they all get the same FrigateApp.detection_queue : multiprocessing.Queue -- they pull a single image in a loop from that and process it. And each camera process also has a handle to the same queue and sends frames to be processed into that queue.

[SuperMaximus1984]

Reading code there is no difference I think. Each detector entry (whatever type) is run in its own process. But they all get the same FrigateApp.detection_queue : multiprocessing.Queue -- they pull a single image in a loop from that and process it. And each camera process also has a handle to the same queue and sends frames to be processed into that queue.

Is it possible to use Coral detector for some cameras and TensorRT for others?
I've tested your models, thank you for the job done. But in my case Yolo7-640.trt did a better job than yolov8s-320. If only Yolov8l could be ran with acceptable inference time on Coral, that would be awesome.

[canadarob]

You can run two frigate instances for that.

[harakas]

Maybe the aggregation was step too far and it's not working like I think it is ;) Might have to revert that stuff.

[harakas]

Yolov8 support for Google Coral has now been merged into the main frigate development branch (#9782).

The development branch is currently very unstable though so not for the faint-hearted to try.

[SuperMaximus1984]

@harakas,
You published YoloV8 .tflite models with 320x320 resolution in your repo. Can you make it 640x640 in YoloV8S and YoloV8N versions?
I am experimenting with 4K cameras with far objects and concluded that 640x640 works way better (comparisons made based on YoloV7 models with TensorRT). Whenever I run something bigger than YoloV8S_320x320 the inference time becomes intolerably slow.
Will 2 x Coral M.2 increase the inference time compared to 1 unit?

[SuperMaximus1984]

@NickM-27 When I become desperate for false positives I start thinking about retraining a custom model too. Not sure Frigate+ could be a solution as I'm on TensorRT.
Nick, would you please point me to a working TRT model training & conversion guide for Frigate's digestive format?
As far as I understood from one of your replies on Reddit, Frigate 0.13.1 is on TensorRT 8.5.3, thus needs Nvidia Container 23.03?

[NickM-27]

I didn't realize you needed trt since this discussion is referring to Google Coral

[SuperMaximus1984]

I didn't realize you needed trt since this discussion is referring to Google Coral

Yeah, since Coral is pretty limited I have no other choice but opting for TRT.
I've tested @harakas YoloV8S_640 model, indeed it makes Coral crash and inference time is 613 ms....

[NickM-27]

I guess my point is I've found my frigate+ model works great and I haven't seen anything that indicates the coral is holding it back. It also has an inference time of 5 ms with the plus model.

[SuperMaximus1984]

As long as Frigate+ is not available to me, I have nothing to object to you :)

[xavan83]

You can try this: yolov8s_640x640_edgetpu.zip. It takes 0.51s to evaluate on my coral -- unbearably slow and so impractical. Also my coral crashed after a bit and is now offline. Need to physically go and replug it to make it work again so can't test more.

You can generate your own using https://github.com/harakas/models/tree/main/ultralytics/yolov8.1 -- have to modify the build_edgetpu.py to add change 320->640 and then make dist. Will take quite a bit of time for full build. You can comment out stuff in Dockerfile to reduce the amount it prepares. Or you can just follow the ultralytics docs to generate them (https://docs.ultralytics.com/modes/export/).

I try your script (I only want to export n model for Coral on 288*288).
I have a dual Coral setup
I comment onnx and openvino builds.
I receive:

=> ERROR [process 6/7] RUN python3 build_edgetpu.py 12.1s

[process 6/7] RUN python3 build_edgetpu.py:
9.681 Exporting yolov8n-oiv7.pt into yolov8/yolov8n-oiv7_288x288_edgetpu.tflite
9.681 Ultralytics YOLOv8.1.12 🚀 Python-3.10.12 torch-2.2.0+cpu CPU (Intel Celeron J4105 1.50GHz)
10.36 [W NNPACK.cpp:64] Could not initialize NNPACK! Reason: Unsupported hardware.
10.43 YOLOv8n summary (fused): 168 layers, 3492527 parameters, 0 gradients, 10.5 GFLOPs
10.89
10.89 PyTorch: starting from 'yolov8n-oiv7.pt' with input shape (1, 3, 288, 288) BCHW and output shape(s) (1, 605, 1701) (6.9 MB)
12.01 Illegal instruction (core dumped)

[harakas]

That's unfortunate. Illegal instruction points that something is not working properly on your hardware. Googling further it's likely torch. Try an older version: change it into torch==1.13.0 in Dockerfile.

If that doesn't help try a different system.

edit: Oldest torch that might work is 1.9.0. Could try that as well.

[xavan83]

Same error with 1.13.0 and 1.9.0 is not found:
2.083 ERROR: Could not find a version that satisfies the requirement torch==1.9.0 (from versions: 1.11.0+cpu, 1.12.0+cpu, 1.12.1+cpu, 1.13.0+cpu, 1.13.1+cpu, 2.0.0+cpu, 2.0.1+cpu, 2.1.0+cpu, 2.1.1+cpu, 2.1.2+cpu, 2.2.0+cpu)

Can you export the n model in a 288*288 size using coco and oiv7 dataset?

[harakas]

Here: yolov8_288x288_models.zip

[xavan83]

Thank you.
I try the latest dev build to be sure having the latest version of the coral detector.
Inference time with 288*288 oiv7 model is more than 100ms, the Coco one is 23ms.

[mariocsp]

can anyone provide docs on how to run the model please?

[blakeblackshear]

@lkaino see here: #10717

You should also remove your public fork as you are likely in violation as well.

[SOUMYAMAFIA]

I want to add my custom trained yolov8 model and I converted successfully it in .tflite format but now I can't configure it in the frigate config.yml file it shows error when give model_path: ....... How can I solve it ?anyone?