[Question] Issue with Image Inference and Model Loading in viplanner_node

[8866lt]

Hi,
I am running the viplanner_node and encountered several issues related to image inference and model loading. Below are the key warnings and errors I observed:

1. Warning: Failed to load image Python extension

/home/slam/.local/lib/python3.8/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 
  warn(f"Failed to load image Python extension: {e}")

This warning indicates that the Python extension for image processing in TorchVision failed to load. While I am not actively doing image inference right now, I am concerned that this might impact performance or cause issues when I do need to process images.
2. Warning: Pre-trained ResNet50 models cannot be used

[Warning] Pre-trained ResNet50 models cannot be used since mask2former not found
This warning suggests that the pre-trained ResNet50 model cannot be used because the mask2former module is missing. It seems that mask2former is required for this task, but I’m unsure whether it should be installed or configured differently.
3. Model weight loading mismatch

Loads checkpoint by local backend from path: /home/slam/catkin_ws/src/viplanner/planner/models/mask2former_r50_lsj_8x2_50e_coco-panoptic_20220326_224516-11a44721.pth
The model and loaded state dict do not match exactly
unexpected key in source state_dict

This message indicates that the model architecture does not match the pre-trained weights I am trying to load. There is a mismatch in the state dictionary, specifically an "unexpected key" error. It seems that the model structure might have changed, or the pre-trained weights are not compatible with the current version of the code.
4. No image inference performed
Despite the model and parameters being loaded, I did not observe any image inference or related processing. It seems that the path planning and image processing steps are not being executed as expected.
Steps I Have Tried:

Reinstalling torchvision: I attempted to resolve the issue by reinstalling torchvision, but the warning persists.

Checking Dependencies: I looked into installing mask2former, but I’m not sure how this module fits into the overall project. I would appreciate clarification on whether this is a mandatory dependency.

Verifying Model Weights: I checked the model weights, but it seems there is a mismatch between the architecture and the pre-trained weights. I am unsure how to resolve this and whether I need a different version of the weights.

Request for Assistance:
I would greatly appreciate your guidance on the following:

How to resolve the "unexpected key" error when loading the model weights. Should I use a different pre-trained model that is compatible with the current architecture?

Installation of mask2former: Can you confirm if mask2former is a required module for the current project? If so, how should I install or configure it?

Image Inference Issues: Why is image inference not being executed? Is there something I need to configure to ensure that image data is processed correctly?

Thank you for your help, and I look forward to your response.
Best regards,

[pascal-roth]

Hi @8866lt,

Thanks for using our work.

Regarding the different points:

1. I am not sure about that issue, didn't experience it. Did the reinstall of torchvision fix it?

2. You can ignore that warning. It is still there for an extensions to work directly with RGB images

3. I assume you run it on the jetson in the docker container? Can you make sure that your versions match the ones mentioned in this issue Jetson deployed with semantic information not outputting correctly #23 (crucial so that your mask2former weights fit to mmdetection)

[pascal-roth]

Hi @8866lt,

did the answers help you to fix the issues?