[Draft]Add Multimodal RAG notebook

[openvino-dev-samples]

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[eaidova]

@openvino-dev-samples for me everything looks good, thanks.

Couple of comments:
Possibly it is better to move on accuracy aware quantization for VLM using optimum-cli, need to provide --weight-format int4 --dataset contextual options for that (fyi @nikita-savelyevv)

Is there any plans to integrate OV Visual Language models directly in llama-index?

[openvino-dev-samples]

@openvino-dev-samples for me everything looks good, thanks.

Couple of comments: Possibly it is better to move on accuracy aware quantization for VLM using optimum-cli, need to provide --weight-format int4 --dataset contextual options for that (fyi @nikita-savelyevv)

Is there any plans to integrate OV Visual Language models directly in llama-index?

Thanks for your review, the integration is already done in llama-index

https://docs.llamaindex.ai/en/stable/examples/multi_modal/openvino_multimodal/

BTW is there an example for phi3-vision's accuracy aware quantization ?

[nikita-savelyevv]

Couple of comments: Possibly it is better to move on accuracy aware quantization for VLM using optimum-cli, need to provide --weight-format int4 --dataset contextual options for that (fyi @nikita-savelyevv)

I'll add that an algorithm itself needs to be specified, e.g. --weight-format int4 --dataset contextual --awq.

Also, the default number of samples of 128 might be too large, so it can be reduced with --num-samples 32.

[openvino-dev-samples]

Couple of comments: Possibly it is better to move on accuracy aware quantization for VLM using optimum-cli, need to provide --weight-format int4 --dataset contextual options for that (fyi @nikita-savelyevv)

I'll add that an algorithm itself needs to be specified, e.g. --weight-format int4 --dataset contextual --awq.

Also, the default number of samples of 128 might be too large, so it can be reduced with --num-samples 32.

Hi, as my test, the accuracy with this configuration is not satisfied:
optimum-cli export openvino --model {vlm_model_id} {vlm_model_path} --trust-remote-code --weight-format int4 --dataset contextual --awq --num-samples 32

[nikita-savelyevv]

Couple of comments: Possibly it is better to move on accuracy aware quantization for VLM using optimum-cli, need to provide --weight-format int4 --dataset contextual options for that (fyi @nikita-savelyevv)

I'll add that an algorithm itself needs to be specified, e.g. --weight-format int4 --dataset contextual --awq.
Also, the default number of samples of 128 might be too large, so it can be reduced with --num-samples 32.

Hi, as my test, the accuracy with this configuration is not satisfied: optimum-cli export openvino --model {vlm_model_id} {vlm_model_path} --trust-remote-code --weight-format int4 --dataset contextual --awq --num-samples 32

Thanks for the information! Have you compared it against the configuration below?

compression_config = {
    "mode": nncf.CompressWeightsMode.INT4_SYM,
    "group_size": 64,
    "ratio": 0.6,
}

Yes, this configuration brings more reasonable responses compared to optimum-cli

[openvino-dev-samples]

--trust-remote-code --weight-format int4 --dataset contextual --awq --num-samples 32

Sorry, I made a mistake before. The result looks good with this accuracy aware config now

However I find it impossible to run this quantization method in a client PC with 32GB RAM

[nikita-savelyevv]

--trust-remote-code --weight-format int4 --dataset contextual --awq --num-samples 32

Sorry, I made a mistake before. The result looks good with this accuracy aware config now

However I find it impossible to run this quantization method in a client PC with 32GB RAM

Could you please verify that NNCF 2.14 is installed? It was released recently and it contains significant improvements in terms of peak RAM during data-aware compression.

[brmarkus]

Could you please verify that NNCF 2.14 is installed? It was released recently and it contains significant improvements in terms of peak RAM during data-aware compression.

At least this pull-request doesn't touch "requirements.txt" and doesn't check to make sure a recent version of NNCF is present. On my 64GB laptop the max. system memory is shortly fully used.