quantization simulation for a LLM model example

[1826133674]

Hello， I'm trying to use AIMET_TORCH to quantize a LLM model, e.g.: llama v2。 where can I find a jupyter NB example which shows quantization simulation for a LLM model?

[DongGeun123]

@1826133674 Hi Did you achieve quantizing llama?

[1826133674]

@1826133674 Hi Did you achieve quantizing llama?

not yet

[xuli-vecml]

I tried to quantize llama1b, but the quantized model output random charactors...
I kind of wondering the different between aimet pro and aimet public version. I see that the llama3b is quantized pretty good with aimet pro by the qualcomm team. But there is no example released of how they do that.