Can all the Visual Tokenizer weights released work for the same Infinity-2B model?

[EternalEvan]

Thanks for your excellent work! I noticed that you have released many Visual Tokenizer weights with different codebook size. I wonder if all these tokenizer work well with the Infinity-2B weight? I have tried the recommended infinity_vae_d32reg.pth and it performers well. Thanks!

[JeyesHan]

@EternalEvan Thanks for your appreciation to Infinity. Infinity-2B weight is trained and therefore works with [infinity_vae_d32reg.pth]. Using other vae weights will generate bad images. If you want to try other vae weights, you can fine-tune Infinity-2B with them. The results will improve very quickly.

[EternalEvan]

@EternalEvan Thanks for your appreciation to Infinity. Infinity-2B weight is trained and therefore works with [infinity_vae_d32reg.pth]. Using other vae weights will generate bad images. If you want to try other vae weights, you can fine-tune Infinity-2B with them. The results will improve very quickly.

ok, I will try tuning Infinity-2B with them. Thanks!

[RealAntonVoronov]

Can you explain, what does _reg mean in VAE_d32? I see in your table with metrics (and confirm by comparing reconstructions from both VAEs) that d32 without _reg works better. What is the reason behind choosing d32_reg for final model?

[JeyesHan]

@RealAntonVoronov We experimentally found that as the vocabulary size increases, VAE relies more on the last few scales. In the model with '_reg', we added some regularizations (to be more specific, adding reconstruction loss to the earlier scales). The '_reg' model shows a slight decrease in reconstruction metrics compared to that one without regularization. However, it reduces the dependence on the last few scales, which is beneficial for generation.