Training results of instruct-pix2pix-lora is bad

[SilverRAN]

Hello! I'm trying to reproduce the results in examples/research_projects/instructpix2pix_lora , and I used totally same hyperparameters with the Readme demo, but the results is not satisfying. After 5000 steps trainig the result is as below (longer training even makes result worse). Could you please check the code in instructpix2pix_lora? Thanks!

[asomoza]

this is SD 1.5 so I can't really tell if the results looks good or bad, it seems similar to the examples in the blogpost, did you compare it with the full finetune one?

Also this is a research project, is not even an example, so the best way is to just experiment with it (when it is a lora it's a lot more fragile than the finetune).

Also ccing @Aiden-Frost as the author of the script.

[Aiden-Frost]

Hello
The results shouldn't look this bad.
I have recently changed the script to support LoRa using the PEFT library.
I could have introduced some bug, I will look into this and keep you posted.

[SilverRAN]

Hello The results shouldn't look this bad. I have recently changed the script to support LoRa using the PEFT library. I could have introduced some bug, I will look into this and keep you posted.

Hello! After visualizing some results I found that when the number of training steps is very little (like 100 steps) the result is good, but the quality is decreasing with training steps increasing. It seems like over-fitting, but it's weird because in my opinion over-fitting shouldn't appear within one training epoch.