Cannot load LCM-LoRA for SDv1.5 (LoRACompatibleConv related ValueError)

[yukyeongleee]

Hi,

ValueError: Module down_blocks.0.attentions.0.proj_in is not a LoRACompatibleConv or LoRACompatibleLinear module.

This kind of error appears when super().load_lora_weights(...) is called.
(Location: File Live2Diff/live2diff/animatediff/pipeline/loader.py line 21, in load_lora_weights)

  File "/c1/username/Live2Diff/live2diff/utils/wrapper.py", line 451, in _load_model                            [32/1823]
    stream.load_lora(few_step_lora)
  File "/c1/username/Live2Diff/live2diff/pipeline_stream_animation_depth.py", line 140, in load_lora
    self.pipe.load_lora_weights(
  File "/c1/username/Live2Diff/live2diff/animatediff/pipeline/loader.py", line 21, in load_lora_weights
    super().load_lora_weights(pretrained_model_name_or_path_or_dict, adapter_name=adapter_name, strict=False, **kwargs) #
 ignore the incompatible layers
  File "/c1/username/anaconda3/envs/live2diff/lib/python3.10/site-packages/diffusers/loaders/lora.py", line 117, in load_
lora_weights
    self.load_lora_into_unet(
  File "/c1/username/anaconda3/envs/live2diff/lib/python3.10/site-packages/diffusers/loaders/lora.py", line 479, in load_
lora_into_unet
    unet.load_attn_procs(
  File "/c1/username/anaconda3/envs/live2diff/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 11
8, in _inner_fn
    return fn(*args, **kwargs)
  File "/c1/username/anaconda3/envs/live2diff/lib/python3.10/site-packages/diffusers/loaders/unet.py", line 294, in load_
attn_procs
    raise ValueError(f"Module {key} is not a LoRACompatibleConv or LoRACompatibleLinear module.")

I printed both state_dict keys and unet... and found out that only the Linear layers from the unet are converted into LoRACompatibleLinear, while Conv2d layers (proj_in and proj_out in BasicTransformerBlock) are still remaining as Conv2d.

unet_warmup

UNet3DConditionWarmupModel(
. . . 
  (down_blocks): ModuleList(
    (0): CrossAttnDownBlock3DWarmup(
      (attentions): ModuleList(
        (0-1): 2 x Transformer3DModel(
          (norm): GroupNorm(32, 320, eps=1e-06, affine=True)
          (proj_in): **Conv2d**(320, 320, kernel_size=(1, 1), stride=(1, 1))
          (transformer_blocks): ModuleList(
            (0): BasicTransformerBlock(
              (attn1): Attention(
                (to_q): **LoRACompatibleLinear**(in_features=320, out_features=320, bias=False)
                (to_k): **LoRACompatibleLinear**(in_features=320, out_features=320, bias=False)
                (to_v): **LoRACompatibleLinear**(in_features=320, out_features=320, bias=False)
                (to_out): ModuleList(
                  (0): **LoRACompatibleLinear**(in_features=320, out_features=320, bias=True)
                  (1): Dropout(p=0.0, inplace=False)
                )
              )
              (norm1): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
              (attn2): Attention(
                (to_q): **LoRACompatibleLinear**(in_features=320, out_features=320, bias=False)
                (to_k): **LoRACompatibleLinear**(in_features=768, out_features=320, bias=False)
                (to_v): **LoRACompatibleLinear**(in_features=768, out_features=320, bias=False)
                (to_out): ModuleList(
                  (0): **LoRACompatibleLinear**(in_features=320, out_features=320, bias=True)
                  (1): Dropout(p=0.0, inplace=False)
                )
              )
              (norm2): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
              (ff): FeedForward(
                (net): ModuleList(
                  (0): GEGLU(
                    (proj): **LoRACompatibleLinear**(in_features=320, out_features=2560, bias=True)
                  )
                  (1): Dropout(p=0.0, inplace=False)
                  (2): **LoRACompatibleLinear**(in_features=1280, out_features=320, bias=True)
                )
              )
              (norm3): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
            )
          )
          (proj_out): **Conv2d**(320, 320, kernel_size=(1, 1), stride=(1, 1))
        )
      )

state_dicts

unet.down_blocks.0.attentions.0.proj_in.lora.down.weight
unet.down_blocks.0.attentions.0.proj_in.lora.up.weight
unet.down_blocks.0.attentions.0.proj_out.lora.down.weight
unet.down_blocks.0.attentions.0.proj_out.lora.up.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_k_lora.down.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_k_lora.up.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_out_lora.down.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_out_lora.up.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_q_lora.down.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_q_lora.up.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_v_lora.down.weight
unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor.to_v_lora.up.weight
. . .

My environment:

• diffusers : 0.25.0
• peft : 0.10.0
• huggingface-hub : 0.19.4