Error: dtype mismatch in matmul_with_alpha, lhs: F16, rhs: BF16

[Aloso]

I'm moving this discussion to a new issue since it is different from #32.

To summarize: When I run diffusion_rs_cli (without CUDA support), it errors right after asking for the prompt.

Click to see output

diffusion_rs_cli --scale 3.5 --num-steps 50 dduf -f FLUX.1-dev-Q4-bnb.dduf
2025-01-09T18:44:36.528527Z  INFO diffusion_rs_core::pipelines: loading from source: dduf file: FLUX.1-dev-Q4-bnb.dduf.
2025-01-09T18:44:36.528909Z  INFO diffusion_rs_core::pipelines: model architecture is: flux
2025-01-09T18:44:36.529194Z  INFO diffusion_rs_core::util::auto_dtype: dtype selected is F16.
2025-01-09T18:44:36.582729Z  INFO diffusion_rs_core::pipelines::flux: loading CLIP model
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 196/196 [00:00<00:00, 2446.43it/s]
2025-01-09T18:44:36.868689Z  INFO diffusion_rs_core::pipelines::flux: loading T5 model
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 138/138 [00:00<00:00, 526.65it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 513/513 [00:02<00:00, 463.28it/s]
2025-01-09T18:44:41.174098Z  INFO diffusion_rs_core::pipelines::flux: loading VAE model
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 244/244 [00:00<00:00, 5353.65it/s]
2025-01-09T18:44:41.484905Z  INFO diffusion_rs_core::pipelines::flux: loading FLUX model
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2672/2672 [00:05<00:00, 605.08it/s]
2025-01-09T18:44:47.498607Z  INFO diffusion_rs_core::pipelines::flux: FLUX pipeline using a guidance-distilled model: true
◇  Height:
│  720
│
◇  Width:
│  1280
│
◇  Prompt:
│  something
│
Error: dtype mismatch in matmul_with_alpha, lhs: F16, rhs: BF16

I built it as following, the last commit was 1d1f0bf (#34):

cargo install --git https://github.com/EricLBuehler/diffusion-rs

• OS: x86_64 Linux 6.11.10-2-MANJARO
• CPU: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics @ 16x 5.137GHz
• GPU: AMD Radeon 780M (radeonsi, gfx1103_r1, LLVM 18.1.8, DRM 3.59, 6.11.10-2-MANJARO)

[Aloso]

I got a backtrace with RUST_BACKTRACE=1:

Error: dtype mismatch in matmul_with_alpha, lhs: F16, rhs: BF16
   0: diffusion_rs_common::core::error::Error::bt
   1: diffusion_rs_common::core::storage::Storage::same_dtype
   2: diffusion_rs_common::core::tensor::Tensor::matmul
   3: <diffusion_rs_backend::bitsandbytes::BnbLinear as diffusion_rs_backend::QuantMethod>::forward
   4: diffusion_rs_backend::QuantMethod::forward_autocast
   5: diffusion_rs_core::models::t5::T5EncoderModel::forward
   6: <diffusion_rs_core::pipelines::flux::FluxPipeline as diffusion_rs_core::pipelines::ModelPipeline>::forward
   7: diffusion_rs_cli::main
   8: std::sys::backtrace::__rust_begin_short_backtrace
   9: main
  10: <unknown>
  11: __libc_start_main
  12: _start


Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: diffusion_rs_cli::main
   2: std::sys::backtrace::__rust_begin_short_backtrace
   3: main
   4: <unknown>
   5: __libc_start_main
   6: _start