add an 8-bit version with bitsandbytes

[lucidrains]

https://github.com/TimDettmers/bitsandbytes/blob/main/compile_from_source.md

[lucidrains]

recent results have tipped me over to recommending this optimizer under the condition of high batch sizes (and mainstream architectures). will put some more work into it

[lucidrains]

bitsandbytes-foundation/bitsandbytes#188

[lucidrains]

if there are any CUDA / 8-bit optimizer experts in the crowd, would welcome a code review over at the bitsandbytes repository

[Qubitium]

@lucidrains Is there a reason behind the 64 per_device_train_batch_size threshold recommendation? With consumer 24gb gpus, it is almost impossible to train or finetune any meaningful model with that high batch size to take advantage of lion. It would be nice if lion is applicable to smaller batches? Thanks

[wukevin]

The original authors note that Lion likely doesn't show significant gains over AdamW under the small batch (<64) regime. See https://arxiv.org/pdf/2302.06675.pdf