LoRA

Low-Ranking Adaptation of Large Language Models is a training method that improves the creation of models based on other larger models.

It speeds up training and consumes less memory when dealing with large models.

It adds pairs of rank-decomposition weight matrices, aka update matrices, to existing weights. It only trains those newly added weights.

Works best for subjects like faces!
Try training a Checkpoint for themes.

Benefits:

• Previous pretrained weights are kept frozen, which avoids catastrophic forgetting.
• Rank-decomposition matrices have less parameters, which means that trained LoRA weights are more easily portable (smaller file size!)
• Lora Matrices are added to the attention layers of the original model.
• Greater memory efficiency means you can run fine-tuning on consumer GPUs!