huggingface

LoRA and QLoRA: Efficient Fine-Tuning on a Single Laptop

LoRA trains only two small matrices per layer and QLoRA does it on a 4-bit base model. The memory arithmetic of each technique and a reproducible example in Docker with no GPU: LoRA trained 3.5% of SmolLM2-135M's parameters with a 2.35 GB peak, against 3.88 GB for full fine-tuning.