LoRA and QLoRA: Efficient Fine-Tuning on a Single Laptop

LoRA cuts fine-tuning cost for large language models by training only small low-rank adaptation matrices instead of every parameter in the base model. QLoRA adds 4-bit quantization on top, cutting required GPU memory by 65-75%, with quality loss of just 1-3% versus full fine-tuning.

October 29, 2024 6 min 277 4.6

Artificial Intelligence

Model Quantization and llama.cpp on Your Laptop

With quantization, model weights are stored with fewer bits (4, 5, or 8 instead of 16), so Llama 2 13B shrinks from 26 GB to about 7.5 GB. With llama.cpp it runs on an ordinary 16GB-RAM laptop with no dedicated GPU, and the quality loss is smaller than intuition suggests.

November 4, 2023 5 min 265 4.5