LoRA and QLoRA: Efficient Fine-Tuning on a Single Laptop

LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA) democratised fine-tuning of large language models by solving the central problem that made it prohibitive: GPU memory. The idea behind LoRA is elegant in its simplicity. Instead of updating all model parameters during training, it adds low-rank adaptation matrices. Those matrices learn the delta between the base model’s behaviour and the desired behaviour.

QLoRA takes that idea further by combining quantization of the base model with LoRA adapters, making fine-tuning of 70 B models viable on consumer hardware.

Key takeaways

• LoRA adds low-rank matrices (rank 4-64) to attention layers; the base model stays frozen.

• QLoRA quantizes the base model to 4 bits (NF4) and keeps only the LoRA adapters in full precision, reducing required memory by 65-75%.

• A Llama 3 8B requires 128 GB of VRAM for full fine-tuning; with QLoRA it fits on a 24 GB RTX 3090.

• Quality loss vs full fine-tuning is 1-3% on standard benchmarks, acceptable for most domain-specific use cases.

• The decision between LoRA/QLoRA and prompt engineering depends on available data volume and domain specificity.

The problem they solve

Traditional fine-tuning of a Llama 3 8B model requires simultaneously storing:

• Model parameters: 8B × 4 bytes = 32 GB just for the weights.

• Training gradients: another 32 GB.

• Adam optimizer state: 64 GB additional (two moments per parameter).

• Total: 128 GB of VRAM, equivalent to four 80 GB A100s.

At 70B scale, the cost multiplies by 8.75x and exceeds what any reasonable experimentation budget can buy.

LoRA solves the problem with a theoretical observation from the original paper by Hu et al.^[1]: the changes fine-tuning induces in model weights have intrinsically low rank. You do not need to update a dense matrix of millions of parameters; you can approximate that change with the product of two much smaller matrices:

ΔW ≈ A × B
where A ∈ ℝ^(d×r) and B ∈ ℝ^(r×k), with r << d and r << k

If rank r is 8 and the original dimension is 4096, the number of trainable parameters drops from 4096² ≈ 16M to 2 × 4096 × 8 = 65k. A 250x reduction in trainable parameters, implying a proportional reduction in gradients and optimizer state.

QLoRA: quantization plus adapters

QLoRA adds one more step: quantizes the base model to NF4 (Normal Float 4-bit), a quantization format that better preserves the distribution of LLM weights than standard integer quantization. Over that quantized model, LoRA adapters are trained in full BFloat16. The paper by Dettmers et al.^[2] that introduced the technique reports fine-tuning a 65B model on a single 48 GB GPU without a drop in task performance versus full 16-bit fine-tuning.

The combination enables:

• Loading a 70B model on four 24 GB GPUs (instead of ten 80 GB).

• Loading a 7B model on a single 8 GB GPU.

• Fine-tuning 13B models on a laptop with a 16 GB RTX 3080.

The quality loss of NF4 vs FP16 is well documented: 1-3% on reasoning and coding benchmarks, acceptable for most domain-specific use cases.

Workflow with PEFT and Hugging Face

The Hugging Face ecosystem makes the workflow surprisingly straightforward. The PEFT^[3] (Parameter-Efficient Fine-Tuning) library manages LoRA adapters; transformers loads the base model; trl adds SFTTrainer for supervised fine-tuning:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

# Load quantized model (QLoRA)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Configure LoRA adapters
lora_config = LoraConfig(
    r=16,                     # matrix rank
    lora_alpha=32,            # scaling factor
    target_modules=["q_proj", "v_proj"],  # layers to adapt
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 5,242,880 || all params: 8,035,885,056 || trainable%: 0.0653

The result: fewer than 0.07% of parameters are trainable, with a proportional memory reduction in gradients and optimizer state.

Full Transformer architecture with encoder and decoder blocks, multi-head attention layers, and feed-forward layers. LoRA inserts its low-rank adaptation matrices into this architecture during fine-tuning (Image: dvgodoy, CC BY 4.0, via Wikimedia Commons)

Choosing hyperparameters

The LoRA hyperparameters that most impact results:

• Rank (r): low values (4-8) for simple style or format tasks; medium values (16-32) for domain adaptation; high values (64-128) for deeper behaviour changes. Higher rank means more trainable parameters and greater capacity, but also greater overfitting risk.

• lora_alpha: 2× the rank as a starting value. Controls the scaling factor of the adapters.

• target_modules: query and value projection layers (q_proj, v_proj) are the minimum; adding key (k_proj) and output (o_proj) projections improves quality at the cost of more parameters.

• Learning rate: 1e-4 to 3e-4, below what you would use for full fine-tuning.

For models you will later serve with vLLM in production, ensure the LoRA adapter format is compatible with the vLLM version you use. Multi-LoRA support lets you serve more than one adapter over the same base model.

LoRA/QLoRA vs prompt engineering

The decision between fine-tuning and prompt engineering depends on three factors:

When fine-tuning pays:

• You have more than 500-1,000 high-quality training examples.

• The domain has specific vocabulary, format, or reasoning the base model does not know.

• The task requires high format consistency (prompt engineering can give unwanted variability).

• Token cost in production is significant; a fine-tuned model needs shorter prompts.

When prompt engineering is sufficient:

• Training data is scarce or hard to obtain.

• The task is reasonably general and the base model handles it well with instructions.

• You need to iterate without training cycles.

• Base model hallucination is acceptable for the use case.

The most common mistake is fine-tuning on insufficient data (<200 examples) expecting large improvements. With little data, the model learns the format but loses generalisation. With LoRA/QLoRA there is little technical friction to experiment, but that does not eliminate the requirement for quality data.

Deploying LoRA adapters

Once trained, the LoRA adapter is saved as a file independent of the base model: 10-200 MB against the gigabytes of the full model. Deployment options:

• Merge: merge_and_unload() merges the adapter weights into the base model, generating a standard model any runtime can serve.

• Multi-LoRA with vLLM: serve more than one adapter over the same base model, switching per request. Efficient when you have more than one specialised fine-tune; the vLLM documentation on LoRA adapters^[4] covers hot-loading and unloading them.

• Separate adapter with transformers: load the base model plus adapter dynamically; useful for experimentation.

For observability of the fine-tuned model in production, the specific challenge of LoRA adapters is tracking which adapter served each inference. Default instrumentation does not emit that field: you add it to the trace yourself.

Conclusion

LoRA and QLoRA eliminated the hardware barrier that made fine-tuning prohibitive without a GPU cluster. With QLoRA, a 7B model fits on an 8 GB GPU, and quality loss vs full fine-tuning is marginal in practice. The real bottleneck is no longer hardware or framework: it is quality training data and rigorous evaluation of whether fine-tuning actually improves what matters for your use case.

This article is also available in Spanish: LoRA y QLoRA: fine-tuning eficiente al alcance de un solo portátil.

Sources:

1. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)^[1]
2. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)^[2]
3. Hugging Face, PEFT documentation^[3]
4. vLLM, LoRA adapters documentation^[4]

Frequently asked questions

How much VRAM do I need to fine-tune a Llama 3 8B with QLoRA?

It fits on a 24 GB RTX 3090. Full fine-tuning of the same model requires 128 GB of VRAM (32 GB of weights, 32 GB of gradients and 64 GB of Adam optimizer state), the equivalent of four 80 GB A100s. QLoRA quantizes the base model to NF4 and trains only the adapters. With it, a 7B model fits on a single 8 GB GPU, a 13B model on a laptop with a 16 GB RTX 3080, and a 70B model on four 24 GB GPUs.

How much quality do I lose with QLoRA compared to full fine-tuning?

Between 1% and 3% on standard reasoning and coding benchmarks, an acceptable loss for most domain-specific use cases. The paper by Dettmers et al. reports fine-tuning a 65B model on a single 48 GB GPU without a drop in task performance versus full 16-bit fine-tuning. The NF4 (Normal Float 4-bit) format preserves the distribution of LLM weights better than standard integer quantization.

How many examples do I need before LoRA fine-tuning pays off over prompt engineering?

More than 500-1,000 high-quality examples, plus a domain with vocabulary, format or reasoning the base model does not know, or a task that requires high format consistency. The most common mistake is fine-tuning on fewer than 200 examples expecting large improvements: with little data the model learns the format but loses generalisation. If training data is scarce or you need to iterate without training cycles, prompt engineering is sufficient.

Sources

1. original paper by Hu et al.
2. paper by Dettmers et al.
3. PEFT
4. vLLM documentation on LoRA adapters