LoRA and QLoRA: Efficient Fine-Tuning on a Single Laptop

Tested with PyTorch 2.14.1 · Transformers 5.21.0 · PEFT 0.21.2 · TRL 1.16.0 · bitsandbytes 0.50.2 · Docker on arm64, no GPU · verified 2026-10-27

Updated: 2026-10-27

LoRA (Low-Rank Adaptation) trains two small matrices per layer instead of every weight in the model, and QLoRA (Quantized LoRA) does the same on a base model quantized to 4 bits. Both attack the problem that made fine-tuning prohibitive: the memory taken by gradients and optimizer state. To check it with my own numbers, I trained the same model with LoRA and with full fine-tuning in a Docker container with no GPU: LoRA used 3.5% of the parameters, a peak of 2.35 GB of RAM against 3.88 GB, and left a 19.6 MB adapter.

QLoRA takes the idea further: the frozen weights are stored in NF4, a 4-bit format, and only the adapters train in 16 bits. That fits on a GPU what would not fit in 16 bits. In the same test, bitsandbytes 0.50.2 also loaded the model in 4 bits on CPU: 109.8 MB against 269 MB in BF16.

Key takeaways

• LoRA freezes the base model and trains a pair of low-rank matrices per chosen layer; at rank 8 on a 4096 × 4096 matrix, trainable parameters drop from 16.8 million to 65,536.

• Full fine-tuning with Adam in FP32 needs about 16 bytes per parameter for weights, gradients and optimizer: 128 GB for an 8-billion-parameter model, before counting activations.

• In the CPU test with SmolLM2-135M, rank-16 LoRA on every linear layer trained 4.88 million parameters and peaked at 2.35 GB of memory, against 3.88 GB for full fine-tuning.

• The QLoRA paper fine-tuned a 65-billion-parameter model on a single 48 GB GPU while preserving full 16-bit fine-tuning performance.

• Without enough data no technique saves the training: Soup, the CLI I tested, refuses below 50 examples and suggests switching path from about 200 clean rows.

The problem they solve

Full fine-tuning keeps in memory, for every parameter, the weight, its gradient and two Adam optimizer moments. In FP32 that is 4 + 4 + 8 = 16 bytes per parameter. For an 8-billion-parameter model such as Llama 3 8B, the arithmetic gives:

• Weights: 8 billion × 4 bytes = 32 GB.

• Gradients: another 32 GB.

• Adam optimizer state: 64 GB (two moments per parameter).

• Total: 128 GB, without activations, which grow with batch size and sequence length.

At 70 billion parameters the same arithmetic multiplies by 8.75 and passes 1 TB. Mixed-precision training in BF16 trims weights and gradients somewhat, but the optimizer is still the heaviest part.

LoRA solves the problem with an observation from the original paper by Hu et al.^[1]: the weight changes fine-tuning induces have low intrinsic rank. You do not need to update a dense matrix of millions of parameters; you can approximate that change with the product of two much smaller matrices:

ΔW ≈ B × A
where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with r << d and r << k

If rank r is 8 and the original matrix is 4096 × 4096, trainable parameters drop from 16.8 million to 2 × 4096 × 8 = 65,536, 256 times fewer. Gradients and optimizer state only exist for those parameters, so they shrink in the same proportion.

QLoRA: quantization plus adapters

QLoRA adds one more step: it quantizes the base model to NF4 (4-bit Normal Float), a format designed for the distribution of LLM weights. On that frozen model, the LoRA adapters train in BF16.

The paper by Dettmers et al.^[2] introduced the technique in May 2023. It fine-tuned a 65-billion-parameter model on a single 48 GB GPU while preserving full 16-bit fine-tuning performance. To get there it added double quantization of the constants and paged optimizers, which absorb memory spikes.

The weight arithmetic is direct. In NF4 each parameter takes half a byte plus a small share of quantization constants. An 8-billion-parameter model goes from 16 GB in BF16 to about 4 GB, and a 70-billion one from 140 to about 35 GB. On top of that come activations, the adapters and their optimizer, which depend on batch and context: measure before buying hardware.

A reproducible example: LoRA on CPU with Docker

To get my own figures, I trained the same model with LoRA and with full fine-tuning on 27 October 2026. I used a Docker container on an arm64 machine with 18 cores and 62 GB of RAM, no GPU.

The model is SmolLM2-135M-Instruct^[3], which needs no license acceptance. The data is the first 320 examples of trl-lib/Capybara^[4], a conversation dataset. The image installs CPU PyTorch and the Hugging Face stack:

FROM python:3.12-slim
RUN pip install --no-cache-dir torch \
      --index-url https://download.pytorch.org/whl/cpu \
 && pip install --no-cache-dir transformers peft trl datasets accelerate
WORKDIR /work

pip resolved PyTorch 2.14.1, Transformers 5.21.0, PEFT 0.21.2, TRL 1.16.0 and Datasets 5.1.0. The training script uses TRL’s SFTTrainer, and the only difference between the two modes is whether it receives a LoRA configuration:

ds = load_dataset("trl-lib/Capybara", split="train[:320]")
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
peft_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                         target_modules="all-linear", task_type="CAUSAL_LM")
args = SFTConfig(output_dir="/out/lora", max_steps=30,
                 per_device_train_batch_size=4, learning_rate=2e-4,
                 max_length=256, logging_steps=5, save_strategy="no",
                 report_to=[], use_cpu=True, seed=0)
tr = SFTTrainer(model=model, args=args, train_dataset=ds,
                processing_class=tok, peft_config=peft_config)
tr.train()
tr.save_model("/out/lora/final")

For full fine-tuning, the same script passes peft_config=None and a 2e-5 learning rate. Peak memory comes from resource.getrusage at the end, and each mode runs in its own container:

docker build -t lora-cpu .
docker run --rm -v "$PWD:/work" -v "$PWD/out:/out" \
  -e MODEL=HuggingFaceTB/SmolLM2-135M-Instruct lora-cpu python train.py lora

print_trainable_parameters() returned 4,884,480 trainable parameters out of 139,399,488, 3.5%. That is more than the classic q_proj and v_proj only example, because all-linear also adapts the key, output and MLP projections. Full fine-tuning used 1.65 times as much memory as LoRA:

The difference, 1.53 GB, is almost exactly what the arithmetic predicts: gradients and Adam state for the base model’s 134.5 million parameters add up to 1.61 GB. The rest of the peak, weights and activations, is paid by both modes. On an 8-billion-parameter model that same avoided share is 96 GB.

The 30 steps took 11.7 minutes with LoRA and 9.8 with full fine-tuning. LoRA was not faster: its advantage is memory, not compute, because the backward pass still runs through the whole model. The CPU shared the machine with other processes, so the times are only a rough reference.

Training loss went from 2.36 to 1.59 with LoRA and from 2.37 to 1.59 with full fine-tuning. The saved adapter takes 19.6 MB, which is those 4.88 million parameters in FP32; the base model, by contrast, is untouched and does not need to be redistributed.

QLoRA also ran on this CPU. With bitsandbytes 0.50.2 and the base model in NF4, the same 30 steps peaked at 2.01 GB and took 10.5 minutes. Loss went from 2.58 to 1.71.

Loss starts higher than with LoRA, probably because of the error quantizing the base model introduces, and in 30 steps it did not catch up. On a 135-million-parameter model, the saving over LoRA is small: 0.34 GB, because the weights weigh little next to the activations. On an 8-billion-parameter one, the weights go from 16 to about 4 GB.

A practical detail: the first attempt, with Qwen2.5-0.5B and 512-token sequences, took about 187 s per step on this CPU. On CPU, the real limit is time, not memory.

The same script with QLoRA

The script above becomes QLoRA by changing only how the model loads. BitsAndBytesConfig quantizes the base model to NF4, and the rest, LoraConfig and SFTTrainer, stays the same. This is the snippet I used in the test; on a GPU, change device_map="cpu" to device_map="auto":

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, quantization_config=bnb_config, device_map="cpu")

The PEFT library^[5] (Parameter-Efficient Fine-Tuning) manages the adapters, transformers loads the base model and TRL^[6] adds SFTTrainer for supervised fine-tuning.

Full Transformer architecture with encoder and decoder blocks, multi-head attention layers, and feed-forward layers. That is where LoRA inserts the low-rank adaptation matrices during fine-tuning (Image: dvgodoy, CC BY 4.0, via Wikimedia Commons)

Choosing hyperparameters

These are the usual starting points for LoRA hyperparameters; tune them by measuring against a validation set:

• Rank (r): low values (4-8) for style or format tasks; medium (16-32) for domain adaptation; high (64-128) for deeper behaviour changes. Higher rank means more trainable parameters and capacity, but also more overfitting risk.

• lora_alpha: twice the rank as a starting value. It controls the adapters’ scaling factor.

• target_modules: q_proj and v_proj are the original paper’s minimum; all-linear adapts every linear layer and, in the test, raised trainable parameters to 3.5% of the model.

• Learning rate: around 2e-4, ten times what you would use for full fine-tuning (2e-5 in the test).

For models you will later serve with vLLM in production, make sure the LoRA adapter format is compatible with your vLLM version. Multi-LoRA support lets you serve more than one adapter on the same base model.

LoRA/QLoRA vs prompt engineering

The choice between fine-tuning and prompt engineering depends on the data you have and on what the base model already solves:

When fine-tuning pays off:

• You have hundreds of clean, reviewed examples of the task, not a few dozen.

• The domain has vocabulary, format or reasoning the base model does not know.

• The task needs a constant output format that the prompt cannot pin down.

• Production token cost matters; a fine-tuned model needs shorter prompts.

When prompt engineering is enough:

• Training data is scarce or hard to obtain.

• The task is reasonably general and the base model handles it well with instructions.

• You need to iterate without training cycles.

• Base-model hallucination is acceptable for the use case.

With little data, the model learns the format but loses generalization. Soup, the fine-tuning CLI that tells you when not to train, puts numbers on that boundary. With 20 examples it recommended not training, because its floor is 50 rows. It suggests moving to fine-tuning from about 200 clean rows. LoRA/QLoRA make experimenting technically cheap, but that does not remove the need for quality data.

Deploying LoRA adapters

Once trained, the LoRA adapter is saved separately from the base model: in the test, 19.6 MB against the 538 MB of the base model weights in FP32. Its size is the number of trainable parameters times the bytes of each, so it grows with the rank and with the layers you adapt. There are three deployment options:

• Merge: merge_and_unload() adds the adapter weights into the base model and produces a standard model any runtime can serve. In the test, the merged model took 538 MB and generated text without trouble, although when asked what LoRA is it described LoRa radio. Thirty steps on generic conversations teach no new knowledge: that is why you need to evaluate with examples from your task.

• Multi-LoRA with vLLM: serve more than one adapter on the same base model, switching per request. Efficient when you have more than one specialized fine-tune; the vLLM LoRA adapter documentation^[7] details how to load and unload them at runtime.

• Separate adapter with transformers: load the base model plus the adapter with PeftModel.from_pretrained; useful for experimentation.

For observability of the fine-tuned model in production, the specific challenge with LoRA adapters is knowing which adapter served each inference. That field does not come from default instrumentation: you add it yourself as a trace attribute.

Conclusion

LoRA and QLoRA removed the hardware barrier that made fine-tuning prohibitive for anyone without a GPU cluster. The arithmetic explains it: full fine-tuning pays 16 bytes per parameter and LoRA only pays them for the small share it trains, as the CPU test confirmed. The real bottleneck is no longer hardware or the framework. It is quality training data and rigorous evaluation of whether fine-tuning improves what matters for your use case.

This article is also available in Spanish: LoRA y QLoRA: fine-tuning eficiente al alcance de un solo portátil.

Sources:

1. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)^[1]
2. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)^[2]
3. Hugging Face, PEFT documentation^[5]
4. Hugging Face, TRL documentation^[6]
5. Hugging Face, SmolLM2-135M-Instruct^[3]
6. Hugging Face, trl-lib/Capybara dataset^[4]
7. vLLM, LoRA adapter documentation^[7]

Frequently asked questions

How much memory do I need to fine-tune a Llama 3 8B with QLoRA?

The model weights in NF4 take about 4 GB, against 16 GB in BF16. On top of that come activations, the adapters and their optimizer, which depend on batch and sequence length, so measure with your own configuration before buying hardware. For reference, full fine-tuning of the same model with Adam in FP32 needs 128 GB just for weights, gradients and optimizer. The QLoRA paper fine-tuned a 65-billion-parameter model on a 48 GB GPU.

Can you fine-tune with LoRA without a GPU?

Yes, with small models. With no GPU, in Docker, rank-16 LoRA on SmolLM2-135M trained 30 steps with a 2.35 GB RAM peak. The adapter took 19.6 MB. I used PyTorch 2.14.1, PEFT 0.21.2 and TRL 1.16.0.

The limit on CPU is time: about 23 s per step with that model, and about 187 s with a 500-million-parameter one and 512-token sequences. QLoRA also worked on CPU with bitsandbytes 0.50.2, but where it really saves memory is on a GPU with large models.

How many examples do I need before LoRA fine-tuning pays off over prompt engineering?

Hundreds of clean examples, not dozens. Soup, an open-source fine-tuning CLI, refuses to train below 50 rows and suggests moving to fine-tuning from about 200 clean ones. The domain should also have vocabulary, format or reasoning the base model does not know, or the task should need a constant format. With little data the model learns the format but loses generalization; if data is scarce, start with prompt engineering.

Sources

1. original paper by Hu et al.
2. paper by Dettmers et al.
3. SmolLM2-135M-Instruct
4. trl-lib/Capybara
5. PEFT library
6. TRL
7. vLLM LoRA adapter documentation