The question "should we fine-tune our own LLM?" reaches architecture tables almost monthly. The short answer, almost always, is not yet. The long answer is that there are legitimate cases, and costs have come down thanks to LoRA and QLoRA, though they remain considerable. Alternatives like RAG or prompt engineering solve 80% of needs without training’s operational overhead.

Key Takeaways

  • Fine-tuning is the third customisation tier: the most expensive and the slowest to iterate.

  • LoRA and QLoRA have lowered the GPU bar from "8 × A100" to "an RTX 4090".

  • The real cost isn’t GPUs: it’s data, evaluation, and ongoing operation.

  • Until style, format or cost hits a ceiling, RAG + prompt engineering covers the case without training anything.

  • Fine-tuning is justified when the style, format, or cost problem has hit a ceiling with the other two approaches.

The three customisation levels

To frame the problem, there are three layers of LLM customisation, from lowest to highest cost:

  • Prompt engineering. Tune instructions, few-shot examples, chain-of-thought. Marginal cost, iteration in minutes. Covers the vast majority of well-defined tasks.

  • Retrieval-Augmented Generation (RAG). Retrieve relevant chunks from a knowledge base and pass them into the model’s context. Medium cost (embeddings + vector store), iteration in days.

  • Fine-tuning. Modify model weights with your own examples. High cost (data, GPUs, validation), iteration in weeks.

Jumping directly to fine-tuning is the most common mistake. Teams that try could have gotten equivalent or better results with a well-designed RAG. See the vector database comparison as a pipeline foundation.

When fine-tuning genuinely makes sense

Three cases where fine-tuning justifies its cost:

  • Specific style or voice. If you need the model to respond with an exact brand personality (idioms, grammar structures, a tone you can’t capture in a long system prompt), fine-tuning internalises it.

  • Structured output format. Models fine-tuned to always return a specific JSON, or to follow a proprietary markup schema, are more reliable than prompted ones: the format becomes "sewn into" the model.

  • Cost and latency reduction with small models. A 7B-parameter model fine-tuned on your domain can match or beat GPT-3.5 for that specific task, at 10–20% of the cost per token and better latency.

Outside these cases, RAG usually wins.

LoRA and QLoRA: Accessible Fine-Tuning

The big recent shift is that fine-tuning went from "you need 8 A100s" to "you can do it on an RTX 4090". The key technique is LoRA[1] (Low-Rank Adaptation): instead of training all weights, you add low-rank matrices over the frozen model. The result is practically identical to full fine-tuning at 1% of the GPU cost.

QLoRA[2] combines LoRA with 4-bit quantisation. It lets you fine-tune 65-billion-parameter models on a single 48 GB VRAM GPU, something previously unthinkable.

Libraries like PEFT[3] from Hugging Face and axolotl[4] wrap these methods with declarative config. A LoRA pipeline over Llama 2 7B fits in a 30-line YAML, directly related to what the post on LLaMA 2 and open models describes.

What actually costs

The real cost of fine-tuning isn’t GPUs. It’s everything else.

  • Preparing the dataset. Between 500 and 5,000 quality examples (prompt + ideal response) require substantial manual investment. Poorly designed examples poison the model with biases and failures.

  • Iteration and evaluation. A bad fine-tune can look good on the happy path and fail catastrophically on edge cases. You need automated evals before and after.

  • Production operation. Your own model means managing inference, updates, and drift monitoring. This isn’t just "calling an API" anymore.

Realistic budget for a first serious fine-tune: 2–3 engineering weeks + 1–5k USD in GPU + a basic MLOps pipeline for evaluation.

Alternatives before deciding

Before committing to fine-tuning, exhaust these options:

  • RAG over your domain. With pgvector or Pinecone plus good reranking, you cover "the model needs to know company-specific data" without training anything.

  • Longer prompts with careful examples. GPT-4 with 16 careful few-shot examples beats a fine-tuned 7B model when the examples are good.

  • Function calling with structured response. If you’re after structured output alone, function calling solves it without training.

  • Existing specialised models. For common tasks (code, medical, legal) the community already has fine-tuned models: CodeLlama[5], Med-PaLM, and others.

Conclusion

Fine-tuning has become technically democratised thanks to LoRA and QLoRA, but operationally it’s still a serious investment. For the vast majority of teams, starting with prompt engineering + RAG is the right path; fine-tuning is reserved for problems where the other two have clearly hit a ceiling. When you do pursue it, rigorous evaluation before and after training is as important as the training itself.

Frequently asked questions

Can I fine-tune an LLM on a single consumer GPU?

Yes, thanks to LoRA and QLoRA the bar has dropped from 8 A100s to an RTX 4090. LoRA adds low-rank matrices over the frozen model instead of training all weights, with a result practically identical to full fine-tuning at 1% of the GPU cost. QLoRA adds 4-bit quantisation and lets you fine-tune 65-billion-parameter models on a single 48 GB VRAM GPU. Libraries like PEFT from Hugging Face and axolotl express it in a roughly 30-line YAML.

What does a first serious fine-tune really cost?

A realistic budget is 2–3 engineering weeks plus 1–5k USD in GPU and a basic MLOps pipeline for evaluation. GPUs are not the main cost: preparing between 500 and 5,000 quality examples is, because poorly designed ones poison the model. So is setting up automated evals before and after, since a bad fine-tune looks good on the happy path and fails on edge cases. And so is operating inference, updates, and drift monitoring in production.

What should I try before committing to fine-tuning?

Exhaust the alternatives, which cover 80% of needs: the first is RAG over your domain with pgvector or Pinecone plus good reranking, if the model needs company-specific data. The second is longer prompts with careful examples, because GPT-4 with 16 well-chosen few-shot examples beats a fine-tuned 7B model. That leaves function calling if you are after structured output, and existing specialised models such as CodeLlama or Med-PaLM. Fine-tuning is justified when the style, format, or cost problem has hit a ceiling with those approaches.

Sources

  1. LoRA
  2. QLoRA
  3. PEFT
  4. axolotl
  5. CodeLlama