The question few companies ask before spending on fine-tuning
When an executive learns that an AI model can be trained on the company’s own data so it “speaks the language of the business,” the usual reaction is enthusiasm. Fine-tuning sounds like the ultimate solution: a model that knows our products, our terminology, our processes.
The problem is that fine-tuning is expensive, needs ongoing maintenance, and in many cases isn’t the right tool for the problem at hand. Before investing in it, it’s worth understanding when it really makes sense and when there are more efficient alternatives.
Three ways to adapt an LLM — and when to use each
Prompt engineering
The fastest, cheapest way to adapt an LLM’s behavior. With carefully designed instructions in the prompt — context, role, response format, examples (few-shot) — you can get very specific behavior without touching the model. It’s the right starting point for almost every use case: if prompt engineering solves the problem, there’s no reason to go further.
RAG (retrieval-augmented generation)
When the model needs access to company-specific information — product catalogs, technical documentation, an internal knowledge base, case history — RAG supplies that context at query time, with no retraining. It’s the right solution for most cases where the problem is access to knowledge, not the model’s style or behavior.
Fine-tuning
Fine-tuning makes sense when neither of the other two is enough. Specifically: when you need the model to adopt a very particular response style that’s hard to capture in prompts, when you want more efficient inference by shrinking the system prompt, or when the use case requires behavior the base model doesn’t have natively and that has to be reinforced with thousands of examples of the right style.
When fine-tuning adds real value
Specialized classifiers and extractors
For very specific tasks — classifying documents into business-specific categories, extracting internally defined entities, or detecting patterns in text that don’t appear in general language — a fine-tuned small model beats large general-purpose models on efficiency (speed, cost, accuracy). Fine-tuning classification models on your own labeled data is the use case with the strongest proven ROI.
Models with a well-defined communication style
If the company has a very particular tone — technical, formal, extremely concise, with its own terminology — and needs the model to reproduce it consistently, fine-tuning on examples of the right style can achieve results that prompt engineering can’t deliver reliably.
Cutting costs at scale
For applications with millions of calls a day, fine-tuning a small model for a specific task can be much cheaper than running a large general-purpose model with long prompts. This calls for a cost analysis at scale; it isn’t relevant at low volumes.
Cost, data, and maintenance: the part nobody mentions
Fine-tuning requires good-quality labeled data (typically hundreds to thousands of well-annotated examples for LLMs, more for vision or classification models), GPU time for training (a direct cost), rigorous evaluation of the resulting model, and periodic retraining as the domain evolves.
A fine-tuned model isn’t a static asset — it needs maintenance. If the product catalog changes, the terminology evolves, or the model’s use cases expand, it has to be retrained. Without an MLOps pipeline that supports this cycle, the model degrades over time.
Frequently asked questions
How many training examples do I need to fine-tune an LLM?
For LLMs such as OpenAI’s models or Llama, supervised fine-tuning can show improvements with as few as one hundred to five hundred high-quality prompt–response pairs for very specific tasks. Robust consistency on complex or highly variable tasks takes several thousand. Quality matters more than volume: well-annotated examples are worth more than thousands of mediocre ones.
Should we fine-tune an open-source model or a commercial API model?
Both are viable. Fine-tuning API models (OpenAI, Anthropic) is technically simpler, but the training data is sent to the provider. Fine-tuning open-source models (Llama, Mistral, Qwen) gives you full control over the data and the resulting model, but requires GPU infrastructure and more technical expertise. For sensitive data, local fine-tuning is usually the right choice.
When should I choose RAG over fine-tuning?
Almost always when the problem is access to knowledge. If the model needs to know a product’s price, the contents of a contract, or the steps in a procedure, RAG is more flexible, easier to update, and more explainable than fine-tuning. Fine-tuning isn’t a database: information baked into the model’s weights is hard to update and can’t be retrieved transparently. RAG keeps the knowledge outside the model, where it can be managed explicitly.