Fine-tuning vs RAG — which one does your research team actually need?

Research teams often arrive at this question after a frustrating experience: they've tried prompting a frontier model with domain-specific questions and gotten answers that are confidently wrong. The model doesn't know about their unpublished datasets, their internal terminology, or the papers published after its training cutoff.

The instinct is to "train the model on our data." But that instinct conflates two very different techniques with very different tradeoffs.

What fine-tuning actually does

Fine-tuning updates the weights of a pre-trained model using your data. You're not teaching the model new facts — you're adjusting its behaviour, tone, and output format to match patterns in your training set.

Fine-tuning is the right choice when:

• You need the model to consistently output a specific format (structured JSON, LaTeX equations, a particular citation style)
• You want the model to adopt a specific writing style or persona
• You're doing classification or extraction tasks where you have thousands of labelled examples
• Latency matters and you want a smaller, faster model that matches a larger model's quality on your specific task

Fine-tuning is not a reliable way to inject new factual knowledge. Models forget, hallucinate, and confabulate even after fine-tuning on factual data. If you fine-tune on your research papers, the model will sound more like your domain — but it won't reliably cite the right paper or quote the right statistic.

What RAG actually does

Retrieval-Augmented Generation keeps the model weights unchanged. Instead, at query time, it retrieves relevant documents from your knowledge base and injects them into the prompt as context. The model answers based on what it can see in the context window — not what it memorised during training.

RAG is the right choice when:

• You need the model to answer questions about specific documents, papers, or datasets
• Your knowledge base changes frequently (new papers, updated datasets, revised protocols)
• You need citations — RAG can return the source document alongside the answer
• You need to audit why the model gave a particular answer
• You don't have thousands of labelled examples for fine-tuning

RAG's limitation is context window size. If the relevant information is spread across 50 documents and you can only fit 5 in the context, you'll miss things. Retrieval quality — how well you find the right documents — determines answer quality.

The decision framework

Ask yourself two questions:

1. Is the problem about behaviour or knowledge? If you want the model to act differently (format, style, task type), fine-tune. If you want the model to know things it doesn't know, use RAG.
2. Does your knowledge base change? If yes, RAG is almost always better — re-indexing documents is cheap, re-training a model is expensive.

For most research teams, the answer is RAG. You have a corpus of papers, reports, and datasets that you want to query intelligently. That corpus grows over time. You need citations. RAG handles all of this; fine-tuning handles none of it.

When to use both

The approaches aren't mutually exclusive. A common production pattern:

• Fine-tune a smaller model (e.g. Llama 3 8B) on your domain's output format and terminology
• Use RAG to inject relevant documents at query time
• The fine-tuned model knows how to talk about your domain; RAG gives it the specific facts to talk about

This combination gives you the best of both: consistent output format from fine-tuning, accurate factual grounding from RAG. It's also more cost-efficient than using a frontier model for everything — a fine-tuned 8B model with RAG often outperforms GPT-4 on domain-specific tasks at 1/20th the cost.

The real cost of fine-tuning

Fine-tuning is often underestimated in terms of effort:

• Data preparation: You need clean, labelled training examples. For a research domain, this typically means 500–5,000 high-quality prompt/completion pairs. Creating these takes weeks.
• Training cost: Fine-tuning GPT-4 via the OpenAI API costs roughly $25–100 per training run depending on dataset size. Open-weight models on your own GPU are cheaper but require infrastructure.
• Evaluation: You need a held-out test set and a way to measure whether the fine-tuned model is actually better. This is harder than it sounds for open-ended research tasks.
• Maintenance: When the base model updates, you may need to re-fine-tune. Your training data can become stale.

RAG, by contrast, requires indexing your documents (a one-time setup of a few hours) and maintaining a vector store. Adding new documents is as simple as uploading a file.

A genomics research team we work with spent three months fine-tuning a model on their paper corpus before switching to RAG. The RAG pipeline took two days to set up and gave better citation accuracy than the fine-tuned model ever did.

Setting up RAG for a research team

The core components of a research RAG pipeline:

1. Document ingestion: PDFs, Word docs, and HTML pages chunked into ~500-token segments
2. Embedding: Each chunk converted to a vector using an embedding model (text-embedding-3-small is a good default)
3. Vector store: Chunks stored in a vector database (OpenSearch, Pinecone, or pgvector) for similarity search
4. Retrieval: At query time, the question is embedded and the top-k most similar chunks are retrieved
5. Generation: Retrieved chunks are injected into the prompt; the LLM generates an answer grounded in those chunks

Lab Maneuver includes a managed RAG pipeline with OpenSearch Serverless as the vector store. You upload documents through the UI or API, and the indexing happens automatically. Your team queries the knowledge base through the same playground interface they use for direct model access.

---

Related posts

• Building a RAG pipeline for your research team — from document upload to intelligent query
• How to stop a $12k AI bill before it happens — per-user token quotas explained