Research Lab 9 min read · July 30, 2025

Building a RAG pipeline for your research team — from document upload to intelligent query

Most research teams have the same problem: years of papers, reports, and internal documentation that nobody can find when they need it. RAG turns that knowledge base into something you can actually query. Here's how to build a pipeline that works in production.

Retrieval-augmented generation (RAG) is the technique of giving an LLM access to a specific set of documents at query time, rather than relying solely on what it learned during training. The result is a system that can answer questions grounded in your team's actual knowledge — not hallucinated generalities.

The concept is simple. The implementation has a lot of places to go wrong. This post walks through each stage of the pipeline and the decisions that matter.

The four stages of a RAG pipeline

Every RAG system has the same four stages, regardless of the tools you use:

Ingestion — load documents and extract text
Chunking — split text into segments that fit in an embedding model's context window
Embedding — convert each chunk into a vector representation
Retrieval + generation — at query time, embed the question, find the most similar chunks, and pass them to the LLM as context

Each stage has decisions that significantly affect quality. Let's go through them.

Stage 1: Ingestion

Research teams typically have documents in PDF, Word, and plain text formats. PDFs are the hardest — they're designed for visual rendering, not text extraction. A naive PDF parser will give you garbled text from multi-column layouts, miss content in tables, and drop figures entirely.

For research papers specifically:

Use a parser that understands academic PDF structure (PyMuPDF or pdfplumber are reasonable starting points)
Strip headers, footers, and page numbers — they add noise to every chunk
Decide upfront whether to include figure captions and table content, or skip them
Store the source document metadata (title, authors, date, URL) alongside each chunk — you'll need it for citations

Stage 2: Chunking

Chunking is where most RAG implementations make their biggest mistake: using fixed-size chunks with no regard for semantic boundaries.

A 512-token chunk that cuts mid-sentence, mid-paragraph, or mid-argument will produce embeddings that don't represent coherent ideas. When retrieved, they'll give the LLM fragments that are hard to reason about.

Better approaches:

Sentence-aware chunking: split on sentence boundaries, then group sentences until you hit your token limit
Paragraph-aware chunking: treat each paragraph as a chunk, with overlap between adjacent paragraphs
Semantic chunking: use an embedding model to detect topic shifts and split there — more expensive but produces the best retrieval quality

Overlap matters. A 10–15% overlap between adjacent chunks ensures that content near a boundary appears in at least one chunk in full context.

Chunk size is a retrieval quality dial, not a performance dial. Smaller chunks (256 tokens) retrieve more precisely but give the LLM less context. Larger chunks (1024 tokens) give more context but retrieve less precisely. Start at 512 with 10% overlap and tune from there.

Stage 3: Embedding

The embedding model converts text into a vector — a list of numbers that represents the semantic meaning of the text. Similar meanings produce similar vectors. This is what makes semantic search possible.

Key decisions:

Model choice: OpenAI's text-embedding-3-small is a strong default. For on-premise or cost-sensitive deployments, sentence-transformers/all-MiniLM-L6-v2 runs locally and performs well on general text.
Consistency: the same model must be used for both indexing and querying. Mixing models produces garbage results.
Re-embedding: if you switch embedding models, you must re-embed your entire document corpus. Plan for this.

Vectors are stored in a vector database — purpose-built storage that supports approximate nearest-neighbour search. OpenSearch, Pinecone, Weaviate, and pgvector (PostgreSQL extension) are common choices.

Stage 4: Retrieval and generation

At query time, the pipeline:

Embeds the user's question using the same model used for indexing
Searches the vector store for the top-k most similar chunks (typically k=5–10)
Constructs a prompt: system instructions + retrieved chunks + user question
Sends the prompt to the LLM and returns the response

The retrieval step is where you can add significant intelligence:

Hybrid search: combine vector similarity with keyword search (BM25). Vector search finds semantically similar content; keyword search finds exact matches. Hybrid outperforms either alone.
Re-ranking: after retrieving top-k chunks, use a cross-encoder model to re-rank them by relevance to the specific query. Adds latency but improves precision.
Fallback: if no chunks score above a confidence threshold, fall back to a web search or return "I don't have information on this" rather than hallucinating.

The fallback chain that actually works

For research teams, a three-tier fallback produces the best results:

Internal documents first — search your team's uploaded corpus. If a high-confidence match is found, answer from it with a citation.
Web search second — if no confident match in internal docs, search the web for recent information. Useful for questions about recent papers or external context.
LLM knowledge last — if neither source has a confident answer, fall back to the LLM's training knowledge with a clear disclaimer that the answer isn't grounded in your documents.

This chain means your team always gets an answer, and always knows how confident to be in it.

What to build vs. what to use

Building a RAG pipeline from scratch — ingestion, chunking, embedding, vector store, retrieval, fallback — takes a competent engineer 2–4 weeks to get to a working prototype, and another month to get to something production-reliable.

For most research teams, that's not the right use of time. The pipeline is infrastructure, not research. Lab Maneuver ships a production RAG vector store as part of the platform — upload documents, configure chunking strategy, connect your embedding model, and start querying. The fallback chain (documents → web → LLM) is built in.

Want to make your team's knowledge base queryable?

Lab Maneuver's built-in RAG vector store handles ingestion, chunking, embedding, and retrieval. Upload your documents and start querying in minutes.

Get Early Access →