← All writing9 min read
LLM

Building a RAG Pipeline That Actually Works

Production-focused guide to building RAG pipelines with Python. Chunking strategies, vector search with pgvector, re-ranking, prompt construction, and the stuff that breaks past the tutorial stage.

· 9 min read

Getting a basic RAG pipeline working is quick. Documents go in, vectors come out, you plug in a model, ask a question and get an answer. The demo looks great. Then you point it at real internal documentation, and it starts confidently telling users about features that no longer exist.

“Load documents, chunk, embed, retrieve, generate” is the easy part. The hard part is everything that breaks quietly. Chunks lose their meaning because they were split mid-thought. Embeddings miss domain abbreviations that any person on the team would understand. Search results look relevant but don’t actually answer the question.

At EGS I’ve built RAG on the OpenAI API, which is what the company uses. We started with quality assurance, then used it for support Q&A. Once that had proven itself, we built RAG over user data. In that version, users fetch and filter their own data through a chatbot, and they can generate new data from what they already have.

Why most tutorials don’t prepare you

Most RAG tutorials follow the same recipe: load documents, split them into chunks, embed them, store them in a vector database, retrieve the top results and put them in the prompt. That works on demo datasets and falls apart on real documents.

Fixed-size chunks split paragraphs in the middle of a sentence. Embeddings miss domain terms. Top-K retrieval can return several slightly different versions of the same information while missing the actual answer. And the prompt is often an afterthought. All of this is fixable, but you have to know where to look.

Chunking that preserves meaning

The real problem I ran into was tokenizing and context. Once data is split into chunks, a chunk can lose the context of what it actually is. A paragraph or a row of values means little on its own if nothing in it says which record or section it came from, and the embedding has nothing to anchor it. The fix is to keep that context with every chunk.

For the splitting itself, fixed-size chunks are easy to implement and poor in practice. I prefer a two-pass approach. First split on structural boundaries such as headings and paragraphs, then merge small pieces and split large ones on sentence boundaries. Each chunk then gets a short header saying what it belongs to, and that header is part of the text that gets embedded.

Python
import re
from dataclasses import dataclass, field
 
 
@dataclass
class Chunk:
    content: str
    token_count: int
    context: str = ""
    metadata: dict = field(default_factory=dict)
 
    @property
    def embedding_text(self) -> str:
        # Embed the context header together with the content,
        # so the chunk still says what it is about on its own
        return f"{self.context}\n\n{self.content}" if self.context else self.content
 
 
def estimate_tokens(text: str) -> int:
    # Rough estimate; use your embedding model's tokenizer for exact counts
    return int(len(text.split()) * 1.3)
 
 
def semantic_chunk(
    text: str,
    context: str = "",
    max_tokens: int = 512,
    min_tokens: int = 100,
) -> list[Chunk]:
    # context is a short header such as "Source: user-guide.md | Section: Billing"
    def _make_chunk(body: str) -> Chunk:
        return Chunk(
            content=body,
            token_count=estimate_tokens(body),
            context=context,
            metadata={"context": context},
        )
 
    sections = re.split(r"\n#{1,3}\s|\n\n", text)
    sections = [s.strip() for s in sections if s.strip()]
 
    chunks: list[Chunk] = []
    buffer = ""
 
    for section in sections:
        if estimate_tokens(section) > max_tokens:
            # Too big: flush what we have, then split this section on sentences
            if buffer:
                chunks.append(_make_chunk(buffer))
                buffer = ""
            sentences = re.split(r"(?<=[.!?])\s+", section)
            current = ""
            for sent in sentences:
                candidate = f"{current} {sent}".strip()
                if current and estimate_tokens(candidate) > max_tokens:
                    chunks.append(_make_chunk(current))
                    current = sent
                else:
                    current = candidate
            buffer = current
        elif estimate_tokens(f"{buffer}\n\n{section}") > max_tokens:
            if buffer:
                chunks.append(_make_chunk(buffer))
            buffer = section
        else:
            buffer = f"{buffer}\n\n{section}".strip()
 
    if buffer:
        chunks.append(_make_chunk(buffer))
 
    return [c for c in chunks if c.token_count >= min_tokens]

A 512-token maximum fits comfortably within embedding model limits, and a 100-token minimum keeps out useless fragments. It also helps to overlap consecutive chunks by a sentence or two, so that a chunk mentioning “a sharp increase” still includes what actually increased.

Attach metadata to every chunk as well, such as the source, section title and page number. The header gives the embedding context, and the metadata lets you trace an answer back to where it came from. Without it you get answers with no provenance.

Vector search with pgvector

Our stack is PostgreSQL with the pgvector extension and OpenAI embeddings, rather than a dedicated vector database. The reasoning is simple: if you already run PostgreSQL, a separate vector database is one more service to deploy and pay for, and for moderate volumes pgvector handles the job.

SQL
CREATE EXTENSION IF NOT EXISTS vector;
 
CREATE TABLE chunks (
    id SERIAL PRIMARY KEY,
    document_id INTEGER REFERENCES documents(id) ON DELETE CASCADE,
    content TEXT NOT NULL,
    content_hash TEXT UNIQUE NOT NULL,
    embedding vector(1536),  -- match your embedding model's dimensions
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMPTZ DEFAULT NOW()
);
 
CREATE INDEX chunks_embedding_idx ON chunks USING hnsw (embedding vector_cosine_ops);

I’d use an HNSW index rather than IVFFlat. It doesn’t need retuning as the data grows, and it usually gives better recall and speed.

It’s worth retrieving more candidates than you plan to use, for example 10 results when only 3 to 5 go into the final prompt. Vector similarity is a rough first pass, and re-ranking is where the real filtering happens.

Re-ranking

This is probably the biggest quality improvement that most pipelines skip.

Vector search finds chunks that are semantically similar to the query, but being similar and actually answering the question are different things. A chunk that describes the problem in similar words can rank above the chunk that contains the solution.

A cross-encoder takes the query and each candidate chunk as a pair and scores how well the chunk answers the query:

Python
from sentence_transformers import CrossEncoder
 
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
 
def rerank(query: str, results: list, top_n: int = 5) -> list:
    pairs = [(query, r.content) for r in results]
    scores = reranker.predict(pairs)
    for result, score in zip(results, scores):
        result.score = float(score)
    return sorted(results, key=lambda r: r.score, reverse=True)[:top_n]

Re-ranking tends to help most with ambiguous queries, where the user’s wording doesn’t match the terminology in the documents.

Prompt structure

Don’t just concatenate every chunk into one large block of context. The model loses track of which chunk says what, and longer context makes hallucination more likely.

Python
def build_prompt(query: str, results: list) -> list[dict]:
    context_blocks = []
    for i, r in enumerate(results, 1):
        source = r.metadata.get("source", "unknown")
        context_blocks.append(f"[Source {i}] ({source})\n{r.content}")
 
    context = "\n\n---\n\n".join(context_blocks)
 
    return [
        {
            "role": "system",
            "content": (
                "Answer based on the provided context only. "
                "Reference sources by number. "
                "If the context doesn't have enough information, say so."
            ),
        },
        {"role": "user", "content": f"Context:\n{context}\n\n---\nQuestion: {query}"},
    ]

Numbering the sources lets the model cite them, and separators help it keep chunks apart. The explicit instruction to say when the context isn’t enough matters most. Without it, the model tends to fill gaps from partial context.

Where it still breaks

The most frustrating failure is when the right chunks are retrieved and the answer is still wrong. The usual cause is contradictory documents, or outdated documents mixed in with current ones. Adding dates to chunk metadata and telling the model to prefer recent sources helps.

Questions whose answer is spread across two chunks are also hard. Sentence overlap helps. For complex documents, parent-child chunking is worth trying: you store a large parent chunk along with smaller children, search against the children, and put the parent into the context.

Users also don’t use the same words as your documents. They say “cancel my account” when the docs say “account deactivation process”. Query expansion, where a fast model generates alternative phrasings before the search, catches many of these mismatches.

What I’d do differently today

If I were building this again, I’d go further on the context problem. A fixed header says where a chunk comes from, but not how it fits into the rest of the document. With contextual retrieval, a cheap model writes a sentence or two for each chunk explaining its place in the whole document, and that text is embedded along with the chunk. It adds one model call per chunk at indexing time, and it goes straight at the problem I hit.

I’d also use hybrid search from the start, running a keyword search such as BM25 alongside the vector search and merging the results. Embeddings are weak on exact terms, codes and abbreviations, and keyword search catches those.

For the user-data chatbot, I’d treat fetching and filtering structured data as a job for tool calls or generated SQL rather than embeddings. When a user asks for records that match certain conditions, a query answers that exactly. Vector search is still the right tool for free text.

And I’d keep a small, fixed set of real questions with known answers, and run it after every change to chunking, prompts or models. Without it, it’s hard to tell whether a change actually helped.

What to expect

In general, factual questions with clear answers work well. Combining information from several sources works reasonably but is less reliable. Complex reasoning across many documents is much weaker and probably needs something like a knowledge graph, not just RAG.

Re-ranking adds latency and every query has a cost, so measure both from the start rather than guessing.

If you’re starting from scratch, put most of your effort into chunking and make sure every chunk carries its context, because bad chunks hurt everything downstream. Use pgvector if you already run PostgreSQL, add re-ranking early, and structure prompts with numbered sources. In production, log each query with its retrieval scores and review the lowest-scoring ones regularly. They show you where the pipeline is failing in ways an eval set won’t.

Comments

Loading comments