RAG That Actually Works: Lessons From a File-Based Retrieval System

The chatbot on this site answers questions about me, and it never makes things up — because it isn't allowed to answer from the model's memory. It answers from my content, retrieved at query time. That's retrieval-augmented generation, and building it taught me that the hard part is never the LLM call.

I built it deliberately small: no vector database, just a JSON file of embeddings and cosine similarity in memory. That constraint forced me to understand every moving part.

The whole pipeline in one breath

At build time, I chunk my knowledge sources and embed each chunk into a vector. At query time, I embed the visitor's question, find the closest chunks by cosine similarity, and hand those chunks to the model as the only context it's allowed to use.

[knowledge files] → chunk → embed → embeddings.json   (build time)
[user question]   → embed → cosine top-k → prompt → LLM  (request time)

No database, no network hop to a vector store. For a corpus this size, an array and a dot product are faster than any hosted index — and there's nothing to provision.

Chunking is where quality is won or lost

The instinct is to dump whole documents in. Don't. Embeddings represent meaning, and a 2,000-word document has too many meanings to compress into one vector — the signal averages out to mush.

I chunk by semantic unit — a section, an experience entry, a project write-up — so each vector stands for one coherent idea. Rules that held up:

• One idea per chunk. A chunk about a job shouldn't also contain my tech stack.
• Keep some context in the text. A chunk that says "Built it with NestJS and Redis" is useless without naming what "it" is. I prepend a short label.
• Don't over-shrink. Tiny chunks retrieve precisely but lose the surrounding context the model needs to answer well.

Embeddings and the one config that matters

An embedding model maps text to a vector where distance means dissimilarity. The subtlety most tutorials skip: many embedding APIs distinguish the task. You embed stored documents one way and the search query another.

const res = await ai.models.embedContent({
  model: EMBED_MODEL,
  contents: chunk.text,
  config: { taskType: "RETRIEVAL_DOCUMENT", outputDimensionality: 768 },
});

At query time I switch that to RETRIEVAL_QUERY. Same model, different mode — and it measurably improves which chunks come back. Get this wrong and retrieval quietly degrades with no error to tell you.

Retrieval is just cosine similarity

Once everything is a vector, finding relevant chunks is embarrassingly simple: score each stored vector against the query vector and take the top few.

export function retrieve(query: number[], k: number) {
  return index
    .map((chunk) => ({ chunk, score: cosine(query, chunk.vector) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k)
    .map(({ chunk }) => chunk);
}

Cosine similarity measures the angle between vectors, ignoring magnitude — which is what you want, since it's direction (meaning) that matters, not vector length.

Grounding: the prompt does the discipline

Retrieval finds the right context. The system prompt is what stops the model from ignoring it. Mine says, in effect: answer only from the context below; if it isn't there, say you don't know and point to the contact page; never invent facts.

The fallback that keeps it alive

The index can be empty — before ingestion runs, or if embedding fails at request time. So the system degrades instead of breaking: when there's no index or retrieval errors out, it falls back to a summary built from structured content. Less precise, still grounded, never a 500.

What I'd tell my past self

RAG isn't a model problem, it's a data problem. The LLM call is three lines. The quality lives entirely in how you chunk, how you embed, and how strictly you ground. Get those right and even a JSON file and a dot product will out-answer a sloppy pipeline sitting on an expensive vector database.