Language models are trained on public data. They don't know your policies, products or internal documents, and when they guess they can sound confident and be wrong. Retrieval-Augmented Generation (RAG) fixes this by finding the most relevant pieces of your own content and giving them to the model as context. It's the technology behind most 'chat with your docs' and smart support assistants we build.
1. How RAG works
RAG has two phases. Ingestion happens once (and whenever content changes); retrieval happens on every question.
- Ingest: split documents into small chunks, convert each chunk into an embedding (a list of numbers that captures meaning) and store it.
- Retrieve: convert the user's question into an embedding and find the chunks with the closest meaning.
- Generate: send the question plus the best chunks to the model, with instructions to answer only from them.
- Cite: return the sources alongside the answer so users can verify it.
“A model doesn't know your business. RAG hands it the right pages at the right moment, and asks it to answer only from them.”— Strategic Takeaway
2. Set up the vector store in Postgres
pgvector is a Postgres extension supported by Supabase, Neon, PlanetScale Postgres and most managed providers. It lets you store embeddings next to your normal data and search them with SQL.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
source TEXT NOT NULL, -- e.g. file name or URL, used for citations
content TEXT NOT NULL,
embedding VECTOR(1536) NOT NULL, -- text-embedding-3-small returns 1536 dimensions
created_at TIMESTAMPTZ DEFAULT now()
);
-- HNSW index for fast similarity search
CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);3. Ingest: chunk and embed your content
Chunks of roughly 500–1,000 characters with a small overlap work well for most documents. Overlap prevents an important sentence from being cut in half between two chunks.
npm install openai @neondatabase/serverlessimport OpenAI from "openai";
import { neon } from "@neondatabase/serverless";
const openai = new OpenAI();
const sql = neon(process.env.DATABASE_URL!);
function chunkText(text: string, size = 800, overlap = 150): string[] {
const chunks: string[] = [];
for (let start = 0; start < text.length; start += size - overlap) {
chunks.push(text.slice(start, start + size));
}
return chunks;
}
export async function ingestDocument(source: string, text: string) {
const chunks = chunkText(text);
// Embed many chunks in one request: faster and cheaper
const { data } = await openai.embeddings.create({
model: "text-embedding-3-small",
input: chunks,
});
for (let i = 0; i < chunks.length; i++) {
await sql.query(
"INSERT INTO documents (source, content, embedding) VALUES ($1, $2, $3::vector)",
[source, chunks[i], JSON.stringify(data[i].embedding)]
);
}
console.log(`Ingested ${chunks.length} chunks from ${source}`);
}4. Retrieve and answer
At question time, we embed the question, fetch the five closest chunks with the pgvector cosine distance operator (<=>), and pass them to the model with strict instructions.
import OpenAI from "openai";
import { neon } from "@neondatabase/serverless";
const openai = new OpenAI();
const sql = neon(process.env.DATABASE_URL!);
export async function ask(question: string) {
// 1. Embed the question
const { data } = await openai.embeddings.create({
model: "text-embedding-3-small",
input: question,
});
// 2. Find the most similar chunks
const matches = await sql.query(
`SELECT source, content, 1 - (embedding <=> $1::vector) AS similarity
FROM documents
ORDER BY embedding <=> $1::vector
LIMIT 5`,
[JSON.stringify(data[0].embedding)]
);
const context = matches
.map((m, i) => `[${i + 1}] (${m.source})\n${m.content}`)
.join("\n\n");
// 3. Answer using only the retrieved context
const response = await openai.responses.create({
model: process.env.OPENAI_MODEL!,
instructions:
"Answer the question using ONLY the context provided. Cite sources like [1]. " +
"If the answer is not in the context, say you don't know and suggest contacting support.",
input: `Context:\n${context}\n\nQuestion: ${question}`,
});
return {
answer: response.output_text,
sources: matches.map((m) => m.source),
};
}5. Making RAG production-ready
A working demo takes a day. A reliable assistant needs a few more steps.
- Re-ingest automatically when content changes (CMS webhook or scheduled job).
- Store metadata (product, language, access level) and filter by it in the SQL query, so users only see what they're allowed to.
- Combine vector search with keyword search (hybrid search) for product codes, names and exact terms.
- Set a minimum similarity threshold and fall back to 'I don't know' below it.
- Log questions with no good answer. They show exactly which content is missing.
- Evaluate answers against a set of real questions before every major change.
6. What it costs
Embeddings are inexpensive. text-embedding-3-small costs $0.02 per million tokens (checked November 2026), so embedding hundreds of pages typically costs cents. Most of the running cost comes from the answer-generation model and database hosting, which depends on the model you choose and how many questions users ask.
Official resources
Principles for deliberate execution
- 1RAG = retrieve relevant chunks of your data, then let the model answer using only those chunks.
- 2You don't need a separate vector database: Postgres with pgvector handles most projects.
- 3Chunking and good metadata matter more than the choice of model.
- 4Always return sources, and tell the model to say 'I don't know' when the context has no answer.