Turns Out · November 7, 2026 · 12 min read

RAG chatbot explained: how it works, what it costs, and when it beats fine-tuning

How retrieval-augmented generation works, what a RAG chatbot costs to run, where it fails, and when it beats fine-tuning for a business website.

Every pitch for an AI assistant on a business website contains the sentence “we’ll train the AI on your data.” It means one of three very different things, and the price tags differ by two orders of magnitude. A RAG chatbot does not train anything. Retrieval-augmented generation (RAG) looks up the relevant pieces of your own content at the moment a question is asked, hands those pieces to a language model, and has the model write an answer from them alone.

That design is why a RAG chatbot can answer about last Tuesday’s price change without anyone retraining a model, and why it can say “I don’t know” when the answer is not in your content. It is also why most of the engineering effort lands in places the vendor demo never shows. We build these for client sites and run one on our own, so here is what we have learned about how it works, what it costs to operate, and where fine-tuning or a giant prompt is the better call.

How a RAG chatbot works, stage by stage

The term comes from a 2020 paper by Lewis and colleagues, who described RAG as models that “combine pre-trained parametric and non-parametric memory for language generation.” In plain words: the model’s general knowledge plus a searchable store of your documents. A production RAG chatbot is a five-stage pipeline, and each stage has its own way of going wrong.

• —Ingestion — pull the content from where it lives: CMS entries, help-center articles, PDFs, product data. Keep the source URL and a last-modified date with every record; you will need both later.
• —Chunking — split each document into passages small enough to be retrieved on their own (a common range is 200–600 tokens), each with the heading it sat under attached. A chunk that reads sensibly without its neighbors is a good chunk.
• —Embeddings — run every chunk through an embedding model, which turns text into a list of numbers that encodes its meaning; two passages about the same thing end up with similar numbers. Store the numbers in a vector database, which is any store that can find the nearest neighbors of a given vector quickly.
• —Retrieval — when a question arrives, embed it the same way, fetch the closest chunks (typically 5 to 10), and often re-rank them with a second, more accurate model.
• —Generation — put the retrieved chunks, the question and a system prompt into the language model, and ask for an answer that cites the chunks it used and declines if they do not contain one.

The vector database sounds like the exotic part. It is not. Postgres with the pgvector extension does nearest-neighbor search with HNSW or IVFFlat indexes, and for a knowledge base of a few thousand chunks the query looks like this:

SELECT url, heading, body
FROM chunks
ORDER BY embedding <=> $1   -- cosine distance to the question vector
LIMIT 8;

Everything else is ordinary web engineering: a queue that re-embeds a page when an editor saves it, a cache in front of the model, logs you can read, and a human handoff when confidence is low.

What a RAG chatbot costs to run

Four drivers set the monthly bill: model tokens, embedding refresh, hosting for the index, and evaluation. People budget for the first and get surprised by the fourth.

Start with tokens, because the numbers are smaller than most people expect. A typical RAG answer sends about 3,000 input tokens (system prompt, eight chunks, a little conversation history, the question) and gets back about 250 output tokens. At the prices published in November 2026, that one answer costs:

• —Small models — Claude Haiku 5.5 is $0.10 per million input tokens and $0.50 per million output tokens; OpenAI’s gpt-6-luna is priced the same. One answer is roughly $0.0004. 10,000 answers a month: about $4.
• —Mid-tier models — gpt-6.1-terra at $1 and $5 per million lands near $0.004 per answer, or about $40 for 10,000.
• —Frontier models — Claude Sonnet 5.5 at $2 and $10 per million is about $0.0085 per answer, or $85 for 10,000. Prompt caching cuts the repeated system prompt to a twentieth of the input price on Sonnet 5.5, so the real figure is lower.

Embedding refresh is cheaper still. A 500-page site is roughly 400,000 tokens. Voyage’s voyage-4 model is $0.06 per million tokens, so the first full index costs about $0.02, and re-embedding the whole site every week for a year is under $1.50. Even on a model priced at ten times that, the line is noise. The reindex cost is not the tokens; it is the developer who wires the CMS save hook to the re-embed queue and the test that proves it fired, which is why it sits in the stale-index section below rather than on this invoice.

Hosting the index is where the first fixed line appears, and it is still modest. Pinecone’s Starter plan is free up to 2 GB and 1 million read units a month, the Builder plan is $20 a month flat, and Standard starts at a $50 monthly minimum. Or use the Postgres you already pay for: pgvector is open source and handles a few hundred thousand chunks without drama. “RAG as a service” products bundle all of this for a per-seat or per-query fee; they are fine for a pilot, and the lock-in question comes later.

Evaluation is the budget line nobody mentions in the pitch. To know whether the chatbot answers correctly, you need a set of real questions with known-good answers, run after every content change and every prompt edit. Writing that set is human work: 50 to 100 questions, a few days of a subject-matter expert’s time, and a developer to wire the run into deployment. On the builds we scope, that work costs more than the first year of tokens.

RAG vs fine-tuning vs long context

The three ways to give a model your knowledge solve different problems, and a vendor who offers you the wrong one usually has a reason that is not yours.

• —RAG — best when the content changes, when you need the answer to cite its source, and when “I don’t know” is an acceptable answer. Cost scales with questions asked, not with the size of the knowledge base. Weakness: it only knows what the retriever found, so a bad chunk or a stale index produces a confident wrong answer.
• —Fine-tuning — training a copy of a model on your examples. It teaches style, format and task behavior well: your tone of voice, a strict JSON output, a classifier for support tickets. It teaches facts badly. Ovadia and colleagues, in a paper first posted in December 2023, found that “RAG consistently outperforms it” for facts the model saw in training and for new ones alike. A fine-tuned model also goes stale the day you change a price, and retraining is a project, not a save button.
• —Long context — skip retrieval and put the whole knowledge base in every prompt. Claude models from 4.6 onward (except Haiku 5.5) take a 1 million token context at standard pricing, so a 400,000-token site fits. It costs $0.80 per question on Sonnet 5.5 uncached, or about $0.04 with a cache hit — five to a hundred times the RAG figure, every single question. Accuracy is the second problem: Liu and colleagues showed that model performance is often “highest when relevant information occurs at the beginning or end of the input context” and falls off sharply when it sits in the middle, which is where most of your content will be.

The market has noticed the fine-tuning problem. OpenAI’s pricing page now says it is “winding down the fine-tuning platform” and is closed to new users, with the last listed training rates at $25 per million tokens for gpt-4.1 and $5 for gpt-4.1-mini. Fine-tuning is not dead, but for a business website it has become the specialist’s tool, not the default.

Our rule for client sites: RAG for knowledge, fine-tuning for behavior, long context for one-off analysis of a document someone just uploaded. The combinations are legitimate too. A fine-tuned small model that answers in your house style from RAG-retrieved chunks is a sensible production setup once the plain version has proven the content is good enough.

Where RAG chatbots fail, and how to test for each

Each RAG chatbot we have audited so far failed in at least one of four ways, and each has a test that takes an afternoon to set up. The Ragas evaluation library names four metrics (faithfulness, response relevancy, context precision and context recall); the discipline is running them. The two that catch most failures are in the list below.

• —Stale index — the editor changed the returns policy; the bot still quotes the old one. Test: pick 20 pages, edit one line on each, ask the bot about each change within an hour. If the reindex is nightly, you will find out. Fix: re-embed on save, triggered by the CMS webhook, and store the last-modified date with the chunk so the prompt can say when the information is from.
• —Bad chunking — the answer is on the page, but the chunk that was retrieved cut it in half, or lost the heading that said which product it applied to. Test: Ragas context recall, the share of facts in a known-good answer that the retrieved chunks actually contained. Anything below about 0.8 on your own question set usually means the problem is chunking or retrieval, not the model. Fix: chunk by heading, not by character count, and prepend the page title and section to every chunk.
• —Hallucinated citations — the bot cites a URL that does not exist or does not say that. Test: Ragas faithfulness, the share of claims in the answer supported by the retrieved context, plus a script that resolves every cited URL. Fix: only allow citations to the chunks that were retrieved, by ID, and render them from your own data rather than letting the model type a URL.
• —Refusing to say “I don’t know” — the retriever found nothing useful and the model answered anyway. Test: 20 questions that are deliberately outside the content (“do you offer financing?” when you do not). The pass rate should be 20 of 20. Fix: a confidence threshold on retrieval similarity below which the bot declines and offers a human.

Run the whole set after every change, as part of deployment, not as a quarterly review. A chatbot that passed in March and has had 40 content updates since is an untested chatbot.

What we would do differently

Our first builds chunked by character count because every tutorial did. The retrieval scores looked fine and the answers were subtly wrong, because a 500-character window does not know which plan’s pricing it is quoting. Chunking by heading, with the breadcrumb attached, fixed more than any model upgrade.

We also used to start with the architecture. We now start with the 50 real questions, pulled from support inboxes and site search logs, before writing a line of code. Half the time that list shows the content itself is the problem: the answer exists only in someone’s head, or in three pages that contradict each other. AI on messy content automates the mess, which is the same finding our free AI readiness audit reports more often than clients expect.

Three things we advise against for a first build:

• —Fine-tuning before RAG works — you will spend the budget teaching a model to sound right before it can be right.
• —Multiple agents before one agent is measured — orchestration multiplies every failure mode above; earn it.
• —“Chat with your PDF” as the product — if nobody asked those questions before, a chatbot will not make them.

And one obligation that is not optional: in the EU, since August 2026, a chatbot talking to the public has to make clear it is an AI. The dates and exceptions are in our EU AI Act Article 50 guide; confirm what applies to you with your lawyer.

Questions readers ask

Is a RAG chatbot the same as uploading files to ChatGPT?

Same idea, different control. Consumer tools retrieve from the files you upload, but you do not choose the chunking, cannot see what was retrieved, cannot wire the bot into your site or CRM, and cannot test it against a fixed question set. A chatbot built for a business website is that mechanism with the dials exposed and the results logged.

Do I need a vector database for a RAG chatbot?

You need vector search; you do not necessarily need a separate product. Postgres with pgvector covers most company sites. A dedicated vector database earns its cost at millions of chunks, heavy metadata filtering, or when the ops team does not want another index to tune.

How long does RAG development take?

For a site with clean, structured content, a first version that answers from the site and hands off to a human is a matter of weeks, and most of that is content mapping and the evaluation set. Messy content adds months, and that work belongs to the content team, not the developer.

Does an AI chatbot for a website stop hallucinating with RAG?

It stops inventing facts about your business if the retrieval is good and the prompt forbids answering beyond the retrieved chunks. It does not stop a model from misreading a chunk, which is why the faithfulness test exists and why the bot should cite what it used.

The live example is our own agent: it states company facts only from retrieved site content, and when nothing relevant is retrieved it says so and hands the question to the team instead of inventing an answer; the knowledge base it reads is checked against the published pages before every reindex. If you are weighing one for your own site, our AI agents service page explains how we scope one, including the parts that happen before any code.