AI Chatbot With Knowledge Base: How to Build One

Three things get called "chatbot" and they behave very differently:

• Scripted bots follow a decision tree. If the user's question isn't on the tree, they fail.
• Plain AI chat (ChatGPT dropped onto a page) answers anything, but from what it learned in training. It doesn't know your refund policy, your product details, or last week's release notes, and it will guess.
• A knowledge-base chatbot sits in between: an AI model writes the answer, but only after the system finds the relevant passages from your content and puts them in front of the model. When the passages don't exist, a well-built bot says so instead of inventing one.

The technique is called retrieval-augmented generation (RAG). In plain terms: look up the right passages, then have the AI answer from them. It is the standard approach behind Intercom Fin, Zendesk AI agents, Chatbase, and most "train a bot on your docs" products. None of them retrain a model on your data; they look up passages and hand them to the AI.

There are two parts: one that runs when your documents change, one that runs on every question.

Preparing your documents (runs when docs change)

1. Collect sources. Help-center articles, PDFs, product docs, a crawled marketing site, Notion pages, policy documents.
2. Split. Each document is split into passages of a few paragraphs (roughly 300–800 tokens) with some overlap, so what a single answer needs fits in one passage. Splitting on headings works better than splitting at a fixed length.
3. Index. Each passage is turned into a list of numbers that captures its meaning, called an embedding. Passages that mean similar things end up with similar numbers.
4. Store. Those numbers, the original text and details like the source link, title and last-updated date go into a search index: one inside your own database, or a hosted search service like Pinecone.

Answering (runs per question)

1. Index the question the same way.
2. Find the 3–8 passages closest in meaning. Many systems also run a plain keyword search and combine the results, which helps with product names and error codes.
3. Brief the AI: instructions ("answer only from these passages; if they don't cover it, say you don't know"), the passages found, the conversation so far, the question.
4. Write the answer, showing it as it is written, and attach citations back to the passages used.
5. Log the question, the passages found, and the answer. This log is how you find gaps in your docs later.

1. Pick the sources and decide the boundary. Start with the content that already answers most support questions, usually a help center and product docs. Leave out anything you wouldn't want quoted verbatim to a customer.
2. Choose the three pieces. An AI model to write answers (Claude, GPT, or an open model you run yourself), an indexing model (OpenAI, Cohere, or one you run yourself), and a search index (one inside your own database is enough for well under a million passages). Keep them swappable; prices and quality shift every few months.
3. Build the document upload. An admin page to upload files and paste links, a background job that reads, splits and indexes them, and an update step that only re-indexes documents whose content actually changed.
4. Build the chat screen. Replies that appear as they are written, clean formatting, a "Sources" block under every answer, per-user conversation history, and a short snippet to paste so the same bot can live on your site or inside your app.
5. Add guardrails and measurement. A refusal path when retrieval returns nothing relevant, an escalation button that creates a ticket with the transcript attached, cost tracking per conversation, and a fixed test set of 30–50 real questions you re-run whenever you change how documents are split, the instructions, or the models.

Steps 3–5 are where the time goes. The prompt at the top of this page generates that shape as a real app, with document upload, the search index, live chat replies and citations already set up, so the work left is your content and your instructions to the AI.

There are three realistic routes. Which one is right depends on how you pay and what you need to own.

Buy if your support team already lives in Intercom or Zendesk, volume is low enough that per-resolution pricing doesn't hurt, and you don't need the bot anywhere except the help widget.

Build or generate if per-resolution pricing scales badly for you, the bot needs to sit inside your own product with your logins and your design, your data has to stay in your database, or you want to change the model when a better or cheaper one ships.

The recurring cost is model usage, and it's small compared to per-resolution pricing.

• Indexing is a one-time cost per document, on the order of a hundredth of a cent per passage. A 100-page documentation set costs cents to index, and re-indexing only what changed keeps it that way.
• Answers cost roughly $0.01–$0.05 per conversation on a small model such as Claude Haiku or GPT-4o-mini for typical support-length exchanges. A thousand conversations a month is $10–$50 in AI usage.
• Hosting is a database with a search index, plus the app itself. On Supabase and Vercel that starts free and stays under typical SaaS seat pricing for a long time.

Compare that with $0.99 per resolved conversation on Intercom Fin: 1,000 resolutions is $990 a month before the platform subscription. Cost is rarely the reason to buy; speed and not having to own anything is.

• No refusal path. If the instructions don't tell the AI to decline when the passages found are empty or off-topic, it will answer anyway, and it will be confidently wrong.
• No citations. Users trust an answer that links to the doc it came from and distrust one that doesn't. Citations also make bad answers easy to trace.
• Passages that are too big or too small. Too big and the relevant sentence is buried in noise; too small and the answer is split across passages that don't all get found. Split on document structure, and test.
• A stale index. Docs change; the search index doesn't know unless something re-indexes them. Re-index when a document changes, not on a fixed schedule.
• Mixing customers' documents. If the bot serves multiple customers or workspaces, every search has to stay within that customer's documents. Forgetting this leaks one customer's documents into another's answers.
• No test set. Without 30–50 fixed real questions and expected answers, every change to how documents are split or to the instructions is a guess. Run the set on every change.

The bot is only as good as what it can retrieve, and some sources retrieve much better than others.

• Works well: help-center articles, product documentation, policy pages, FAQs, release notes, anything already written to answer a question.
• Needs preparation: long PDFs (split by section and keep headings), spreadsheets (add a row-per-record description), and marketing pages (strip menus and repeated page text before splitting).
• Usually not worth it: scanned PDFs without text recognition, Slack or email threads where the answer is spread across twenty messages, and internal wikis nobody has updated in a year. The bot will cite them, and the citation will be wrong.

A useful starting rule: if a human support agent would paste a link to it, add it. If they'd say "let me check with someone," it isn't ready to be a source.