How to build a company knowledge assistant that cites its sources

What is a RAG knowledge assistant, in plain terms?

A RAG knowledge assistant answers a question in 2 moves. First it searches your approved documents for the passages most likely to hold the answer. Then a language model writes an answer from those passages only, with links back to them. Your documents are the source of truth; the model's general knowledge is not.

The technique is called retrieval-augmented generation, a term from a 2020 research paper by Lewis and colleagues (opens in a new tab). For a business, the useful part is simple: answers can be checked, because every claim points at a paragraph someone in your company wrote.

A working assistant has more parts than the demo videos suggest:

Common first uses: policy and HR questions, product and pricing specs for sales, past proposals and bids, support macros and troubleshooting guides. Our support knowledge assistant and contract review assistant blueprints show 2 of them end to end.

When does RAG beat fine-tuning or a long prompt?

Use RAG when answers depend on facts that change, when people need to see sources, or when different people may see different documents. Fine-tuning teaches a model a style or a narrow task, and it is a poor way to load facts. If all your content is small, skip retrieval and put it in the prompt.

The size threshold is lower than most people expect. In its contextual retrieval write-up (opens in a new tab) (September 2024), Anthropic suggests that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt, with prompt caching keeping the cost down. Long prompts have a catch, though. Research by Liu and colleagues (opens in a new tab) found that models used information at the start or end of a long context more reliably than information buried in the middle. Above a few hundred pages, or with any permissions, retrieval wins.

Counting and comparison questions are a separate case. "Which suppliers raised prices this year?" is a database or reporting question. RAG retrieves a handful of passages, so it cannot see every row. Put that data in a table and query it, or give an agent a reporting tool.

How do you prepare documents for RAG?

Decide which documents are allowed to answer, remove or label the outdated ones, and give each source an owner and a review date. This is the main schedule risk in every RAG build. In our 6-week plan, the first 2 weeks are mostly documents.

An assistant can only be as current as the documents behind it. If the shared drive holds 3 versions of the travel policy, the assistant will cite whichever one retrieval finds first. Our preparation checklist:

1. Choose sources that have an owner. A folder nobody maintains will give confident, outdated answers. Leave it out until someone owns it.
2. Kill duplicates and old versions. Mark one current version of each document, and archive or exclude the rest. File names such as "policy v3 FINAL (2)" are the warning sign.
3. Add metadata. At minimum: owner, last reviewed date, department, audience and permission group. The assistant uses dates to prefer newer passages and shows them next to citations.
4. Fix formats. Scanned PDFs need OCR. Tables need to survive parsing. Slide decks with the text inside images need a text version or exclusion.
5. Find the gaps. List the 50 questions staff ask most, and check that each has a document that answers it. Missing answers become a writing task for the owner, not a model problem.
6. Set a review cadence. Quarterly for policies, monthly for pricing and product facts, on every release for support content.

None of this needs AI expertise. It needs a person with authority to say which version is true. An AI readiness audit measures this before anyone commits to a build.

How does retrieval work: chunking, embeddings and hybrid search?

Documents are split into passages (chunking), each passage gets an embedding so text with similar meaning can be found, and each question is matched on both meaning and exact keywords (hybrid search). A reranker then puts the strongest candidates first.

Chunking decides what the model sees, so it deserves more care than it usually gets. Split on the document's own structure (headings, sections, clauses) rather than every 500 characters. Keep the heading path with each passage, so a chunk from the international section of the travel policy still says which policy and section it came from. Keep tables whole.

Embeddings let "maternity leave" find a section titled "parental leave". They are weak on exact strings: product codes, policy numbers, error messages and people's names. Keyword search (BM25 is the usual algorithm) catches those. Running both and merging the results is hybrid search, and we use it by default.

Anthropic's contextual retrieval experiments put numbers on these choices. On their test sets, adding a short description of each chunk's context before embedding cut the top-20 retrieval failure rate by 35%. Combining that with contextual keyword search cut it by 49%, and adding a reranker on top cut it by 67%. Your documents will produce different numbers, which is why we measure on your own question bank. Hybrid search plus reranking is still our default over plain vector search.

Finally, retrieve more than you show. Pull 20 to 50 candidates, rerank, and pass the best 5 to 15 passages to the model, strongest first.

How do you stop the assistant showing people documents they should not see?

Enforce permissions at retrieval. Each passage carries the access groups of its source document, and the search filters by the asking person's groups before the model sees anything. Instructions in a prompt ("do not reveal HR files") are not access control.

The controls we put in every build:

• Single sign-on. People ask through Slack, Teams or a page behind company login, so the assistant knows who is asking.
• Permission sync. Access lists are copied from SharePoint, Google Drive or Confluence with every index refresh. When someone loses access to a folder, their results change within the sync interval, and that interval is written into the runbook.
• Separate or excluded collections. HR cases, board papers and legal matters either get their own index with a narrow audience or stay out.
• Read-only by default. A knowledge assistant answers. It does not send email or update records, which removes most of the damage a bad answer can do.
• Defense against injected instructions. Documents and web pages can contain text written to steer a model. The OWASP Gen AI Security Project lists this as indirect prompt injection (opens in a new tab), its top risk for LLM applications. Retrieved text is treated as data, never as instructions, and the assistant has no tool that can send data out. See prompt injection.
• Log retention. Questions can be sensitive too. Set how long logs are kept and who can read them.

How do you test answers before launch?

Build a test set of real questions with known answers and sources, run it before launch and after every change, and grade each answer for correctness, grounding, citation and refusal. Agree the pass mark with the content owner before the first run, so the bar does not move to fit the results.

We collect about 100 questions (50 to 150 depending on scope) from help desk tickets, Slack threads and email, with the owner writing the reference answer and the source for each. Around 15 should have no answer in the documents, because a good assistant says so. Add a few permission tests: questions asked as a user who should not see the source.

Test retrieval separately from answers. If the right passage was not in the top 10, no prompt will fix the answer, and you know to work on chunking or search. If the passage was there and the answer was still wrong, the fix is in the prompt or the model.

A model can grade answers at scale, a practice usually called LLM-as-judge, but people should spot-check its grades every week. The LLM evaluation entry covers the methods. After the test set passes, run a pilot with 10 to 20 real users and a thumbs up or down on every answer, and read every thumbs down.

What does a RAG assistant cost to build and run?

Our RAG knowledge systems work costs $15,000 to $75,000 to build. Running costs are hosting, the search index, model usage and someone keeping documents current. We pass infrastructure and model usage through at cost.

For an outside reference point, RaftLabs' RAG cost guide (opens in a new tab) (updated August 2025) puts simple document Q&A at $12,000 to $30,000, production multi-source systems at $30,000 to $60,000 and enterprise builds at $70,000 to $120,000 or more, with infrastructure at $300 to $2,500 a month. Our bands sit in the same range.

What pushes the price up: scanned documents, more than 3 or 4 sources, permissions that vary below the folder level, several languages, and any request for the assistant to take actions (at which point it becomes an agent, priced as custom AI agents).

After launch, someone has to watch answer quality, review thumbs-down answers and re-run the test set when documents or models change. You can do that in-house with the runbook we hand over, or through AI ops monitoring at $1,500 to $5,000 a month with tooling at cost.

What does a 6-week rollout look like?

Six weeks is realistic for one well-kept source and one interface, in the middle of the 5 to 7 week range. Several sources with permission sync take 8 to 12 weeks, mostly because permission mapping and expert review of test answers take longer. The order matters more than the speed: questions first, documents second, retrieval third, answers fourth.

You see working software in a demo every week, and everything runs on your accounts: the index, the model keys, the code and the prompts.

When should you not build a RAG assistant?

Skip a custom build when the content is small, the documents are wrong, or the questions are really database queries. We would rather tell you that on a scoping call than build something nobody trusts.

• Your content is small. Under about 500 pages, a long prompt or the assistant built into a suite you already pay for (Microsoft 365 Copilot, Gemini in Google Workspace or Notion AI, for example) may be enough. Test that first.
• Nobody owns the documents. If no one can say which version of a policy is true, the assistant will spread the confusion faster. Fix ownership first.
• The questions are about numbers across many records. Totals, trends and comparisons belong in reporting tools.
• Answers carry legal or clinical weight without review. An assistant can find the clause or the guideline. A qualified person should make the call.
• Few people will ask. A handful of questions a week does not justify $15,000. A well-organized intranet page might.

If your case passes these tests, the build is mostly careful document work followed by disciplined testing. On a 30-minute scoping call we can tell you which scope band you are in.