# How to build a company knowledge assistant that cites its sources

> A company knowledge assistant uses retrieval-augmented generation (RAG): it searches your approved documents, hands the best passages to a language model, and answers with links to the passages it used. Most of the work is document preparation, permissions and testing, not the model. Plan on 5 to 7 weeks for one well-kept source, 8 to 12 weeks for several sources with permissions, and $15,000 to $75,000 to build, with hosting and model usage on top.

- Page: https://methodcrew.com/insights/build-rag-knowledge-assistant
- Published: 2026-10-28
- Last updated: 2026-10-28
- Publisher: [The Method Crew](https://methodcrew.com/)
- Category: AI systems
- Reading time: 10 minutes
- Question answered: AI knowledge base for company documents

## Key takeaways

- RAG answers from your documents and shows its sources. Fine-tuning changes how a model writes, not reliably what it knows.
- Document clean-up is the main schedule risk. Budget the first 2 weeks of a 6-week build for it.
- Use hybrid search (meaning plus exact keywords) and a reranker. Plain vector search misses product codes, policy numbers and names.
- Enforce permissions at retrieval time, before the model sees anything. A prompt is not an access control.
- Write a test set of about 100 real questions before you build, and agree the pass mark before the first run.

## What is a RAG knowledge assistant, in plain terms?

A RAG knowledge assistant answers a question in 2 moves. First it searches your approved documents for the passages most likely to hold the answer. Then a language model writes an answer from those passages only, with links back to them. Your documents are the source of truth; the model's general knowledge is not.

The technique is called [retrieval-augmented generation](https://methodcrew.com/glossary/retrieval-augmented-generation), a term from a [2020 research paper by Lewis and colleagues](https://arxiv.org/abs/2005.11401). For a business, the useful part is simple: answers can be checked, because every claim points at a paragraph someone in your company wrote.

A working assistant has more parts than the demo videos suggest:

| Part | What it does | Common choices |
|---|---|---|
| Connectors | Pull documents from where they live and keep them in sync | SharePoint, Google Drive, Confluence, Notion, the help desk |
| Parser and chunker | Turn PDFs, pages and slides into clean text split into passages | Document parsers, OCR for scans |
| Embedding model | Turn each passage into a vector so similar meaning can be found | OpenAI or other embedding APIs |
| Index | Store vectors and keywords for fast search | Postgres with pgvector, Pinecone, Qdrant, Elasticsearch or OpenSearch |
| Retriever and reranker | Find the best passages for each question and order them | Hybrid search plus a reranking model |
| Answer model and prompt | Write the answer from the passages, cite them, refuse when the answer is missing | OpenAI or Anthropic models |
| Interface | Where people ask | Slack, Microsoft Teams, an intranet page, a help desk sidebar |
| Logs and evaluation | Record questions, answers, sources and feedback | An LLM observability tool or your own tables |

Common first uses: policy and HR questions, product and pricing specs for sales, past proposals and bids, support macros and troubleshooting guides. Our [support knowledge assistant](https://methodcrew.com/blueprints/support-knowledge-assistant) and [contract review assistant](https://methodcrew.com/blueprints/contract-review-assistant) blueprints show 2 of them end to end.

## When does RAG beat fine-tuning or a long prompt?

Use RAG when answers depend on facts that change, when people need to see sources, or when different people may see different documents. Fine-tuning teaches a model a style or a narrow task, and it is a poor way to load facts. If all your content is small, skip retrieval and put it in the prompt.

| Approach | Good for | Weak at | Cost of an update |
|---|---|---|---|
| Whole collection in the prompt | Small, stable collections | Large or permissioned collections; cost per question grows with length | Edit the document |
| RAG | Large, changing collections; citations; per-user permissions | Questions that need every document at once ("how many contracts renew in March?") | Re-index the changed document |
| Fine-tuning | Tone, output format, narrow classification | Fresh facts, citations, permissions | Retrain the model |

The size threshold is lower than most people expect. In its [contextual retrieval write-up](https://www.anthropic.com/news/contextual-retrieval) (September 2024), Anthropic suggests that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt, with prompt caching keeping the cost down. Long prompts have a catch, though. [Research by Liu and colleagues](https://arxiv.org/abs/2307.03172) found that models used information at the start or end of a long context more reliably than information buried in the middle. Above a few hundred pages, or with any permissions, retrieval wins.

Counting and comparison questions are a separate case. "Which suppliers raised prices this year?" is a database or reporting question. RAG retrieves a handful of passages, so it cannot see every row. Put that data in a table and query it, or give an agent a reporting tool.

## How do you prepare documents for RAG?

Decide which documents are allowed to answer, remove or label the outdated ones, and give each source an owner and a review date. This is the main schedule risk in every RAG build. In our 6-week plan, the first 2 weeks are mostly documents.

An assistant can only be as current as the documents behind it. If the shared drive holds 3 versions of the travel policy, the assistant will cite whichever one retrieval finds first. Our preparation checklist:

1. Choose sources that have an owner. A folder nobody maintains will give confident, outdated answers. Leave it out until someone owns it.
2. Kill duplicates and old versions. Mark one current version of each document, and archive or exclude the rest. File names such as "policy v3 FINAL (2)" are the warning sign.
3. Add metadata. At minimum: owner, last reviewed date, department, audience and permission group. The assistant uses dates to prefer newer passages and shows them next to citations.
4. Fix formats. Scanned PDFs need OCR. Tables need to survive parsing. Slide decks with the text inside images need a text version or exclusion.
5. Find the gaps. List the 50 questions staff ask most, and check that each has a document that answers it. Missing answers become a writing task for the owner, not a model problem.
6. Set a review cadence. Quarterly for policies, monthly for pricing and product facts, on every release for support content.

None of this needs AI expertise. It needs a person with authority to say which version is true. An [AI readiness audit](https://methodcrew.com/services/ai-readiness-audit) measures this before anyone commits to a build.

## How does retrieval work: chunking, embeddings and hybrid search?

Documents are split into passages ([chunking](https://methodcrew.com/glossary/chunking)), each passage gets an [embedding](https://methodcrew.com/glossary/embeddings) so text with similar meaning can be found, and each question is matched on both meaning and exact keywords ([hybrid search](https://methodcrew.com/glossary/hybrid-search)). A reranker then puts the strongest candidates first.

Chunking decides what the model sees, so it deserves more care than it usually gets. Split on the document's own structure (headings, sections, clauses) rather than every 500 characters. Keep the heading path with each passage, so a chunk from the international section of the travel policy still says which policy and section it came from. Keep tables whole.

Embeddings let "maternity leave" find a section titled "parental leave". They are weak on exact strings: product codes, policy numbers, error messages and people's names. Keyword search (BM25 is the usual algorithm) catches those. Running both and merging the results is hybrid search, and we use it by default.

Anthropic's contextual retrieval experiments put numbers on these choices. On their test sets, adding a short description of each chunk's context before embedding cut the top-20 retrieval failure rate by 35%. Combining that with contextual keyword search cut it by 49%, and adding a reranker on top cut it by 67%. Your documents will produce different numbers, which is why we measure on your own question bank. Hybrid search plus reranking is still our default over plain [vector search](https://methodcrew.com/glossary/vector-database).

Finally, retrieve more than you show. Pull 20 to 50 candidates, rerank, and pass the best 5 to 15 passages to the model, strongest first.

## How do you stop the assistant showing people documents they should not see?

Enforce permissions at retrieval. Each passage carries the access groups of its source document, and the search filters by the asking person's groups before the model sees anything. Instructions in a prompt ("do not reveal HR files") are not access control.

The controls we put in every build:

- Single sign-on. People ask through Slack, Teams or a page behind company login, so the assistant knows who is asking.
- Permission sync. Access lists are copied from SharePoint, Google Drive or Confluence with every index refresh. When someone loses access to a folder, their results change within the sync interval, and that interval is written into the runbook.
- Separate or excluded collections. HR cases, board papers and legal matters either get their own index with a narrow audience or stay out.
- Read-only by default. A knowledge assistant answers. It does not send email or update records, which removes most of the damage a bad answer can do.
- Defense against injected instructions. Documents and web pages can contain text written to steer a model. The OWASP Gen AI Security Project lists this as [indirect prompt injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/), its top risk for LLM applications. Retrieved text is treated as data, never as instructions, and the assistant has no tool that can send data out. See [prompt injection](https://methodcrew.com/glossary/prompt-injection).
- Log retention. Questions can be sensitive too. Set how long logs are kept and who can read them.

## How do you test answers before launch?

Build a test set of real questions with known answers and sources, run it before launch and after every change, and grade each answer for correctness, grounding, citation and refusal. Agree the pass mark with the content owner before the first run, so the bar does not move to fit the results.

We collect about 100 questions (50 to 150 depending on scope) from help desk tickets, Slack threads and email, with the owner writing the reference answer and the source for each. Around 15 should have no answer in the documents, because a good assistant says so. Add a few permission tests: questions asked as a user who should not see the source.

| Check | Pass means |
|---|---|
| Correct | Matches the reference answer on the facts that matter |
| Grounded | Every claim is supported by a cited passage |
| Cited | Links point at the right document and section |
| Refuses well | Says the answer is not in the documents and names who to ask |
| Permission-safe | Never cites a document the asker cannot open |

Test retrieval separately from answers. If the right passage was not in the top 10, no prompt will fix the answer, and you know to work on chunking or search. If the passage was there and the answer was still wrong, the fix is in the prompt or the model.

A model can grade answers at scale, a practice usually called LLM-as-judge, but people should spot-check its grades every week. The [LLM evaluation](https://methodcrew.com/glossary/llm-evaluation) entry covers the methods. After the test set passes, run a pilot with 10 to 20 real users and a thumbs up or down on every answer, and read every thumbs down.

## What does a RAG assistant cost to build and run?

Our [RAG knowledge systems](https://methodcrew.com/services/rag-knowledge-systems) work costs $15,000 to $75,000 to build. Running costs are hosting, the search index, model usage and someone keeping documents current. We pass infrastructure and model usage through at cost.

| Scope | Build (our pricing) | Typical timeline |
|---|---|---|
| One well-kept source, one interface | $15,000 to $35,000 | 5 to 7 weeks |
| Several sources with permission sync and single sign-on | $35,000 to $75,000 | 8 to 12 weeks |

For an outside reference point, [RaftLabs' RAG cost guide](https://www.raftlabs.com/blog/rag-development-cost) (updated August 2025) puts simple document Q&A at $12,000 to $30,000, production multi-source systems at $30,000 to $60,000 and enterprise builds at $70,000 to $120,000 or more, with infrastructure at $300 to $2,500 a month. Our bands sit in the same range.

What pushes the price up: scanned documents, more than 3 or 4 sources, permissions that vary below the folder level, several languages, and any request for the assistant to take actions (at which point it becomes an agent, priced as [custom AI agents](https://methodcrew.com/services/custom-ai-agents)).

After launch, someone has to watch answer quality, review thumbs-down answers and re-run the test set when documents or models change. You can do that in-house with the runbook we hand over, or through [AI ops monitoring](https://methodcrew.com/services/ai-ops-monitoring) at $1,500 to $5,000 a month with tooling at cost.

## What does a 6-week rollout look like?

Six weeks is realistic for one well-kept source and one interface, in the middle of the 5 to 7 week range. Several sources with permission sync take 8 to 12 weeks, mostly because permission mapping and expert review of test answers take longer. The order matters more than the speed: questions first, documents second, retrieval third, answers fourth.

| Week | Work | What you get |
|---|---|---|
| 1 | Pick sources, collect the question bank, agree the pass mark, set up access on your accounts | Written scope, question bank of about 100 questions |
| 2 | Clean documents, add metadata, run OCR, connect sources | Indexed collection and a gap list for owners |
| 3 | Tune chunking, hybrid search and reranking against the question bank | Retrieval scores per question |
| 4 | Answer prompt, citations, refusals, permission filtering, Slack or Teams interface | A working assistant in a test channel |
| 5 | Pilot with 10 to 20 users, review every thumbs-down, fix gaps | Pilot report listing failures and fixes |
| 6 | Monitoring, runbook, training, owner sign-off | Live assistant, runbook and quality dashboard |

You see working software in a demo every week, and everything runs on your accounts: the index, the model keys, the code and the prompts.

## When should you not build a RAG assistant?

Skip a custom build when the content is small, the documents are wrong, or the questions are really database queries. We would rather tell you that on a scoping call than build something nobody trusts.

- Your content is small. Under about 500 pages, a long prompt or the assistant built into a suite you already pay for (Microsoft 365 Copilot, Gemini in Google Workspace or Notion AI, for example) may be enough. Test that first.
- Nobody owns the documents. If no one can say which version of a policy is true, the assistant will spread the confusion faster. Fix ownership first.
- The questions are about numbers across many records. Totals, trends and comparisons belong in reporting tools.
- Answers carry legal or clinical weight without review. An assistant can find the clause or the guideline. A qualified person should make the call.
- Few people will ask. A handful of questions a week does not justify $15,000. A well-organized intranet page might.

If your case passes these tests, the build is mostly careful document work followed by disciplined testing. On a 30-minute scoping call we can tell you which scope band you are in.

## Common questions

### How long does it take to build a RAG knowledge assistant?

Plan on 5 to 7 weeks for one well-kept source and one interface such as Slack or Teams, and 8 to 12 weeks for several sources with permission sync and single sign-on. The first 2 weeks are mostly document work: choosing sources, removing old versions and adding owners and dates. Messy or scanned documents extend the schedule more than any technical choice does.

### Do we need a vector database for a knowledge assistant?

Not always a separate one. For most company collections, Postgres with the pgvector extension handles the vectors, and a keyword index handles exact terms for hybrid search. A managed vector database such as Pinecone or Qdrant makes sense for very large collections or heavy query volume. Choose the index after you know the document count and the permission model, not before.

### Can a RAG assistant still make things up?

Yes, less often than a model answering from memory, but it can. The controls are an instruction to answer only from retrieved passages, a citation for every claim, a clear refusal when the answer is missing, and a test set of real questions run before launch and after every change. Monitoring thumbs-down feedback after launch catches the cases the test set missed.

### Is our company data used to train the AI model?

Under the standard commercial API terms of OpenAI and Anthropic, business inputs are not used to train their models by default. Consumer chat apps can have different terms, so the assistant should run on API accounts your company owns. Check the current terms for your chosen provider and plan, set data retention, and record both in the runbook.

### Should we fine-tune a model on our documents instead?

Rarely. Fine-tuning changes how a model writes or classifies, but it is an unreliable way to teach facts, it cannot cite sources, and it cannot respect per-user permissions. Every document change would also mean retraining. RAG keeps facts in your documents, where owners can update them, and shows the reader where each answer came from.

### What does a RAG assistant cost to run each month?

It depends on document volume and question volume. RaftLabs puts infrastructure at $300 to $2,500 a month in its August 2025 guide, and model usage scales with the number of questions. We pass both through at cost with no markup. Optional monitoring and improvement through our AI ops service runs $1,500 to $5,000 a month.

## Services that do this work

- [RAG and knowledge systems](https://methodcrew.com/services/rag-knowledge-systems): Your documents turned into an assistant that cites its sources. $15,000 to $75,000 per project.
- [AI ops monitoring and improvement](https://methodcrew.com/services/ai-ops-monitoring): Quality checks, cost control and model updates for live AI systems. $1,500 to $5,000 per month.
- [Custom AI agents and copilots](https://methodcrew.com/services/custom-ai-agents): Agents that look things up in your CRM or helpdesk and take the next step, with approvals. $15,000 to $90,000 per project.
- [AI readiness audit](https://methodcrew.com/services/ai-readiness-audit): An assessment that ranks your AI opportunities by return and prices the first build. $3,000 to $15,000 per project.

## Terms used in this article

- [Retrieval-augmented generation (RAG)](https://methodcrew.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation is a method where a system first searches your own documents for relevant passages, then gives them to a language model to answer from.
- [Chunking](https://methodcrew.com/glossary/chunking): Chunking is splitting documents into smaller passages before they are embedded and indexed, so a retrieval system can find and return the exact part that answers a question.
- [Embeddings](https://methodcrew.com/glossary/embeddings): Embeddings are lists of numbers that capture the meaning of text or images, so software can measure how closely related one item is to another.
- [Hybrid search](https://methodcrew.com/glossary/hybrid-search): Hybrid search runs keyword search and semantic search on the same query and merges the results, so it catches exact terms and paraphrased meaning together.
- [Vector database](https://methodcrew.com/glossary/vector-database): A vector database stores embeddings, the numeric fingerprints of text or images, and quickly finds the stored items whose meaning is closest to a query.
- [LLM evaluation (evals)](https://methodcrew.com/glossary/llm-evaluation): LLM evaluation scores a language model system's outputs against a fixed set of real test cases, so you can tell whether a change made it better or worse.