RAG for business – how AI answers from company documents and what it costs

An ordinary AI chat answers from what it learned from public texts. It does not know your price list, procedures or supplier contracts. Asked about them, it guesses or answers in generalities.

RAG works like an employee who checks the files before answering. The system first searches the company’s documents for a few passages that match the question. Then the language model – the program that writes the answers – composes an answer from them and points to the source document.

As a result, a new document is available as soon as it is added to the collection. There is no need to retrain the model, and every answer can be checked against its source.

For example, an engineer asks in Microsoft Teams about the permissible dimensional tolerance for a particular product. The assistant finds the right process sheet and gives the value with a link to the document. Our case study of a knowledge assistant for production engineers shows how this works in practice.

RAG pays off when people often look for answers in a large collection of documents that change. When there are few questions or only a handful of documents, a simpler solution is enough.

RAG is often confused with further training a model on your own data (fine-tuning). Fine-tuning changes how the model writes, but it does not guarantee that it will give a current fact. For questions about procedures, price lists and contracts, RAG works better, because you only need to swap the document.

The table below helps you match the solution to the situation.

In Microsoft’s off-the-shelf tools, most of these elements work without your involvement. In a custom-built solution, each one is a separate design decision. Below we describe them in the order a document passes through them.

Most quality problems arise in the first two steps. If a table in a PDF is badly converted to text, no model will fix it later.

For most mid-sized companies working in Microsoft 365, the sensible order is an off-the-shelf tool on one set of documents first, and only then, if needed, a custom build. A custom solution gives more control, but it has to be maintained for years.

The prices in the table are vendors’ list prices from October 2026; for Microsoft, with annual billing. Check the current price list before you decide.

Let us show the costs for a hypothetical company that wants to give employees access to 20,000 documents averaging 5 pages. An AI model counts text in tokens, which are fragments of words. Let us cautiously assume 1,000 tokens per page, because Polish text splits into more tokens than English.

The whole collection is therefore 20,000 × 5 × 1,000 = 100 million tokens. At USD 0.13 per million tokens, the one-off conversion into a searchable form costs USD 13, or USD 2 with the cheaper model. You will pay the same again to reprocess everything, e.g. after changing the model.

Questions also need processing, but this costs next to nothing. 200 questions a day of 50 tokens each over 22 working days gives 220,000 tokens, or about USD 0.03 a month.

A bigger item is the model that writes the answers. With each question you send the passages found, e.g. five of 500 tokens each, about 3,000 tokens in total. Per month, that is 4,400 questions × 3,000 = 13.2 million tokens, which you multiply by the current price of the chosen model.

The biggest cost has no price in any price list. It is the time of the people who choose the documents, remove old versions, sort out permissions and check the answers. The table below brings all the components together.

The quality of answers depends more on the documents than on the choice of AI model. The list below is the minimum worth doing before launch.

These steps are useful even without AI. An ordinary search engine will also search a well-organised set of procedures faster.

Before launch, prepare a set of several dozen questions that employees really ask. For each one, record the answer approved by the owner of the collection.

Assess three things separately: whether the system found the right document, whether the answer matches it, and whether the assistant admits it does not know when the answer is not in the documents.

The last point is the most important. An assistant that says “I could not find this in the documents” is safer than one that always gives some answer. In Copilot Studio, you can switch off answers that are not based on sources.

Repeat the test after every change: a new model, new documents, a different way of splitting the text. Providers regularly retire older models, so you will face such a change sooner or later.

The assistant should not show employees more than they can see in their own folders. Microsoft’s off-the-shelf tools check the permissions of each person signed in through Microsoft Entra ID. In a custom-built solution, you have to design and test this yourself.

For services outside Microsoft 365, find out in which region the data is stored and whether the provider uses it to train models. If the documents contain personal data, you need a data processing agreement.

If the assistant is to answer customers, from 2 August 2026 it must tell them they are talking to AI. This is an obligation under Article 50 of the AI Act, not just good practice.