How an AI assistant answers from your own documents.
An AI assistant that answers from your documents is not retrained on them. Your documents are split into passages and indexed. For each question the system searches that index and hands the passages it finds to the model, together with the question. It needs readable, current documents, a person who maintains them, and a rule for who may see what.
How it works
The language model is not changed. Your manuals, contracts and offers are stored in a search index. The method is called retrieval-augmented generation, RAG for short. The term comes from a 2020 paper by Patrick Lewis and colleagues, who gave a language model a searchable index of Wikipedia.
OpenAI documents these steps for searching uploaded files.
- Each file is cut into pieces of 800 tokens by default, the unit in which models count text, and neighbouring pieces overlap by 400 tokens
- Each piece is indexed by its meaning, so the search also finds text that shares few or no words with the question
- A question is searched in the index, which returns the best-matching pieces, 10 by default, each with its file
- The pieces and the question go to the model, which writes the answer from them
What decides the quality
The model only sees the passages the search hands over. If the right passage is not in the index, or the search misses it, the answer is wrong or empty. Current documents and a good search matter more than the choice of model.
Every answer should show the passage it comes from, so the reader can check it. ThesisCheck, my research product, shows a claim only if its quote appears word for word in the stored source.
An assistant also has to say when it found nothing. ThesisCheck states what the filings do not cover, and I set up a company assistant the same way.
What it needs from you
- The documents as text a computer can read: Word files, presentations, text files or PDFs that contain real text
- One person who owns the content, removes old versions and adds new ones
- A decision on who may see which documents, before anything is indexed
Where it goes wrong
- Outdated documents. If two versions of a price list are in the index, the search can return the old one
- Scanned PDFs. A scan is a picture of a page. Outside its Enterprise plan, ChatGPT reads the digital text of a file and not the images in it, so a scan needs text recognition first
- Tables. A table cut into pieces can lose its column headings. Microsoft's guide to building such systems lists reformatting tables as a preparation step
- Questions that need calculation. The search returns only the best passages, and counting the contracts that end this year needs every contract. The adding up belongs in code as well. In InkDraft the model reads the amounts from the sales call, and code adds them up
- Permission leaks. Microsoft Copilot shows a person only content that person may at least view. A folder shared with everyone by mistake can then appear in anyone's answers
Ready-made or built
ChatGPT, Claude and Microsoft Copilot already do this with documents you give them. ChatGPT works with files you attach to a chat or add to a project. A Claude project on a paid plan switches to search by itself once its documents exceed what the model can read at once. Microsoft Copilot searches the documents, emails and chats in your Microsoft 365 that the person asking may see.
A team with a few dozen documents can start there.
A built system makes sense when the documents sit in systems these tools do not reach, when customers read the answers, or when every answer has to pass a check before it is shown. InkDraft is an example: it writes proposals from a company's own documents and case studies.
What you can do now
Start small, with tools you already pay for.
- Pick one set of documents and one group of users, such as the service manual for the support team
- Name one person who keeps those documents current
- Remove old versions, turn scans into text and settle who may see which folder
- Write down 20 questions with the correct answers and test with them
- Require the source passage with every answer, and “not found” when there is none
Questions
How many documents can such an assistant handle?
ChatGPT allows 5 files per project on the Free plan, 25 on Go and Plus, and 40 on Edu, Pro, Business and Enterprise. Anthropic says a Claude project in search mode holds up to ten times more content than before. An index built on OpenAI's API is billed by size: free up to 1 GB, then 0.10 US dollars per GB and day.
Would it be better to train a model on our documents?
No. The 2020 paper names two open problems of a model that holds its knowledge inside itself: showing where a statement comes from, and updating what it knows. A search index addresses both. A changed document is indexed again, and the answer can show its passage.
Can customers use it, or only our staff?
Both can, and the mechanism is the same. Microsoft writes about Copilot that its responses are not guaranteed to be 100% factual and that users should review them. Staff can do that and a customer often cannot. For customers, index only public documents and show the source with every answer.
Do our documents leave the company?
With a cloud tool the text goes to the provider at indexing and with every question, unless it already sits there, as with Microsoft 365. OpenAI states that it does not use content from its business offerings, such as the API and ChatGPT Enterprise, to improve its models, and that for consumer services this depends on your data settings. Microsoft states that prompts, responses and the data Copilot accesses are not used to train its foundation models. Where no public AI service is allowed, a system can run on a self-hosted model, as InkDraft can.