In short
When a company starts using language models with its own data, the first architecture tends to be the same. Documents are split into passages, each passage becomes a numeric vector and, for each question, the system retrieves the most similar passages and sends them to the model along with the question. This technique is called RAG, short for retrieval-augmented generation. It solves an important part of the context problem and is one of the sources that feed the Company Brain, the layer that knows the company: what data exists, where it lives, how it connects and who may see what.
The term was introduced in 2020, in a paper by researchers from Facebook AI Research, University College London and New York University that proposed combining a generative model with a document index queried for every answer. The core idea is to separate what the model learned in training from the knowledge it consults at the moment of the question.
In enterprise practice, the pipeline has four steps:
Passage retrieval has one clear strength: questions whose answer is written in some document. What the refund policy is, what a product's onboarding manual says, what the procedure is for opening a security ticket. In these cases the right passage contains the answer, and the model only needs to rephrase it.
It is also relatively simple to deploy. A document base can be indexed without changing the source systems, which explains why RAG shows up in so many pilots.
Many business questions depend on data that is not in a document. How many of this customer's orders were late this quarter, which SLA was contracted for this type of ticket, which accounts have a renewal coming up and a critical ticket open. The answer is in records in the ERP, the CRM and the helpdesk, and in the relationships between them.
Similarity search was designed to find text that resembles the question. A question about late orders may retrieve the passage from the delivery terms policy, which looks like the question and does not contain the number. The model receives context that is relevant in form and insufficient in content, and fills the gap with the most likely continuation.
There is also fragmentation. A contract split into passages loses the link between the discount clause and the appendix that defines the price table. The retrieved passage is correct, and the answer comes out incomplete.
Variations of the technique, such as hybrid search, which combines keywords and vectors, and reranking of the retrieved passages, improve precision. They make RAG better at finding the right text and still depend on the answer being written down somewhere.
The first RAG project is usually designed for one base and one type of question. On the second use case, the company discovers it needs another base, another chunking strategy, other filters and other access control. Each new project rebuilds the integration from scratch.
This rework has a direct cost in tokens. To make up for the lack of precision, pipelines increase the number of passages sent to the model, and the prompt grows. More text per call means a higher cost per answer and, often, the need for a more expensive model to handle the volume.
The effect also shows up in maintenance. Each indexed base has to be updated when the documents change, and a RAG that indexed the previous version of a policy keeps answering with it until the next reindexing. With several independent bases, nobody knows for sure which version each assistant is using.
A vector index stores passages without the context of who may see each one. If the salary policy and the onboarding manual are in the same index, the search may retrieve the restricted passage for someone who should not have access to it. Passage-level access control, kept in sync with the permissions of the source systems, is usually left out of the RAG pilot and comes back as an obstacle in the security review.
Document search is still useful. In the Company Brain, it works alongside the knowledge graph, the business ontology and the semantic layer. When the question depends on text, the semantic layer triggers passage retrieval. When it depends on a record, it queries the structured source. When it depends on both, such as knowing which clause applies to a customer's late order, it combines the two and links the contract passage to the right entity in the graph.
Permissions are inherited from the source system on both paths, and the context is served to the model over MCP with the source of each piece of information. The prompt receives only the slice the question needs, which cuts tokens and makes it possible to run the same task on lower-cost models.
For companies that already have a RAG in production, the starting point is to list the questions it gets wrong. Often they depend on a structured record or on the relationship between two systems, and they show where the first Company Brain slice should begin.
