Announcing US$ 3.2M pre-seed round · OneVC · Maya · Norte Ventures Read →
Back to blog

Six audit questions to ask an AI context platform

In short: Before signing with an AI context platform, companies in regulated sectors should take six questions to the meeting with the vendor: where the data and credentials live, when the permission is applied, whether the answer cites document and version, who can read the query log, whether the data can become training and how the platform detects stale context. The answers say more about governance, compliance and risk than any demo.

In conversations with security, legal and compliance teams, AI evaluation usually starts with answer quality. In an audit, the questions are why the agent answered that, based on which document, and who else could have seen the same information. A context platform is the layer that connects a company's systems and delivers that content to agents, and it is where AI governance is either settled or lost.

Company Brain (the business's operational brain): the layer that connects a company's systems once, organises data, memory and knowledge, and delivers that context to any AI agent with each person's permissions.

At Strattum, we built the Company Brain from these six questions.

1. Where do the data, credentials and keys live?

"In Brazil" is an incomplete answer. The buyer needs to know where the records, the vector indexes, the credentials for the sources, the encryption keys and the logs live. Many platforms run only the connector in the customer's cloud and process the rest in the vendor's environment, which creates a new processor of personal data under Brazil's data protection law (LGPD). In the Company Brain, the whole platform runs in the customer's cloud.

A query log treated as sensitive data that also needs access control
The query log needs access control too

2. Is the permission applied before or after retrieval?

When the filter comes after ranking, the forbidden passage has already been retrieved and can influence the answer or appear in an intermediate log. When it comes before, the passage never reaches the model. It is also worth asking whether the permission is read from the source or copied into a database of its own, because the copy falls out of date when someone moves to another team.

3. Does the answer cite document, version and passage?

Without the version, the auditor cannot confirm whether the policy cited was the one in force on the date of the decision. The citation has to point to the source paragraph, and the lineage has to take each attribute back to its source.

4. Who can read the query log, and how long is it kept?

Every query should produce a record of who asked, what context they received and what answer came out. The log itself contains sensitive data, so the next question is who has access to it and where it is stored.

5. Can the data be used to train models?

The contract has to answer for both the platform vendor and the model provider. The architecture helps: when the context is assembled inside the tenant, only the passage needed goes on to the model. At Strattum, customer data is never used to train models.

A revoked policy and the policy in force, with versioned knowledge
Versioning knowledge avoids citing a revoked policy

6. How does the platform detect stale or contradictory context?

An agent that cites a revoked policy creates a compliance problem that looks like a correct answer. The platform needs to detect contradictions between documents, flag recurring questions with no answer and version the knowledge it delivers.

I wrote a longer guide applying these criteria to regulated sectors: How to evaluate a private context platform for AI in regulated sectors. To talk to the Strattum team about an evaluation, go to strattum.ai/contato.

Get the next articles on AI governance and security