Announcing US$ 3.2M pre-seed round · OneVC · Maya · Norte Ventures Read →
Back to blog

Data catalog:
what it is and what it lacks to serve AI

In short

A data engineer needs to build a returns report and finds four tables with similar names in the data warehouse: orders, orders_final, orders_new and orders_bkp. She does not know which one is kept up to date, who maintains each of them or whether any of them holds personal data. The data catalog exists to answer those questions. It is also a valuable source for AI projects and one of the starting points of the Company Brain, the layer that knows the company: what data exists, where it lives, how it connects and who may see what.

The data catalog is the inventory of assets with their metadata

A data catalog lists the company's tables, views, reports, files and pipelines and records, for each asset, the information that lets people use it safely:

  • Description: what the asset contains and what it is for;
  • Owner: the person or team accountable for it;
  • Schema: columns, types and keys;
  • Sensitivity classification: whether it holds personal, financial or confidential data;
  • Lineage: where the data comes from and where it goes;
  • Quality: tests, refresh frequency and incidents;
  • Usage: who queries it and how often.

The structure follows an open standard. The W3C maintains DCAT, a vocabulary for describing datasets and services in catalogs published on the web, whose version 3 became a recommendation in 2024. Corporate catalogs apply the same logic inside the company.

Data analysts and engineers are the original audience

The catalog was designed for people looking for the trusted table. An analyst searches for a term, reads the description, checks the date of the last refresh and, when in doubt, writes to the owner. The judgement on whether to use that asset stays with her.

That origin explains the format. The catalog describes datasets in text and in structured fields, for human reading and for search. Part of the content is collected automatically by connectors that read schemas, queries and execution logs. The other part, such as descriptions and business definitions, is written by people, and that is the part that ages fastest.

Typical functions range from discovery to LGPD classification

  • Discovery: search by name, term or owner, with the certified table flagged.
  • Business glossary: definitions of terms such as return or average ticket, linked to the columns that calculate them.
  • Lineage: the path of the data from source to report, used for impact analysis.
  • LGPD classification: tagging the columns that hold personal data, the basis for the data mapping the law requires.
  • Governance: policies, access request workflows and asset certification.
Metadata recorded in a data catalog for each table
Each asset in the data catalog carries a description, owner, schema, sensitivity, lineage, quality and usage

The data catalog's owners, classifications and definitions already help AI

Part of what the catalog records is useful to AI agents without any adaptation. The owner shows who to turn to when an answer looks wrong. The sensitivity classification flags which columns need care before they reach a model. The glossary definitions guide how business terms are interpreted, and certification shows which of the four orders tables to use.

The agent needs records, relationships and permissions at the moment of the question

The catalog describes tables. An agent answering whether a customer's order was delivered has to work at the record level. To serve agents, the data catalog has to be complemented with four elements:

  • the specific record, with the current value of each field;
  • the relationship between records in different systems, such as the order in the ERP linked to the invoice and to the helpdesk ticket;
  • permissions applied per person at the moment of the question, row by row;
  • delivery over a protocol the agent consumes, such as MCP.

Catalog access policies usually work at the table level and define who can request access to a dataset. The agent needs a filter applied to every query, on behalf of whoever asked, with the same slice that person has in the source system.

The difference between what a data catalog describes and what an AI agent needs
The data catalog describes the table, and the agent needs the record with its relationships and permissions

The Company Brain reuses the existing data catalog

A company that already maintains a data catalog has part of the work done. The Company Brain imports descriptions, owners, classifications and glossary terms and uses them as a starting point. The glossary feeds the business ontology. Lineage strengthens the traceability of each answer. Classification complements the access rules, with permissions inherited from the source system.

From there, the knowledge graph links records across systems, the semantic layer translates the question into the right query and the slice is served to the model via MCP. The AI project starts from what the data team has already documented, and the first use case reaches production sooner.

The Company Brain reusing the data catalog as a source of metadata
The Company Brain starts from the metadata the data catalog has already organised

An outdated catalog becomes an official source of errors

When the description says the table is refreshed every day and the load process stopped months ago, an attentive analyst notices it from the date of the last record. An agent that consumes that metadata repeats it with the authority of an official source, and the error starts circulating in answers that cite the data catalog as their origin.

That is why reusing the catalog requires metadata with an owner and a review date, automatic collection of whatever can be collected, such as schema, lineage and usage, and human review of the business definitions.

The starting point is to cross the catalog with the real questions

The first step is to gather the questions of the first AI use case and list the tables and terms they touch. Next, check in the data catalog whether each asset has an owner, a current definition and a classification. The gaps show the work that comes first, and what is complete goes into the Company Brain as a foundation.

Get the next articles on the Company Brain