Databricks Lakebase represents a significant evolution in data architecture, specifically in the domain of OLTP (Online Transactional Processing) databases. It isn’t just another cloud database, but a fundamental redefinition of how transactional data is stored, processed, and integrated with analytics and artificial intelligence ecosystems. This new architecture aims to close the historical gap between low-latency operational needs and the scale and flexibility demands of data lakes.

Traditionally, OLTP databases and analytical systems operate in separate silos, requiring complex, fragile ETL pipelines to move data between them. Lakebase proposes a unified solution, combining the robustness of a relational database with the scalability and economics of cloud storage. Its design is inherently optimized for the era of AI agents, offering capabilities that legacy systems can’t provide.

To understand Lakebase’s impact, it’s essential to look at its technical pillars and how they solve long-standing challenges in data engineering. The innovation lies in how it handles latency, data isolation, and integration, all while supporting intensive operational workloads and the emerging demands of artificial intelligence.

Low-Latency Engineering on Object Storage

One of the biggest challenges in decoupling storage and compute in OLTP databases, using object storage (such as S3 or ADLS), is latency. Lakebase tackles this with a sophisticated architecture that uses state layers (soft-state) and intelligent caching mechanisms. While cold data lives in low-cost object storage, active Postgres pages are kept in high-performance intermediate layers. This approach lets Lakebase deliver single-digit millisecond latency and support millions of transactions per second, performance once considered out of reach for architectures based purely on data lakes. It’s the combination of cloud storage efficiency with the performance that critical transactional applications need.

Instant Branching: The “Git Checkout” for Operational Data

Lakebase’s branching mechanism, based on copy-on-write, is a transformative feature. It lets you create exact replicas of a production database, including schema and data, instantly and at marginal cost. This is possible because Lakebase isolates the metadata. When a new branch is created, it initially points to the same original data files. Changes are written only to new blocks, ensuring full isolation for:

  • High-fidelity testing: Developers can test new features in an environment identical to production with no risk.
  • Complex schema migrations: Schema changes can be tested and validated in an isolated environment before being applied to production.
  • AI agent experimentation: Agents can operate on their own isolated instances for experimentation and validation, without affecting production or other agents.

This capability removes the need for expensive, slow staging environments, speeding up the development cycle and protecting the integrity of production data.

Optimized for AI Agents: Why Lakebase Is Essential

The real driving force behind Lakebase’s design is robust support for autonomous AI agents. It isn’t just branching that makes it ideal, but a combination of characteristics that meet the intrinsic demands of operational artificial intelligence:

  1. Low-Latency Persistent State: AI agents, especially in real-time scenarios such as product recommendation or fraud detection, need access to operational data with millisecond latency. Lakebase delivers this by keeping active Postgres pages in high-performance layers, even with the underlying storage in object storage. This means agents can make fast decisions based on the latest data, without the performance bottlenecks that would limit traditional systems.
  2. Elastic Scalability (Serverless): AI agents can have unpredictable demand spikes. A traditional database would require overprovisioning to handle those spikes, resulting in high costs. Lakebase, with its serverless architecture, scales elastically down to zero when it isn’t in use and expands instantly to handle millions of transactions per second. This optimizes costs and ensures agents always have the resources they need, without manual intervention.
  3. Native Unification with Unity Catalog: Data governance is crucial for AI agents. Lakebase integrates natively with Unity Catalog, extending the same Lakehouse permissions and security policies to operational data. This simplifies access management and ensures agents operate within compliance boundaries, while also making operational data immediately visible for training and monitoring AI models in the Lakehouse, without complex ETL pipelines.

These pillars, combined with branching for safe experimentation, make Lakebase the ideal state infrastructure for the security, reliability, and agility of autonomous artificial intelligence. It provides the foundation for AI agents to experiment, learn, and operate in production with confidence.

Real Unification: The End of the Gap Between OLTP and OLAP

Historically, moving data from operational databases to analytical systems (the Lakehouse) was an arduous process, involving fragile ETL pipelines, manual schema management, and significant latency. Lakebase, with its native Unity Catalog integration, removes that complexity.

Operational data is synchronized in near real time with the analytical layer, without duplicated infrastructure or data egress costs. Governance is unified: the same permissions and security policies defined in Unity Catalog for the Lakehouse are extended to Lakebase. The result is a truly unified data platform, where operational data is immediately available for analysis and AI model training, without friction.

Reference Architecture: Integrating Lakebase with Existing Ecosystems

To understand how Lakebase fits into a modern data architecture, it’s crucial to visualize its integration with legacy systems and the Lakehouse. The approach isn’t traditional ETL, but rather Zero-ETL integration or federated access, which minimizes data movement and complexity.

Imagine the following scenario:

  1. Legacy OLTP Systems (Data Sources): Your organization has existing transactional databases (PostgreSQL, MySQL, SQL Server, Oracle, etc.) holding critical operational data. These systems remain the primary source of truth for your legacy applications.
  2. Lakehouse (Unified Analytics Layer): The Databricks Lakehouse, built on Delta Lake and Unity Catalog, acts as your unified platform for analytical data, data warehousing, BI, and Machine Learning. It ingests data from many sources, including the legacy OLTP systems, through mechanisms such as Lakehouse Federation or data streaming (e.g., Kafka, Change Data Capture - CDC) into Delta Live Tables.
  3. Lakebase (Operational Serving Layer for AI Agents): This is where Lakebase comes in. It serves as the high-performance, low-latency operational data layer specifically optimized for AI agents. Lakebase can be fed in two main ways: New Operational Data: For new applications or microservices that require Lakebase’s unique capabilities (low latency on object storage, branching, serverless scalability), data can be written directly to Lakebase, and Data Synchronized from the Lakehouse: For data that originates in legacy systems and is ingested into the Lakehouse, Lakebase can consume that data in an optimized way. Thanks to the native integration with Unity Catalog, Lakehouse tables can be easily accessed and, if needed, materialized or referenced in Lakebase so AI agents have low-latency operational access. This creates a bidirectional or consumption data flow, where Lakebase acts as an intelligent operational cache or as a primary datastore for the agents’ operations.

Lakebase’s Role for AI Agents:

In this architecture, Lakebase doesn’t replace the Lakehouse, it complements it, providing the transactional speed and agility that AI agents require. It’s the agents’ “short-term memory” and “workspace,” where they can:

  • Read operational data in real time: To make instant decisions (e.g., recommendation, fraud detection).
  • Write and update data: To carry out actions (e.g., adjust inventory, process an order).
  • Experiment safely: Use branching to test new agent logic or models without impacting production.

This architecture lets the organization keep its legacy systems running while modernizing its data layer to support the next generation of applications and AI agents, all within a unified structure governed by Unity Catalog.

Data Modeling and Retention Strategy in Lakebase

Data modeling and retention strategy are crucial to optimizing Lakebase for AI agents and ensuring its efficiency in the Lakehouse ecosystem. The approach here differs from traditional analytical databases.

1. Data Modeling: Pure Relational for Agent Operations

Yes, the recommendation is a pure relational model, similar to PostgreSQL, for the tables inside Lakebase. This is because of:

  • Familiarity and Maturity: The relational model is widely understood and optimized for transactional operations (OLTP), which are the focus of AI agents that need to read and write data quickly.
  • Referential Integrity: The ability to enforce primary and foreign keys ensures data consistency, which is essential for the reliability of agents’ decisions.
  • Optimized Point Queries: Agents frequently run point queries or specific transactions (e.g., “what’s the inventory for product X?”, “update the status of order Y?”). The relational model is highly efficient for these kinds of operations.

It’s important to stress that, while Lakebase keeps the relational model for the operational layer, the Lakehouse (Delta Lake) remains the ideal place for dimensional data models (Star Schema, Snowflake Schema) optimized for complex analytics and BI.

2. Ingestion Decision Matrix: What Goes Into Lakebase?

The decision about which data should be materialized in Lakebase isn’t arbitrary; it’s guided by operational criticality for the AI agents. It isn’t an indiscriminate copy of all legacy data, but a strategic selection based on three main criteria:

Write and Update Frequency (Volatility):

  • Question: Does the AI agent need to update this data in milliseconds or seconds? Does the information change so quickly that asynchronous replication or a higher-latency cache in the Lakehouse would be insufficient for operational consistency?
  • Example: Product inventory in an e-commerce business is highly volatile, critical data. A recommendation or order management agent needs to know the exact inventory right now to avoid selling an unavailable product. Data such as the customer’s shipping address, while important, doesn’t require the same update frequency by an agent and can be accessed with higher latency from the Lakehouse.

State Dependency for Immediate Decisions:

  • Question: Does the AI agent need this data to make an immediate, critical decision in its workflow? Would the absence or staleness of this information prevent the agent from completing its task or lead to an incorrect decision?
  • Example: For a fraud detection agent, a user’s recent transaction history is crucial for identifying anomalous patterns at the moment of the transaction. The full purchase history for trend analysis, on the other hand, can live in the Lakehouse, since it isn’t needed for the immediate decision to approve or deny a transaction.

Access Volume and Granularity:

  • Question: Does the agent access this data at high granularity (individual records) and with a large volume of point requests? Or is access more for aggregated or batch analysis?
  • Example: A personalization agent may need to access a user’s individual preference profile (high granularity, many point requests). Data on past marketing campaigns (low granularity, batch access for analysis) would be better suited to the Lakehouse.

How Unity Catalog Helps with the Decision:

Unity Catalog acts as the central point of governance. It lets you define and manage access to data, whether it lives in Lakebase or in the Lakehouse. For data that needs to be materialized in Lakebase, you can use data synchronization mechanisms (such as CDC) that bring only the critical tables or columns from the Lakehouse into Lakebase, keeping lineage and governance unified. Alternatively, for new applications, Lakebase can be the primary destination for this critical operational data.

In short, the decision to “copy” (or materialize) data into Lakebase is a data engineering exercise focused on performance and operational criticality. Only the data that is truly the “active state” and that requires millisecond latency for agent operations should live in Lakebase, while the vast history and less volatile data stay in the Lakehouse.

3. Storage: Optimized for Low Latency and Efficiency

Lakebase doesn’t indiscriminately “copy” all legacy data. The storage strategy is smarter and focused on the “active state” the agents need:

  • Active Operational Data: Lakebase is designed to store the subset of operational data that AI agents need to access and modify in real time. This can be a replication of specific tables from legacy systems (via CDC or streaming into the Lakehouse and then into Lakebase) or data generated directly by new applications that use Lakebase as their primary database.
  • Selective Materialization: For data that lives primarily in the Lakehouse (originating from legacy systems), Lakebase can materialize specific views or tables that are relevant to the agents’ operations. This means only the data needed for the agents’ real-time decisions is kept in Lakebase’s low-latency layer, while the full history and cold data stay in the Lakehouse.
  • Object Storage with Intelligent Caching: Although data is stored in low-cost object storage, Lakebase uses caching and soft-state layers to ensure that the data agents access most is available with millisecond latency. This avoids massive data duplication and optimizes storage costs.

Retention Recommendation: Active State vs. Cold History

The retention strategy in Lakebase should be driven by the AI agents’ operational needs:

  • Lakebase: Short-Term Retention (Active State): Lakebase should retain only the active state relevant to the agents’ real-time operations. That means data the agents frequently read and modify. For example, for a recommendation agent, current product inventory, the user’s recent interaction history, and the status of orders in progress. Retention periods can range from days to a few weeks, depending on the data’s volatility and how critical it is to the agent’s decisions.
  • Lakehouse: Long-Term Retention (Cold History): The full history and cold data (no longer needed for real-time decisions, but valuable for retrospective analysis, model training, and compliance) should be offloaded to the Lakehouse (Delta Lake). Lakebase’s native integration with Unity Catalog makes this process easier, allowing data to be moved or synchronized to the Lakehouse efficiently and under governance.

This clear division of responsibilities ensures that Lakebase stays agile and low-latency for the agents, while the Lakehouse offers scalability and economy for long-term storage and complex analytics. It’s an approach that balances operational performance with cost efficiency and data governance.

Use Case: Real-Time Product Recommendations with AI Agents in E-commerce

Imagine a large e-commerce platform that wants to offer hyper-personalized, real-time product recommendations that adapt instantly to user behavior and to inventory and price changes. Traditional systems struggle with the latency and complexity of managing transactional and analytical data for this purpose.

Challenge:

  • Latency: Delays in ingesting click, view, and purchase data result in outdated recommendations.
  • Experimentation: Testing new recommendation algorithms or pricing strategies in production is risky and slow.
  • Scalability: Handling millions of events per second while maintaining the transactional database’s performance.
  • AI Agents: Training and operating AI agents that need fast, consistent access to operational data to make decisions in milliseconds.

Solution with Databricks Lakebase:

  1. Operational Data in Lakebase: All user events (clicks, views, add-to-cart actions, purchases) and product catalog data (price, inventory) are stored in Lakebase. Low latency ensures this data is available almost instantly.
  2. AI Agents for Recommendation: AI agents are developed to monitor user behavior in real time. They access Lakebase data to identify patterns and generate personalized recommendations.
  3. Branching for Safe Experimentation: Before launching a new recommendation algorithm, a branch of the Lakebase database is created. AI agents can be trained and tested in this isolated environment, simulating production scenarios without impacting real customers. If the new algorithm produces hallucinations or inappropriate recommendations, the branch is simply discarded.
  4. Integration with the Lakehouse: Lakebase data is synchronized in real time with the Lakehouse via Unity Catalog. This lets Data Science teams use the full interaction history to train more complex AI models and run in-depth business analyses, while the agents operate on the latest data.

Benefits:

  • Real-Time Recommendations: Lakebase’s low latency ensures recommendations are always based on the most current data.
  • Accelerated Innovation: Branching lets you test and iterate quickly on new AI algorithms with no risk to production.
  • Improved Customer Experience: More accurate, personalized recommendations lead to higher engagement and sales.
  • Unified Governance: Unity Catalog ensures that all data, from operational to analytical, sits under the same security and compliance umbrella.

Step-by-Step Implementation Guide: AI Agents with Agent Bricks and Lakebase

To demonstrate what Lakebase can do together with Agent Bricks to build robust AI agents, let’s walk through a technical workflow in detail. The focus will be on creating a recommendation agent that interacts with operational data in Lakebase and uses branching for safe experimentation.

1. Initial Setup in Databricks and Lakebase

First, make sure your Databricks environment is set up with Unity Catalog and with the Agent Bricks Preview enabled. You’ll need a Databricks workspace and permissions to create catalogs and schemas and to use Agent Bricks.

-- Criar um catálogo no Unity Catalog para o Lakebase
CREATE CATALOG IF NOT EXISTS ecommerce_catalog;
USE CATALOG ecommerce_catalog;

-- Criar um esquema (database) para os dados operacionais do e-commerce
CREATE SCHEMA IF NOT EXISTS operational_db;
USE SCHEMA operational_db;

-- Tabela de Pedidos (OLTP) no Lakebase
CREATE TABLE orders (
    order_id STRING NOT NULL PRIMARY KEY,
    user_id STRING NOT NULL,
    product_id STRING NOT NULL,
    quantity INT NOT NULL,
    order_timestamp TIMESTAMP NOT NULL,
    status STRING NOT NULL
) USING LAKEBASE;

-- Tabela de Produtos (OLTP) no Lakebase
CREATE TABLE products (
    product_id STRING NOT NULL PRIMARY KEY,
    product_name STRING NOT NULL,
    category STRING NOT NULL,
    price DECIMAL(10, 2) NOT NULL,
    stock_quantity INT NOT NULL
) USING LAKEBASE;

-- Inserir dados de exemplo
INSERT INTO products VALUES
(\'prod_001\', \'Laptop Gamer\', \'Eletronicos\', 1500.00, 100),
(\'prod_002\', \'Mouse Sem Fio\', \'Acessorios\', 50.00, 500),
(\'prod_003\', \'Teclado Mecanico\', \'Acessorios\', 120.00, 200);

INSERT INTO orders VALUES
(\'order_001\', \'user_a\', \'prod_001\', 1, \'2026-02-27 10:00:00\', \'completed\'),
(\'order_002\', \'user_b\', \'prod_002\', 2, \'2026-02-27 10:05:00\', \'pending\'),
(\'order_003\', \'user_a\', \'prod_003\', 1, \'2026-02-27 10:15:00\', \'completed\');

2. Building a Recommendation Agent with Agent Bricks

Let’s create a simple agent that, given a user_id, recommends products based on previous purchases and current inventory. Agent Bricks lets you define agents declaratively. This example simulates the logic inside a Databricks notebook, which would be the basis for an Agent Bricks agent.

# Exemplo de código Python em um notebook Databricks para um agente de recomendação
# Este código simula a lógica que seria encapsulada por um Agent Bricks

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import Statement

# Inicializar o cliente do Workspace (assumindo autenticação configurada)
w = WorkspaceClient()

def get_user_orders(user_id: str, branch: str = "main"):
    # Conectar ao Lakebase usando o branch especificado
    # Em um ambiente real, o Agent Bricks gerencia a conexão e o contexto do branch
    query = f"USE BRANCH {branch}; SELECT product_id FROM orders WHERE user_id = \'{user_id}\'"
    result = w.statement_execution.execute_statement(
        statement=query,
        warehouse_id="<seu_warehouse_id>", # Substitua pelo ID do seu SQL Warehouse
        catalog="ecommerce_catalog",
        schema="operational_db"
    ).result.data_array
    return [row[0] for row in result]

def get_product_stock(product_id: str, branch: str = "main"):
    query = f"USE BRANCH {branch}; SELECT stock_quantity FROM products WHERE product_id = \'{product_id}\'"
    result = w.statement_execution.execute_statement(
        statement=query,
        warehouse_id="<seu_warehouse_id>",
        catalog="ecommerce_catalog",
        schema="operational_db"
    ).result.data_array
    return result[0][0] if result else 0

def recommend_products(user_id: str, branch: str = "main"):
    purchased_products = get_user_orders(user_id, branch)
    recommendations = []
    # Lógica de recomendação simplificada: recomendar produtos com estoque > 0 que não foram comprados
    all_products_query = f"USE BRANCH {branch}; SELECT product_id, product_name FROM products WHERE stock_quantity > 0"
    all_products_result = w.statement_execution.execute_statement(
        statement=all_products_query,
        warehouse_id="<seu_warehouse_id>",
        catalog="ecommerce_catalog",
        schema="operational_db"
    ).result.data_array

    for prod_id, prod_name in all_products_result:
        if prod_id not in purchased_products:
            recommendations.append(prod_name)
    return recommendations

# Exemplo de uso do agente (simulado)
user_to_recommend = "user_a"
print(f"Recomendações para {user_to_recommend} (branch main): {recommend_products(user_to_recommend)}")

3. Safe Experimentation with Branching and Agent Bricks

Now let’s use Lakebase branching to test a new recommendation strategy that involves an aggressive promotion on a specific product, which may affect inventory. Agent Bricks can be configured to operate on specific branches for testing.

-- Criar um novo branch para testar a promoção
CREATE BRANCH promo_test ON TABLE orders;
CREATE BRANCH promo_test ON TABLE products;

An AI agent configured for the promo_test branch can now simulate the changes. In Agent Bricks, you would specify the branch in the agent’s execution context.

# Simulação de um agente de IA operando no branch \'promo_test\'
# O Agent Bricks abstrairia a gestão do branch para o desenvolvedor do agente

# Agente de IA decide aplicar um desconto e vender um Laptop Gamer
# Esta operação ocorre APENAS no branch \'promo_test\'
update_query = "UPDATE products SET stock_quantity = stock_quantity - 1 WHERE product_id = \'prod_001\'"
w.statement_execution.execute_statement(
    statement=update_query,
    warehouse_id="<seu_warehouse_id>",
    catalog="ecommerce_catalog",
    schema="operational_db",
    branch="promo_test" # Especifica o branch para a operação
)

# O agente verifica o estoque no branch de teste
print(f"Estoque de Laptop Gamer no branch promo_test: {get_product_stock(\'prod_001\', branch=\'promo_test\')}")

# O estoque no branch principal permanece inalterado
print(f"Estoque de Laptop Gamer no branch main: {get_product_stock(\'prod_001\', branch=\'main\')}")

If the result of the simulation on promo_test is positive (for example, an increase in simulated sales without depleting inventory in a harmful way), the changes can be merged into the main branch. Otherwise, the promo_test branch can be discarded without affecting production.

-- Se os testes forem bem-sucedidos, mesclar as alterações (exemplo conceitual, a mesclagem pode ser mais complexa)
-- MERGE BRANCH promo_test INTO main ON TABLE orders;
-- MERGE BRANCH promo_test INTO main ON TABLE products;

-- Descartar o branch de teste após a conclusão (seja mesclado ou não)
DROP BRANCH promo_test ON TABLE orders;
DROP BRANCH promo_test ON TABLE products;

4. Continuous Integration with the Lakehouse (Unity Catalog)

Regardless of the operational branches, Lakebase data is continuously synchronized with the Lakehouse via Unity Catalog. This lets Data Science and Analytics teams keep training more complex AI models and running in-depth business analyses on the full interaction history, while the agents operate on the latest data and in their isolated branches.

-- Exemplo de como uma equipe de Data Science pode consultar dados operacionais
-- diretamente do Lakehouse via Unity Catalog para treinamento de modelos
SELECT
    o.user_id,
    p.product_name,
    o.quantity,
    o.order_timestamp,
    p.category
FROM
    ecommerce_catalog.operational_db.orders AS o
JOIN
    ecommerce_catalog.operational_db.products AS p
ON
    o.product_id = p.product_id
WHERE
    o.order_timestamp >= current_date() - INTERVAL \'30 days\';

Conclusion

Databricks Lakebase, together with Agent Bricks, offers a powerful platform for building and operating intelligent applications and AI agents. Its ability to deliver low latency on object storage, the branching mechanism for safe experimentation, and seamless integration with Unity Catalog for unified governance position Lakebase as the ideal technical foundation for the next generation of transactional and analytical systems. For organizations looking to innovate quickly with AI, Lakebase and Agent Bricks provide the tools needed to turn operational data into actionable intelligence securely and at scale.

References