Deploying data projects across different environments has always been a challenge comparable to trying to replicate a gourmet dish in different kitchens. Even with the same ingredients, small variations in settings, in the order of preparation or in the tools used can lead to completely different experiences, sometimes with disastrous results.
The Databricks Asset Bundle (DAB) arrives as a true professional chef’s kit for data engineers. Just as a Michelin-starred chef can bring their own set of knives, spices and detailed recipes to guarantee the same culinary quality in any kitchen in the world, DAB lets data teams package, version and deploy their projects consistently to any Databricks workspace, whether development, test or production.
This revolutionary approach transforms how we manage projects on the Databricks platform, bringing modern DevOps practices into the data world. By adopting the concept of “Infrastructure as Code” (IaC), DAB allows the entire project (including notebooks, workflows, cluster configurations and other resources) to be defined in declarative files, versioned in systems such as Git and deployed automatically through CI/CD pipelines.
The Perfect Recipe (What the Databricks Asset Bundle Is)
Imagine you’re a chef responsible for making sure the same dish is served perfectly in different restaurants of an international franchise. You’d need an extremely detailed recipe, one that leaves no room for interpretation or unwanted variation. That’s exactly what the Databricks Asset Bundle offers for data projects.
The heart of DAB is the databricks.yml file, comparable to a recipe meticulously written by a Michelin-starred chef. This YAML file contains every instruction needed to “cook” your Databricks project, from defining the “ingredients” (resources) to the method (configurations) and the final plating (deployment).
Simplified example of a databricks.yml
bundle:
name: meu-projeto-analytics
variables:
data_path:
default: /data/bronze
description: Caminho para os dados brutos
targets:
dev:
workspace:
host: https://dev-workspace.cloud.databricks.com
variables:
data_path: /data/dev/bronze
prod:
workspace:
host: https://prod-workspace.cloud.databricks.com
resources:
jobs:
etl_diario:
name: ETL Diário
schedule:
quartz_cron_expression: "0 0 2 * * ?"
tasks:
ingestao:
notebook_task:
notebook_path: /Notebooks/ingestao
job_cluster_key: cluster_padrao
job_clusters:
cluster_padrao:
spark_version: 13.3.x-scala2.12
node_type_id: Standard_DS3_v2
autoscale:
min_workers: 1
max_workers: 4
Just as a well-written recipe leaves no doubt about quantities, times and techniques, the databricks.yml file removes ambiguity from the project configuration. Each section has a specific purpose:
- Bundle metadata: The project’s name and basic information
- Variables: Parameters that can be referenced throughout the file
- Environments (targets): Specific configurations for different workspaces
- Resources: Definitions of jobs, clusters, notebooks and other components
The Premium Ingredients (Resources Managed by DAB)
A chef knows that the quality of the final dish depends directly on the quality of the ingredients. In the context of the Databricks Asset Bundle, the “ingredients” are the various resources that make up your project:
- Notebooks: The equivalent of the individual recipes that make up a menu
- Workflows (Jobs): The preparation sequence, like a tasting menu
- Clusters: The ovens and equipment needed for cooking
- Secrets: The special ingredients that need to be protected
- Pipelines: The advanced preparation techniques
DAB lets you manage all these “ingredients” centrally, making sure they are combined in the right proportions and in the proper order. Just as a chef wouldn’t leave the choice of ingredients to chance, DAB doesn’t leave resource configuration at the mercy of error-prone manual processes.
One of the big differentiators of this approach is the ability to version these “ingredients” along with the code. Imagine being able to keep not only the recipes, but also the exact specification of the ingredients and equipment used in each version of your menu. That’s what DAB delivers by integrating with version control systems such as Git.
Cooking in Any Kitchen (Consistency Across Environments)
One of the biggest challenges for international chefs is adapting to different kitchens while keeping their dishes consistent. In the same way, data teams face the challenge of making sure their projects behave the same way in development, test and production environments.
The Databricks Asset Bundle solves this problem by letting you define environment-specific configurations while keeping the project’s basic structure intact. It’s like having a base recipe that can be slightly adapted for different occasions without losing its essence.
targets:
dev:
workspace:
host: https://dev-workspace.cloud.databricks.com
variables:
cluster_size: small
data_retention: 7
qa:
workspace:
host: https://qa-workspace.cloud.databricks.com
variables:
cluster_size: medium
data_retention: 14
prod:
workspace:
host: https://prod-workspace.cloud.databricks.com
variables:
cluster_size: large
data_retention: 30
This capacity for controlled adaptation brings significant benefits:
- Predictability: If it works in development, it will work in production
- Traceability: Every difference between environments is explicit and documented
- Reproducibility: Environments can be recreated exactly as they were at any point in time
- Governance: Specific policies can be applied to each environment
The Automated Sous-Chefs (CI/CD Integration)
In high-end restaurants, the head chef relies on trusted sous-chefs who follow their instructions to the letter, making sure every dish comes out perfect even when the chef isn’t directly involved in every step. In the world of the Databricks Asset Bundle, CI/CD pipelines take on that role.
Integrating DAB with tools such as GitHub Actions, Azure DevOps or GitLab CI lets you automate the entire deployment process, from validating the “recipe” to “plating the final dish” in the target environment.
Here’s an example of what a GitHub Actions workflow to deploy a DAB project would look like:
name: Deploy Databricks Assets
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install Databricks CLI
run: pip install databricks-cli
- name: Deploy to Dev
if: github.event_name == 'pull_request'
run: |
databricks bundle deploy -t dev
env:
DATABRICKS_HOST: ${{ secrets.DEV_DATABRICKS_HOST }}
DATABRICKS_TOKEN: ${{ secrets.DEV_DATABRICKS_TOKEN }}
- name: Deploy to Prod
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
run: |
databricks bundle deploy -t prod
env:
DATABRICKS_HOST: ${{ secrets.PROD_DATABRICKS_HOST }}
DATABRICKS_TOKEN: ${{ secrets.PROD_DATABRICKS_TOKEN }}
These “automated sous-chefs” bring several benefits:
- Consistency: The deployment process is always the same, no matter who kicks it off
- Speed: Deployments happen quickly, with no manual intervention
- Auditing: Every deployment is logged, including who started it and which changes were applied
- Security: Credentials and secrets are managed securely, with no direct exposure
Tasting Before Serving (Validation and Testing)
No responsible chef would serve a dish without tasting it first. In the same way, the Databricks Asset Bundle lets you “taste” your project before deploying it to shared environments.
The Databricks CLI offers specific commands to validate and test your bundle locally:
# Validar a configuração do bundle
databricks bundle validate -t dev
# Implantar localmente para teste
databricks bundle deploy -t dev
This up-front validation capability is crucial for catching problems before they affect other users or, even worse, production data. It’s like having a test kitchen where you can try out new recipes before adding them to the official menu.
Some examples of problems that can be caught during validation:
- References to nonexistent resources
- Invalid or incompatible configurations
- Syntax problems in the YAML file
- Inconsistencies between different parts of the project
Step-by-Step Recipe (Hands-On Implementation)
Now that we understand the concepts and benefits of the Databricks Asset Bundle, let’s see how to implement it in a real project, following a step-by-step recipe:
1. Preparing the “Kitchen” (Development environment)
First, we need to install the necessary tools:
# Para Windows
winget install Databricks.DatabricksCLI
# Para Mac
brew tap databricks/tap
brew install databricks
Next, we configure authentication with the Databricks workspace:
databricks configure --profile DEFAULT
2. Creating the “Base Recipe” (Bundle initialization)
With the tools installed, we can start our first bundle:
# Criar um novo diretório para o projeto
mkdir meu-projeto-databricks
cd meu-projeto-databricks
# Inicializar o bundle
databricks bundle init
This command will create a basic databricks.yml file that we can customize.
3. Customizing the “Recipe” (Bundle configuration)
Now we edit the databricks.yml file to include all the necessary resources:
bundle:
name: meu-projeto-analytics
variables:
env:
default: dev
description: Ambiente atual (dev, qa, prod)
data_path:
default: /data/bronze
description: Caminho para os dados brutos
targets:
dev:
workspace:
host: https://dev-workspace.cloud.databricks.com
variables:
env: dev
qa:
workspace:
host: https://qa-workspace.cloud.databricks.com
variables:
env: qa
prod:
workspace:
host: https://prod-workspace.cloud.databricks.com
variables:
env: prod
resources:
jobs:
etl_diario:
name: "ETL Diário - ${variables.env}"
schedule:
quartz_cron_expression: "0 0 2 * * ?"
tasks:
ingestao:
notebook_task:
notebook_path: /Notebooks/ingestao
base_parameters:
data_path: "${variables.data_path}"
env: "${variables.env}"
job_cluster_key: cluster_padrao
job_clusters:
cluster_padrao:
spark_version: 13.3.x-scala2.12
node_type_id: Standard_DS3_v2
autoscale:
min_workers: 1
max_workers: 4
4. “Tasting” (Local validation and testing)
Before sharing our recipe, let’s test it locally:
# Validar a configuração
databricks bundle validate -t dev
# Implantar para teste
databricks bundle deploy -t dev
5. Getting Ready for “Service” (CI/CD configuration)
Finally, we set up a CI/CD pipeline to automate deployment. If you’re using GitHub, create a .github/workflows/deploy.yml file:
name: Deploy Databricks Assets
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install Databricks CLI
run: pip install databricks-cli
- name: Validate Bundle
run: databricks bundle validate
- name: Deploy to Dev
if: github.event_name == 'pull_request'
run: |
databricks bundle deploy -t dev
env:
DATABRICKS_HOST: ${{ secrets.DEV_DATABRICKS_HOST }}
DATABRICKS_TOKEN: ${{ secrets.DEV_DATABRICKS_TOKEN }}
- name: Deploy to QA
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
run: |
databricks bundle deploy -t qa
env:
DATABRICKS_HOST: ${{ secrets.QA_DATABRICKS_HOST }}
DATABRICKS_TOKEN: ${{ secrets.QA_DATABRICKS_TOKEN }}
Conclusion
The Databricks Asset Bundle represents a significant evolution in how we manage projects on the Databricks platform. Just as a professional chef’s kit transforms the culinary experience, DAB transforms the development lifecycle of data projects.
By adopting this approach, data teams can:
- Ensure consistency across different environments
- Automate deployments through CI/CD pipelines
- Version the entire configuration along with the code
- Reduce manual errors through standardized processes
- Speed up the development cycle with automated validations
As with any new culinary technique, there’s an initial learning curve. However, the time invested in mastering the Databricks Asset Bundle quickly pays off in gains in productivity, quality and governance.
The next time you’re struggling with manual deployments or inconsistencies between environments, remember: with the right chef’s kit, even the most complex data dishes can be prepared with precision and confidence in any Databricks kitchen.
See you in the next article!
