https://julianamarialop.github.io/en/tags/databricks/Recent content in Databricks on Ahead of the AIHugoen-USTue, 03 Mar 2026 12:31:00 +0000https://julianamarialop.github.io/en/articles/databricks-lakebase-definitive-guide-operational-data-and-ai-agents/Tue, 03 Mar 2026 12:31:00 +0000https://julianamarialop.github.io/en/articles/databricks-lakebase-definitive-guide-operational-data-and-ai-agents/Databricks Lakebase represents a significant evolution in data architecture, specifically in the domain of OLTP (Online Transactional Processing) databases. It isn’t just another database…https://julianamarialop.github.io/en/articles/from-etl-to-etl-a-the-rise-of-autonomous-pipelines/Wed, 21 Jan 2026 17:30:00 +0000https://julianamarialop.github.io/en/articles/from-etl-to-etl-a-the-rise-of-autonomous-pipelines/Imagine you need to get from point A to point B. A few decades ago, your only option was a car with a manual transmission, demanding full attention to every gear change and every move in traffic. Over time…https://julianamarialop.github.io/en/articles/ai-data-engineer-the-evolution-of-the-data-engineer-in-the-age-of-intelligent-agents/Wed, 07 Jan 2026 14:21:00 +0000https://julianamarialop.github.io/en/articles/ai-data-engineer-the-evolution-of-the-data-engineer-in-the-age-of-intelligent-agents/The data engineering profession is going through its biggest transformation since the move to the cloud. For years, the focus was on building efficient pipelines, moving data from one point to another and making sure…https://julianamarialop.github.io/en/articles/microsoft-fabric-at-ignite-2025-the-complete-guide-to-whats-new/Tue, 25 Nov 2025 15:53:00 +0000https://julianamarialop.github.io/en/articles/microsoft-fabric-at-ignite-2025-the-complete-guide-to-whats-new/Microsoft Ignite 2025 wasn’t an event about future promises; it was about concrete deliveries that mark the maturity of Microsoft Fabric. The platform stopped being a collection of promising tools and became…https://julianamarialop.github.io/en/articles/building-a-data-skyscraper-how-to-ensure-quality-in-your-bi/Thu, 18 Sep 2025 17:14:00 +0000https://julianamarialop.github.io/en/articles/building-a-data-skyscraper-how-to-ensure-quality-in-your-bi/Imagine you’ve been hired to build the tallest skyscraper in the city. It would be an incredible project, right? But what if I told you that you had to use cracked bricks, expired cement and rusty iron…https://julianamarialop.github.io/en/articles/la-traviata-of-data-how-fabric-failed-to-win-the-market/Mon, 25 Aug 2025 14:21:00 +0000https://julianamarialop.github.io/en/articles/la-traviata-of-data-how-fabric-failed-to-win-the-market/In May 2023, Microsoft announced Microsoft Fabric as a revolutionary solution to unify every analytical workload on a single platform. The promise was ambitious: lakehouse, data warehouse…https://julianamarialop.github.io/en/articles/databricks-one-the-superman-of-data-democratizes-information/Thu, 14 Aug 2025 13:06:00 +0000https://julianamarialop.github.io/en/articles/databricks-one-the-superman-of-data-democratizes-information/Just as Clark Kent had to turn into Superman so ordinary people could be saved, the data world has always required business users to turn into “super-techies” to access…https://julianamarialop.github.io/en/articles/fabric-hero-complete-arsenal-new-powers-and-gear-july-2025/Wed, 16 Jul 2025 14:19:00 +0000https://julianamarialop.github.io/en/articles/fabric-hero-complete-arsenal-new-powers-and-gear-july-2025/July 2025 arrived with a shower of updates that turned Microsoft Fabric into an even more powerful superhero. Like any self-respecting hero, Fabric never stops evolving, picking up new powers and…https://julianamarialop.github.io/en/articles/the-new-digital-office-how-ai-agents-became-the-most-efficient-employees/Tue, 01 Jul 2025 18:09:00 +0000https://julianamarialop.github.io/en/articles/the-new-digital-office-how-ai-agents-became-the-most-efficient-employees/Imagine an office where the employees never get tired, work 24 hours a day, are always learning and carry out tasks with surgical precision. This scenario, which seemed like science fiction just a few years ago…https://julianamarialop.github.io/en/articles/perfect-blueprint-for-analytics-on-azure-well-architected-review/Fri, 06 Jun 2025 15:58:00 +0000https://julianamarialop.github.io/en/articles/perfect-blueprint-for-analytics-on-azure-well-architected-review/In today’s world, where data is the new oil (or maybe the new bricks?), building robust, efficient Analytics solutions on Azure has become a complex engineering task. Platforms like Microsoft Fabric…https://julianamarialop.github.io/en/articles/spark-land-4-0-data-amusement-park-just-opened-its-gates/Thu, 29 May 2025 12:56:00 +0000https://julianamarialop.github.io/en/articles/spark-land-4-0-data-amusement-park-just-opened-its-gates/For more than a decade, Apache Spark has been the data processing platform of choice for data engineers and data scientists around the world. With its ability to process large volumes of information in…https://julianamarialop.github.io/en/articles/chefs-kit-for-data-databricks-asset-bundles-transform-deployment/Thu, 22 May 2025 17:01:00 +0000https://julianamarialop.github.io/en/articles/chefs-kit-for-data-databricks-asset-bundles-transform-deployment/Deploying data projects across different environments has always been a challenge comparable to trying to replicate a gourmet dish in different kitchens. Even with the same ingredients, small variations in settings, in the…https://julianamarialop.github.io/en/articles/back-to-the-future-of-pipelines-azure-data-factory-databricks-jobs/Mon, 19 May 2025 17:54:00 +0000https://julianamarialop.github.io/en/articles/back-to-the-future-of-pipelines-azure-data-factory-databricks-jobs/For years, data engineers have been building pipelines that connect Azure Data Factory (ADF) to Databricks, a powerful combination that often required manual tweaks and complex configuration. It was like…https://julianamarialop.github.io/en/articles/the-expanding-databricks-universe-new-constellations-of-may-2025/Mon, 19 May 2025 16:58:00 +0000https://julianamarialop.github.io/en/articles/the-expanding-databricks-universe-new-constellations-of-may-2025/The Databricks universe is constantly expanding. Like a living, pulsing galaxy, every month brings new “constellations” of features, integrations, and acquisitions that light the way for data professionals and…https://julianamarialop.github.io/en/articles/scrooge-mcduck-changing-his-mind-real-estate-tokenization-databricks/Sat, 10 May 2025 22:19:00 +0000https://julianamarialop.github.io/en/articles/scrooge-mcduck-changing-his-mind-real-estate-tokenization-databricks/For decades, Scrooge McDuck was the ultimate symbol of concentrated wealth. Owner of a colossal fortune, he preferred to keep his money in a giant vault where he could literally swim in gold coins. Only he…https://julianamarialop.github.io/en/articles/azure-databricks-sql-pipe-syntax-master-query-elements-like-avatar/Fri, 09 May 2025 14:15:00 +0000https://julianamarialop.github.io/en/articles/azure-databricks-sql-pipe-syntax-master-query-elements-like-avatar/Anyone who works with SQL knows that, powerful as the language is, writing and understanding complex queries, especially those with multiple nested subqueries, can be a challenge. We often feel…https://julianamarialop.github.io/en/articles/monsters-inc-no-more-ingestion-scares-databricks-lakeflow-connect/Mon, 05 May 2025 13:16:00 +0000https://julianamarialop.github.io/en/articles/monsters-inc-no-more-ingestion-scares-databricks-lakeflow-connect/Any Monsters, Inc. fans out there? Remember how the whole factory depended on the energy from children’s screams (and later their laughter!) to keep Monstropolis running? In our data universe, we could say that…https://julianamarialop.github.io/en/articles/azure-databricks-delta-sharing-and-the-data-buffet/Thu, 06 Feb 2025 15:11:00 +0000https://julianamarialop.github.io/en/articles/azure-databricks-delta-sharing-and-the-data-buffet/Imagine a world where sharing data between companies is as simple as opening a delivery app and ordering your favorite meal. Instead of complicated processes, slow transfers, and worries…https://julianamarialop.github.io/en/articles/egyptian-gods-and-ai-build-an-llm-on-azure-databricks-rag-fine-tuning-mosaicml/Fri, 31 Jan 2025 14:29:00 +0000https://julianamarialop.github.io/en/articles/egyptian-gods-and-ai-build-an-llm-on-azure-databricks-rag-fine-tuning-mosaicml/If the ancient Egyptians could build an artificial intelligence, they would probably call on their gods to make the system wiser, faster and more efficient. Just as Ra lit up the world, Thoth recorded…https://julianamarialop.github.io/en/articles/azure-databricks-dqx-the-doctor-strange-of-your-data-pipeline/Mon, 20 Jan 2025 13:45:00 +0000https://julianamarialop.github.io/en/articles/azure-databricks-dqx-the-doctor-strange-of-your-data-pipeline/These days, data-driven decision-making is at the heart of every company. However, the effectiveness of those decisions depends directly on the quality of the data used. Inconsistent, incomplete, or…https://julianamarialop.github.io/en/articles/like-a-well-organized-beehive-exploring-azure-databricks-clusters/Wed, 15 Jan 2025 13:45:00 +0000https://julianamarialop.github.io/en/articles/like-a-well-organized-beehive-exploring-azure-databricks-clusters/Databricks is a robust platform for processing and analyzing large volumes of data, supporting workloads that range from data engineering to data science and business analytics. One of the…https://julianamarialop.github.io/en/articles/azure-databricks-and-watsonx-data-in-the-heroes-arena/Mon, 13 Jan 2025 13:30:00 +0000https://julianamarialop.github.io/en/articles/azure-databricks-and-watsonx-data-in-the-heroes-arena/In recent years, the need for robust platforms to analyze large volumes of data has become clear for companies looking to become more data-driven. Two of the most talked-about solutions…https://julianamarialop.github.io/en/articles/modernize-cobol-processes-azure-databricks-pave-recipe-jcl-vsam/Fri, 10 Jan 2025 19:12:00 +0000https://julianamarialop.github.io/en/articles/modernize-cobol-processes-azure-databricks-pave-recipe-jcl-vsam/After writing my previous article on migrating from SAS to Databricks as a cake recipe, I enjoyed the experience so much that I decided to repeat the idea, but this time with COBOL! Since COBOL is a language…https://julianamarialop.github.io/en/articles/how-to-migrate-from-sas-to-azure-databricks-a-foolproof-two-step-recipe/Thu, 09 Jan 2025 13:17:00 +0000https://julianamarialop.github.io/en/articles/how-to-migrate-from-sas-to-azure-databricks-a-foolproof-two-step-recipe/Migrating a well-established platform like SAS to Databricks can be challenging, but when we follow a well-structured recipe, everything becomes simpler and more efficient. In this recipe, we’ll split the process into…https://julianamarialop.github.io/en/articles/turn-your-pipelines-into-superheroes-5-villains-azure-databricks/Thu, 09 Jan 2025 00:13:00 +0000https://julianamarialop.github.io/en/articles/turn-your-pipelines-into-superheroes-5-villains-azure-databricks/How about comparing the mistakes that sabotage your data pipelines to the villains who keep getting in the way of the superheroes’ mission? Picture your pipeline as the hero of the story, tasked with carrying the data to its final destination…https://julianamarialop.github.io/en/articles/rub-the-lamp-and-get-insights-a-guide-to-ai-bi-genie-on-azure-databricks/Mon, 21 Oct 2024 15:47:00 +0000https://julianamarialop.github.io/en/articles/rub-the-lamp-and-get-insights-a-guide-to-ai-bi-genie-on-azure-databricks/Imagine being able to talk to your data as if you were chatting with a coworker or, better yet, as if you were dealing with the genie of a magic lamp. That’s exactly what AI/BI Genie makes possible!…https://julianamarialop.github.io/en/articles/from-zero-to-bronze-build-a-notebook-without-coding-databricks-assistant/Wed, 09 Oct 2024 21:31:00 +0000https://julianamarialop.github.io/en/articles/from-zero-to-bronze-build-a-notebook-without-coding-databricks-assistant/In the data world, building a bronze layer, which organizes raw data into an accessible structure for later transformations, has traditionally involved a lot of coding. Recently, however, I explored a…https://julianamarialop.github.io/en/articles/may-the-analytical-force-be-with-you-exploring-databricks-star-wars-style/Mon, 23 Sep 2024 15:56:00 +0000https://julianamarialop.github.io/en/articles/may-the-analytical-force-be-with-you-exploring-databricks-star-wars-style/In recent years, Databricks has established itself as a powerful platform, much like the Force that guides the Jedi on their missions, helping companies accelerate their data and analytics projects. Just as the union between the…https://julianamarialop.github.io/en/articles/clash-of-the-titans-databricks-vs-snowflake-in-the-mortal-data-battle/Tue, 30 Jul 2024 15:00:00 +0000https://julianamarialop.github.io/en/articles/clash-of-the-titans-databricks-vs-snowflake-in-the-mortal-data-battle/Databricks and Snowflake are two big names in cloud data solutions. Both platforms have been key to helping companies generate value from their internal and external data assets. Each platform…https://julianamarialop.github.io/en/articles/data-analytics-on-scorched-earth-the-decline-of-azure-synapse/Fri, 26 Jul 2024 22:23:00 +0000https://julianamarialop.github.io/en/articles/data-analytics-on-scorched-earth-the-decline-of-azure-synapse/Azure Synapse is Microsoft’s Data Warehouse solution on the Azure cloud. However, it has been facing internal competition from Databricks for some time and, since last year, from Microsoft Fabric as well.https://julianamarialop.github.io/en/articles/demystifying-databricks-unity-catalog-simplifying-data-governance-and-auditing/Thu, 18 Jul 2024 14:32:00 +0000https://julianamarialop.github.io/en/articles/demystifying-databricks-unity-catalog-simplifying-data-governance-and-auditing/Databricks Unity Catalog makes it easier for your team to understand its data. It automatically generates information about how data is used, based on cluster consumption logs. This reflects how…https://julianamarialop.github.io/en/articles/connected-future-unity-catalog-and-microsoft-fabric-working-together/Wed, 10 Jul 2024 13:28:00 +0000https://julianamarialop.github.io/en/articles/connected-future-unity-catalog-and-microsoft-fabric-working-together/This article reflects my personal experiences and points of view, not the official position of Microsoft or Databricks. In addition, although this post describes potential scenarios, it does not necessarily reflect the…https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-4-v-order/Mon, 08 Jul 2024 16:55:00 +0000https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-4-v-order/We’ve reached the end of the “Integrating Azure Databricks and Microsoft Fabric” series. Throughout this journey, we explored how these two powerful tools can be combined to optimize data analytics processes…https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-iii-onelake-gold-layer/Fri, 05 Jul 2024 17:26:00 +0000https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-iii-onelake-gold-layer/Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-2-reporting-layer/Thu, 04 Jul 2024 14:18:00 +0000https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-2-reporting-layer/Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-a-meeting-of-giants-part-1/Tue, 02 Jul 2024 20:17:00 +0000https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-a-meeting-of-giants-part-1/Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.https://julianamarialop.github.io/en/articles/how-to-calculate-real-cost-azure-databricks-sql-warehouse-instances/Sat, 29 Jun 2024 22:21:00 +0000https://julianamarialop.github.io/en/articles/how-to-calculate-real-cost-azure-databricks-sql-warehouse-instances/Shall we learn how to calculate the cost of SQL Warehouse instances in Microsoft Azure Databricks the easy way? Then follow along with this article.https://julianamarialop.github.io/en/articles/databricks-sql-and-dimensional-modeling-optimizing-your-data-warehouse/Thu, 23 May 2024 14:06:00 +0000https://julianamarialop.github.io/en/articles/databricks-sql-and-dimensional-modeling-optimizing-your-data-warehouse/In the dynamic landscape of information storage, the methodologies used to structure data play a crucial role in uncovering valuable insights for informed decision-making. Since the…https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-exploring-practical-examples/Thu, 25 Apr 2024 14:23:00 +0000https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-exploring-practical-examples/Following up on the previous article, “Databricks Lakehouse Federation: An Alternative to Zero ETL?”, let’s explore some practical examples of combining data from different sources.https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-an-alternative-to-zero-etl/Wed, 24 Apr 2024 20:48:00 +0000https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-an-alternative-to-zero-etl/Released in December 2023, the Lakehouse Federation feature is currently in Public Preview, but it has enormous potential. In today’s article, I’ll explore some interesting perspectives, such as the ability to…https://julianamarialop.github.io/en/articles/databricks-best-practices-managing-bronze-silver-gold-medallion/Thu, 05 Oct 2023 13:00:00 +0000https://julianamarialop.github.io/en/articles/databricks-best-practices-managing-bronze-silver-gold-medallion/Many of the clients I work with implement a Medallion architecture to logically organize their data in a Lakehouse. As data flows, it moves through several stages or layers. The best-known design…https://julianamarialop.github.io/en/articles/azure-data-factory-and-azure-databricks-best-practices/Fri, 29 Sep 2023 13:00:00 +0000https://julianamarialop.github.io/en/articles/azure-data-factory-and-azure-databricks-best-practices/Hello everyone, this week we’ll go over some best practices for a data ingestion pipeline:https://julianamarialop.github.io/en/articles/key-differences-between-azure-synapse-and-databricks/Thu, 21 Sep 2023 15:31:00 +0000https://julianamarialop.github.io/en/articles/key-differences-between-azure-synapse-and-databricks/Hi everyone, in this article I’ll talk a bit about these data platforms.https://julianamarialop.github.io/en/articles/databricks-tip-track-data-source-by-file-input-file-name/Thu, 22 Jun 2023 13:24:00 +0000https://julianamarialop.github.io/en/articles/databricks-tip-track-data-source-by-file-input-file-name/Wouldn’t it be a dream to have a column in your Bronze table (or Landing, whatever you prefer to call it) telling you which file each record came from?https://julianamarialop.github.io/en/articles/databricks-the-swiss-army-knife-of-your-big-data-platform/Tue, 23 Apr 2019 16:07:00 +0000https://julianamarialop.github.io/en/articles/databricks-the-swiss-army-knife-of-your-big-data-platform/Imagine a team made up of a SQL developer, a Python professional and a Scala contributor. They all work together to ingest data into your Datalake, your Datawarehouse or simply…