https://julianamarialop.github.io/en/tags/data-engineering/Recent content in Data Engineering on Ahead of the AIHugoen-USMon, 22 Jun 2026 11:48:00 +0000https://julianamarialop.github.io/en/articles/rayfin-the-missing-mechanic-to-get-your-car-on-the-podium/Mon, 22 Jun 2026 11:48:00 +0000https://julianamarialop.github.io/en/articles/rayfin-the-missing-mechanic-to-get-your-car-on-the-podium/Imagine you’ve hired the most talented driver in Formula 1. He’s fast, intuitive, and can get the most out of any car. But when he gets to the starting grid, the vehicle you handed him is a prototype with no…https://julianamarialop.github.io/en/articles/brain-behind-the-magic-anthropic-claude-professor-xavier-of-analytics/Mon, 08 Jun 2026 12:03:00 +0000https://julianamarialop.github.io/en/articles/brain-behind-the-magic-anthropic-claude-professor-xavier-of-analytics/If you’ve ever watched the X-Men, you know Professor Xavier’s power isn’t just reading minds. His real power lies in Cerebro, the machine that amplifies his abilities and lets him find the needle in the…https://julianamarialop.github.io/en/articles/beast-didnt-build-cerebro-in-a-day-claude-managed-agents/Mon, 20 Apr 2026 21:51:00 +0000https://julianamarialop.github.io/en/articles/beast-didnt-build-cerebro-in-a-day-claude-managed-agents/When the X-Men go into battle, the world sees Wolverine, Storm, Cyclops. Nobody sees Hank McCoy, the Beast, working on the systems underneath. But he’s the one who designed Cerebro. He’s the one who keeps the Danger Room…https://julianamarialop.github.io/en/articles/storm-didnt-learn-to-fly-vs-code-with-claude-without-being-a-developer/Thu, 16 Apr 2026 11:30:00 +0000https://julianamarialop.github.io/en/articles/storm-didnt-learn-to-fly-vs-code-with-claude-without-being-a-developer/There’s a belief in the data world that few people question: the IDE is developer territory. VS Code, with its folders, files, and terminal, would be a hostile environment for anyone who doesn’t write code every day.https://julianamarialop.github.io/en/articles/databricks-lakebase-definitive-guide-operational-data-and-ai-agents/Tue, 03 Mar 2026 12:31:00 +0000https://julianamarialop.github.io/en/articles/databricks-lakebase-definitive-guide-operational-data-and-ai-agents/Databricks Lakebase represents a significant evolution in data architecture, specifically in the domain of OLTP (Online Transactional Processing) databases. It isn’t just another database…https://julianamarialop.github.io/en/articles/the-force-awakens-from-data-engineer-to-ai-data-engineer-in-90-days-with-aws/Tue, 24 Feb 2026 11:56:00 +0000https://julianamarialop.github.io/en/articles/the-force-awakens-from-data-engineer-to-ai-data-engineer-in-90-days-with-aws/In the vast Star Wars universe, the Force connects everything, but only the most dedicated Jedi manage to master it to build and wield lightsabers and intelligent holocrons. In our technology universe, the transition…https://julianamarialop.github.io/en/articles/leap-of-faith-data-engineer-to-ai-engineer-in-90-days-microsoft/Fri, 20 Feb 2026 13:40:00 +0000https://julianamarialop.github.io/en/articles/leap-of-faith-data-engineer-to-ai-engineer-in-90-days-microsoft/In the Assassin’s Creed universe, the Leap of Faith is an act of courage, technique and trust. It isn’t a blind jump, but a calculated move that demands knowledge of the terrain and precision to reach a new level. In the…https://julianamarialop.github.io/en/articles/from-etl-to-etl-a-the-rise-of-autonomous-pipelines/Wed, 21 Jan 2026 17:30:00 +0000https://julianamarialop.github.io/en/articles/from-etl-to-etl-a-the-rise-of-autonomous-pipelines/Imagine you need to get from point A to point B. A few decades ago, your only option was a car with a manual transmission, demanding full attention to every gear change and every move in traffic. Over time…https://julianamarialop.github.io/en/articles/the-floor-plan-of-software-demystifying-function-point-analysis/Thu, 08 Jan 2026 21:19:00 +0000https://julianamarialop.github.io/en/articles/the-floor-plan-of-software-demystifying-function-point-analysis/Have you ever tried to explain to a client how much a software project will cost and heard: “But why does it take so long?” or “Why does it cost so much?”. The difficulty lies in translating functionality into size, and size into…https://julianamarialop.github.io/en/articles/ai-data-engineer-the-evolution-of-the-data-engineer-in-the-age-of-intelligent-agents/Wed, 07 Jan 2026 14:21:00 +0000https://julianamarialop.github.io/en/articles/ai-data-engineer-the-evolution-of-the-data-engineer-in-the-age-of-intelligent-agents/The data engineering profession is going through its biggest transformation since the move to the cloud. For years, the focus was on building efficient pipelines, moving data from one point to another and making sure…https://julianamarialop.github.io/en/articles/microsoft-fabric-at-ignite-2025-the-complete-guide-to-whats-new/Tue, 25 Nov 2025 15:53:00 +0000https://julianamarialop.github.io/en/articles/microsoft-fabric-at-ignite-2025-the-complete-guide-to-whats-new/Microsoft Ignite 2025 wasn’t an event about future promises; it was about concrete deliveries that mark the maturity of Microsoft Fabric. The platform stopped being a collection of promising tools and became…https://julianamarialop.github.io/en/articles/mission-to-mars-protect-data-pipelines-with-aerospace-precision/Tue, 16 Sep 2025 17:05:00 +0000https://julianamarialop.github.io/en/articles/mission-to-mars-protect-data-pipelines-with-aerospace-precision/In aerospace engineering, failure is not an option. Every rocket launch, every orbital maneuver, and every landing on a distant planet is the result of thousands of hours of testing, simulations, and…https://julianamarialop.github.io/en/articles/la-traviata-of-data-how-fabric-failed-to-win-the-market/Mon, 25 Aug 2025 14:21:00 +0000https://julianamarialop.github.io/en/articles/la-traviata-of-data-how-fabric-failed-to-win-the-market/In May 2023, Microsoft announced Microsoft Fabric as a revolutionary solution to unify every analytical workload on a single platform. The promise was ambitious: lakehouse, data warehouse…https://julianamarialop.github.io/en/articles/perfect-blueprint-for-analytics-on-azure-well-architected-review/Fri, 06 Jun 2025 15:58:00 +0000https://julianamarialop.github.io/en/articles/perfect-blueprint-for-analytics-on-azure-well-architected-review/In today’s world, where data is the new oil (or maybe the new bricks?), building robust, efficient Analytics solutions on Azure has become a complex engineering task. Platforms like Microsoft Fabric…https://julianamarialop.github.io/en/articles/spark-land-4-0-data-amusement-park-just-opened-its-gates/Thu, 29 May 2025 12:56:00 +0000https://julianamarialop.github.io/en/articles/spark-land-4-0-data-amusement-park-just-opened-its-gates/For more than a decade, Apache Spark has been the data processing platform of choice for data engineers and data scientists around the world. With its ability to process large volumes of information in…https://julianamarialop.github.io/en/articles/chefs-kit-for-data-databricks-asset-bundles-transform-deployment/Thu, 22 May 2025 17:01:00 +0000https://julianamarialop.github.io/en/articles/chefs-kit-for-data-databricks-asset-bundles-transform-deployment/Deploying data projects across different environments has always been a challenge comparable to trying to replicate a gourmet dish in different kitchens. Even with the same ingredients, small variations in settings, in the…https://julianamarialop.github.io/en/articles/back-to-the-future-of-pipelines-azure-data-factory-databricks-jobs/Mon, 19 May 2025 17:54:00 +0000https://julianamarialop.github.io/en/articles/back-to-the-future-of-pipelines-azure-data-factory-databricks-jobs/For years, data engineers have been building pipelines that connect Azure Data Factory (ADF) to Databricks, a powerful combination that often required manual tweaks and complex configuration. It was like…https://julianamarialop.github.io/en/articles/the-expanding-databricks-universe-new-constellations-of-may-2025/Mon, 19 May 2025 16:58:00 +0000https://julianamarialop.github.io/en/articles/the-expanding-databricks-universe-new-constellations-of-may-2025/The Databricks universe is constantly expanding. Like a living, pulsing galaxy, every month brings new “constellations” of features, integrations, and acquisitions that light the way for data professionals and…https://julianamarialop.github.io/en/articles/monsters-inc-no-more-ingestion-scares-databricks-lakeflow-connect/Mon, 05 May 2025 13:16:00 +0000https://julianamarialop.github.io/en/articles/monsters-inc-no-more-ingestion-scares-databricks-lakeflow-connect/Any Monsters, Inc. fans out there? Remember how the whole factory depended on the energy from children’s screams (and later their laughter!) to keep Monstropolis running? In our data universe, we could say that…https://julianamarialop.github.io/en/articles/holy-upgrade-batman-robin-f2-sku-now-has-copilot-in-fabric/Thu, 01 May 2025 21:38:00 +0000https://julianamarialop.github.io/en/articles/holy-upgrade-batman-robin-f2-sku-now-has-copilot-in-fabric/Hey there, data community! You know that feeling of being in Gotham City… I mean, in our Data City? It’s vibrant, full of opportunities, but also packed with challenges: we’ve got the villains of…https://julianamarialop.github.io/en/articles/how-to-migrate-from-sas-to-microsoft-fabric-2-layer-banoffee-recipe/Sat, 19 Apr 2025 16:13:00 +0000https://julianamarialop.github.io/en/articles/how-to-migrate-from-sas-to-microsoft-fabric-2-layer-banoffee-recipe/Migrating an established platform like SAS can look like a complex challenge. But, just like making an irresistible Banoffee, if we follow the steps calmly and methodically, the result will be delicious! Inspired…https://julianamarialop.github.io/en/articles/streaming-in-microsoft-fabric-the-production-line-of-the-future/Thu, 13 Feb 2025 18:06:00 +0000https://julianamarialop.github.io/en/articles/streaming-in-microsoft-fabric-the-production-line-of-the-future/Picture a modern production line where products are manufactured in real time, without interruption. Now swap the products for data, and you’ll have an idea of what data streaming is. Welcome to the factory…https://julianamarialop.github.io/en/articles/dbt-the-personal-trainer-that-turns-your-data-into-elite-athletes/Mon, 10 Feb 2025 12:46:00 +0000https://julianamarialop.github.io/en/articles/dbt-the-personal-trainer-that-turns-your-data-into-elite-athletes/Imagine your data is like a couch potato who decided to get in shape. DBT (Data Build Tool) is the personal trainer who takes that data, builds the right workout plan (SQL transformations) and takes it to a new…https://julianamarialop.github.io/en/articles/azure-databricks-dqx-the-doctor-strange-of-your-data-pipeline/Mon, 20 Jan 2025 13:45:00 +0000https://julianamarialop.github.io/en/articles/azure-databricks-dqx-the-doctor-strange-of-your-data-pipeline/These days, data-driven decision-making is at the heart of every company. However, the effectiveness of those decisions depends directly on the quality of the data used. Inconsistent, incomplete, or…https://julianamarialop.github.io/en/articles/like-a-well-organized-beehive-exploring-azure-databricks-clusters/Wed, 15 Jan 2025 13:45:00 +0000https://julianamarialop.github.io/en/articles/like-a-well-organized-beehive-exploring-azure-databricks-clusters/Databricks is a robust platform for processing and analyzing large volumes of data, supporting workloads that range from data engineering to data science and business analytics. One of the…https://julianamarialop.github.io/en/articles/turn-your-pipelines-into-superheroes-5-villains-azure-databricks/Thu, 09 Jan 2025 00:13:00 +0000https://julianamarialop.github.io/en/articles/turn-your-pipelines-into-superheroes-5-villains-azure-databricks/How about comparing the mistakes that sabotage your data pipelines to the villains who keep getting in the way of the superheroes’ mission? Picture your pipeline as the hero of the story, tasked with carrying the data to its final destination…https://julianamarialop.github.io/en/articles/from-zero-to-bronze-build-a-notebook-without-coding-databricks-assistant/Wed, 09 Oct 2024 21:31:00 +0000https://julianamarialop.github.io/en/articles/from-zero-to-bronze-build-a-notebook-without-coding-databricks-assistant/In the data world, building a bronze layer, which organizes raw data into an accessible structure for later transformations, has traditionally involved a lot of coding. Recently, however, I explored a…https://julianamarialop.github.io/en/articles/may-the-analytical-force-be-with-you-exploring-databricks-star-wars-style/Mon, 23 Sep 2024 15:56:00 +0000https://julianamarialop.github.io/en/articles/may-the-analytical-force-be-with-you-exploring-databricks-star-wars-style/In recent years, Databricks has established itself as a powerful platform, much like the Force that guides the Jedi on their missions, helping companies accelerate their data and analytics projects. Just as the union between the…https://julianamarialop.github.io/en/articles/clash-of-the-titans-databricks-vs-snowflake-in-the-mortal-data-battle/Tue, 30 Jul 2024 15:00:00 +0000https://julianamarialop.github.io/en/articles/clash-of-the-titans-databricks-vs-snowflake-in-the-mortal-data-battle/Databricks and Snowflake are two big names in cloud data solutions. Both platforms have been key to helping companies generate value from their internal and external data assets. Each platform…https://julianamarialop.github.io/en/articles/microsoft-fabric-pause-resume-scale-capacity-with-data-factory-pipeline/Mon, 15 Jul 2024 14:30:00 +0000https://julianamarialop.github.io/en/articles/microsoft-fabric-pause-resume-scale-capacity-with-data-factory-pipeline/There are several ways to pause, resume, or scale a Microsoft Fabric Capacity. Note that this applies only to F-SKUs in Azure, since Premium Capacity (P-SKU) can only be scaled.https://julianamarialop.github.io/en/articles/databricks-sql-and-dimensional-modeling-optimizing-your-data-warehouse/Thu, 23 May 2024 14:06:00 +0000https://julianamarialop.github.io/en/articles/databricks-sql-and-dimensional-modeling-optimizing-your-data-warehouse/In the dynamic landscape of information storage, the methodologies used to structure data play a crucial role in uncovering valuable insights for informed decision-making. Since the…https://julianamarialop.github.io/en/articles/best-practices-for-using-data-fabric-workspaces-with-git/Mon, 20 May 2024 16:55:00 +0000https://julianamarialop.github.io/en/articles/best-practices-for-using-data-fabric-workspaces-with-git/Effective management of data pipelines is key to ensuring continuous development, testing, and deployment processes. Integrating Git with Microsoft Fabric workspaces offers a…https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-exploring-practical-examples/Thu, 25 Apr 2024 14:23:00 +0000https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-exploring-practical-examples/Following up on the previous article, “Databricks Lakehouse Federation: An Alternative to Zero ETL?”, let’s explore some practical examples of combining data from different sources.https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-an-alternative-to-zero-etl/Wed, 24 Apr 2024 20:48:00 +0000https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-an-alternative-to-zero-etl/Released in December 2023, the Lakehouse Federation feature is currently in Public Preview, but it has enormous potential. In today’s article, I’ll explore some interesting perspectives, such as the ability to…https://julianamarialop.github.io/en/articles/continuous-integration-and-delivery-cicd-in-azure-data-factory/Thu, 12 Oct 2023 13:00:00 +0000https://julianamarialop.github.io/en/articles/continuous-integration-and-delivery-cicd-in-azure-data-factory/CI is the practice of automatically testing every code change, while CD deploys those changes to staging or production systems. In Azure Data Factory, CI/CD involves moving pipelines between…https://julianamarialop.github.io/en/articles/azure-data-factory-and-azure-databricks-best-practices/Fri, 29 Sep 2023 13:00:00 +0000https://julianamarialop.github.io/en/articles/azure-data-factory-and-azure-databricks-best-practices/Hello everyone, this week we’ll go over some best practices for a data ingestion pipeline:https://julianamarialop.github.io/en/articles/hands-on-with-microsoft-fabric-a-beginners-guide-part-1/Fri, 22 Sep 2023 13:00:00 +0000https://julianamarialop.github.io/en/articles/hands-on-with-microsoft-fabric-a-beginners-guide-part-1/Traditional large-scale data analytics solutions rely on data warehouses and SQL queries to store and retrieve data. However, the rise of big data, characterized by large #volumes …https://julianamarialop.github.io/en/articles/azure-synapse-analytics-vs-microsoft-fabric-another-migration-project/Tue, 30 May 2023 17:44:00 +0000https://julianamarialop.github.io/en/articles/azure-synapse-analytics-vs-microsoft-fabric-another-migration-project/In May 2023, Microsoft announced Microsoft Fabric. This new solution extends the integration promise made in Azure Synapse Analytics to cover every analytics workload, and it can be used…https://julianamarialop.github.io/en/articles/big-data-architecture-choose-the-right-plumbing-for-your-pipelines/Mon, 15 Jun 2020 16:40:00 +0000https://julianamarialop.github.io/en/articles/big-data-architecture-choose-the-right-plumbing-for-your-pipelines/Last week, I wrote an article about data pipelines (check it out here) covering the main characteristics of a robust and optimized build, ideally using a…https://julianamarialop.github.io/en/articles/data-pipelines-is-your-company-plumbing-clean-water-or-sewage/Fri, 12 Jun 2020 20:21:00 +0000https://julianamarialop.github.io/en/articles/data-pipelines-is-your-company-plumbing-clean-water-or-sewage/Pipeline is an English term that can be translated into Portuguese as “tubagem” or “canalização” (piping or plumbing). In our language, the concept is used to refer to a computing architecture. These virtual pipes are created…