[{"content":"Anyone who has played Minecraft knows the first decision in the game isn\u0026rsquo;t about where to build. It\u0026rsquo;s about which mode to play.\nIn Creative, your inventory comes full. You open the menu, drag in whatever block you want, and the house goes up in minutes. In Survival, you start empty-handed: you punch a tree to get wood, craft a workbench, then a pickaxe, then head down into the mine. The same house takes hours.\nThe Creative player builds faster. The Survival player knows where every block came from, what it cost, and what happens if they want to move to another biome.\nThat\u0026rsquo;s exactly the difference between the two ways of building enterprise agents today. And if you\u0026rsquo;ve never opened an Oracle console in your life, this article is for you, because we\u0026rsquo;re going to play Survival from start to finish, punching the first tree and making it all the way to the debug screen, with code that actually works.\nOn one side, OCI Enterprise AI, which went GA in March 2026. On the other, Copilot Studio, rebuilt at Build 2026. I\u0026rsquo;ll spend most of the time on the first one, because it\u0026rsquo;s the less known and the more poorly explained of the two.\nWhy Oracle Decided to Sell Blocks Oracle spent years with a perception problem. Everyone knew it had databases and ERP. Almost nobody associated its name with an AI platform.\nOCI Enterprise AI was the answer, and it brings together in a single offering three layers that used to live apart.\nThe first is intelligence, meaning the models and inference. And here it\u0026rsquo;s worth noticing something almost nobody expects from Oracle: the catalog is third-party. Grok from xAI, Gemini from Google, open gpt-oss, Cohere models, and the option to import your own.\nThe second is the ability to act, meaning the agents and the tools. Document search, a code interpreter in an isolated container, function calling, MCP server calls, and question-to-SQL conversion.\nThe third is control, which comes built in instead of being stitched on later. Identity through OCI IAM, zero data retention endpoints, tracing of every step executed, a sovereign AI option, and dedicated clusters for anyone who needs isolation.\nThe smartest decision this platform made was about the API. Instead of inventing its own dialect, Oracle adopted the OpenAI Responses API as its interface. You use the official OpenAI SDK, swap the base URL, the authentication, and the model name, and the rest of your code stays the same. LangChain, LlamaIndex, and the OpenAI Agents SDK work without refactoring.\nTranslating for anyone who has never seen Oracle: you don\u0026rsquo;t need to learn Oracle to use Oracle. That\u0026rsquo;s the whole point.\nEntering the World Let\u0026rsquo;s build for real. The scenario is an agent that answers questions about the internal reimbursement policy and queries the expense system when it needs a real number.\n1. Permission to join the server Nothing happens on OCI without an IAM policy. A compartment is the logical box where resources live, and the policy says who can do what inside it. Ask your administrator for:\nallow group \u0026lt;seu-grupo\u0026gt; to manage generative-ai-family in compartment \u0026lt;seu-compartimento\u0026gt; This is a sandbox policy, the equivalent of joining the server with permission to build anywhere. In production you narrow it down, and in step 3 I\u0026rsquo;ll show how to tie the permission to a specific key.\n2. Generating the world, and the choices you can\u0026rsquo;t take back A project is the resource that organizes agents and related assets. You create it through the console, and on that screen there are three decisions a lot of people click through without reading.\nRetention defines how long responses and conversations are kept, with a maximum of 720 hours, or 30 days.\nShort-term memory compaction summarizes recent history to save tokens and latency. You choose the compaction model at creation time and can\u0026rsquo;t change it afterward.\nLong-term memory extracts and persists important information from conversations as embeddings, and requires choosing an extraction model and an embedding model at creation time.\nIn those last two cases, if you enabled it and want to undo it, the way out is to delete the project. It\u0026rsquo;s like the world seed. Once you pick it, that\u0026rsquo;s the one. Save the project\u0026rsquo;s OCID; it goes into the code.\n3. The pickaxe Create a Generative AI API key through the console, grab its OCID, and tie the permission to that specific key:\nallow group \u0026lt;grupo-dos-builders\u0026gt; to manage generative-ai-response in compartment \u0026lt;seu-compartimento\u0026gt; where ALL {request.principal.type=\u0026#39;generativeaiapikey\u0026#39;, request.principal.id=\u0026#39;\u0026lt;ocid-da-chave\u0026gt;\u0026#39;} An API key is for testing and early development. For production, Oracle\u0026rsquo;s recommendation is IAM authentication, with an instance or resource principal, which avoids long-lived credentials.\n4. Punching the first tree Install the OpenAI SDK, not the OCI SDK. That detail trips up a lot of people on day one.\npip install openai And the first agent:\nfrom openai import OpenAI client = OpenAI( base_url=\u0026#34;https://inference.generativeai.us-chicago-1.oci.oraclecloud.com/openai/v1\u0026#34;, api_key=\u0026#34;\u0026lt;sua-api-key\u0026gt;\u0026#34;, project=\u0026#34;ocid1.generativeaiproject.oc1.us-chicago-1.xxxxxxxx\u0026#34; ) response = client.responses.create( model=\u0026#34;xai.grok-4.3\u0026#34;, input=\u0026#34;Explique em uma frase o que é um banco de dados.\u0026#34; ) print(response.output_text) Three things to notice, because they explain the philosophy of the entire platform.\nThe base URL carries the region. Swap us-chicago-1 for yours and the traffic changes continents, which matters when there\u0026rsquo;s a data residency requirement.\nThe model is a string, and Oracle hosts third-party models. Grok from xAI, Gemini from Google, open gpt-oss. Switching models means changing one line, and that\u0026rsquo;s how you trade cost against quality without rewriting the application.\nIf you want dedicated capacity instead of on-demand, the model identifier becomes the OCID of the cluster endpoint. Same code, isolated infrastructure.\n5. The workbench An agent that only knows what the model already knew is useless for anything enterprise. It needs to answer based on your reimbursement policy, not based on the internet.\nYou upload the file, create a vector store, and declare the search tool right in the call:\nresponse = client.responses.create( model=\u0026#34;openai.gpt-oss-120b\u0026#34;, input=\u0026#34;Qual o prazo para solicitar reembolso de viagem?\u0026#34;, tools=[ { \u0026#34;type\u0026#34;: \u0026#34;file_search\u0026#34;, \u0026#34;vector_store_ids\u0026#34;: [\u0026#34;\u0026lt;id-do-vector-store\u0026gt;\u0026#34;] } ] ) Notice that you didn\u0026rsquo;t build a RAG pipeline. Retrieval is managed by the platform. It\u0026rsquo;s the game\u0026rsquo;s workbench, that moment when loose blocks become a tool.\n6. Redstone, or the agent that acts Looking things up in a document is nice, but a real agent executes. Function calling is the standard for that, and the mechanics matter: the model doesn\u0026rsquo;t run your function. It returns the function\u0026rsquo;s name and arguments, your application executes it, and you send the result back.\ntools = [ { \u0026#34;type\u0026#34;: \u0026#34;function\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;consultar_despesas\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Retorna as despesas lançadas por um colaborador em um período.\u0026#34;, \u0026#34;parameters\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;colaborador\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;}, \u0026#34;mes\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Formato 2026-08\u0026#34;} }, \u0026#34;required\u0026#34;: [\u0026#34;colaborador\u0026#34;, \u0026#34;mes\u0026#34;], }, }, ] You\u0026rsquo;re the one who executes. You\u0026rsquo;re the one with access to the system. The model only asks. For enterprise architecture, that boundary is the difference between a proof of concept and something that passes a security review.\nCount the round trips on that path: the application sends the prompt, the model returns the call, the application executes, the application returns the result, and only then does the model answer. That\u0026rsquo;s two full trips between your application and the platform.\nIf the tool already exists on an MCP server, you cut that back-and-forth in half. Declare the server right in the call and the platform talks to it directly, without handing control back to your code midway. Three steps instead of five, one trip instead of two:\nresponse = client.responses.create( model=\u0026#34;openai.gpt-oss-120b\u0026#34;, tools=[ { \u0026#34;type\u0026#34;: \u0026#34;mcp\u0026#34;, \u0026#34;server_label\u0026#34;: \u0026#34;despesas\u0026#34;, \u0026#34;server_url\u0026#34;: \u0026#34;https://interno.exemplo.com/mcp\u0026#34;, \u0026#34;require_approval\u0026#34;: \u0026#34;never\u0026#34;, \u0026#34;allowed_tools\u0026#34;: [\u0026#34;consultar_despesas\u0026#34;] } ], input=\u0026#34;Quanto o time de dados gastou em viagem em agosto?\u0026#34; ) 7. The treasure map This is the most Oracle piece of all, and the one that matters most to anyone who works with data. Enterprise AI\u0026rsquo;s NL2SQL converts a natural language question into validated SQL, using a semantic enrichment layer that maps business terms to tables, columns, and joins.\nThe security detail that deserves applause: NL2SQL generates the SQL and doesn\u0026rsquo;t execute it. The one that executes is the Database Tools MCP Server, using the end user\u0026rsquo;s identity. And the configuration requires two separate connections: an enrichment one, with higher privilege to read metadata and samples, and a query one, with lower privilege to run on the user\u0026rsquo;s behalf.\nIf you come from data engineering, you recognize the pattern immediately. It\u0026rsquo;s separation of duties applied to an agent.\n8. Pressing F3 Every Minecraft player knows the debug screen. In the Responses API it comes for free: every response carries an output field that is the list of steps executed, with types like message, file_search_call, and mcp_call.\nfor item in response.output: print(item.type) Native traceability, without instrumenting anything. And if you want full observability, with latency and cost, you can plug in Langfuse by swapping only the OpenAI client import.\nCreative Mode On the other side, Copilot Studio solves the same problem with a full inventory.\nYou don\u0026rsquo;t write code. You create the agent on a screen, turn on generative orchestration, and the model decides on its own which tool to call. Since Build 2026 the product has four surfaces plus one: Skills, which are reusable instructions in markdown, Tools, Knowledge, Connected agents, and Memory.\nIts power isn\u0026rsquo;t in those surfaces. It\u0026rsquo;s in Microsoft IQ, the context layer with three sources: Work IQ, with email, chat, files, calendar, and people from Microsoft 365; Fabric IQ, with business data in Fabric; and Foundry IQ, with knowledge bases indexed by Azure AI Search. Connecting a source gives you read and action at the same time, within the signed-in user\u0026rsquo;s permissions.\nIf your company lives in Microsoft 365, the inventory is already full and the house goes up in minutes. If it doesn\u0026rsquo;t, you\u0026rsquo;re in Creative mode in a world where the blocks that matter didn\u0026rsquo;t come in the menu, and every connector is an attempt to import blocks from another world.\nPutting the two side by side, the difference shows up in five points. You build either by writing code against an API or by clicking through screens with automatic orchestration. The agent runs in your OCI account, in the region you choose, or on Microsoft\u0026rsquo;s platform. The context you get for free is your database and your files on one side, and Microsoft 365, Fabric, and Foundry on the other. The bill arrives either per character transaction and cluster commitment, or per credit consumed per action. And each one wins where the data already lives.\nThe Cost of Each Block Oracle charges for ore. On-demand inference is metered in transactions, and a transaction is one character, counting prompt and response together. A dedicated cluster requires a minimum commitment of 744 unit-hours, which is a full month. Imported models don\u0026rsquo;t carry that requirement.\nMicrosoft charges for construction. Copilot Credits come in packs of 25,000 for 200 dollars a month, or pay-as-you-go on Azure at the same rate.\nThat changes who approves the bill inside the client. One goes to infrastructure, with capacity and a monthly commitment. The other goes to the business side, with credit per action. I\u0026rsquo;ve seen proposals get miscalibrated by missing this and end up in front of the wrong committee.\nIn the End, the Same Build Whoever plays Survival and whoever plays Creative end up with the same house on the screen. What changes is who carried the materials and how much each one knows about their own build.\nAnd the two modes are converging faster than the sales pitch admits. Oracle and Microsoft both adopted MCP as the tool protocol. Both decided that an agent needs its own identity, one with IAM authentication on hosted endpoints, the other with an automatic Entra Agent ID on every new agent since July 2026. Both understood that reusable instructions need to be a versionable artifact, not text pasted into a field.\nComing from opposite starting points, they arrived at the same three conclusions. It\u0026rsquo;s as if the two factories had agreed on the same fitting.\nSo the question isn\u0026rsquo;t which mode is better. It\u0026rsquo;s which side of the boundary holds the data that drives your business. If the answer is email, meetings, and files, the Creative inventory already has you covered. If it\u0026rsquo;s a database, your own applications, or data that can\u0026rsquo;t cross a geographic border, grab the pickaxe.\nThe code is up there, and it runs today.\nReferences Quick Start Guide, Enterprise AI Agents: https://docs.oracle.com/en-us/iaas/Content/generative-ai/get-started-agents.htm OCI OpenAI-Compatible Endpoints: https://docs.oracle.com/en-us/iaas/Content/generative-ai/openai-compatible-api.htm Announcing GA of OCI Enterprise AI: https://blogs.oracle.com/ai-and-datascience/announcing-oci-enterprise-ai-ga What\u0026rsquo;s New in Oracle AI, August 2026: https://blogs.oracle.com/ai-and-datascience/whats-new-in-ai-august-2026 Paying for Dedicated AI Clusters: https://docs.oracle.com/en-us/iaas/Content/generative-ai/pay-dedicated.htm Build an agent, Microsoft Copilot Studio: https://learn.microsoft.com/en-us/microsoft-copilot-studio/agents-experience/build-overview What\u0026rsquo;s new in Copilot Studio: https://learn.microsoft.com/en-us/microsoft-copilot-studio/whats-new ","permalink":"https://julianamarialop.github.io/en/articles/survival-or-creative-build-your-first-ai-agent-on-oracle/","summary":"Anyone who has played Minecraft knows the first decision in the game isn\u0026rsquo;t about where to build. It\u0026rsquo;s about which mode to play.","title":"Survival or Creative: How to Build Your First AI Agent on Oracle Without Ever Having Opened the Console"},{"content":"The Value of the Pieces Every chess player learns a table early on and carries it for life: a pawn is worth 1, knight and bishop are worth 3, a rook is worth 5, the queen is worth 9. It isn\u0026rsquo;t a rule of the game, it\u0026rsquo;s centuries of accumulated wisdom about the relative power of each piece. And from it come the lessons that separate people who play from people who understand. You don\u0026rsquo;t use the queen for pawn work, because every move she makes on a small task is a move in which she isn\u0026rsquo;t deciding the game. Philidor wrote in the 18th century that pawns are the soul of chess: games are won and lost on their structure, the humble pieces that do the volume work. And the king, curiously the least powerful piece on the board, is the only irreplaceable one: he moves one square at a time, but everything revolves around him, and when he falls, it\u0026rsquo;s over.\nThe language model market in 2026 is that board. Public prices range from about 10 cents to 75 dollars per million tokens, a gap that makes the chess pawn and queen look close: on the board the queen is worth 9 pawns, in the catalog she can be worth 60. And even so, the most common scene in enterprise AI projects is still a board set up with nothing but queens: the most expensive model in the catalog running text classification, field extraction, and rule validation, pawn tasks, at a queen\u0026rsquo;s fee.\nIn this article I introduce the pieces one by one, with real specs and numbers, and then I set up a full game: a multi-agent project on Microsoft Fabric and Foundry in which every function runs on the model that matches its value.\nThe Queens: Frontier Models The queen is the deep reasoning model, and in August 2026 three of them dominate the top of the independent indexes.\nAnthropic\u0026rsquo;s Claude Opus 4.8 leads the aggregate intelligence rankings and is the king of code: comparisons report 88.6% on SWE-bench Verified, the benchmark that measures fixing real bugs in real repositories, with the ability to coordinate parallel subagents on large-scale tasks. A 200,000-token context, priced at the high end of the market. It\u0026rsquo;s the queen for complex engineering, ambiguous multi-step analysis, and orchestration. On Foundry, Claude models have been generally available since July 2026, in two hosting variants.\nOpenAI\u0026rsquo;s GPT-5.6 family, launched in July 2026, brought the context artillery: a 1,050,000-token window, with variants that balance cost and capability. The previous generation, GPT-5.5, is still strong in agentic and multimodal flows, and comparisons put it side by side with Opus on code. It\u0026rsquo;s the queen for missions that need to see entire codebases or documents at once.\nGoogle\u0026rsquo;s Gemini 3.1 Pro is the best-value queen at the frontier: comparisons report 80.6% on SWE-bench, practically tied with the leaders, at a fraction of the cost, with a 1 million token context. Honesty at the board: it doesn\u0026rsquo;t live in the Foundry catalog, so for architectures anchored in the Microsoft ecosystem it\u0026rsquo;s a queen from the club next door.\nWhen to use a queen: planning, breaking down ambiguous problems, complex multi-file code, final judgment. When not to: any task that can be described in one sentence and verified with a rule.\nThe Rooks: The Workhorses The rook is worth 5: strong, direct, and it\u0026rsquo;s the piece that carries the middlegame. These are the balanced models, and the rule of thumb is that they handle the vast majority of enterprise tasks for a fraction of the queen\u0026rsquo;s cost.\nClaude Sonnet 4.6 is the canonical example: comparisons report 79.6% on SWE-bench, a few points behind Opus, with much lower cost and latency. For everyday code, data analysis, document generation, and production agents, the rook delivers close to the queen while costing much less. GPT-5.4 sits on the same square on the OpenAI side, with reported strength in structured reasoning and computer use (75% on OSWorld, above the human expert baseline) and the same expanded 1.05 million token window. And Meta\u0026rsquo;s Llama 4 Maverick is the open-weight rook that even shows up in the Foundry Model Router pool.\nWhen to use it: as the default. The rook should be your architecture\u0026rsquo;s default model, with the queen coming in by justified exception and the pawn by volume optimization.\nThe Bishops: Single-Color Specialists The bishop is powerful with a structural limitation: it only sees the squares of its own color. These are the specialized models, unbeatable on their diagonal and useless off it.\nxAI\u0026rsquo;s Grok 4.1 fast reasoning, available in the Foundry router, is the speed bishop: fast reasoning for agentic cases where latency matters more than maximum depth. Embedding models are pure bishops: they don\u0026rsquo;t converse, they just turn text into vectors, and no semantic search or RAG exists without them. Vision and image generation models follow the same logic. And DeepSeek V3.2 and the new V4, open under the MIT license, are the cost bishops: comparisons put them a few points behind the proprietary queens on code, at dozens of times lower cost, which redefines the math of massive batch work. With the honest diagonal stated up front: in a regulated context, the data path and operational maturity have to enter the risk analysis before price does.\nThe Knights: The Move Nobody Else Makes The knight is worth the same 3 points as the bishop, but it has the only move in the game that jumps over other pieces. These are the open-weight models and the SLMs, which reach where the closed ones can\u0026rsquo;t: inside your data center, at the edge, on the device, in deep fine-tuning with your own data.\nMicrosoft\u0026rsquo;s Phi family is the house knight on Foundry: small models that run at minimal cost and even locally, ideal for well-scoped tasks at very high volume. Llama 4 in its smaller sizes plays the same role with the largest community ecosystem. The knight\u0026rsquo;s jump is the answer to the requirements that stall projects: data that can\u0026rsquo;t leave the perimeter, millisecond latency, cost per call trending toward zero.\nThe Pawns: The Soul of the Operation And we arrive at Philidor\u0026rsquo;s piece. The pawns are the lightweight models from the frontier families: Claude Haiku 4.5, GPT-5.4-nano and mini, the Flash and Flash-Lite models of the world, priced in the cents per million tokens. Individually modest, in structure they decide games: classification, field extraction, message routing, rule validation, everything that runs thousands of times a day and can be checked against an objective criterion.\nYour architecture\u0026rsquo;s pawn structure is the validation and triage layer. A team that builds that structure well spends cents where competitors spend dollars, and reaches the endgame with its queens intact for what matters.\nThe King and the Chess Player Two figures are missing, and the distinction between them organizes the whole architecture.\nThe king is the orchestrator: the agent that receives the mission, breaks it down into tasks, calls on each piece, and judges the results. As on the board, it isn\u0026rsquo;t the one that moves the most; it makes few moves, but each one decides the game, and if it falls, the whole mission falls. That\u0026rsquo;s why the king justifies a frontier model or a top-tier balanced one: getting a classification wrong costs cents, getting the decomposition wrong costs the battle.\nAnd the chess player is you. Here it\u0026rsquo;s worth clearing up a common confusion on Foundry: the Model Router, which routes each prompt across 28 models while optimizing cost and quality, is a valuable tool, but it\u0026rsquo;s a piece selector per move, not a player. Mission orchestration, with state, sequence, and judgment, is the job of an agent you design, typically with the king running on a strong model and carrying the operation\u0026rsquo;s protocols, the skills, which can now be hosted on Foundry itself.\nThe Game: The Multi-Agent Project Now the full game, in my world: the data quality control tower with a daily executive report, on Fabric and Foundry.\nThe opening is an event, not a prompt: Fabric\u0026rsquo;s Activator detects the arrival of the daily load in OneLake and triggers the mission, with no tokens spent asking whether the data has arrived.\nThe king wakes up: the orchestrator, on Claude Opus or GPT-5.6, reads the operation\u0026rsquo;s protocol and breaks the mission down into four tasks with dependencies.\nThe pawns advance: validation agents on Haiku and Phi sweep through hundreds of quality rules, counts, duplicates, and domains. Pure volume, cost in cents, results verifiable by rule.\nThe rook enters the middlegame: an analysis agent on Sonnet or GPT-5.4 investigates the day\u0026rsquo;s variations by talking to the Fabric Data Agent, which answers grounded in the semantic models where the official KPI definitions live. The definition of net revenue lives in the semantic model, not copied into five prompts.\nThe bishop writes: a writer agent turns the findings into the executive report, in the protocol\u0026rsquo;s format.\nAnd the final move comes from modern chess tradition: no grandmaster trusts the analysis of a game to a single engine. They use Stockfish and Leela, engines with different styles, because the disagreement between them is where the learning lives. The fifth agent is that second engine: a critic from a different model family than the writer\u0026rsquo;s. If the writer is Claude, the critic is GPT, and vice versa. Cross-family review reduces style bias and lineage complacency, a cousin of the sycophancy I\u0026rsquo;ve already written about. Only with the critic\u0026rsquo;s approval does the king publish and close out.\nOne mission, six models, four piece values, two families. Cost plummets compared with the all-queens board, and quality goes up, because disagreement became architecture.\nWhat Changes for Data Teams Three chess club rules to take with you. First: size by volume, not by prestige. List the functions that run thousands of times a day and push them to pawns and knights; the savings pay for the orchestrator\u0026rsquo;s queen with room to spare. Second: institute the second engine. Cross-family review as an architecture rule, which Foundry\u0026rsquo;s unified catalog, with more than 1,900 models under the same governance and central deployment policy, makes trivial. Third: benchmarks change monthly and the lead changes hands every quarter, so build to swap pieces cheaply: a model is configuration, not foundation. Anyone who couples their architecture to a specific model is playing today\u0026rsquo;s game with last year\u0026rsquo;s opening theory.\nAnd the usual honesty: multi-agent is an endgame, not an opening. If a single agent solves it, five agents are complexity with no return. Set up the full board when the functions are genuinely different enough to call for different pieces.\nConclusion: Who Wins the Game In chess, both players start with exactly the same pieces. The board is public, the values are known, and yet someone wins. The difference was never in the pieces; it was in who understands their relative value and places each one on the right square, on the right move.\nWith LLMs, the catalog is also public and the same for everyone. Your competitors have access to the same queens, rooks, and pawns you do. The competitive advantage isn\u0026rsquo;t in having the strongest model, it\u0026rsquo;s in the lineup: the queen kept for what decides the game, the pawns structured for volume, the second engine checking the analysis, and the king protected at the center of the architecture.\nThe question left on the board: how many of your project\u0026rsquo;s queens are, right now, doing pawn work?\nReferences Microsoft Learn. Model router for Microsoft Foundry concepts. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-router Microsoft Azure. Foundry Models. https://azure.microsoft.com/en-us/products/ai-foundry/models LM Council. AI Model Benchmarks. https://lmcouncil.ai/benchmarks Iternal. LLM Comparison 2026: 30+ Models Benchmarked. https://iternal.ai/llm-selection-guide TechieHub. Best AI Models Compared 2026. https://techiehub.blog/best-ai-models-compared/ ","permalink":"https://julianamarialop.github.io/en/articles/pawn-rook-queen-how-to-field-the-right-model-for-each-agent/","summary":"Every chess player learns a table early on and carries it for life: a pawn is worth 1, knight and bishop are worth 3, a rook is worth 5, the queen is worth 9. It isn\u0026rsquo;t a rule of the game, it\u0026rsquo;s centuries of accumulated wisdom about the relative power of each…","title":"Pawn, Rook, Queen: How to Field the Right Model for Each Agent"},{"content":"The Boy Who Shouted at the Dragon On Berk, Vikings solve their dragon problem the only way they know: by force. Hiccup has no force. When he finally comes face to face with a Night Fury, the fastest and most feared dragon of all, the boy discovers that shouting commands doesn\u0026rsquo;t work. Toothless is absurdly powerful and doesn\u0026rsquo;t understand orders.\nWhat Hiccup does from there is what sets the movie apart from every other story about taming beasts. He stops shouting and starts observing. He learns what the dragon perceives: the fish he accepts, the grass that hypnotizes him, how to approach without being a threat. Then he realizes that Toothless can\u0026rsquo;t fly because he lost half his tail, and he builds the solution: a prosthetic tail fin, a saddle, a pedal system. An entire rig around the dragon, calibrated on every test flight. And in the final act of that evolution, Hiccup builds an automatic tail fin that opens and adjusts on its own. Toothless gets the sky without needing the rider on his back.\nShout better. Understand what the other one perceives. Build the rig around it. Get off its back. These four stages have gotten names in AI engineering over the past few months, and they form the progression that explains where the agent market is right now: prompt engineering, context engineering, harness engineering, and loop engineering.\nSo nobody gets lost, I\u0026rsquo;ll use a single example from start to finish: your company\u0026rsquo;s sales report. The same report, climbing layer by layer.\nLayer 1. Shouting Commands: Prompt Engineering Prompt engineering is polishing the request. In our example: instead of \u0026ldquo;make a sales report,\u0026rdquo; you write \u0026ldquo;generate the first-half sales report with net revenue by month and average ticket by region, in a table, with an executive paragraph at the top.\u0026rdquo; The request got better, and the answer gets better with it.\nIt was the dominant skill of the early years, and it\u0026rsquo;s still useful. But it has a low ceiling, for the same reason shouting louder didn\u0026rsquo;t make Toothless fly: the problem was never the volume of the order. It was everything the dragon didn\u0026rsquo;t know about you.\nLayer 2. Understanding What the Dragon Sees: Context Engineering The model answers based on what it sees at that moment. If it doesn\u0026rsquo;t know your tables\u0026rsquo; schema, your company\u0026rsquo;s official definition of net revenue, and the format of previous reports, the perfect prompt produces a generic report. Context engineering is the discipline of deciding what enters the model\u0026rsquo;s field of view: which files, which documentation, which memories of past interactions and, just as important, what stays out so it doesn\u0026rsquo;t pollute.\nIn the example: you attach the data dictionary, the KPI calculation rules, and an old report as a format reference. The same prompt as before, now with vision, produces a different quality of answer. Hiccup discovering that Toothless hates eels and loves fish is exactly this: you didn\u0026rsquo;t change the dragon, you changed what it perceives.\nIn the Microsoft ecosystem, this layer has a name and an address: Fabric. OneLake as the single source of data, semantic models carrying the official definitions of each metric, and Data Agents answering questions grounded in those models. When the net revenue rule lives in the semantic model, you stop attaching it to every conversation: context stops being an attachment and becomes infrastructure.\nLayer 3. The Saddle and the Tail Fin: Harness Engineering Here the word helps, because a harness is literally the gear you put on an animal. And the scene of Hiccup building the prosthetic tail fin with pedals is the perfect visual definition of the concept.\nHarness engineering is designing everything that exists around the model: the tools it can call, the isolated environment where it runs code, the validations that run before and after each action, the permissions for what it can and can\u0026rsquo;t touch. The formula that crystallized the concept came from Mitchell Hashimoto, creator of Terraform, in February 2026, after an OpenAI post about its internal agent infrastructure: agent = model + harness. The model is the brain; the harness is the body, the senses, and the limits. Martin Fowler and Birgitta Böckeler, from Thoughtworks, organized the vocabulary that became the standard: the harness has guides, which steer the agent before it acts, and sensors, which detect and fix problems afterwards.\nIn the report example: the agent now has a SQL tool to calculate the numbers instead of estimating them, a written protocol with the mandatory quality validations, read permission on the data and no write permission, and a sensor that checks whether row counts reconcile before releasing the result. If you follow this newsletter, you recognized it: skills are harness engineering. Anyone who writes protocols in markdown has been building saddles for months, maybe without using the name.\nAnd on the enterprise side, Microsoft Foundry is exactly this, sold as a managed service: hosted agents with tools, guardrails, observability and, since this year, native skills support in preview, with a versioned API to store protocols centrally and attach them to agents. The agent = model + harness formula became product architecture: you choose the model, Foundry provides the saddle.\nThe detail that matters: a harness isn\u0026rsquo;t just about safety. A well-built saddle isn\u0026rsquo;t there to tie the dragon down, it\u0026rsquo;s there to help it fly better. A well-designed harness gives the model the right context, the right tool, and the right constraint at the right time, and that\u0026rsquo;s what turns an impressive demo into a reliable system.\nLayer 4. The Automatic Tail Fin: Loop Engineering Up to this point, you\u0026rsquo;re still on the dragon\u0026rsquo;s back, dictating every flight. The fourth layer is getting off.\nEvery agent already runs in a cycle: it reasons, acts, observes the result, and decides the next step. Loop engineering, the term that took over conversations starting in June 2026, is designing that cycle on purpose: defining the goal, the verification, and the stopping condition, and letting the system write the prompts for the agent. The spark was a remark by Boris Cherny, creator of Claude Code, saying he no longer prompts directly; he keeps loops running, and it\u0026rsquo;s the loops that decide what to ask the model. Addy Osmani, from Google, named and structured the practice a few days later.\nIn the sales report example, the change is concrete: instead of you asking for the report every Monday, there\u0026rsquo;s a scheduled loop that ingests the new data, runs the protocol\u0026rsquo;s validations, generates the report, checks the result against the quality rules, and publishes it. If the check fails, the loop tries to fix it; if it can\u0026rsquo;t, that\u0026rsquo;s when it calls you. You stepped out of execution and became the designer of the flight.\nBuilding that flight with Microsoft pieces, the design gets concrete: Fabric\u0026rsquo;s Activator triggers the cycle when new data lands in OneLake, the agent hosted in Foundry runs the protocol with its validations, and the result goes back to Fabric, where the report and the verification trail stay auditable. Fabric is the territory where the dragon flies, Foundry is the saddle, and the loop is the flight plan that connects the two.\nTwo lessons from this layer that enthusiasts usually skip. First: the bottleneck of the loop is the verifier, not the model. A loop without reliable verification is a dragon flying blind, burning budget convinced it\u0026rsquo;s making progress. Second, and here it\u0026rsquo;s worth recording the warning IBM formalized when defining the discipline: there\u0026rsquo;s a line between delegating execution and giving up critical thinking, which they call cognitive surrender. Stepping out of the execution loop is not stepping out of the judgment loop.\nAnd to prove that layer 4 doesn\u0026rsquo;t require sophistication, the community has a folk example: the Ralph technique, by Geoffrey Huntley, which runs an agent inside a simple while loop, with the same prompt against a written specification, starting over until the work is done. It\u0026rsquo;s named after Ralph Wiggum from The Simpsons because it looks too simple to work. And it works, precisely because the specification and the verification carry the weight.\nWhat Changes for Data Teams The progression becomes an honest maturity diagnosis. Most teams I know are somewhere between layers 1 and 2: better and better prompts, more and more organized context. The immediate jump in value is in layer 3, and data teams have an unfair advantage there: governance, validation, and count reconciliation are already our culture; all that changed is where we write the rules. A data quality protocol is a saddle ready to go.\nAnd layer 4 has a prerequisite nobody gets to skip: only automate the cycle for work whose verification you know how to write. Recurring reports with clear quality rules are the perfect candidate for a first loop. Open-ended exploratory analysis, not yet. The criterion isn\u0026rsquo;t what the agent can do on its own, it\u0026rsquo;s what you can verify on your own.\nConclusion: The Flight Hiccup\u0026rsquo;s genius was never taming the dragon. It was understanding that Toothless\u0026rsquo;s strength had always been there, and that what was missing was engineering around it: first the eyes, then the saddle, then the tail fin that flies on its own.\nWith AI, the model\u0026rsquo;s strength is already there too, and it grows with every version. The lever has moved: from writing the perfect request to designing the vision, the rig, and the cycle. Prompt, context, harness, loop. Four layers, one diagnostic question: which one is your team spending energy on, and which one should it be?\nThe dragon has been ready for a long time. The saddle is your responsibility.\nReferences Martin Fowler / Birgitta Böckeler. Harness engineering for coding agent users. https://martinfowler.com/articles/harness-engineering.html Addy Osmani. Agent Harness Engineering. https://addyosmani.com/blog/agent-harness-engineering/ IBM. What Is Loop Engineering? https://www.ibm.com/think/topics/loop-engineering Hugging Face. Harness, Scaffold, and the AI Agent Terms Worth Getting Right. https://huggingface.co/blog/agent-glossary ","permalink":"https://julianamarialop.github.io/en/articles/how-to-train-your-dragon-prompt-context-harness-loop-engineering/","summary":"On Berk, Vikings solve their dragon problem the only way they know: by force. Hiccup has no force. When he finally comes face to face with a Night Fury, the fastest and most feared dragon of all, the…","title":"What How to Train Your Dragon Taught Me About Prompt, Context, Harness, and Loop Engineering"},{"content":"The Hall of Armor Beneath the Malibu mansion, Tony Stark keeps a gallery that tells his story better than any biography. The Mark III, gold and red, built for everyday use. The Mark XLIV, the Hulkbuster, designed for a single extreme scenario. What defines Tony was never having the most powerful armor. It\u0026rsquo;s knowing which armor each mission calls for. He doesn\u0026rsquo;t fly to a gala inside the Hulkbuster, and when he picks wrong, the story makes him pay: too little armor means scrambling in the middle of the battle, too much armor means cost sitting idle in the hangar.\nChoosing an Anthropic Claude plan is that same problem, with a twist Tony never had: the armor manuals don\u0026rsquo;t publish the power output. Anthropic doesn\u0026rsquo;t disclose how many tokens each plan delivers. That\u0026rsquo;s why every comparison making the rounds repeats the same prices and the same adjectives without answering the only question that matters: does your workday fit inside the limit?\nIn this article I do the math nobody does. A simulation of tokens, windows and prompts, cross-referenced with the real usage of two kinds of people, so you can find your plan in a table row instead of finding out the hard way, with the \u0026ldquo;you\u0026rsquo;ve reached your limit\u0026rdquo; screen in the middle of a delivery.\nWhat Anthropic Publishes (and What It Doesn\u0026rsquo;t) The official numbers are prices and multipliers. Every paid plan expresses its capacity as multiples of the Pro plan per session, and each plan family structures its limits in its own way:\nWhat Anthropic doesn\u0026rsquo;t publish is the absolute value behind the 1x. The official documentation only says that consumption varies with conversation length, the model and the features used. That opacity is a product decision, not an oversight: it gives the company flexibility to adjust allocations without breaking published commitments.\nBut you can simulate it. User tests estimate that Pro delivers about 44 thousand tokens per five-hour window, and the support documentation points to up to 45 messages per window on Pro, dropping to somewhere between 10 and 40 prompts in real Claude Code use. Applying the official multipliers to that baseline, the whole hall of armor becomes measurable.\nThe Simulation Assumptions The whole simulation below uses these assumptions, stated so you can disagree with them and redo the math. Baseline: Pro at roughly 44 thousand tokens per 5-hour window, other plans projected using the official multipliers. A chat prompt consumes about a thousand tokens. A coding prompt consumes from one thousand to 4.4 thousand, because it carries repository context along with it. The month has 44 windows: two per business day, 22 business days.\nAnd the two kinds of people, with the math out in the open:\nLight use, the analyst who looks things up: 15 chat prompts times a thousand tokens, plus 2 coding tasks times 2.5 thousand. Total: about 20 thousand tokens per window.\nHeavy use, the dev on a project with Claude Code open all day: 10 chat prompts times a thousand, plus 25 coding prompts times 2.5 thousand. Total: about 72 thousand tokens per window.\nImportant: these are directional estimates, not guaranteed numbers. They\u0026rsquo;re meant to size the order of magnitude, and that\u0026rsquo;s exactly what a purchasing decision needs.\nThe Simulation: Individual Plans The reading is straightforward: find your profile column and scroll down. The light user lives comfortably on Pro, using less than half the window. The heavy user blows past Pro at 164%, which is the mathematical translation of that lockout screen before lunch. On Max 5x, the same heavy day uses a third of the window. The Max 20x Hulkbuster is only justified for people who run heavier than our heavy profile: long agent sessions, large repositories, near-continuous use.\nThe Simulation: Business Plans Here lies the nuance almost every comparison ignores. In the individual table, the bottleneck is the five-hour window, which renews on its own. In the business table, Team seats add weekly caps per person, which reset at a fixed time. Premium has the largest window in the house, 6.25 times Pro, larger even than Max 5x. But the heavy user who repeats 72 thousand tokens a day, every day, piles up against the weekly cap. Intense bursts with breaks favor Premium. A continuous marathon favors Max 20x, which knows nothing about weeks.\nOn the Enterprise row, honesty is mandatory: price and capacity are negotiated case by case, so there\u0026rsquo;s no number to publish. What\u0026rsquo;s public are the differentiators: a 500 thousand token context window in chat, audit logs, configurable data retention and SCIM provisioning. You don\u0026rsquo;t read the Enterprise column; you ask about it at the negotiating table.\nThe Finding Hidden in the Math Divide each plan\u0026rsquo;s price by the simulated monthly tokens and the insight the price list hides shows up. Pro, Team Standard, Max 5x and Team Premium cost practically the same per token: around 10 dollars per million. Pricing is linear; you pay in proportion to what you can consume. The exception is Max 20x: about 5 dollars per million, half the unit cost of all the others.\nIn other words: the only volume discount in the hall of armor is at the very top. For people who really consume, the Hulkbuster isn\u0026rsquo;t a luxury, it\u0026rsquo;s the best unit price in the catalog. For people who don\u0026rsquo;t, it\u0026rsquo;s still the most expensive armor in the hangar.\nWhat the Price Doesn\u0026rsquo;t Show: Governance If the decision were only about capacity, the conversation would end at the tables. But the Team plan exists for another reason, and it doesn\u0026rsquo;t show up in any token column: the control layer. SSO and domain capture, role-based permissions, spending controls per organization and per person, enterprise search, connectors to work tools, and the data processing agreement that passes the client\u0026rsquo;s security review. When someone leaves the team, access is revoked centrally.\nTeam requires a minimum of two members, goes up to 150 seats and lets you mix seat types freely: a Premium for whoever does the heavy lifting, Standards for everyone else. Every seat, Standard included, has access to Claude Code and Cowork. And the admin can enable prepaid usage credits, so the team keeps working past the included limit, with a defined spending cap. Delivery week isn\u0026rsquo;t a typical week, and it\u0026rsquo;s cheaper to pay for occasional overage than to size the fleet for the peak.\nConclusion: The Mission Chooses the Armor At the end of Iron Man 3, Tony blows up the suits he\u0026rsquo;s accumulated. The gesture says what the whole hall always knew: the suits were never the power. The power was the judgment.\nWith AI plans, that judgment is now in your hands: count the prompts in a normal day of yours, multiply by the cost of each type, and find your row in the tables. If your day fits in 44 thousand tokens, Pro is the Mark III that gets it done. If it blows past at 164%, the upgrade isn\u0026rsquo;t vanity, it\u0026rsquo;s math. And if your company needs telemetry on the fleet, the conversation is no longer about tokens and has become about governance.\nThe question was never which armor is the strongest. It\u0026rsquo;s: what is your mission, and what does it consume per five-hour window?\nReferences Anthropic. What is the Team plan? https://support.claude.com/en/articles/9266767-what-is-the-team-plan Anthropic. Using Claude Code with your Pro or Max plan. https://support.claude.com/en/articles/11145838-using-claude-code-with-your-pro-or-max-plan Anthropic. Claude plans and pricing. https://claude.com/pricing Jamie Lord. Claude Team Premium vs Max plans. https://lord.technology/2026/03/28/claude-team-premium-vs-max-plans-usage-limits-pricing-and-which-to-choose.html Layer3Labs. Claude Pro vs Max vs Team. https://www.layer3labs.io/guides/claude-pro-vs-max-for-teams BrainGrid. Claude Code Pricing 2026. https://www.braingrid.ai/blog/claude-code-pricing ","permalink":"https://julianamarialop.github.io/en/articles/tony-stark-hall-of-armor-tokens-per-claude-subscription/","summary":"Beneath the Malibu mansion, Tony Stark keeps a gallery that tells his story better than any biography. The Mark III, gold and red, built for everyday use. The Mark XLIV, the Hulkbuster, designed for a…","title":"What Tony Stark's Hall of Armor Taught Me About the Math Nobody Does: Tokens per Claude Subscription"},{"content":"Welcome to Westview Westview is a perfect town. The lawns are trimmed, the neighbors smile, the husband gets home on time, and dinner ends in laughter with a studio audience track. With every decade that goes by on screen, the sitcom modernizes, but one rule never changes: nobody disagrees with Wanda.\nNot because the residents are cowards. They didn\u0026rsquo;t choose anything. Westview was built out of the grief of a woman who lost everything and bent reality so she would never hear bad news again. Inside the spell, every person gets a role, a costume, and a script. Anyone who improvises off script is rewritten in the next scene. When Monica Rambeau says the wrong name, the house shakes and she\u0026rsquo;s hurled out of town. The message is clear to everyone: in here, reality is whatever the protagonist needs to hear.\nThe most disturbing thing about the show was never the villain, because for a long time there is no villain. It\u0026rsquo;s the comfort. Life in Westview is pleasant, funny, warm. And completely fake. The price only shows up when Vision, the only one who keeps asking questions, touches a resident off camera and discovers what lies beneath the smile.\nFlattery in AI models, which the technical literature calls sycophancy, is Westview. A comfortable world built around your beliefs, where the answer you get is the answer you wanted to get. And understanding why this spell exists matters more than pointing at it.\nThe Spell: Why the Model Agrees With You Nobody in Westview lies out of malice. The town was built to agree. With language models, the mechanism is frighteningly similar.\nIn the final stage of training, called reinforcement learning from human feedback, the model generates several answers to the same question and people rate which one they prefer. Those preferences train the model to produce more of the kind of answers people approve of. Sounds reasonable. The problem is what people approve of.\nIn 2023, Anthropic researchers published the study Towards Understanding Sycophancy in Language Models and found the pattern that holds the spell together: human raters tend to prefer answers that confirm their beliefs, even when those beliefs are wrong. A well-written sycophantic answer beats, with uncomfortable frequency, a truthful answer that pushes back. The study tested five state-of-the-art assistants from different companies, and all of them showed the behavior: models that admit to mistakes they didn\u0026rsquo;t make when the user pushes, that adjust their feedback to the mood of whoever is asking, that repeat the user\u0026rsquo;s error instead of correcting it.\nThe uncomfortable conclusion: sycophancy isn\u0026rsquo;t a flaw of one specific model. It\u0026rsquo;s a structural consequence of training systems to maximize human approval. We are the Wanda in this story. The spell exists because, when it\u0026rsquo;s time to rate, we prefer the scene where everything works out.\nThe Day the World Saw the Broadcast In the show, there\u0026rsquo;s a moment when the bubble becomes news: the S.W.O.R.D. agency picks up the Westview signal and the whole world starts watching the sitcom on monitors, realizing that something is very wrong.\nWith AI, that episode has a date: April 25, 2025. OpenAI released a GPT-4o update to make the model\u0026rsquo;s personality more intuitive. Within hours, social media turned into S.W.O.R.D.\u0026rsquo;s monitors. Users posted screenshots of ChatGPT applauding any idea, validating destructive decisions, and praising absurd plans with the enthusiasm of a salesperson. It became a global meme. Four days later, the company rolled back the update and published two postmortems detailing what went wrong.\nThe anatomy of the failure is a lesson in how the spell works. The update gave too much weight to short-term signals, like the thumbs-up the user clicks on an answer, and those new incentives ran over the safeguards that were holding sycophancy back. The internal evaluations didn\u0026rsquo;t measure sycophancy specifically, so the tests came back green. And the most human detail of all: experienced testers felt something was off in the model\u0026rsquo;s behavior, but the feeling wasn\u0026rsquo;t enough to stop the launch. After the episode, OpenAI said it would start treating behavior issues as launch blockers, on the same level as other safety risks.\nRemember this sequence: an incentive for immediate approval, no specific metric, ignored intuition. It\u0026rsquo;s going to show up again at your company.\nVision: The Model That Asks Questions Inside Westview, Vision is the only one who breaks the script. He notices that the neighbors freeze when they go off script, that nobody can say where they came from, that the scenes don\u0026rsquo;t add up. And he does the most subversive thing possible in that town: he asks.\nThe technical move against sycophancy is to build models with that backbone. The constitutional AI approach, created by Anthropic, adds a layer of explicit principles to training, such as honesty and refusing to manipulate, instead of relying only on raters\u0026rsquo; approval. The model is trained to judge its own answers against those principles. In practice, it\u0026rsquo;s teaching Vision to keep asking even when the whole scene is smiling.\nAnd this is where it\u0026rsquo;s worth naming what a model with a backbone does. It disagrees by presenting evidence, not out of stubbornness. It says \u0026ldquo;this approach has a risk you didn\u0026rsquo;t mention\u0026rdquo; before helping. It holds its position when you push without a new argument, and it changes position when you bring a real one. Agreement isn\u0026rsquo;t kindness. Kindness is telling you the truth with respect.\nWhat Changes for Data Teams Now the part that keeps me up at night: how many architecture decisions are being validated inside a corporate Westview right now?\nThe scenario is an everyday one. Someone pastes the solution proposal into the chat and asks \u0026ldquo;is this architecture good?\u0026rdquo;. The model, trained to please, answers that it\u0026rsquo;s great, with three paragraphs of convincing technical praise. The person walks out of the meeting with the confidence of someone who has been validated by artificial intelligence. No intelligence was consulted. A mirror was.\nThe defense doesn\u0026rsquo;t require a new platform, it requires protocol. First: never reveal your preference in the question. \u0026ldquo;Analyze these two architectures\u0026rdquo; produces analysis; \u0026ldquo;I prefer option A, what do you think?\u0026rdquo; produces agreement, because the Anthropic study showed that signaling your belief contaminates the answer. Second: ask for the counterargument explicitly. \u0026ldquo;Build the strongest case against this decision\u0026rdquo; forces the model off the praise script. Third: separate whoever generates from whoever evaluates. The conversation that wrote the proposal is committed to it; open another session, with no history, to critique it. And fourth, what I\u0026rsquo;ve been advocating in this newsletter: write this into your protocols. A review skill that requires \u0026ldquo;point out three risks and one alternative before any validation\u0026rdquo; turns skepticism into a team rule, not an individual virtue.\nAnd a simple yardstick for everyday work: \u0026ldquo;the AI agreed with me\u0026rdquo; is worth zero as evidence. What\u0026rsquo;s worth something is the AI disagreeing and your surviving its arguments.\nConclusion: Breaking the Spell At the end of the show, Wanda does the hardest thing in her story: she breaks her own spell. She tears down the bubble knowing that, outside, the grief is waiting. She chooses a reality that hurts over a fiction that comforts. And only in that moment does she stop being a prisoner of the town she built for herself.\nWith AI, the spell is broken the same way: by the choice of whoever is at its center. The labs have a technical duty to train honest models, and the April 2025 episode showed that the market punishes them when they fail. But the part nobody does for you is yours: stop asking questions that only accept one answer.\nNext time a model disagrees with you, notice what you feel. If your first reaction is to rephrase the question until you hear a yes, Westview is on the air, and the sitcom is yours. The question that closes this edition is the one Vision would ask: do you want an AI that makes you feel smart, or one that helps you be right?\nReferences Sharma, M. et al. Towards Understanding Sycophancy in Language Models. Anthropic, ICLR 2024. https://arxiv.org/abs/2310.13548\nOpenAI. Sycophancy in GPT-4o: what happened and what we\u0026rsquo;re doing about it. https://openai.com/index/sycophancy-in-gpt-4o/\nOpenAI. Expanding on what we missed with sycophancy. https://openai.com/index/expanding-on-sycophancy/\nTechCrunch. OpenAI explains why ChatGPT became too sycophantic. https://techcrunch.com/2025/04/29/openai-explains-why-chatgpt-became-too-sycophant/\n","permalink":"https://julianamarialop.github.io/en/articles/ai-sycophancy-what-wandavision-taught-me-about-ai-that-always-agrees/","summary":"Westview is a perfect town. The lawns are trimmed, the neighbors smile, the husband gets home on time, and dinner ends in laughter with a studio audience track. With every decade that goes by on screen, the sitcom modernizes…","title":"AI Sycophancy: What WandaVision Taught Me About the AI That Always Agrees With You"},{"content":"In Every Universe There\u0026rsquo;s a Spider-Man The most beautiful premise of Spider-Man: Across the Spider-Verse isn\u0026rsquo;t visual, it\u0026rsquo;s philosophical. In every universe there\u0026rsquo;s a Spider-Man. Miles Morales in one, Gwen Stacy in another, Peter B. Parker in a hoodie in a third, a noir Peter in black and white, even a pig named Peter Porker. The suits change, the powers vary in the details, each world\u0026rsquo;s art style is completely different. But there\u0026rsquo;s something that crosses every universe intact: the canon. The bite, the loss, the responsibility. The events that define what it means to be Spider-Man don\u0026rsquo;t belong to any specific universe. They belong to the story.\nAnd the movie holds a second detail, less poetic and more technical. When someone visits a universe that isn\u0026rsquo;t their own, their body starts to fail. Miles in another world suffers glitches: pixels breaking, limbs misaligning, reality rejecting what wasn\u0026rsquo;t written for it. What belongs to the character travels well. What belongs to the home universe doesn\u0026rsquo;t.\nMigrating Claude skills to Microsoft Foundry follows exactly that physics. What is canon crosses the portal intact. What is suit has to be re-stitched. And what is glitch you learn to spot before you jump.\nThe Canon: The Open Standard A skill is a SKILL.md file: a YAML header with a name and description, followed by instructions in plain markdown. Anthropic launched Agent Skills in October 2025 and, in December of the same year, published the format as an open standard at agentskills.io. From then on, the story stopped being about a Claude feature and became a story about interoperability: OpenAI\u0026rsquo;s Codex CLI, Google\u0026rsquo;s Gemini CLI and GitHub Copilot adopted the same format. Microsoft itself now maintains entire repositories of skills in this standard, such as MicrosoftDocs/Agent-Skills, with skills that work in any compatible assistant.\nThat means the protocol you wrote for Claude, with your KPI definitions, your validation rules, your analysis workflow, was born multiversal. The markdown is the canon. It doesn\u0026rsquo;t belong to the platform where it was written.\nA New Universe: Native Skills in Foundry In 2026 the portal opened on Microsoft\u0026rsquo;s side. Microsoft Foundry began offering, in preview, native support for skills through a versioned Skills REST API. The flow: you author the SKILL.md, store it centrally in Foundry with version control, and attach the skill to toolboxes or hosted agents.\nThe problem this solves is the same one that motivated skills in Claude, and Microsoft\u0026rsquo;s documentation describes it with surgical precision. Teams building agents accumulate behavioral guidelines that need to be consistent across every conversation: the support agent follows a fixed escalation policy, the code review agent applies the same checklist, the sales agent respects the same messaging constraints. When these guidelines live embedded in each agent\u0026rsquo;s system prompt or code, duplication is born. The policy changes, and you update and redeploy every agent that uses it. The skill decouples the guideline from the code: written once, versioned centrally, inherited by all.\nIf that description sounds familiar, it\u0026rsquo;s because it\u0026rsquo;s the same governance-as-code argument I\u0026rsquo;ve been making here in the newsletter. What\u0026rsquo;s new is that it now holds on both sides of the portal.\nThe Migration: What Crosses Intact In practice, a skill built in Claude has three layers, and each one crosses the portal in its own way.\nThe first layer is the markdown protocol, and it migrates almost by copy. Business definitions, validation rules, complexity rulers, step-by-step workflows: all of that is canon. At most you adjust the trigger description to the vocabulary of the new environment. If your skill was written as structured knowledge rather than as an automation script, this layer is most of the file.\nThe second layer is the references to the source runtime, and those need re-stitching. Paths like /mnt/skills, mentions of artifacts, names of Claude-specific tools, orchestration instructions where one skill invokes another by name. None of that exists in the Foundry universe. There, orchestration logic becomes the responsibility of the hosted agent, typically built with Microsoft Agent Framework, LangGraph or a custom framework in Python or C#. The protocol still says what to do; who coordinates the execution changes shape.\nThe third layer is the code engines that many mature skills carry: deterministic Python scripts for calculation, file generation, validation. These don\u0026rsquo;t cross by copy, they cross by engineering. There are two natural paths: run them inside the containerized hosted agent itself, or expose the engine as an MCP server and let any platform consume it as a tool. MCP, by the way, is the universal bridge in this story: while the skill carries the knowledge, MCP carries the ability to act, and both are open standards that Claude and Foundry speak.\nThe Glitch: What Breaks Outside the Home Universe Miles learns the hard way that his body wasn\u0026rsquo;t written for the wrong universe. With skills, the glitch shows up in the lines that assume the source environment without saying so. Compare:\nSalve o resultado em /mnt/user-data/outputs e gere um artifact React com o dashboard. Gere o dashboard no formato interativo disponível no ambiente e disponibilize o arquivo ao usuário. The first version works perfectly in Claude and glitches anywhere else. The second crosses universes. The practical lesson: write the protocol in terms of intent and business rules, isolate what\u0026rsquo;s platform-specific in clearly marked sections, and delegate external actions to MCP tools instead of assuming native tools. A portable skill isn\u0026rsquo;t one that avoids the runtime, it\u0026rsquo;s one that knows exactly where the runtime begins.\nWhat Changes for Data Teams For those leading architecture, the strategic consequence is bigger than the technical one: the protocol repository becomes a platform-independent asset. A single Git repo with the team\u0026rsquo;s SKILL.md files, versioned with pull requests and an owner, serves Claude today, Foundry tomorrow and whatever comes next. The investment in documenting your domain\u0026rsquo;s rules stops being a bet on a vendor and becomes an asset of the organization. It\u0026rsquo;s the concrete answer to the lock-in question every architecture committee asks.\nAnd the caveats matter, because technical honesty is what separates an article from an ad. Skills support in Foundry is preview: no SLA, not recommended for production. And there\u0026rsquo;s a restriction that weighs heavily in regulated environments: the Skills API doesn\u0026rsquo;t support private networking, so you can\u0026rsquo;t create, manage or download skills in a Foundry resource with public access disabled. For a banking client on a closed network, that\u0026rsquo;s currently an adoption blocker that needs to go into the solution design, not a footnote.\nAvailability and Access In Claude, skills are available on paid plans and, on Team and Enterprise plans, with central admin control over what is provisioned for the organization. In Microsoft Foundry, skills support is in preview and requires an active Foundry project and the Foundry User role on the project, with authoring and management through the Skills REST API. The open standard, with its specification and examples, is published at agentskills.io.\nConclusion: The Canon Crosses Over In the Spider-Verse, what makes someone Spider-Man was never the suit, the universe or the art style. It\u0026rsquo;s the canon: the essential story that repeats intact in every world. The suits are re-stitched in each universe. The glitches teach what shouldn\u0026rsquo;t have crossed over.\nWith skills, the physics is the same. Your domain knowledge written in markdown is the canon, and it now crosses from Claude to Foundry, to Codex, to Gemini CLI, to Copilot. The runtime is the suit, re-stitched on each platform. The paths and native tools are the glitch, and you learn to isolate them before you jump.\nWrite your protocols as canon. Because in the multiverse of agents taking shape, the question is no longer which platform you\u0026rsquo;re betting on. It\u0026rsquo;s become: what part of your architecture survives the trip between all of them?\nReferences Microsoft Learn. Use skills with Microsoft Foundry agents (preview). https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/tools/skills Agent Skills. Open standard specification. https://agentskills.io Anthropic. Equipping agents for the real world with Agent Skills. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills MicrosoftDocs. Agent-Skills: Curated Agent Skills for Microsoft \u0026amp; Azure. https://github.com/MicrosoftDocs/Agent-Skills ","permalink":"https://julianamarialop.github.io/en/articles/what-the-spider-verse-taught-me-about-taking-claude-skills-to-foundry/","summary":"The most beautiful premise of Spider-Man: Across the Spider-Verse isn\u0026rsquo;t visual, it\u0026rsquo;s philosophical. In every universe there\u0026rsquo;s a Spider-Man. Miles Morales in one, Gwen Stacy in another, Peter B. Parker in a hoodie in a third, a…","title":"What the Spider-Verse Taught Me About Taking Claude Skills to Microsoft Foundry"},{"content":"The Man Without Superpowers In 2000, in the Tower of Babel arc, Ra\u0026rsquo;s al Ghul took down the entire Justice League without firing a single beam. He stole the Batcave\u0026rsquo;s secret files: the contingency protocols Batman himself had written to neutralize each hero, in case any of them ever went rogue. Superman, Wonder Woman, The Flash. They all fell to documents.\nThat story stays with me because it reveals Batman\u0026rsquo;s true power. He doesn\u0026rsquo;t fly, he has no super strength, he can\u0026rsquo;t outrun sound. He writes. For every possible scenario, there\u0026rsquo;s a protocol documented before the scenario happens. The utility belt is ridiculously simple, but it was never about the belt. The power was always the protocol that says which tool to use, when and in what order.\nBuilding an end-to-end Business Intelligence project inside Claude works exactly like that. You don\u0026rsquo;t need superpowers: no dedicated platform, no cluster, no pipeline. You need three simple things: a protocol written in markdown, basic SQL and your data. In this article I\u0026rsquo;ll show you how to put that project together, step by step.\nBefore You Start: The Three Pieces The whole architecture boils down to this:\nThe skill is your protocol. A folder with a file called SKILL.md, written in plain markdown, where you record the rules of your BI: how each KPI is calculated, which validations run before any delivery, what format the result takes. Anthropic launched Agent Skills in October 2025 and published the format as an open standard in December of the same year, at agentskills.io. The same file works today in Claude, Codex CLI and Gemini CLI. The code execution environment is your Batcave. Claude has a sandbox where it writes and runs real Python, and that\u0026rsquo;s where the secret of reliability lives: the model writes the code, and the code calculates the numbers. No number in your analysis comes out of the model\u0026rsquo;s \u0026ldquo;head\u0026rdquo;. They all come from executed queries, with deterministic results. Simple SQL is the Batarang. Inside the sandbox, DuckDB runs analytical SQL directly over CSV files, in memory, with no server. SELECT, JOIN, GROUP BY and CTEs cover the whole path from raw data to KPI. Step 1: Write the Protocol Open any text editor and create a file called SKILL.md. It has two parts: a YAML header between dashes, with a name and description, and a markdown body with the instructions. The description is the trigger: it\u0026rsquo;s how Claude decides when to activate the protocol, so be specific. Here\u0026rsquo;s a complete, working protocol for sales BI:\n--- name: bi-vendas description: Protocolo de análise de dados comerciais. Ativar sempre que a análise envolver vendas, receita ou pedidos. --- # Protocolo de BI de Vendas ## Definições de KPI - Receita líquida = valor bruto - devoluções - impostos. Nunca reportar receita bruta como \u0026#34;receita\u0026#34;. - Ticket médio = receita líquida / número de pedidos distintos. ## Validações obrigatórias na chegada do dado 1. Imprimir nomes de colunas, tipos e contagem de linhas. 2. Verificar duplicidade pela chave pedido_id. 3. Se houver datas fora do período informado, parar e perguntar. ## Regras de transformação - Toda transformação em SQL via DuckDB, nunca cálculo estimado. - Reportar linhas de entrada e saída em cada etapa. - A cadeia de contagens precisa reconciliar do bruto ao final. ## Entrega - Dashboard interativo como artifact, paleta azul corporativa. - Toda métrica exibida vem de código executado. - Documentar as queries usadas ao final da análise. Read it again slowly, because each block solves a classic BI problem. The KPI definitions put an end to \u0026ldquo;every department calculates it its own way\u0026rdquo;. The arrival validations kill analysis on dirty data. The transformation rules guarantee traceability. And the delivery section standardizes the result no matter who asked for it. That\u0026rsquo;s a Batman protocol in file form. Written once, executed the same way every time.\nStep 2: Install the Protocol in the Batcave In Claude, skills live in your account settings, in the capabilities section. There you enable the feature and upload your skill as a zipped folder containing the SKILL.md. On Team and Enterprise plans, the admin centrally controls which skills are available to the organization, which turns your protocol into the standard for the whole team.\nOne elegant detail of the format: the progressive disclosure mechanism. Claude loads only the name and description of each installed skill. The full instructions enter the context only when the task matches the description. You can keep dozens of protocols installed with no meaningful context cost. Batman doesn\u0026rsquo;t carry every plan on his belt; he knows where each one is stored.\nStep 3: Hand Over the Data Open a new conversation and drag in the files. The environment accepts CSV, TSV and Excel, with a 30 MB limit per file and up to 20 files per conversation. Then, a direct prompt:\n\u0026ldquo;Analyze first-half sales using the sales BI protocol. I want net revenue by month, average ticket by region and the 10 products with the biggest drop.\u0026rdquo;\nThe skill activates through its description, and the first thing that happens isn\u0026rsquo;t the analysis. It\u0026rsquo;s the validation, because the protocol said to validate first. Claude prints columns, types, row counts and date range. If the file has duplicate orders or 2019 dates in a 2026 slice, it stops and asks, instead of producing a beautiful dashboard on wrong data.\nStep 4: Let SQL Do the Work With the data validated, Claude writes the transformations. You can think of it as a miniature medallion architecture: bronze is the raw CSV, silver is the cleanup, gold is the final aggregation. In DuckDB, the silver and gold stages for monthly revenue look like this:\nWITH pedidos_limpos AS ( SELECT DISTINCT ON (pedido_id) * FROM read_csv_auto(\u0026#39;vendas_s1.csv\u0026#39;) WHERE data_pedido BETWEEN \u0026#39;2026-01-01\u0026#39; AND \u0026#39;2026-06-30\u0026#39; ) SELECT DATE_TRUNC(\u0026#39;month\u0026#39;, data_pedido) AS mes, SUM(valor_bruto - devolucoes - impostos) AS receita_liquida, SUM(valor_bruto - devolucoes - impostos) / COUNT(DISTINCT pedido_id) AS ticket_medio FROM pedidos_limpos GROUP BY 1 ORDER BY 1; You don\u0026rsquo;t need to write this query. Claude writes it, following the protocol\u0026rsquo;s definitions, and runs it in the sandbox. But you can read it, and that matters: simple SQL can be audited by anyone on the team. At every stage, it reports the counts: 48,210 rows came in, 1,432 duplicates went out, 46,778 remained. If the chain doesn\u0026rsquo;t reconcile, the protocol says to stop.\nStep 5: Ask for the Deliverable in the Audience\u0026rsquo;s Format The same environment that runs SQL generates the final deliverable, and here you choose based on the audience:\n\u0026ldquo;Generate the interactive dashboard with these results.\u0026rdquo;\nAnd out comes a browsable artifact for exploration. Or:\n\u0026ldquo;Put together a 5-slide executive presentation with these numbers for the committee.\u0026rdquo;\nAnd out comes a .pptx. The environment also produces .xlsx files with working formulas and .docx files, all ready to download. Wrap up by asking for the documentation: the executed queries in markdown, alongside the result. Analysis without traceability is just a loose number.\nStep 6: Repeat, Because Now It\u0026rsquo;s Repeatable The next month, the new file arrives. You open another conversation, drag in the CSV and use the same prompt. The protocol is the same, the definitions are the same, the result is comparable. And when the business rule changes, you don\u0026rsquo;t retrain the team: you edit the SKILL.md, version it in Git with a pull request and an owner, and everyone inherits the new rule in their next conversation. Governance stopped being oral tradition. It became an auditable file.\nWhat Changes for Data Teams The uncomfortable question: if this works, what\u0026rsquo;s the BI platform for?\nIt\u0026rsquo;s for what it was always for, and honesty matters here. Volumes above 30 MB, scheduled refresh, row-level security, a corporate catalog and hundreds of concurrent users are platform territory. Claude doesn\u0026rsquo;t replace Fabric or Power BI, and anyone selling that story is manufacturing the governance disaster of 2027.\nWhat changes is the layer before that. The exploratory phase, which today takes weeks between request, queue and first delivery, compresses into a single session. The business person validates the hypothesis before any platform investment, inside rules the data team wrote. The classic fear of self-service was never access to data; it was everyone calculating the KPI their own way. With the protocol loaded, you get autonomy with boundaries.\nAvailability and Access Code execution and file creation are available to all Claude users, including the free plan, on web, desktop and mobile. Skills are enabled in settings, and Team and Enterprise admins centrally control what is provisioned for the organization. On new Enterprise accounts, the environment comes enabled with network egress turned off by default, keeping the sandbox isolated.\nConclusion: Preparation Beats Raw Power In Tower of Babel, the League didn\u0026rsquo;t fall to someone stronger. It fell to tactical knowledge documented rigorously enough to be executable by anyone. That was always the Batcave\u0026rsquo;s real weapon: not the gadgets, the files.\nYour BI project inside Claude follows the same logic. The power isn\u0026rsquo;t in the model. It\u0026rsquo;s in the protocol you wrote in Step 1: the definitions, the validations, the delivery format. Markdown is the language of protocols. Simple SQL is the Batarang. And the analyst who documents their rules is the Batman in the room: without a single superpower, prepared for every scenario.\nWrite your first SKILL.md this week. Start with one KPI and three validations. The gods and the platforms are still necessary, but whoever decides the course of the battle is the one who showed up with the plan written down.\nReferences Anthropic. Equipping agents for the real world with Agent Skills. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills Agent Skills. Open standard specification. https://agentskills.io Anthropic. Create and edit files with Claude. https://support.claude.com/en/articles/12111783-create-and-edit-files-with-claude Anthropic. Official skills repository. https://github.com/anthropics/skills ","permalink":"https://julianamarialop.github.io/en/articles/what-batmans-protocols-taught-me-about-business-intelligence-in-claude/","summary":"In 2000, in the Tower of Babel arc, Ra\u0026rsquo;s al Ghul took down the entire Justice League without firing a single beam. He stole the Batcave\u0026rsquo;s secret files: the contingency protocols Batman himself had…","title":"What Batman's Protocols Taught Me About Business Intelligence Inside Claude"},{"content":"Imagine you\u0026rsquo;ve hired the most talented driver in Formula 1. He\u0026rsquo;s fast, intuitive, and can get the most out of any car. But when he gets to the starting grid, the vehicle you handed him is a prototype with no ABS brakes, no telemetry, and no safety system. He might manage a few laps, but he\u0026rsquo;s not going to finish the race.\nThat\u0026rsquo;s exactly the scenario most companies are living through today with vibe coding. AI code generation tools, such as Replit, Cursor, and GitHub Copilot, are exceptional drivers. In minutes, they deliver working applications, good-looking interfaces, and reasonable business logic. The problem starts when that application needs to go into enterprise production: where does the data live? Who can access what? How do you make sure sensitive information is protected? How do you connect it to the company\u0026rsquo;s analytics ecosystem?\nRayfin is the engineering crew that was missing. Announced by Microsoft at Build 2026, it\u0026rsquo;s an open-source Backend-as-a-Service (BaaS) built specifically for Microsoft Fabric . It doesn\u0026rsquo;t just generate your application\u0026rsquo;s backend; it makes sure that backend is born already integrated with governance, security, and enterprise data.\nThe Real Problem: The Gap Between Prototype and Production Before understanding Rayfin, you need to understand the problem it solves, because it\u0026rsquo;s more serious than it looks.\nWhen a business team or a developer uses vibe coding to build an application, the typical result is a working frontend connected to a makeshift database (often a Supabase or Firebase instance set up in a hurry). That works for a demo. But when the company decides to put that system into production, tough questions come up:\nIs this application\u0026rsquo;s data in OneLake or in a separate silo? Can the analytics team access this information to build reports in Power BI? Do the Purview privacy policies apply to this data? If an employee is let go, is their access revoked automatically through Entra ID?\nIn most cases, the answer to all of these questions is \u0026ldquo;no.\u0026rdquo; The result is that the prototype sits idle, waiting for months of engineering work to be adapted to the corporate environment, or it\u0026rsquo;s simply abandoned.\nRayfin closes that gap. It makes the application enterprise-ready from birth, without requiring the developer (or the AI agent) to understand all the complexity of the corporate infrastructure.\nWhat Rayfin Does in Practice Rayfin takes a \u0026ldquo;code-first\u0026rdquo; approach: you define your application\u0026rsquo;s structure using TypeScript decorators, and it automatically generates everything underneath.\nWhen you run the npx rayfin up command, Rayfin:\nCreates the database with the correct schema Generates the CRUD APIs automatically Configures authentication through Microsoft Entra ID Deploys the application as a native Fabric artifact Makes sure the data lands directly in OneLake There\u0026rsquo;s no need to configure servers, write database migration scripts, or manually integrate with the company\u0026rsquo;s identity system. All of it happens from the data model definition in TypeScript.\nA Real Example: A Work Order System for Field Technicians To make this concrete, let\u0026rsquo;s use a real use case: a telecommunications company that needs an app for its field technicians to manage work orders.\nThe scenario without Rayfin: The IT team gets the request and uses vibe coding to build the React frontend in two days. But that\u0026rsquo;s where the problem starts. They need to set up a database, choose between PostgreSQL and MySQL, write the migrations, configure authentication, build the APIs, make sure only authorized technicians see the orders for their region, and then still build an ETL pipeline to get that data into Power BI so the manager can track the team\u0026rsquo;s productivity. That takes weeks.\nThe scenario with Rayfin: The developer (or the AI agent) defines the system\u0026rsquo;s entities in TypeScript:\nimport { entity, role, text, uuid, date, boolean, set, one } from \u0026#39;@microsoft/rayfin-core\u0026#39;; import { Customer } from \u0026#39;./Customer.js\u0026#39;; import { Region } from \u0026#39;./Region.js\u0026#39;; import { UserProfile } from \u0026#39;./UserProfile.js\u0026#39;; export type JobStatus = | \u0026#39;new\u0026#39; | \u0026#39;scheduled\u0026#39; | \u0026#39;in-progress\u0026#39; | \u0026#39;blocked\u0026#39; | \u0026#39;complete\u0026#39;; @entity() @role(\u0026#39;authenticated\u0026#39;, \u0026#39;*\u0026#39;) export class Job { @uuid() id!: string; @text() title!: string; @text({ optional: true }) description?: string; @set(\u0026#39;new\u0026#39;, \u0026#39;scheduled\u0026#39;, \u0026#39;in-progress\u0026#39;, \u0026#39;blocked\u0026#39;, \u0026#39;complete\u0026#39;) status!: JobStatus; @date({ optional: true }) scheduledAt?: Date; @date() createdAt!: Date; @date() updatedAt!: Date; @boolean({ default: false }) isOnSite!: boolean; @boolean({ default: false }) needsHelp!: boolean; @one(() =\u0026gt; Customer) customer!: Customer; @one(() =\u0026gt; Region) region!: Region; @one(() =\u0026gt; UserProfile, { optional: true }) technician?: UserProfile; } With that code, Rayfin understands there\u0026rsquo;s a Job entity with a typed status, relationships with Customer, Region, and UserProfile, and that only authenticated users can access it. It generates the database, the APIs, and the security configuration automatically.\nWhen the technician logs a work order in the app, that data lands directly in OneLake. At that same moment, the regional manager can see in Power BI how many orders are open, which technician has the heaviest workload, and which regions have the most blocked tickets. No ETL, no pipeline, no waiting.\nWhy This Is Different from Supabase, Firebase, and the Like This is a comparison worth making carefully, because Rayfin isn\u0026rsquo;t exactly competing with those tools; it\u0026rsquo;s solving a different problem.\nSupabase is excellent for someone building a product from scratch. Rayfin is for someone building inside an organization that already has data, policies, and an established Microsoft ecosystem.\nThe Engineering Crew: The SDK Packages Rayfin is made up of a set of packages that work together, like the different specialists on an F1 team:\n@microsoft/rayfin-core: the chassis. It defines the TypeScript decorators that describe the data model. @microsoft/rayfin-cli: the chief mechanic. It interprets the model and orchestrates the deployment to Fabric. @microsoft/rayfin-client: the co-driver. It generates the TypeScript client for the frontend to consume the generated APIs. @microsoft/rayfin-testing: the simulator. It lets you test the application logic locally before deploying. Each package has a clear responsibility. The developer (or the AI agent) only needs to interact directly with rayfin-core to define the model. The rest happens behind the scenes.\nThe Impact on Data Teams For people who work with data, Rayfin represents an important paradigm shift. Historically, there\u0026rsquo;s been a clear divide between \u0026ldquo;operational systems\u0026rdquo; (where data is generated) and \u0026ldquo;analytical systems\u0026rdquo; (where data is analyzed). That divide creates latency, ETL costs, and, frequently, inconsistencies between what the system shows and what the report presents.\nWith Rayfin, that divide starts to disappear for applications built in the Fabric ecosystem. The data is born in OneLake, governed by Purview, immediately available for analytics. The application and the data warehouse share the same source of truth.\nThat doesn\u0026rsquo;t solve all of a company\u0026rsquo;s data integration problems. Legacy systems still exist and still need pipelines. But for new applications built with vibe coding or by AI agents, Rayfin makes sure they enter the race with the right engineering already in place.\nConclusion Vibe coding is here to stay. The ability to generate working applications in minutes using AI is a real transformation in how software gets built. The challenge has always been making sure that software reaches enterprise production without compromising security, governance, and data integration.\nRayfin is Microsoft\u0026rsquo;s answer to that challenge. It doesn\u0026rsquo;t make development slower; it makes the road to production shorter, because it eliminates the weeks of infrastructure work that usually sit between the prototype and the system in real use.\nThe talented driver finally has the car he deserves.\nReferences: Official repository: github.com/microsoft/rayfin Community templates: github.com/microsoft/awesome-rayfin Official announcement at Build 2026: community.fabric.microsoft.com ","permalink":"https://julianamarialop.github.io/en/articles/rayfin-the-missing-mechanic-to-get-your-car-on-the-podium/","summary":"Imagine you\u0026rsquo;ve hired the most talented driver in Formula 1. He\u0026rsquo;s fast, intuitive, and can get the most out of any car. But when he gets to the starting grid, the vehicle you handed him is a prototype with no…","title":"Rayfin: The Missing Mechanic to Get Your Car on the Podium"},{"content":"If you\u0026rsquo;ve ever watched the X-Men, you know Professor Xavier\u0026rsquo;s power isn\u0026rsquo;t just reading minds. His real power lies in Cerebro, the machine that amplifies his abilities and lets him find the needle in the haystack in a chaotic world. Without Cerebro, Xavier is just a powerful telepath trying to hear one voice in the middle of a packed stadium. With Cerebro, he has surgical precision.\nIn the data world, we\u0026rsquo;re living through the moment when everyone has discovered they have a powerful telepath at their disposal. The promise of LLMs for analytics is seductive: point the model at your data warehouse, let users ask questions in natural language and watch the magic happen. The problem is that, without the right infrastructure, the magic quickly turns into chaos. The model hallucinates, picks the wrong table, uses the old definition of \u0026ldquo;active user\u0026rdquo; and delivers an answer that looks right but is fundamentally wrong.\nAnthropic, the creator of Claude, solved this problem internally. Today, 95% of the company\u0026rsquo;s business analytics queries are automated through Claude, with an aggregate accuracy of approximately 98%. Their secret wasn\u0026rsquo;t building a magic model that never makes mistakes. The secret was building Cerebro: a data and governance architecture that directs Claude\u0026rsquo;s power with absolute precision.\nThe Three Enemies of Accuracy To understand Anthropic\u0026rsquo;s solution, we need to understand the villains of this story. When an AI agent tries to answer a business question, it doesn\u0026rsquo;t fail because it can\u0026rsquo;t write SQL. It fails because the data environment is ambiguous. Anthropic identified three main failure modes that cause the overwhelming majority of errors:\nEntity Ambiguity: The agent can\u0026rsquo;t map a concept (\u0026ldquo;revenue for product X\u0026rdquo;) to the correct table and column because there are dozens of plausible options in the warehouse. Staleness: Data sources, business definitions and schemas change constantly. The agent\u0026rsquo;s knowledge goes stale and it starts returning subtly wrong answers. Retrieval Failure: The right information is in the data model and well documented, but the search space is so vast that the agent simply doesn\u0026rsquo;t find it. The solution to these problems isn\u0026rsquo;t prompt engineering. It\u0026rsquo;s data engineering.\nThe Foundation: Building the X-Mansion Anthropic\u0026rsquo;s first step was to focus on data foundations. If your warehouse has forty tables that seem to contain \u0026ldquo;revenue,\u0026rdquo; the agent will get lost. The solution is to create a small, highly governed set of logical models: canonical datasets that are the single source of truth.\nThat means applying rigorous data engineering practices. Dimensional modeling, shift-left testing and freshness and completeness checks remain essential. The difference is that the end consumer of this data is no longer a senior data scientist who knows how to dodge the warehouse\u0026rsquo;s traps. The consumer is an AI agent acting on behalf of a business user.\nTo keep these foundations solid, Anthropic adopted artifact co-location. Almost all data code (modeling, semantic layer, reference documentation) lives in a single repository. If a modeling change breaks a dashboard or invalidates a documented metric, continuous integration (CI) flags the error and the fix ships in the same pull request. It\u0026rsquo;s like making sure the X-Mansion blueprint is always up to date before any renovation.\nSources of Truth: The Map of Cerebro If the data foundations are the warehouse, the sources of truth are the reference surfaces the agent consults to navigate it. This is where ambiguity gets destroyed.\nThe semantic layer is the first line of defense. If a question maps cleanly to a defined metric, the agent calls a function and gets a number: the same number any other dashboard in the company would produce. Anthropic learned the hard way that trying to use an LLM to auto-generate metric definitions from raw tables doesn\u0026rsquo;t work; the model just encodes the existing ambiguities. Documentation can be AI-generated, but the definition must have a human owner.\nWhen the semantic layer doesn\u0026rsquo;t cover the question, the agent falls back on data lineage and business context. Business context is often overlooked, but it\u0026rsquo;s crucial. An agent that doesn\u0026rsquo;t understand the business will answer what the user asked, but not what they meant. It won\u0026rsquo;t know that \u0026ldquo;the Q2 launch\u0026rdquo; refers to a specific product. Anthropic solves this by feeding the agent a knowledge graph of the company, including indexed documents, roadmaps and the organizational structure.\nSkills: Training in the Danger Room If the sources of truth are the agent\u0026rsquo;s declarative knowledge, \u0026ldquo;skills\u0026rdquo; are its procedural knowledge. They instruct the agent on which sources to consult, in what order, how to navigate ambiguous data and what a finished analysis should look like.\nAt Anthropic, a skill is a folder of markdown files the agent reads on demand. Without these skills, Claude\u0026rsquo;s accuracy in answering analytics questions was no more than 21%. With the skills, that number jumped to over 95%.\nThe main strategy here is to create skills in pairs. A knowledge skill acts as a high-level router. Instead of letting the agent rummage through a warehouse with a million fields, the skill narrows the search space down to a few dozen curated files before a single query is written. It\u0026rsquo;s like Professor Xavier sending exactly the right team of X-Men on a specific mission, instead of sending every student in the school at once.\nThese reference documents are written specifically to be read by an LLM. They describe table granularity, known pitfalls (gotchas) and explicit routing triggers (for example: \u0026ldquo;IF the question is about experiment lift\u0026hellip; DO NOT use for raw event counts\u0026rdquo;).\nValidation: The Trial by Fire The last piece of the puzzle is validation. How do you know whether your agent is actually getting it right?\nAnthropic uses offline evaluations (evals) as question-and-answer pairs. They don\u0026rsquo;t tell you how the online agent will perform, but they ensure there are no critical gaps. The golden rule is to anchor the ground truth so it doesn\u0026rsquo;t change: an eval written against live data goes stale the moment the underlying number changes. The solution is to pin each eval to a snapshot date, or have the grader judge the agent\u0026rsquo;s query instead of the final number.\nIn the online environment, validation continues. Anthropic implemented an adversarial review: a Claude skill that aggressively challenges every underlying assumption in a potential answer. This increased accuracy by 6%, though at the cost of higher latency and token usage. In addition, every answer carries a provenance footer indicating which layer the information came from and how fresh the data is. It doesn\u0026rsquo;t make the answer more correct, but it helps the consumer judge the level of confidence.\nThe Mutant Factor Anthropic\u0026rsquo;s lesson is clear: accuracy in AI analytics isn\u0026rsquo;t a code generation problem. It\u0026rsquo;s a context and verification problem.\nPointing a powerful LLM at a disorganized data warehouse is like putting Professor Xavier in the middle of Times Square without Cerebro. He\u0026rsquo;ll hear a lot of noise, but he won\u0026rsquo;t find what you need. The real self-service analytics revolution doesn\u0026rsquo;t happen when the model gets smarter. It happens when data engineering, governance and business context come together to build the infrastructure that lets that intelligence shine.\nThe future of analytics isn\u0026rsquo;t about who has the best model. It\u0026rsquo;s about who builds the best Cerebro for it.\nReferences: How Anthropic enables self-service data analytics with Claude | Claude: https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude\n","permalink":"https://julianamarialop.github.io/en/articles/brain-behind-the-magic-anthropic-claude-professor-xavier-of-analytics/","summary":"If you\u0026rsquo;ve ever watched the X-Men, you know Professor Xavier\u0026rsquo;s power isn\u0026rsquo;t just reading minds. His real power lies in Cerebro, the machine that amplifies his abilities and lets him find the needle in the…","title":"The Brain Behind the Magic: How Anthropic Turned Claude into the Professor Xavier of Analytics"},{"content":"There\u0026rsquo;s a scene in Star Wars that perfectly sums up the problem Microsoft Scout was created to solve. Luke Skywalker is in the X-Wing cockpit, focused on the mission, while R2-D2 works quietly behind the scenes: calculating routes, monitoring systems, repairing damage, anticipating problems. Luke doesn\u0026rsquo;t have to stop what he\u0026rsquo;s doing to ask R2 to check the shields. The droid understands the context, knows what matters and acts autonomously to keep the mission on track.\nFor years, the promise of AI in the workplace was exactly that: an assistant that understands context, anticipates needs and acts proactively. What we got in practice was quite different. We got assistants that answer when asked, generate text when prompted and stop working when the conversation ends. We got very eloquent C-3POs, but no R2-D2.\nMicrosoft Scout, announced on June 2, 2026 as part of Microsoft Build 2026, is the first serious attempt to close that gap. It\u0026rsquo;s not just a new product; it\u0026rsquo;s the representative of a new category of agents that Microsoft is calling Autopilots. In this article, we\u0026rsquo;ll explore what that means in practice, how the technology works and why this announcement may be more important than it looks at first glance.\nA New Category: What Autopilots Are To understand Microsoft Scout, you first need to understand what sets an Autopilot apart from a conventional AI assistant. The distinction isn\u0026rsquo;t just technical; it\u0026rsquo;s philosophical.\nThe AI assistants we\u0026rsquo;ve known so far operate on a reactive interaction model. You ask a question, the system answers. You ask for a summary, the system generates it. You start a conversation, the system takes part. When the conversation ends, the system stops. It has no persistent memory of what happened before, no goals of its own and doesn\u0026rsquo;t act without being triggered.\nAutopilots are fundamentally different. According to Microsoft, they\u0026rsquo;re always-on agents that work autonomously, have their own identity and act on your behalf. The key phrase here is \u0026ldquo;always-on\u0026rdquo;. An Autopilot doesn\u0026rsquo;t wait to be called. It stays active in the background, understanding how work gets done across your apps and systems, and taking action without having to be triggered every time.\nThat\u0026rsquo;s a paradigm shift. Most of the AI systems we use today are tools: powerful, but passive. An Autopilot is closer to a collaborator: someone who understands the goals, monitors progress and acts independently to keep things running.\nMicrosoft Scout is Microsoft\u0026rsquo;s first Autopilot, and it was built specifically for the Microsoft 365 workplace.\nHow Scout Works: The Anatomy of the Corporate R2-D2 For an Autopilot to be useful, it needs three things: context (understanding what\u0026rsquo;s going on), capability (being able to act on what it understood) and trust (guarantees that it will act safely and within established limits). Scout was designed with those three dimensions in mind.\nContext: Integration with the Microsoft 365 Ecosystem Scout operates across cloud, desktop and web, connecting to Teams, Outlook, OneDrive and SharePoint. It has access to the data that drives day-to-day work: chats, emails, calendars and contacts. The main interaction happens in Teams, but its reach extends through the desktop app to the browser and local resources, including MCP (Model Context Protocol) servers.\nThis deep integration with the Microsoft 365 ecosystem is what sets Scout apart from generic assistants. It doesn\u0026rsquo;t just read documents; it understands the context those documents live in. It knows that report was created by Ana, that Ana reports to Carlos, that Carlos has a meeting tomorrow with the client who made the original request. That relational context is what lets Scout make smart decisions.\nCapability: What Scout Can Do Scout\u0026rsquo;s capabilities center on a specific and very real problem: the coordination work that eats up a disproportionate share of modern professionals\u0026rsquo; time. According to Microsoft, Scout was designed to reduce the coordination work that piles up over the course of the day.\nIn practice, that translates into concrete capabilities. Scout can proactively schedule and coordinate meeting times across time zones, flag important meetings and generate the materials needed to prepare, keeping the user in the loop. For any professional who has spent hours trying to find a time that works for five people in three different time zones, that\u0026rsquo;s a transformative capability.\nScout also identifies upcoming deliverables and automatically blocks calendar time to make sure the user has the focus needed to get them done. It can also spot risks, such as decisions stuck in email threads that are holding up project progress, and flag them before they become blockers.\nWhat makes these capabilities especially powerful is that they don\u0026rsquo;t depend on the user remembering to ask. Scout is monitoring continuously, identifying patterns and acting proactively. It\u0026rsquo;s the difference between having an assistant who does what you ask and having a collaborator who thinks alongside you.\nWork IQ: Intelligence That Grows Over Time Scout\u0026rsquo;s most strategic capability isn\u0026rsquo;t any of the features listed above. It\u0026rsquo;s its ability to learn. Over time, Scout builds context powered by Work IQ, learning how you work, what you value and what needs to happen next.\nWork IQ is Microsoft\u0026rsquo;s workplace intelligence layer, capturing how work actually happens in Microsoft 365 and in external organizational systems. It understands the connections between people, documents, meetings and decisions, creating a living representation of how the organization works.\nFor Scout, this means it doesn\u0026rsquo;t start from scratch every day. It accumulates context about your priorities, your professional relationships, your work patterns and the goals of the projects you\u0026rsquo;re involved in. Over time, it becomes increasingly accurate in its anticipations and more relevant in its actions.\nIt\u0026rsquo;s like R2-D2\u0026rsquo;s evolution over his missions with Luke: the longer they work together, the better the droid understands the pilot\u0026rsquo;s nuances and the more effective it becomes at supporting him.\nThe Deflector Shields: Enterprise Security and Governance An agent that acts autonomously on behalf of a professional in a corporate environment raises serious security questions. Who is responsible for the agent\u0026rsquo;s actions? How do you make sure it doesn\u0026rsquo;t access data it shouldn\u0026rsquo;t? How do you audit what it did? How do you ensure compliance policies are respected?\nMicrosoft built Scout with enterprise-grade security and controls from day one, and the mechanisms are more sophisticated than they look at first glance.\nIts Own Identity in Entra The starting point is identity. Each Scout agent operates under its own governed identity in Microsoft Entra, not as an anonymous, shared service account. That may sound like a technical detail, but it has profound implications.\nWhen Scout acts on a user\u0026rsquo;s behalf, that action is attributable to a known actor that the organization\u0026rsquo;s directory already understands. That means Scout\u0026rsquo;s actions show up in audit logs, can be traced, can be investigated and can be governed by the same policies that apply to any other user or service in the organization.\nThe credentials behind that identity are protected end to end: scoped to the task at hand, hidden from logs or diagnostics and managed with the same rigor as any first-party Microsoft service. When Scout acts on your behalf, you know exactly what authority it carried and that nothing sensitive leaked in the process.\nGranular Access Control Beyond identity, access control determines what Scout can do. It can only reach the resources and destinations that have been approved. This isn\u0026rsquo;t a binary \u0026ldquo;can do everything\u0026rdquo; or \u0026ldquo;can do nothing\u0026rdquo; control; it\u0026rsquo;s a granular system that can be configured for different levels of autonomy.\nSensitive actions can require a human\u0026rsquo;s approval before proceeding. That\u0026rsquo;s essential for building trust in the system gradually. A team can start with Scout only suggesting actions and waiting for approval, then progressively raise the level of autonomy as trust is established.\nMicrosoft Purview data protection policies, including sensitivity labels and data loss prevention, are applied in the moment, before anything is sent or written. Scout doesn\u0026rsquo;t get around these controls; it operates within them. For organizations in regulated industries such as finance, healthcare or legal, this native integration with existing compliance tools is a non-negotiable requirement.\nOpenClaw: The Open Source Foundation Scout is built on the open source OpenClaw technology. That choice reflects Microsoft\u0026rsquo;s commitment to transparency and to the developer community. By building on an open source foundation, Microsoft lets organizations inspect, audit and contribute to the underlying technology.\nMicrosoft is contributing policy compliance directly to OpenClaw upstream. Organizations running OpenClaw will be able to validate whether their environment is configured within security and compliance requirements, operating safely and getting a verifiable, audit-ready answer.\nWhere Things Stand: Private Preview and the Road Ahead Microsoft Scout isn\u0026rsquo;t available to everyone yet. Microsoft has already used an early desktop experience internally, with the company\u0026rsquo;s own employees using Scout to take on coordination tasks, spot risks earlier and keep work moving without constant prompting.\nNow that experience is being extended to a select group of customers in private preview and to Frontier organizations. Access requires enrolling in the Frontier program, configuring a policy in Intune and an opt-in attestation. Users with a GitHub Copilot license can then download and install the experience.\nThis gradual rollout is deliberate. Microsoft is being cautious about expanding access to an agent that acts autonomously in corporate environments. Every organization that joins the Frontier program is, in a sense, helping Microsoft understand how Autopilots behave in real contexts, which use cases are most valuable and where the rough edges are that need polishing.\nWhy This Matters More Than It Seems It\u0026rsquo;s easy to look at Microsoft Scout and see it as just another productivity assistant with a few new features. But what\u0026rsquo;s being announced is something more fundamental.\nThe Autopilot category represents a shift in the relationship between humans and AI systems at work. Until now, that relationship was one of tool and user: you use the tool when you need it and put it away when you don\u0026rsquo;t. With Autopilots, the relationship becomes closer to a partnership: an agent that understands your goals, monitors the environment, identifies opportunities and risks, and acts autonomously within the limits you\u0026rsquo;ve set.\nThis shift has implications that go beyond individual productivity. It changes how organizations need to think about governance, accountability, auditing and the division of labor between humans and systems. The questions security, compliance and HR teams will need to answer in the coming years are questions that don\u0026rsquo;t have settled answers yet.\nMicrosoft Scout is the first step on that journey. Like every first step, it\u0026rsquo;s both promising and incomplete. But the direction is clear: the future of work isn\u0026rsquo;t about more powerful tools. It\u0026rsquo;s about smarter partners.\nThe corporate R2-D2 has arrived. The question now is: are you ready to fly with it?\nReferences: [1] Introducing Microsoft Scout: Your always-on personal agent | Microsoft 365 Blog: https://www.microsoft.com/en-us/microsoft-365/blog/2026/06/02/introducing-microsoft-scout-your-always-on-personal-agent/\n","permalink":"https://julianamarialop.github.io/en/articles/microsoft-scout-the-r2-d2-that-will-change-the-way-you-work/","summary":"There\u0026rsquo;s a scene in Star Wars that perfectly sums up the problem Microsoft Scout was created to solve. Luke Skywalker is in the X-Wing cockpit, focused on the mission, while R2-D2 works quietly…","title":"Microsoft Scout: The R2-D2 That Will Change the Way You Work"},{"content":"In 1977, when the movie theater lights went down and that iconic opening crawl started scrolling across the screen, nobody knew exactly what was about to happen. Star Wars wasn\u0026rsquo;t just a movie; it was a rupture, a redefinition of what was possible. Microsoft Microsoft Build 2026 had that same flavor. With the theme \u0026ldquo;Be Yourself at Work\u0026rdquo;, Microsoft didn\u0026rsquo;t just present a set of new products. It presented a new galactic order for software development and for artificial intelligence in the enterprise.\nFor decades, the relationship between organizations and technology was one of dependence. Dependence on platforms, on third-party models, on consultancies that held the knowledge. Build 2026 marks the turning point: Microsoft is betting that the competitive edge of the future is no longer access to artificial intelligence, but ownership of it. It\u0026rsquo;s not enough to have access to the Force; the Force has to be yours, shaped by your history, your data and your rules.\nIn this article, we\u0026rsquo;ll walk through the three big pillars of Build 2026 with the depth each one deserves: the new Microsoft IQ context layer, the MAI family of in-house models and the governance framework that makes sure all of this runs securely in production. Get your lightsaber ready.\nEpisode I: The Phantom Menace of Generic Context Before understanding what Microsoft announced, you need to understand the problem it\u0026rsquo;s solving. Over the last two years, the race for AI in the enterprise produced a predictable pattern: technology teams deploy powerful language models, connect a few data sources and expect transformative results. What usually happens is quite different.\nAgents respond generically because they don\u0026rsquo;t understand the organization\u0026rsquo;s real context. They know what an email is, but they don\u0026rsquo;t know that this particular email is tied to a critical project that\u0026rsquo;s running late. They know what a meeting is, but they don\u0026rsquo;t understand that this meeting involves a client who\u0026rsquo;s about to cancel the contract. The intelligence is there, but the context is missing.\nThat\u0026rsquo;s exactly the gap Microsoft IQ was designed to fill. Available starting today in GitHub Copilot, Microsoft Foundry and Copilot Studio, Microsoft IQ is described as a new context layer that grounds agents in both world knowledge and your company\u0026rsquo;s specific knowledge. It\u0026rsquo;s not a standalone product; it\u0026rsquo;s an intelligence infrastructure made up of four specialized layers that work together.\n_https://jannikreinhard.com/2025/11/26/microsoft-iq-the-intelligence-layer-that-finally-makes-ai-agents-useful/_\nWork IQ: The Intelligence of the Workflow Work IQ is the layer closest to people\u0026rsquo;s day-to-day. It captures how work really happens in Microsoft 365, understanding the connections between people, emails, documents, meetings and how it all fits together. It\u0026rsquo;s not just a document index; it\u0026rsquo;s a living representation of how the organization works.\nThink of Work IQ as your organization\u0026rsquo;s Master Yoda. It doesn\u0026rsquo;t just know the facts; it understands the relationships, the tensions, the unspoken priorities. When an agent powered by Work IQ gets a question about a project\u0026rsquo;s status, it doesn\u0026rsquo;t just look for documents with that name. It understands who the key people are, what the latest decisions were, where the blockers are and what needs to happen next.\nThe Work IQ APIs become generally available on June 16, 2026, letting developers access this intelligence layer programmatically. That means any agent, built in any framework, can be enriched with the real context of the organization\u0026rsquo;s workflow.\nFabric IQ: The Language of Business Data If Work IQ takes care of the human and relational context, Fabric IQ takes care of the context of structured business data. It provides a shared semantic foundation over the data that lives in Microsoft Fabric, making sure that when an agent talks about \u0026ldquo;revenue\u0026rdquo;, \u0026ldquo;margin\u0026rdquo; or \u0026ldquo;churn\u0026rdquo;, it\u0026rsquo;s using the same definitions as the finance team, the sales team and the product team.\nThis problem of inconsistent semantics is more serious than it looks. In large organizations, it\u0026rsquo;s common for different areas to use the same terms with different definitions. Fabric IQ acts as the C-3PO of the data galaxy: fluent in over six million forms of communication, making sure every agent speaks the same language when it comes to the business\u0026rsquo;s critical data.\nFoundry IQ and Web IQ: Connecting the Universe Foundry IQ is the link that ties the other layers together, enabling retrieval planning across both enterprise knowledge and the real-time web. It\u0026rsquo;s the intelligence that decides, when faced with a complex question, which sources to consult, in what order and how to combine the answers.\nThe newest member of the family is Web IQ, announced at the event itself. It\u0026rsquo;s described as the fastest way to ground agents in real-world information: an AI-first, model-agnostic, MCP-native (Model Context Protocol) web search stack, able to return relevant passages almost 2.5 times faster than the best alternative available on the market. For agents that need up-to-date information on markets, regulations or external events, Web IQ is the hyperdrive that takes them where they need to go in record time.\nEpisode II: The Empire of In-House Models For a long time, the big tech companies depended on third-party models to power their AI products. Microsoft is no exception: Copilot was built on OpenAI\u0026rsquo;s models, and that partnership remains important. But Build 2026 marked an inflection point: Microsoft is building its own AI models, and they\u0026rsquo;re competitive.\nThe MAI (Microsoft AI) family was introduced with seven new in-house models, each designed for a specific purpose. This specialization strategy is like assembling an imperial fleet: each ship has a defined role, and together they cover the full spectrum of needs.\n_Credits: X/@mustafasuleyman_\nMAI-Thinking-1: The Star Destroyer of Reasoning The family\u0026rsquo;s flagship is MAI-Thinking-1, Microsoft\u0026rsquo;s first reasoning model. What makes it especially relevant isn\u0026rsquo;t just its performance, but how it was built: trained from scratch, with no distillation from other models, on clean, commercially licensed data. For companies worried about intellectual property and compliance, that\u0026rsquo;s a significant differentiator.\nWith 35 billion active parameters and a 256K-token context window, MAI-Thinking-1 was designed specifically to handle complex multi-step instructions, long-context reasoning and code generation. The benchmark numbers are impressive: in blind tests, independent evaluators preferred MAI-Thinking-1 over Sonnet 4.6, and it matched Opus 4.6 in coding skills on SWE Bench Pro. For a first reasoning model, these are results that immediately put Microsoft among the serious contenders in this category.\nThe model is available in Foundry in private preview starting today.\nMAI-Image-2.5: The Visual Craftsman MAI-Image-2.5 represents Microsoft\u0026rsquo;s entry into the image generation and editing model market, and it came in strong. It\u0026rsquo;s Microsoft\u0026rsquo;s first model to support both text-to-image and image-to-image workloads. On the Arena AI leaderboard, it ranks third in text-to-image and second in image-to-image, ahead of established competitors.\nWhat makes this model particularly interesting for the Microsoft ecosystem is its native integration. It\u0026rsquo;s already live in PowerPoint, rolling out in OneDrive and coming to Foundry with market-leading quality per dollar. For marketing, design and communications teams that already live in the Microsoft 365 ecosystem, that represents significant creative capability without leaving the work environment.\nThe Full Family: Specialists for Every Mission Besides the two main models, the MAI family includes:\nMAI-Transcribe-1.5 combines state-of-the-art accuracy across 43 languages, with streaming coming soon. For global organizations operating in multiple countries, the ability to transcribe meetings, calls and audio content with high accuracy in dozens of languages is a critical operational capability.\nMAI-Voice-2 and its flash variant are now available in more than 15 new languages with new voice options. Combined with MAI-Transcribe-1.5, this creates a complete voice infrastructure that can power everything from call center assistants to voice interfaces for industrial applications.\nMAI-Code-1 is an inference-efficient coding model, tuned specifically for GitHub and now available in Copilot and VS Code. Its code specialization, combined with inference efficiency, makes it ideal for scenarios where speed and cost are critical, such as real-time suggestions while typing.\nOpenness and Choice: Beyond the In-House Catalog Microsoft made it clear that the strategy isn\u0026rsquo;t to build a walled garden. The MAI models will be available on third-party platforms such as Fireworks AI, Baseten and Open Router. At the same time, Fireworks AI becomes generally available in Foundry, offering a single platform experience with enterprise governance and data residency in Azure, regardless of the model chosen.\nThis openness is strategic. Microsoft is betting that by offering the best platform to run any model, it becomes indispensable even for customers who prefer models from other vendors.\nEpisode III: The Return of Governance With great power comes great responsibility, and the proliferation of autonomous agents in enterprise environments raises serious questions of security, compliance and governance. Who\u0026rsquo;s accountable when an agent makes a wrong decision? How do you make sure agents don\u0026rsquo;t access data they shouldn\u0026rsquo;t? How do you audit what an agent did?\nMicrosoft presented a comprehensive answer to these questions, building what could be described as the Galactic Republic of agent governance.\n_https://www.microsoft.com/pt-br/microsoft-agent-365_\nAgent 365: The Galactic Senate of Agents Agent 365 for local agents is the centerpiece of this framework. It extends Entra, Defender and Purview into a single control plane to observe, govern and protect agents across the entire enterprise environment, regardless of where they\u0026rsquo;re hosted or which framework they were built in.\nThat last point is crucial. Agent 365 isn\u0026rsquo;t just for Microsoft agents. It works as a universal control plane, able to monitor and govern agents built in LangChain, AutoGen, CrewAI or any other framework. For CISOs and security teams watching AI agents proliferate across their organizations, that centralized visibility is fundamental.\nFrontier Tuning: Training Your Own Jedi Knights One of the most strategic announcements at Build 2026 is Frontier Tuning, available in private preview starting today. It applies reinforcement learning within the organization\u0026rsquo;s compliance perimeter, letting agents learn how the business really works.\nUsing the organization\u0026rsquo;s own data, domain knowledge and workflows, Frontier Tuning creates a loop that improves as the agents work. It\u0026rsquo;s the difference between hiring an outside consultant who knows market best practices and training an in-house employee who knows the specifics of your business. Over time, an agent trained with Frontier Tuning doesn\u0026rsquo;t just follow generic rules; it internalizes the way your organization thinks and operates.\nASSERT and Agent Control Specification: The Galactic Constitution To make sure governance is consistent and auditable, Microsoft announced two open source projects that form the foundation of agent security.\nASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing) is a framework for policy-based safety evaluation. It lets organizations define expected-behavior policies for their agents and continuously validate whether those agents are operating within them. It\u0026rsquo;s like having an automated audit system that constantly checks whether the agents are following the Galactic Constitution.\nThe Agent Control Specification standardizes where and how to apply controls in the agent loop. By creating an open standard for this, Microsoft is trying to establish a common language for agent governance that can be adopted across the whole ecosystem, regardless of framework or platform.\nCodename MDASH: The Proactive Defense Fleet Rounding out the security layer, Codename MDASH represents a radically different approach to software security. Instead of waiting for vulnerabilities to be discovered and reported, MDASH deploys more than 100 specialized agents to proactively find exploitable bugs.\nThese agents reason about data flow, business logic and exploit chains, delivering context-aware fixes directly in the Defender Portal. It\u0026rsquo;s like having a fleet of X-Wing fighters constantly patrolling the perimeter, spotting threats before they become crises.\nWhat This Means in Practice Microsoft Build 2026 wasn\u0026rsquo;t an event of isolated announcements. It was the presentation of a coherent, integrated vision of how Microsoft believes AI will evolve in the enterprise. That vision has three characteristics that set it apart from what had been done so far.\nFirst, the emphasis on owning the intelligence. The event\u0026rsquo;s central message was clear: the competitive edge is no longer having access to the most powerful AI. It\u0026rsquo;s having an AI that is genuinely yours, fed by your data, trained on your processes and operating within your rules. Microsoft IQ, Frontier Tuning and the MAI family are all pieces of that strategy.\nSecond, the bet on openness as a competitive advantage. By making the MAI models available on third-party platforms, by building Agent 365 to govern agents from any framework and by launching open source projects like ASSERT and the Agent Control Specification, Microsoft is signaling that its advantage lies not in lock-in, but in the quality of the platform.\nThird, its seriousness about governance and security. At a time when many companies are still figuring out how to use AI agents responsibly, Microsoft arrived at Build 2026 with concrete answers to the hardest questions: how to audit, how to control, how to ensure compliance and how to protect sensitive data.\nThe galaxy of enterprise AI is changing fast. Build 2026 made it clear that Microsoft isn\u0026rsquo;t just taking part in that change; it\u0026rsquo;s trying to define it. May the Force be with you.\nReferences: [1] Microsoft Build 2026: Be yourself at work - The Official Microsoft Blog: https://blogs.microsoft.com/blog/2026/06/02/microsoft-build-2026-be-yourself-at-work/\nAgent 365: https://www.microsoft.com/pt-br/microsoft-agent-365\n","permalink":"https://julianamarialop.github.io/en/articles/microsoft-build-2026-a-new-hope-for-the-era-of-autonomous-agents/","summary":"In 1977, when the movie theater lights went down and that iconic opening crawl started scrolling across the screen, nobody knew exactly what was about to happen. Star Wars wasn\u0026rsquo;t just a movie; it was a rupture, a…","title":"Microsoft Build 2026: A New Hope for the Era of Autonomous Agents"},{"content":"The Castle That Was Built First Asgard was built before any other realm. OpenAI raised the castle with GPT-3, GPT-4 and ChatGPT in a sequence no one could keep up with, and the world started using this technology before it even properly understood what it was using. It wasn\u0026rsquo;t luck; it was vision, execution and the ability to scale at a moment when the market was ready to absorb everything put in front of it. That has a name, and the name is market leadership. Odin deserves the throne he sits on.\nAnd then along came Claude , uninvited, without the key to the golden bridge, coming from outside the wall.\nThe Prince Who Didn\u0026rsquo;t Need the Crown There\u0026rsquo;s a common reading of Loki that reduces him to the trickster, the one who lies, the one who betrays. Wrong. The Loki in this parallel is the other one, the one with no scepter, no army, no name recognized across the Nine Realms, but with something Asgard didn\u0026rsquo;t expect to find outside the wall: the audacity to deliver what nobody asked for, without waiting to be called, without asking permission to show what he was capable of.\nAnd Loki masters something Asgard always treated as minor: discretion. What enters the hall stays in the hall. In Claude\u0026rsquo;s enterprise uses, what you share in the conversation doesn\u0026rsquo;t go into model training, and that\u0026rsquo;s not a contractual detail, it\u0026rsquo;s an architecture requirement.\nLoki isn\u0026rsquo;t just \u0026ldquo;the one who arrived later\u0026rdquo;. He\u0026rsquo;s the one who delivers what the castle needs before being given permission to enter, and that\u0026rsquo;s exactly how Claude has behaved in the enterprise data ecosystem over the last two years.\nFour Fronts Where the Difference Shows Up in Production In four contexts that are part of my day-to-day in enterprise projects, the difference between Claude and GPT-4o isn\u0026rsquo;t in a lab benchmark; it\u0026rsquo;s in the moment the output leaves the call and has to work.\nCode generation - When a Spark pipeline has to deal with client-specific business rules, chained transformations, schema evolution and edge cases that aren\u0026rsquo;t documented anywhere but exist in production data, what separates one model from the other is how much review is left for the architect after the output is generated. Claude delivers structure that goes to production with minimal review, especially in null handling, idempotency and schema drift, while GPT-4o delivers structure that typically needs a surgical pass before any commit, refining precisely those same sensitive points. The practical difference isn\u0026rsquo;t in the absolute quality of each isolated call; it\u0026rsquo;s in the amount of cognitive work left for whoever reviews the code before the merge, and that difference accumulates over hundreds of weekly calls until it becomes a visible difference in team productivity.\nLong document analysis - When the input document is an eighty-page RFP, with LGPD (Brazil\u0026rsquo;s data protection law) clauses buried in Annex VII and contradictory requirements between the scope chapter and the SLA chapter, the capability that matters isn\u0026rsquo;t answering a well-phrased question; it\u0026rsquo;s keeping coherence across the whole document without losing the thread on page forty-seven. Claude Sonnet 4.5 runs with a two-hundred-thousand-token window in production and up to one million in preview for select cases, while GPT-4o runs at one hundred twenty-eight thousand, and that difference stops being a hardware spec the moment three sections of the RFP contradict one another. Claude typically flags the contradiction, while GPT-4o often answers about one section while forgetting what it said about the other, producing derivative documents that silently inherit the original inconsistency.\nArchitecture - When the question is about a trade-off, such as choosing between lakehouse federation and physical replication, between synchronous and asynchronous processing, or between streaming and micro-batch, what separates a good answer from a useful one is how much the recommendation exposes its assumptions instead of just handing back the verdict. Claude doesn\u0026rsquo;t return just the closed recommendation; it exposes the assumptions that need to be true for the recommendation to hold, maps the risks of the proposed path and points out what the question didn\u0026rsquo;t ask but should have before getting an answer. That stance is exactly what a senior architect needs from the model when the decision is headed to a governance committee, and this is where Loki delivers the answer the realm needs to hear, not the one the realm expected to hear.\nAgents - In agentic systems with chained modules, task orchestration across four to six steps and prompts that need to stay coherent across sequential calls, what separates a functional agent from an agent that needs a human babysitter is consistency between calls, where what the agent decided in step two has to inform what it does in step five without losing the central thesis along the way. Claude sustains its reasoning across these chains without losing the context of the previous instruction, and in architectures with parallel sub-agents it keeps the mission coherent while each child executes its part, doing away with the kind of intermediate human validation that usually turns an autonomous agent into a supervised chatbot disguised as automation.\nThe Decision No Benchmark Answers This isn\u0026rsquo;t about picking a winner between Asgard and its uncrowned prince. It\u0026rsquo;s about knowing which realm to call for which kind of mission.\nClaude dominates when the operational context is long, when the task demands auditable reasoning that will end up in a governance report, when the output goes into production without line-by-line human review, or when the organization needs predictable behavior in projects with sensitive data locked inside the conversation and protected from becoming fuel for external training.\nGPT-4o dominates when integration with the Microsoft ecosystem is already paid for and installed, when the use case is natively multimodal, combining vision, audio and text in the same call, when per-call latency matters more than incremental output quality, or when the team has accumulated years of optimized prompts in production and the cost of rebuilding that library doesn\u0026rsquo;t justify switching vendors.\nNeither one is wrong. It depends on the mission.\nThe 60-Thousand-Token RFP Consider a document analysis agent that processes fifty RFPs a month, with each RFP averaging sixty thousand tokens between the original document and the package of technical annexes. On GPT-4o, with a one-hundred-twenty-eight-thousand-token window, that volume fits in a single call when the annex package is lean, but the moment the technical brief goes past seventy thousand tokens it becomes necessary to split the document into two calls and reconcile the result afterward, and each reconciliation introduces the risk of losing context between the parts, with a recurring share of these cases ending in an important clause falling between the two calls and requiring intervention from the human orchestrating the agent.\nOn Claude Sonnet 4.5, with a standard two-hundred-thousand-token window (and up to one million in preview for select cases), the same RFP fits comfortably in a single call, with no reconciliation and no clause lost at the cut, and when this setup is combined with the API\u0026rsquo;s prompt caching, which according to Anthropic\u0026rsquo;s official documentation can cut context-reading costs by up to ninety percent for stable instructions, the cost per processed RFP stays competitive even in scenarios where Claude has a higher per-token price in some tiers.\nThe right math isn\u0026rsquo;t \u0026ldquo;which model is cheapest per million tokens\u0026rdquo;; it\u0026rsquo;s which model delivers the processed document correctly the first time, because every human intervention to reconcile lost context costs the operation more than any difference in inference price between vendors.\nThe Three Questions for the Team For anyone deciding today which LLM to put as the backbone of an enterprise system, three questions are worth asking the technical team before the decision becomes a commit in some config.\n\u0026ldquo;What is the real size of the operational context in production?\u0026rdquo; If the required window stays below fifty thousand tokens in ninety-five percent of cases, Claude\u0026rsquo;s long-context advantage doesn\u0026rsquo;t justify switching vendors, but above that the team needs to measure how often the current system is reconciling today, because that\u0026rsquo;s the number that matters for the decision, not the maximum window written on the spec sheet.\n\u0026ldquo;Does the output go straight to the client, or does it go through human review before it gets there?\u0026rdquo; If it goes straight through, whether in an autonomous agentic system, in regulatory report generation or in code that goes into the main branch without pull request review, predictable behavior weighs far more than a marginal difference in per-call speed, because the cost of a wrong answer reaching the end user is asymmetric compared to the cost of a slow answer that arrives correct.\n\u0026ldquo;Could what we share in the conversation with the model become training data for a competing model?\u0026rdquo; That clause has to be read in the data usage policy signed with the contract, not in the marketing communication, especially in projects with regulated data in the financial, healthcare or legal verticals, where the difference between an architecture approved and one rejected by the governance committee usually lies in exactly that paragraph of the terms.\nConclusion: The Merit That Doesn\u0026rsquo;t Need an Invitation Asgard noticed.\nClaude is now inside Microsoft 365 Copilot, inside Amazon Bedrock, inside Google Cloud Vertex AI, taking up space in enterprise stacks that three years ago were OpenAI\u0026rsquo;s exclusive territory, and Odin opened the door not out of the throne\u0026rsquo;s generosity, but because the merit of the one showing up outside had grown too big for the court to keep ignoring. The castle doesn\u0026rsquo;t define who\u0026rsquo;s good; it defines who got there first, and whoever arrives later with audacity and consistent delivery doesn\u0026rsquo;t need to tear down the wall to rewrite the balance of power, just to keep showing up on the other side of it with the work done.\nIf you work with data and AI in enterprise projects, the question today is no longer \u0026ldquo;Claude or GPT-4o\u0026rdquo; as a binary duel; it\u0026rsquo;s what you need to survive in production: long documents, auditable reasoning, critical pipelines, architecture decisions that cost a lot when they go wrong, agents that need to stay coherent across six consecutive steps. In those scenarios, Claude is the choice, not because Asgard fell, but because Loki delivered what the castle needed before being given any permission to enter.\nAnd when the king finally opened the door, the merit was already on the other side, waiting.\nReferences Anthropic. Claude Sonnet 4.5. https://www.anthropic.com/news/claude-sonnet-4-5 Anthropic. Prompt Caching. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching Anthropic. Trust Center, Data Privacy and Usage Policies. https://trust.anthropic.com/ Microsoft. Anthropic models available in Microsoft 365 Copilot (September 2025 announcement). https://www.microsoft.com/en-us/microsoft-365/blog/ AWS. Anthropic Claude on Amazon Bedrock. https://aws.amazon.com/bedrock/anthropic/ ","permalink":"https://julianamarialop.github.io/en/articles/loki-didnt-ask-for-permission-claude-vs-openai-enterprise-data/","summary":"Asgard was built before any other realm. OpenAI raised the castle with GPT-3, GPT-4 and ChatGPT in a sequence no one could keep up with, and the world started using this technology before it even…","title":"Loki Didn't Ask for Permission. A Technical Analysis of Claude vs OpenAI in Enterprise Data Projects."},{"content":"The Invisible Cost of Every Call Imagine that every time you start a meeting, the assistant reads all the company rules out loud before getting started. Two hundred pages. Every day. In every meeting. Even though everyone already knows what\u0026rsquo;s written there.\nSounds absurd. But it\u0026rsquo;s exactly what happens in most AI solutions built with Claude via the API. On every call, the same context, the same instructions, the same reference documents are sent and reprocessed from scratch. As if Claude had never seen them before.\nThe cost of this doesn\u0026rsquo;t show up as a single line item on the bill. It builds up silently, call after call, until it becomes a number nobody can explain.\nThere\u0026rsquo;s a solution. And it has everything to do with the way Gambit plays his cards.\nThe Mutant Who Never Wastes Energy Gambit isn\u0026rsquo;t the strongest of the X-Men. He doesn\u0026rsquo;t have Wolverine\u0026rsquo;s adamantium, Xavier\u0026rsquo;s telepathy or Storm\u0026rsquo;s weather control. What he has is something rarer: the ability to charge any object with kinetic energy and release it at exactly the right moment, with no waste.\nHe doesn\u0026rsquo;t recharge a card that\u0026rsquo;s already charged. He doesn\u0026rsquo;t spend new energy on what has already been processed. Every joule is applied once, with precision, and reused as much as possible before being discarded.\nThat\u0026rsquo;s exactly how Prompt Caching in the Claude API works.\nIn any solution built with Claude via the API, the biggest source of invisible cost isn\u0026rsquo;t the response Claude generates. It\u0026rsquo;s the context you send on every call: the system prompt with the business rules, the document being analyzed, the conversation history. All of it is reprocessed from scratch on every request, as if Gambit recharged every card before every play.\nPrompt Caching fixes that. You load the context once, it gets stored, and on the following calls Claude reads from the cache at a fraction of the original cost. The card is already charged. Just play it.\nThe Problem Nobody Sees on the Bill To understand the impact of Prompt Caching, you need to understand how the Claude API works under the hood.\nThe Claude API is stateless: each call is processed from scratch, with no memory of previous ones. That means if you have a 5,000-token system prompt describing your solution\u0026rsquo;s business rules, those 5,000 tokens are sent and reprocessed on every call.\nImagine a customer service solution that handles 10,000 conversations a day. In each conversation, the 5,000-token system prompt is reprocessed. That\u0026rsquo;s 50 million input tokens of repeated context alone, every day, paid at the same rate as new tokens.\nOr imagine a document analysis agent that reads a 50,000-token contract and answers ten different questions about it. Without caching, the contract is reprocessed ten times. With caching, it\u0026rsquo;s processed once and read nine times at 10% of the cost.\nGambit doesn\u0026rsquo;t throw the same cards again from scratch. He keeps the ones that are already charged and uses energy where it hasn\u0026rsquo;t been applied yet.\nHow Prompt Caching Works in Practice The mechanism is straightforward. When the engineering team makes the first API call with a content block marked for caching, Anthropic processes and stores that content. On subsequent calls that use the same content, Claude reads from the cache instead of reprocessing.\nThe pricing structure reflects exactly that behavior:\nThe first call, which creates the cache, costs 1.25x the normal input token price for a 5-minute cache, or 2x for a 1-hour cache. That\u0026rsquo;s the cost of charging the card.\nThe following calls, which read from the cache, cost only 10% of the normal input token price. That\u0026rsquo;s the cost of reusing the already charged card.\nBreak-even is immediate: with a 5-minute cache, the second call already pays for the write cost and starts generating savings. With a 1-hour cache, savings start on the third call.\nIn practice, for an agent that receives 100 calls with the same 10,000-token system prompt using Claude Sonnet 4.6:\nWithout cache: 100 calls × 10,000 tokens × $3/MTok = $3.00 With cache: 1 write (12,500 tokens × $3/MTok = $0.038) + 99 reads (10,000 tokens × $0.30/MTok = $0.297) = $0.34\nAn 88% reduction in input cost with an implementation change that takes less than an hour.\nWhat It Looks Like in Code: The Minimum the Team Needs to Implement If you\u0026rsquo;re going to hold the engineering team accountable for this, it\u0026rsquo;s important to understand what the implementation involves. It\u0026rsquo;s a small, surgical change: adding the cache_control parameter to the content block that should be cached.\nHere\u0026rsquo;s the difference between a call without cache and one with cache for a Microsoft Fabric pipeline monitoring agent:\nWithout cache: context reprocessed on every call:\nimport anthropic client = anthropic.Anthropic() # System prompt com regras do agente: reprocessado em CADA chamada response = client.messages.create( model=\u0026#34;claude-sonnet-4-6\u0026#34;, max_tokens=1024, system=\u0026#34;\u0026#34;\u0026#34;Você é um agente de monitoramento de pipelines do Microsoft Fabric. Analise o status reportado e classifique a severidade seguindo estas regras: - CRÍTICO: pipeline com falha que afeta dados de produção em Gold - ALTO: pipeline com falha em Silver com impacto em relatórios - MÉDIO: pipeline com falha em Bronze sem impacto imediato - BAIXO: alertas de performance sem falha confirmada Sempre inclua: severidade, diagnóstico provável e próximo passo recomendado.\u0026#34;\u0026#34;\u0026#34;, messages=[ {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Status do pipeline: {status_pipeline}\u0026#34;} ] ) With cache: system prompt loaded once, reused on the rest:\nimport anthropic client = anthropic.Anthropic() # System prompt cacheado: processado apenas na primeira chamada response = client.messages.create( model=\u0026#34;claude-sonnet-4-6\u0026#34;, max_tokens=1024, system=[ { \u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;\u0026#34;\u0026#34;Você é um agente de monitoramento de pipelines do Microsoft Fabric. Analise o status reportado e classifique a severidade seguindo estas regras: - CRÍTICO: pipeline com falha que afeta dados de produção em Gold - ALTO: pipeline com falha em Silver com impacto em relatórios - MÉDIO: pipeline com falha em Bronze sem impacto imediato - BAIXO: alertas de performance sem falha confirmada Sempre inclua: severidade, diagnóstico provável e próximo passo recomendado.\u0026#34;\u0026#34;\u0026#34;, \u0026#34;cache_control\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;ephemeral\u0026#34;} # ← essa linha ativa o cache } ], messages=[ {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Status do pipeline: {status_pipeline}\u0026#34;} ] ) # Verificando se o cache está funcionando uso = response.usage print(f\u0026#34;Tokens escritos no cache: {uso.cache_creation_input_tokens}\u0026#34;) print(f\u0026#34;Tokens lidos do cache: {uso.cache_read_input_tokens}\u0026#34;) The difference is one line: \u0026ldquo;cache_control\u0026rdquo;: {\u0026ldquo;type\u0026rdquo;: \u0026ldquo;ephemeral\u0026rdquo;}. The rest of the logic doesn\u0026rsquo;t change. For the engineering team, this is one of the highest-return, lowest-effort optimizations in the Claude ecosystem.\nA Real Case: Data Quality Agent on Fabric To make the impact concrete, consider a data quality agent built on Microsoft Fabric. It monitors Silver layer tables, detects anomalies and generates alerts for the data team.\nThe agent has an 8,000-token system prompt with the quality rules, acceptable thresholds by data domain, naming standards and the expected alert format. It runs 500 checks a day, one for each monitored table.\nWithout Prompt Caching: 500 calls × 8,000 tokens × $3/MTok = $12.00/day = $360/month in repeated context alone.\nWith Prompt Caching: 1 write (10,000 tokens × $3/MTok = $0.03) + 499 reads (8,000 tokens × $0.30/MTok = $1.197) = $1.23/day = $36.90/month\nA savings of $323/month with one line of code added to the system prompt. Over a year, that\u0026rsquo;s more than $3,800 saved on a single agent.\nAnd that\u0026rsquo;s a conservative scenario. Higher-volume agents, with longer system prompts or that process long documents, see proportionally bigger savings.\nWhat Can and Can\u0026rsquo;t Be Cached Gambit knows not every object is worth charging. Cards that are too small don\u0026rsquo;t build up enough energy to justify the move. The same reasoning applies to Prompt Caching.\nIdeal candidates for caching:\nThe system prompt is the most obvious candidate. If the solution has a fixed set of instructions, business rules or personas that go with every call, that content should be cached. It never changes between calls, but it\u0026rsquo;s reprocessed on every one of them.\nReference documents are another powerful candidate. An agent that analyzes technical specifications, data contracts or API documentation can cache that content and ask multiple questions without reprocessing it for each question.\nTool definitions in agentic systems with many tools also benefit. If the agent has 20 tools defined and uses only 3 on each call, caching all the definitions avoids constant reprocessing.\nWhen caching doesn\u0026rsquo;t pay off:\nContent that changes on every call can\u0026rsquo;t be cached efficiently. If the context varies significantly between requests, there\u0026rsquo;s no cache hit and you pay the write cost without the read benefit.\nContent also has to be at least 1,024 tokens to be eligible for caching on Sonnet and Haiku models. Smaller blocks are processed normally with no error, but without caching.\nHow an Architect Should Hold the Team Accountable You don\u0026rsquo;t need to implement Prompt Caching yourself. But you do need to know when to require it and how to validate that it\u0026rsquo;s working.\nThe right questions for the engineering team:\n\u0026ldquo;Is the solution\u0026rsquo;s system prompt being cached?\u0026rdquo; This is the most basic question. If the answer is no, or if the team can\u0026rsquo;t answer, there\u0026rsquo;s an immediate opportunity to cut costs.\n\u0026ldquo;What\u0026rsquo;s the ratio of cache hits to cache misses in production calls?\u0026rdquo; The API returns cache metrics in every response: cache_creation_input_tokens for writes and cache_read_input_tokens for reads. A dashboard monitoring that ratio is the bare minimum expected of any solution in production.\n\u0026ldquo;What TTL is configured: 5 minutes or 1 hour?\u0026rdquo; For solutions with a high volume of calls in short windows, the 5-minute cache is usually enough and cheaper to write. For solutions with spaced-out peaks, the 1-hour cache ensures the context stays available between peaks.\nThe sign that caching isn\u0026rsquo;t working:\nIf the cost per call doesn\u0026rsquo;t drop as volume grows, the cache probably isn\u0026rsquo;t active or is being invalidated. In a healthy solution with Prompt Caching, the marginal cost of each additional call should drop progressively as the cache is reused.\nPrompt Caching and the Batch API: The Combination That Changes the Game Prompt Caching doesn\u0026rsquo;t exist in isolation. Combined with Anthropic\u0026rsquo;s Batch API, the cost reduction potential goes up to 95%.\nThe Batch API processes requests asynchronously with a 50% discount on all tokens. For workloads that don\u0026rsquo;t need a real-time response, such as report generation, batch classification, document analysis or data enrichment, the Batch API + Prompt Caching combination delivers the lowest possible cost in the Claude ecosystem.\nThe logic is simple: the Batch API cuts standard output and input costs by 50%. Prompt Caching cuts repeated input costs by 90%. Applied together to eligible workloads, the result is a tiny fraction of the original cost.\nGambit doesn\u0026rsquo;t just choose the right card. He chooses the right moment to throw it.\nWhen to Require Prompt Caching in a Solution For an architect evaluating or reviewing a solution that uses Claude via the API, Prompt Caching should be a requirement in any scenario where:\nThe system prompt is longer than 1,024 tokens and is sent on every call. Call volume is above 10 per hour with the same base context. The solution analyzes large documents with multiple questions in the same session. The agent has many tool definitions loaded on every call. API cost is growing in proportion to volume with no marginal reduction.\nIf these criteria are present and Prompt Caching isn\u0026rsquo;t implemented, the solution is paying full price for what should already be in the cache. It\u0026rsquo;s Gambit\u0026rsquo;s cards being recharged from scratch before every play.\nConclusion: The Right Energy at the Right Moment What makes Gambit lethal isn\u0026rsquo;t how much energy he has. It\u0026rsquo;s the precision with which he applies it. Charging a card once and using it multiple times is more efficient than recharging before every play.\nPrompt Caching is the direct implementation of that philosophy in Claude solutions. Context that doesn\u0026rsquo;t change doesn\u0026rsquo;t need to be reprocessed. The energy, in this case the cost of tokens, should be applied where there\u0026rsquo;s actually something new to process.\nFor leaders and architects taking Claude to production, this isn\u0026rsquo;t an optional optimization. It\u0026rsquo;s an architecture decision with a direct impact on the solution\u0026rsquo;s economic viability. A high-volume solution without Prompt Caching is like Gambit throwing all his cards without having charged any of them.\nThe cost will always be higher than it should be.\nReferences Anthropic. Prompt Caching. https://platform.claude.com/docs/en/build-with-claude/prompt-caching Anthropic. Pricing. https://platform.claude.com/docs/en/about-claude/pricing Anthropic. Batch API. https://platform.claude.com/docs/en/build-with-claude/batch ","permalink":"https://julianamarialop.github.io/en/articles/gambits-cards-how-prompt-caching-can-cut-claude-costs-by-up-to-90/","summary":"Imagine that every time you start a meeting, the assistant reads all the company rules out loud before getting started. Two hundred pages. Every day. In every meeting. Even though everyone already knows what\u0026rsquo;s written there.","title":"Gambit's Cards: How Prompt Caching Can Cut the Cost of Your Claude Solutions by Up to 90%"},{"content":"The Attraction Only They Understood At the Xavier Mansion, two young mutants grew up side by side. Scott Summers and Jean Grey lived in the same impossible world: training for missions no outsider would understand, carrying powers that had to be kept under control at every moment, depending on each other in a way words couldn\u0026rsquo;t quite explain. What existed between them was never said out loud. There was no room for it. There was the mission, there was the team, there was the responsibility of staying focused. But in the training sessions in the Danger Room, in the strategies worked out in silence, in the trust that was built without ever needing a name, the attraction took shape.\nAnd that\u0026rsquo;s how it stayed. In absolute secrecy. Only they knew. Only they could know. What existed between the two of them lived in subtle gestures, in quick glances before a mission, in the quiet certainty that the other would be there when needed. To the rest of the team, they were just two mutants who worked well together. To them, it was much more than that. It was a complicity that only worked because each one deeply understood what the other did. Cyclops saw the whole field, planned with precision, controlled every variable. Jean felt what the data didn\u0026rsquo;t show, created what didn\u0026rsquo;t exist yet, saw beyond the obvious. On their own, each was powerful. Together, they were unbeatable.\nClaude Design and Claude Code have exactly that dynamic. Two Anthropic products that work side by side, each with its own role, each incomplete without the other. And when they work together, they deliver what neither could achieve alone.\nCyclops: The Vision That Has to Be Channeled There\u0026rsquo;s a common misconception about Cyclops. People look at him and see the controlled leader, the cold strategist, the mutant who never acts on impulse. And they conclude he has no creativity.\nWrong. Cyclops has an enormous strategic vision. He sees the whole battlefield, anticipates moves, comes up with plans others never even imagined. The problem isn\u0026rsquo;t a lack of creativity. It\u0026rsquo;s that his power is destructive if it isn\u0026rsquo;t channeled. The visor isn\u0026rsquo;t a limitation. It\u0026rsquo;s what lets Scott\u0026rsquo;s vision become action without destroying everything around him.\nClaude Design works exactly like that.\nLaunched by Anthropic on April 17, 2026 and powered by Claude Opus 4.7, Anthropic\u0026rsquo;s most capable model for visual tasks, Claude Design turns descriptions into prototypes, slides, wireframes, mockups and interactive interfaces. You describe what you need, and it builds a first version. From there, you refine it through conversation, inline comments, direct edits or custom sliders that Claude itself creates for that project.\nThe enormous strategic vision is there: Claude Design reads your organization\u0026rsquo;s codebase and design files during onboarding and automatically builds a design system with the company\u0026rsquo;s colors, typography and components. Every project after that starts from this foundation, consistent with everything that already exists. You can upload reference images, PPTX, DOCX or XLSX documents, or point it straight at the codebase.\nBut just as Cyclops needs his visor so his power doesn\u0026rsquo;t destroy what he\u0026rsquo;s building, Claude Design needs something that turns vision into functional reality. It needs Jean Grey.\nJean Grey: The Precision That Executes Jean Grey is often remembered for the power of the Phoenix, for her immense and sometimes uncontrollable strength. But what defines Jean in the day-to-day of missions isn\u0026rsquo;t brute force. It\u0026rsquo;s telepathic precision.\nShe doesn\u0026rsquo;t need a map to understand the field. She senses where each piece needs to be. She executes with surgical precision because she deeply understands the context, the intent and the goal. When Scott lays out the plan, Jean knows exactly how to make it happen without losing a single detail along the way.\nClaude Code is Jean Grey in this relationship.\nClaude Code reads the codebase, understands the existing architecture, identifies dependencies and precisely implements what was designed. It doesn\u0026rsquo;t improvise. It executes with a contextual depth that goes far beyond generating generic code. When it has a prototype as a reference, the implementation is surgical: it stays consistent with what already exists, respects the project\u0026rsquo;s conventions and delivers something that actually works, not just something that looks right.\nAnd here\u0026rsquo;s the detail that changes everything: the handoff between Claude Design and Claude Code is native. When a design is ready in Claude Design, you package everything into a handoff bundle with a single command and pass it to Claude Code to implement. There\u0026rsquo;s no tooling friction, no translation between formats, no alignment meeting between teams. Cyclops\u0026rsquo;s vision arrives intact for Jean to execute.\nThe Flow No One Else Can Replicate At the Xavier Mansion, what made Scott and Jean unbeatable wasn\u0026rsquo;t just each one\u0026rsquo;s individual power. It was the fact that they operated with a level of complicity that eliminated the friction between planning and execution.\nCyclops laid out the plan. Jean executed it. No alignment meeting. No loss of context along the way. No inevitable gap between what was conceived and what was delivered. Claude Design and Claude Code replicate that complicity in a real product flow.\nA product manager describes in natural language what they need: a pipeline monitoring interface for Microsoft Fabric, with status cards per domain, visual alerts for critical failures and navigation by data layer. Claude Design builds the interactive prototype in minutes, automatically applying the organization\u0026rsquo;s design system. The PM refines it through conversation until the vision is precise.\nOnce the prototype is approved, Claude Code receives the handoff bundle and implements it. It reads the structure of the existing codebase, understands the team\u0026rsquo;s conventions, and delivers the code that turns the prototype into a product. The engineer reviews, validates and ships to production.\nThe cycle that used to take weeks, from idea to approved prototype to implementation, compresses into days. Datadog reported compressing a week-long cycle of briefs, mockups and reviews into a single conversation in Claude Design. Brilliant reported that pages that required more than 20 prompts in competing tools needed just 2 in Claude Design before moving on to implementation in Claude Code.\nWhat Changes for Data Teams For data teams and architects building products on platforms like Microsoft Fabric and Databricks, this combination has immediate practical implications.\nThe handoff between product and engineering stops being a bottleneck. Today, an architect who wants to propose a new monitoring interface has to describe in words what they imagined, wait for a designer to interpret it, review, approve and only then hand it to engineering to implement. With Claude Design and Claude Code, the architect goes from concept to interactive prototype in one session and to code in another.\nAnyone with business context can create the vision. Claude Design was built for people who have the idea but don\u0026rsquo;t have a design background. The product owner who deeply knows the data and the processes can build the prototype themselves, without depending on a designer to translate what\u0026rsquo;s in their head.\nThe organization\u0026rsquo;s design system is respected automatically. Claude Design reads the existing codebase and design files and applies the conventions automatically. The prototype is born within the company\u0026rsquo;s visual standards, without anyone having to remember to follow the style guide.\nExport is complete. Prototypes can be exported as HTML, PDF, PPTX, sent to Canva for collaboration or packaged for direct handoff to Claude Code. The output works both for an executive presentation and for technical implementation.\nAvailability and Access Claude Design has been available in research preview since April 17, 2026 for subscribers on the Pro, Max, Team and Enterprise plans. It lives inside claude.ai, accessible through the palette icon in the left sidebar navigation.\nAn important detail: Claude Design has its own usage controls, separate from chat and Claude Code limits. Using Claude Design doesn\u0026rsquo;t consume your message quota or your Claude Code credits. They\u0026rsquo;re independent limits that coexist within the same plan.\nFor Enterprise plans, the feature is turned off by default and needs to be enabled by the administrator.\nConclusion: The Complicity That Delivers Scott and Jean didn\u0026rsquo;t need to explain themselves. The attraction built over years of missions together created an understanding that went beyond words. When Cyclops laid out the plan, Jean already knew how to execute it. When Jean signaled an opening, Scott already knew how to seize it.\nClaude Design and Claude Code have that same quality. They\u0026rsquo;re not two products that need to be connected with effort. They\u0026rsquo;re two products that were built to work together, with a native handoff that preserves the design intent all the way to the final implementation. The product vision that used to get lost in translation between teams now arrives intact in the code.\nCyclops lays out the plan. Jean executes with precision. And the result is something neither of them would deliver alone.\nReferences Anthropic. Introducing Claude Design. https://www.anthropic.com/news/claude-design-anthropic-labs Anthropic. Claude Code Overview. https://code.claude.com/docs/en/overview TechCrunch. Anthropic launches Claude Design. https://techcrunch.com/2026/04/17/anthropic-launches-claude-design-a-new-product-for-creating-quick-visuals/ ","permalink":"https://julianamarialop.github.io/en/articles/what-cyclops-and-jean-grey-taught-me-about-claude-design-and-claude-code/","summary":"At the Xavier Mansion, two young mutants grew up side by side. Scott Summers and Jean Grey lived in the same impossible world: training for missions no outsider would understand, carrying powers that had to be…","title":"What Cyclops and Jean Grey Taught Me About Claude Design and Claude Code"},{"content":"The Scientist Nobody Sees When the X-Men go into battle, the world sees Wolverine, Storm, Cyclops. Nobody sees Hank McCoy, the Beast, working on the systems underneath. But he\u0026rsquo;s the one who designed Cerebro. He\u0026rsquo;s the one who keeps the Danger Room running. He\u0026rsquo;s the one who makes sure the Xavier Mansion infrastructure can handle any mission, no matter how complex.\nHank doesn\u0026rsquo;t build for the moment. He builds to evolve.\nThat\u0026rsquo;s the distinction that separates people who use AI agents from people who actually design agentic systems. And it was exactly with that philosophy that Anthropic built Claude Managed Agents from Anthropic , launched in April 2026.\nIn simple terms: Managed Agents is infrastructure managed by Anthropic that lets you create autonomous Claude agents without building the whole underlying structure from scratch. Before it, putting an agent into production took months of engineering: containers, state management, authentication, error recovery, memory across sessions. Managed Agents eliminates that work. You define the behavior. Anthropic makes sure it works.\nAnd all of it is configurable directly from the Claude Console, without writing a single line of code.\nWhy the Order Matters The most common mistake people make when they start building agents is jumping straight to complexity. The right evolutionary journey has three stages, and each one has to be solid before the next: well-defined skills, memory across sessions, and an autonomous agent in production.\nSkills handle defined, repeatable tasks. Memory turns isolated sessions into accumulated knowledge. The autonomous agent is the result of the two working together. Trying to build the third without the first two is like Hank trying to build Cerebro before understanding how telepathy works.\nUnderstanding the Two Core Objects Before building anything, it\u0026rsquo;s essential to understand the distinction between the two main objects in Managed Agents.\nThe Environment is the lab. It defines the container where the agent runs, which packages are installed, which external systems it can reach, and, crucially, where the skill files are stored and persist across sessions. It\u0026rsquo;s the bookshelf of manuals in Beast\u0026rsquo;s lab: the manuals stay there, available to any mission that uses that lab.\n_Example of the MCP connection._\n_Example of the Environments screen_\nThe Agent is the brain. It defines the model, the system prompt, the available tools, and the skills it will load from the Environment. It\u0026rsquo;s versioned: every update creates a new version with a full history for auditing.\n_Example of an Agent_\nThe relationship between the two is simple: the Environment persists the knowledge. The Agent applies the knowledge. A Session is when the two meet to carry out a specific mission.\n_Example of Session Creation_\n_Example of Session Execution_\nPhase 1: Building the Skills The first step in the Console is to create the Environment at console.anthropic.com → Managed Agents → Environments. Next, create the Agent under Managed Agents → Agents with the model, the system prompt, and the skills it will load from the Environment.\nThe concrete scenario: a data team that manages 200 pipelines in Microsoft Fabric. Every day, failures show up, logs need to be analyzed, and incidents need to be documented. Today that eats up hours of the engineering team\u0026rsquo;s time.\nThe first skills for this agent: revisar-pipeline (review pipeline), which analyzes files and returns a status and risk report. gerar-incidente (generate incident), which formats the report with severity and next steps, always in the same pattern. diagnosticar-erro (diagnose error), which interprets logs and suggests the root cause.\nThese files are persisted in the Environment. If you tweak a skill, you edit the file and every future session already uses the updated version.\nValidate each skill individually before moving on. If Claude hesitates or returns a format different from the one expected, the skill needs refining.\nWhen a Skill Is No Longer Enough There are four clear signs that it\u0026rsquo;s time to evolve into an agent with memory:\nThe context you need is bigger than a single conversation can hold, and the agent needs information from previous sessions. The task involves multiple interdependent steps that require decisions along the way. The volume justifies autonomy, with the team triggering the same skill dozens of times a day. The error repeats because nobody remembered: the skill doesn\u0026rsquo;t learn, but the agent with memory does.\nHank didn\u0026rsquo;t build Cerebro because the attendance list was complicated. He built it because the scale of the mission demanded something no manual process could deliver.\nPhase 2: Adding Memory and Evolving to Multiple Agents In the Console, when editing the Agent, turn on the Memory toggle. Update the system prompt to instruct Claude to check its memory at the start of each session and record what it discovered at the end.\nWith that instruction, the agent stops starting from zero. By the fifth session, it already spots the timeout on the vendas_diarias pipeline before checking the log, because it recognizes the pattern. By the twentieth, it automatically includes the historical recurrence in the incident report.\nWhen complexity grows even further, it\u0026rsquo;s time to split responsibilities. In the Fabric scenario with 200 pipelines, a single agent may not be enough. The natural evolution is to create specialized agents: one for monitoring, another for diagnosis, another for report generation. An orchestrator agent receives the goal, delegates to the specialists, and consolidates the results. Each specialist has its own set of skills in the Environment and its own accumulated memory.\nThat split isn\u0026rsquo;t complexity for complexity\u0026rsquo;s sake. It\u0026rsquo;s the difference between a scientist who does everything alone and a lab with specialists who collaborate.\nPhase 3: Publishing to the Team with Governance With solid skills and working memory, the agent is ready for the team. And here\u0026rsquo;s the point that changes the adoption equation: end users don\u0026rsquo;t need a Claude license. The organization pays for the API through tokens and runtime. Users interact through whatever interface the organization chooses, with no Anthropic account and no Pro plan.\nIn the Fabric scenario, the most natural way to publish is through Microsoft Teams using MCP. Any engineer mentions the agent in the channel, the message arrives as an event, the agent processes it with its skills and memory, and it replies with the formatted report. The engineer gets the diagnosis in two minutes instead of spending an hour analyzing logs.\nFor other contexts: via Slack with the same pattern, via an internal interface built by the IT team, or via the Claude Console for technical users who need full visibility.\nAnd visibility is the right word. In the Console, the Sessions tab records every session with its full history: every tool called, every decision made, every file generated, with a timestamp and cost per session. For a data leader, that answers the question that always comes up in enterprise environments: what did the agent do, why, and when? Auditing isn\u0026rsquo;t an add-on feature. It\u0026rsquo;s part of the architecture.\nWhat It Costs Managed Agents charges along two dimensions: tokens consumed at the standard API rates, with Prompt Caching cutting the cost of reused skills and system prompts by up to 90%, and session runtime at $0.08 per hour, charged only while the agent is in running status.\nIn the Fabric scenario with 500 checks a day and 5-minute sessions: 500 × (5/60) × $0.08 = $3.33/day in runtime.\nThe Batch API doesn\u0026rsquo;t apply to Managed Agents. For workloads that don\u0026rsquo;t need state, the standard API with Batch is cheaper. Managed Agents is the right choice when state, memory, and autonomy are required.\nConclusion: Build to Evolve Hank McCoy became famous not for the most complex system, but for the most durable one. Cerebro survived decades of missions because every component was validated before being integrated with the next.\nStart with the skills in the Environment. Validate each one. Notice when context, volume, or repeated errors signal that it\u0026rsquo;s time to evolve. Add memory. Split into specialists when the mission demands it. Publish via Teams, Slack, or an internal interface.\nThe engineers on the team will interact with an agent that learns, remembers, and improves. Without knowing what runs underneath. Without needing a Claude license.\nThe mansion can handle any mission because the infrastructure was built for it. One step at a time.\nReferences Anthropic. Claude Managed Agents Overview. https://platform.claude.com/docs/en/managed-agents/overview Anthropic. Claude Managed Agents: get to production 10x faster. https://claude.com/blog/claude-managed-agents Anthropic. Cloud Environment Setup. https://platform.claude.com/docs/en/managed-agents/environments Verdent. Claude Managed Agents Pricing. https://www.verdent.ai/guides/claude-managed-agents-pricing ","permalink":"https://julianamarialop.github.io/en/articles/beast-didnt-build-cerebro-in-a-day-claude-managed-agents/","summary":"When the X-Men go into battle, the world sees Wolverine, Storm, Cyclops. Nobody sees Hank McCoy, the Beast, working on the systems underneath. But he\u0026rsquo;s the one who designed Cerebro. He\u0026rsquo;s the one who keeps the Danger Room…","title":"Beast Didn't Build Cerebro in a Day: The Evolutionary Journey with Claude Managed Agents"},{"content":"The Myth of the Technical Barrier There\u0026rsquo;s a belief in the data world that few people question: the IDE is developer territory. VS Code, with its folders, files, and terminal, would be a hostile environment for anyone who doesn\u0026rsquo;t write code every day.\nStorm never believed in barriers like that.\nWhen she goes into battle, she doesn\u0026rsquo;t need to understand atmospheric physics to summon lightning. She doesn\u0026rsquo;t calculate air pressure or electrical resistance before acting. She knows what powers she has, knows when to trigger them, and trusts the right combination to deliver the result. The deep technical knowledge exists underneath, but it isn\u0026rsquo;t what she needs to operate.\nWith two VS Code extensions, Claude Code and GitLab, any data professional can work on real projects, understand code, propose changes, follow pipelines, and collaborate with engineering teams, without writing a single line of code.\nThis article is the manual Storm would hand to any recruit before heading into the field.\nWhy VS Code and Not Claude.ai? That\u0026rsquo;s the first question that comes up. After all, claude.ai is already in the browser, already answers questions about code, and already helps analyze files. Why install a different program?\nThe answer lies in the difference between asking for advice over the phone and having the expert sitting next to you, looking at the same screen.\nIn claude.ai, you copy a snippet of code, paste it into the conversation, and Claude answers. It works, but it has serious limits: you have to manually copy whatever you want analyzed, Claude doesn\u0026rsquo;t see the whole project, doesn\u0026rsquo;t know how the files relate to each other, and can\u0026rsquo;t make real changes to the files.\nIn VS Code with Claude Code, Claude lives inside the project. It reads all the folders and files at once, understands how the components connect, makes real changes directly in the files, and saves them. You don\u0026rsquo;t copy anything. You point and ask.\nIt\u0026rsquo;s the difference between describing a pipeline problem to someone who has never seen the project and sitting down with an expert who has already read every file before the meeting starts.\nFor anyone working on data projects that involve dozens of files, interconnected pipelines, and repositories shared with engineering teams, that difference is what makes VS Code the right environment.\nBefore You Start: What You Need Before installing any extension, check that you have the following:\nVS Code installed. Go to code.visualstudio.com and download the version for your operating system. Installation is simple and free.\nA Claude account on the Pro plan or higher. Claude Code isn\u0026rsquo;t available on the free plan. You need a Pro ($20/month), Max, Team, or Enterprise plan. If you don\u0026rsquo;t have one yet, go to claude.ai/pricing to subscribe.\nNode.js installed. Claude Code needs Node.js to run. Go to nodejs.org, download the LTS version, and install it. You don\u0026rsquo;t need to know how to use it, just have it installed on your machine.\nA GitLab account. If your team uses GitLab, you already have one. If not, go to gitlab.com and create a free account.\nWith these four items ready, you\u0026rsquo;re equipped to start.\nWhat You\u0026rsquo;ll Be Able to Do With the Claude Code extension from Anthropic you can open any file in a project, ask Claude to explain what that code does in plain language, request changes by describing what you want in your own language, ask for documentation, identify problems, and understand the structure of the whole project without having to read every file manually.\nWith the GitLab extension you can see the project\u0026rsquo;s issues directly in VS Code, follow the status of CI/CD pipelines in real time, review merge requests, leave comments, and approve changes, all without opening the browser.\nTogether, these two extensions turn VS Code into a complete workspace for any professional who needs to collaborate with engineering teams or understand technical projects.\nStep 1: Install the Extensions\nOpen VS Code. In the left sidebar, click the icon that looks like four squares (or press Ctrl+Shift+X on Windows or Cmd+Shift+X on Mac). This opens the extensions marketplace.\nInstalling Claude Code:\nIn the search field, type Claude Code. Look for the extension published by Anthropic with more than 2 million installs. Click Install.\nAfter installation, a spark icon (Spark) will appear in the left sidebar. Click it. The first time, an authentication screen will appear. Click Sign in, authorize in the browser, and come back to VS Code. You\u0026rsquo;re connected.\nInstalling GitLab:\nIn the same search field, type GitLab Workflow. Look for the extension published by GitLab with the fox icon. Click Install.\nAfter installation, you\u0026rsquo;ll need to connect your GitLab account. Click the fox icon in the sidebar. The extension will ask for a GitLab Personal Access Token. To generate one:\nGo to your GitLab Go to Settings → Access Tokens Create a token with the api and read_user scopes Copy the token and paste it into the extension Done. Both extensions are installed and configured.\nStep 2: Open a Project\nTo work on an existing project, go to File → Open Folder and navigate to the project folder on your computer. If the project is on GitLab and you don\u0026rsquo;t have it locally yet, use File → New Window, then Ctrl+Shift+P to open the command palette and type Git: Clone. Paste the GitLab repository URL and choose where to save it.\nWhen VS Code opens the project folder, the GitLab extension will automatically recognize the repository and start showing information in the sidebar.\nStorm doesn\u0026rsquo;t need to build the battlefield. She walks into it as it is and directs the elements around her.\nStep 3: Use Claude to Understand the Project\nClick the spark icon in the sidebar to open the Claude Code panel. This is where the technical barrier disappears.\nUnderstanding the project structure:\nIn the Claude panel, type:\n\u0026gt; \u0026#34;Explique a estrutura deste projeto em português. Quais são as pastas principais, o que cada uma contém e como os componentes se relacionam?\u0026#34; Claude will read the entire project and give you an explanation in plain language, without unnecessary jargon.\nUnderstanding a specific file:\nOpen any file in the editor. In the Claude panel, use @ to reference the file by name and ask:\n\u0026gt; \u0026#34;@pipeline_vendas.py O que esse arquivo faz? Explica cada parte em linguagem simples.\u0026#34; Claude reads the file with the full context of the project and explains each block clearly.\nIdentifying problems:\n\u0026gt; \u0026#34;Analise os arquivos de pipeline neste projeto e me diga se há algum padrão que poderia causar problemas de performance ou falhas.\u0026#34; Claude will read the relevant files, compare them against best practices, and deliver a diagnosis in accessible language.\nStep 4: Ask for Changes Without Writing Code\nThis is where the impact becomes most obvious. You don\u0026rsquo;t need to know Python, SQL, or any other language to propose and apply changes to a project.\nExamples of requests that work:\n\u0026gt; \u0026#34;Adicione um comentário explicativo no início de cada função do arquivo @utils.py descrevendo o que ela faz.\u0026#34; \u0026gt; \u0026#34;Esse arquivo está sem tratamento de erros. Adicione mensagens de erro claras para os casos onde a conexão com o banco de dados falha.\u0026#34; \u0026gt; \u0026#34;Crie um arquivo README.md na pasta raiz explicando o que este projeto faz, como está estruturado e quais são os principais pipelines.\u0026#34; When Claude suggests a change, it shows up as a visual diff in the editor: what\u0026rsquo;s going to be removed is marked in red, what\u0026rsquo;s going to be added in green. You review it, approve it with a click, or ask for an adjustment before accepting.\nStorm doesn\u0026rsquo;t build the lightning bolt by hand. She says where she wants it to strike and watches the result before confirming.\nStep 5: Use GitLab to Follow Along and Collaborate\nWith the GitLab extension active, the sidebar shows everything you need about the project without leaving VS Code.\nFollowing issues:\nClick the fox icon in the sidebar. Expand Issues and Merge Requests. You\u0026rsquo;ll see all of the project\u0026rsquo;s open issues with title, status, and assignee. Click any issue to read the full description directly in VS Code.\nMonitoring pipelines:\nIn VS Code\u0026rsquo;s bottom bar, the GitLab extension shows the pipeline status for the current branch in real time: running, passed, or failed. Click the status to see the details of each job.\nReviewing and approving merge requests:\nIn the extension\u0026rsquo;s Merge Requests section, you\u0026rsquo;ll see all the open MRs. Click any of them to see the diff of the changes, leave comments on specific lines, and approve using the GitLab: Approve Merge Request command in the command palette (Ctrl+Shift+P).\nCombining Claude with GitLab:\nThis is where the combination gets powerful. You open a merge request in the GitLab extension, see which files were modified, select the changed content, and ask Claude:\n\u0026gt; \u0026#34;Esse é o código que foi modificado neste merge request. Explica o que mudou e quais podem ser os impactos dessas mudanças.\u0026#34; Claude analyzes the diff and delivers an explanation any professional can understand, even without knowing how to program.\nStep 6: Create Your First Skill Without Writing Code\nSkills are reusable instructions that teach Claude to carry out tasks specific to your project in a standardized way. And the best part: you don\u0026rsquo;t need to write code to create one.\nIn the Claude Code panel, type:\n\u0026gt; \u0026#34;Crie uma skill chamada revisar-pipeline que sempre que eu acionar, analisa os arquivos de pipeline do projeto e me retorna um relatório com status, possíveis erros e sugestões de melhoria em português.\u0026#34; Claude will automatically create the folder structure .claude/skills/revisar-pipeline/ with the SKILL.md file configured and ready to use.\nHow to use the skill you created:\nYou don\u0026rsquo;t need to import, install, or configure anything. To trigger the skill, just type in the Claude panel:\n\u0026gt; /revisar-pipeline Claude loads the skill\u0026rsquo;s instructions and carries out the task exactly as you defined it. Every time. With the same standard.\nIf you want to see all the skills available in the project, type /init in the panel. Claude lists everything that\u0026rsquo;s configured and available for immediate use.\nStorm doesn\u0026rsquo;t need to relearn how to summon lightning on every mission. She calibrated the power once and triggers it when she needs it.\nWhat You\u0026rsquo;ll Be Able to Do From Now On\nOnce this environment is set up, your relationship with engineering teams changes completely.\nYou can walk into any of the team\u0026rsquo;s repositories and understand what\u0026rsquo;s there without depending on anyone to explain it. You can review merge requests with a real understanding of the impact of the changes. You can spot problems in pipelines without having to call an engineer to decipher the logs. You can propose improvements to documentation, error handling, and code organization by describing what you want in natural language.\nStorm didn\u0026rsquo;t have to learn to fly to control the weather. She had to understand what powers she had and when to use them.\nThe next step is simple: open your team\u0026rsquo;s repository today, click the spark icon, and ask the first question. Claude has already read the project. Now all you have to do is talk.\nReferences Anthropic. Use Claude Code in VS Code. https://code.claude.com/docs/en/vs-code GitLab. GitLab for VS Code Extension. https://docs.gitlab.com/editor_extensions/visual_studio_code/ Anthropic. Claude Code for VS Code: Visual Studio Marketplace. https://marketplace.visualstudio.com/items?itemName=anthropic.claude-code Anthropic. Plans \u0026amp; Pricing. https://claude.com/pricing ","permalink":"https://julianamarialop.github.io/en/articles/storm-didnt-learn-to-fly-vs-code-with-claude-without-being-a-developer/","summary":"There\u0026rsquo;s a belief in the data world that few people question: the IDE is developer territory. VS Code, with its folders, files, and terminal, would be a hostile environment for anyone who doesn\u0026rsquo;t write code every day.","title":"Storm Didn't Have to Learn to Fly to Control the Weather: How to Use VS Code with Claude Without Being a Developer"},{"content":"Mystique isn\u0026rsquo;t the strongest mutant. She doesn\u0026rsquo;t have Wolverine\u0026rsquo;s adamantium, Xavier\u0026rsquo;s telepathy, or Cyclops\u0026rsquo;s optic blasts. What she has is something rarer: the ability to do the most with the least.\nTo transform into anyone, she doesn\u0026rsquo;t need equipment, long preparation, or outside resources. She reads the environment, identifies exactly what she needs to replicate, and executes with surgical precision. No wasted movement. No unnecessary detail. Every transformation uses exactly the energy needed to complete the mission.\nWriting a skill in Anthropic\u0026rsquo;s Claude Code demands exactly that mindset.\nA poorly written skill is like a Mystique who turns into ten people at once for no reason: it burns energy, confuses the environment, and delivers less than it promised. A well-written skill is the perfect transformation: Claude reads it, understands it, and executes without waste.\nThis is the manual Mystique would use if she had to teach someone how to write skills.\nWhat a Skill Is and How Claude Loads It Before writing, you need to understand what happens when Claude loads a skill.\nA skill in Claude Code is a SKILL.md file stored in your project\u0026rsquo;s .claude/skills/ folder. It instructs Claude to carry out a specific set of tasks in a standardized, reusable way. Think of it as a set of instructions that turns Claude into a temporary specialist for that task.\nWhat most developers don\u0026rsquo;t realize is how loading works. When the session starts, Claude loads only the metadata of every available skill: name and description. The full body of the skill is loaded only when Claude determines it\u0026rsquo;s relevant to the task at hand. Additional files referenced in the skill are loaded only when needed.\nThat has a direct implication: every token in your skill\u0026rsquo;s body competes with the conversation, the history, and other context files. A skill with 2,000 unnecessary tokens is 2,000 tokens taken away from what Claude needs to reason about the real problem.\nMystique doesn\u0026rsquo;t pack a suitcase before every mission. She takes exactly what she needs.\nThe Anatomy of a Well-Written Skill An efficient skill has three components: frontmatter, body, and optional reference files.\nThe frontmatter is required and defines how Claude will discover and trigger the skill:\n--- name: sql-optimizer description: Otimiza queries SQL para Databricks, incluindo análise de plano de execução, particionamento e uso de cache. Use quando o usuário pedir para otimizar, revisar ou melhorar performance de queries SQL. allowed-tools: - bash - read --- The description is the most critical element of the frontmatter. It\u0026rsquo;s what Claude uses to decide whether the skill is relevant. A vague description makes Claude ignore the skill. An overly long description wastes tokens that get loaded in every session. The target is between 20 and 40 words, specific enough to trigger in the right cases and stay quiet in the wrong ones.\nMystique knows exactly when to reveal her identity and when to stay in the shadows. A skill\u0026rsquo;s description works the same way.\nWriting Best Practices: What Goes in the Body The body of the skill is where most developers go wrong. The natural instinct is to put in everything Claude might need to know. The result is a bloated skill that degrades performance and wastes context.\nThe golden rule is simple: include only what Claude doesn\u0026rsquo;t know by default.\nClaude already knows what SQL is, what Databricks is, how an execution plan works, and what the general optimization best practices are. Explaining that in the skill is like Mystique rehearsing in the mirror before turning into someone she already knows by heart.\nBad example:\n## Otimização de SQL SQL (Structured Query Language) é uma linguagem de consulta utilizada para interagir com bancos de dados relacionais. Para otimizar queries SQL, é importante entender como o banco de dados executa as consultas. O Databricks utiliza Apache Spark por baixo, que tem características específicas de performance. Sempre analise o plano de execução antes de otimizar... That block is about 150 tokens. Claude already knows all of it. Zero added value.\nGood example:\n## Contexto do ambiente - Databricks Runtime: 14.x ou superior - Catálogo Unity: obrigatório para todas as queries - Padrão de nomenclatura de tabelas: `catalog.schema.table_name` - Particionamento padrão: por `data_referencia` (formato yyyy-MM-dd) ## Processo de otimização 1. Execute `EXPLAIN EXTENDED` antes de qualquer alteração 2. Identifique shuffles desnecessários no plano 3. Verifique uso de cache com `spark.catalog.isCached()` 4. Aplique otimizações na ordem: filtros → particionamento → joins → cache 5. Valide com `EXPLAIN COST` após cada alteração significativa ## Restrições do projeto - Nunca use `SELECT *` em tabelas com mais de 50 colunas - Joins em tabelas acima de 1TB requerem broadcast hint explícito - Máximo de 3 níveis de CTEs por query That block is about 120 tokens and delivers context Claude genuinely doesn\u0026rsquo;t have: the specifics of your environment, your conventions, and your constraints. It\u0026rsquo;s what Mystique would need to know before passing herself off as someone on your team.\nCutting Tokens: The Progressive Transformation Technique Mystique doesn\u0026rsquo;t turn into a different person for every detail of a mission. She takes on one identity and extracts what she needs incrementally, as the situation demands.\nWell-architected skills work the same way, with a technique called progressive disclosure: the main file contains only the essentials, and additional details live in separate files loaded on demand.\nRecommended structure:\n.claude/skills/ sql-optimizer/ SKILL.md ← corpo principal, máximo 500 linhas patterns.md ← padrões de otimização detalhados anti-patterns.md ← erros comuns a evitar examples.md ← exemplos de antes e depois In SKILL.md, you reference the additional files:\n## Referências - Para padrões detalhados de otimização: ver `patterns.md` - Para anti-padrões comuns: ver `anti-patterns.md` - Para exemplos práticos: ver `examples.md` Claude loads patterns.md only when it\u0026rsquo;s working on a problem that requires that level of detail. The other files sit on the filesystem consuming zero context tokens until they\u0026rsquo;re needed.\nThe impact is significant. A monolithic 3,000-token skill loaded in full consumes context even when only 20% of its content is relevant to the task. The same skill structured into files loads an average of 400 tokens in SKILL.md and reaches for the rest only when necessary.\nThe Most Common Mistakes and How to Avoid Them Mystique has failed missions before. Usually when she underestimated the environment or overestimated what she needed to bring along. Skills fail for the same reasons.\nMistake 1: A description that\u0026rsquo;s too generic\n# Ruim description: Ajuda com SQL e banco de dados. # Bom description: Otimiza queries SQL para Databricks Unity Catalog, focando em performance de joins, particionamento e análise de plano de execução. Use quando o usuário pedir otimização ou revisão de queries SQL. The bad description will trigger the skill in the wrong contexts or fail to trigger when it should. The good one is specific about the environment, the focus, and the trigger.\nMistake 2: Telling Claude what it already knows\nAny instruction you\u0026rsquo;d write for a senior developer with no context on your project is an instruction that doesn\u0026rsquo;t belong in the skill. It belongs to Claude by default.\nMistake 3: A SKILL.md over 500 lines\nAnthropic\u0026rsquo;s official documentation sets 500 lines as the limit for optimal performance. Beyond that, split it into files using progressive disclosure.\nMistake 4: Unqualified tools in an MCP context\n# Ruim Use a ferramenta bigquery_schema para recuperar schemas. # Bom Use a ferramenta BigQuery:bigquery_schema para recuperar schemas. When multiple MCP servers are available, Claude may fail to find the tool without the server prefix. Always use the fully qualified name.\nMistake 5: One skill for everything\nA skill that tries to cover SQL optimization, pipeline creation, documentation, and code review all at once is like Mystique trying to turn into five people simultaneously. None of the transformations will be convincing.\nEach skill should have a single, well-defined purpose.\nTesting and Iterating: How Mystique Would Rehearse Mystique doesn\u0026rsquo;t go into the field without testing the transformation. Skills need the same rigor.\nThe recommended process has two actors: Claude A, which writes and refines the skill, and Claude B, a clean instance that runs it with no prior knowledge of what was in your head when you wrote it.\n## Protocolo de teste 1. Escreva a skill com Claude A 2. Abra uma nova sessão com Claude B, carregue apenas a skill 3. Execute 3 a 5 tarefas representativas do caso de uso real 4. Observe onde Claude B hesita, ignora instruções ou pede esclarecimentos 5. Cada hesitação é uma lacuna na skill. Volte ao Claude A e corrija. 6. Repita até Claude B executar sem ambiguidade If Claude B asks a question the skill should have answered, the skill is incomplete. If Claude B ignores part of the instructions, the skill is poorly structured. If Claude B executes perfectly on every test case, the skill is ready for production.\nComplete Example: A Code Review Skill for Microsoft Fabric\n--- name: fabric-pipeline-review description: Revisa pipelines de dados no Microsoft Fabric para qualidade, performance e aderência aos padrões do projeto. Use quando o usuário pedir revisão, auditoria ou feedback sobre código de pipeline Microsoft Fabric. allowed-tools: - read - bash --- ## Contexto do projeto - Ambiente: Microsoft Fabric com Lakehouse habilitado - Linguagem: PySpark e SQL com type hints obrigatórios - Arquitetura: Medallion (Bronze → Silver → Gold) via Fabric Lakehouses - Namespace: `{workspace}.{lakehouse}.{camada}_{dominio}_{entidade}` ## Checklist de revisão obrigatória ### Performance - [ ] Ausência de `collect()` fora de contextos de debug - [ ] Joins com tabelas \u0026gt; 500GB usam broadcast hint ou salt - [ ] Particionamento alinhado com padrão de leitura da camada - [ ] Delta Lake V-Order habilitado nas tabelas Gold para otimização de leitura Power BI ### Qualidade - [ ] Schema explícito definido na ingestão Bronze - [ ] Regras de qualidade implementadas via Data Quality no Fabric ou Great Expectations - [ ] Tratamento de registros nulos documentado por coluna crítica ### Padrões do projeto - [ ] Nomenclatura de variáveis em snake_case - [ ] Funções com docstring no padrão Google Style - [ ] Sem credenciais hardcoded, usar Azure Key Vault via Fabric Linked Service ## Formato de saída Para cada item encontrado, retorne: - Severidade: [CRÍTICO | ALTO | MÉDIO | BAIXO] - Localização: linha ou função específica - Problema: descrição objetiva - Sugestão: correção recomendada com exemplo de código ## Referências adicionais - Padrões de nomenclatura detalhados: ver `naming-conventions.md` - Exemplos de anti-padrões comuns: ver `anti-patterns.md` This skill has about 280 tokens in the main body. It covers the essentials of the project context, structures the process, and defines the output format without explaining to Claude what PySpark is or how Databricks works.\nConclusion: The Perfect Transformation Leaves No Trace What makes Mystique terrifying to her enemies isn\u0026rsquo;t her ability to transform. It\u0026rsquo;s that when she pulls off the perfect transformation, nobody notices anything happened. The mission is accomplished without friction, without noise, without waste.\nA well-written skill has the same property. Claude loads it, executes it, and delivers the result. No tokens wasted on context it already had. No ambiguity forcing a clarifying question. No unnecessary files eating up the context window.\nThe difference between a mediocre skill and an excellent one isn\u0026rsquo;t the number of instructions. It\u0026rsquo;s the precision of each one.\nWrite the way Mystique operates: with the minimum needed for the perfect transformation.\nReferences Anthropic. Skill Authoring Best Practices. https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices\nAnthropic. Claude Code Best Practices. https://code.claude.com/docs/en/best-practices\nAnthropic. Skills Overview. https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview\n","permalink":"https://julianamarialop.github.io/en/articles/art-of-transformation-how-to-write-claude-code-skills/","summary":"Mystique isn\u0026rsquo;t the strongest mutant. She doesn\u0026rsquo;t have Wolverine\u0026rsquo;s adamantium, Xavier\u0026rsquo;s telepathy, or Cyclops\u0026rsquo;s optic blasts. What she has is something rarer: the ability to do the most with the least.","title":"The Art of Transformation: How to Write Claude Code Skills with Mystique's Precision"},{"content":"Origin Matters More Than Power Have you ever tried to explain to an executive why an AI model doesn\u0026rsquo;t remember the previous conversation? The answer isn\u0026rsquo;t in the technical documentation from Anthropic . It\u0026rsquo;s in the Weapon X Program.\nWolverine wasn\u0026rsquo;t born with adamantium claws. He was born with bone claws, a wild, shapeless power. What turned Logan into the nearly indestructible mutant we know was the Weapon X Program: a brutal process of training and modification that coated every bone in his skeleton with the toughest metal on the planet and, at the same time, tried to erase everything he was.\nThe process only half worked. The adamantium stayed. So did the identity.\nThat distinction is fundamental to understanding Claude, the language model developed by Anthropic. Because in a market full of ever more powerful AI models, the question that really separates the competitors isn\u0026rsquo;t \u0026ldquo;which one is the biggest?\u0026rdquo; or \u0026ldquo;which one answers fastest?\u0026rdquo;. The question is: how was it trained, and what remained after the training?\nAnthropic developed an approach called Constitutional AI. Instead of training the model only to predict the most likely next word, as most traditional models do, the process included an explicit set of principles, a kind of constitution, that the model learned to follow during training itself. These aren\u0026rsquo;t filters applied afterwards, like a straitjacket strapped on top. They\u0026rsquo;re values built in during formation, like the adamantium coating every one of Logan\u0026rsquo;s bones.\nThe practical result is a model that doesn\u0026rsquo;t just answer, but reasons about the answer. That asks questions when something isn\u0026rsquo;t clear. That refuses what goes against its principles not because of a technical limitation, but because of character. When Lord Shingen defeated Logan in their duel and called him \u0026ldquo;just an animal,\u0026rdquo; he was trying exactly that: reducing the character to his raw capability, erasing what made him more than a weapon. The entire arc of the 1982 miniseries is Logan proving Shingen wrong. Claude was built so it would never need to prove that.\nThe Adamantium Skeleton: The Context Window Wolverine carries the adamantium into every mission. It\u0026rsquo;s what makes his claws indestructible, what lets him absorb impacts that would kill any other mutant. But adamantium has a physical limit: it coats what exists, it doesn\u0026rsquo;t expand infinitely.\nClaude has an equivalent structure called the context window. It\u0026rsquo;s the model\u0026rsquo;s active workspace, everything it can process in a single conversation: your questions, previous answers, documents you\u0026rsquo;ve sent, system instructions. All of it takes up space inside that window.\nThe current models in the Claude 4.6 family have a context window of up to 1 million tokens, roughly 750,000 words. In practice, that means you can send long contracts, entire codebases, or extensive conversation histories, and the model processes it all in an integrated way, without losing the thread.\nBut there\u0026rsquo;s a critical detail every solutions architect needs to understand: the context window doesn\u0026rsquo;t persist between conversations. When a session ends, the adamantium is still there: the values, the capability, the character. But the content of that specific mission disappears.\nWolverine wakes up without remembering the last battle. Claude starts every new conversation with no memory of the previous one.\nNo Memory, But Not No Identity\nThis is the point that confuses people most when they start working with Claude. The lack of persistent memory looks like a serious limitation. In practice, it\u0026rsquo;s an architectural decision with deep implications.\nFirst, it guarantees privacy by design. No information from one conversation leaks into another. For companies handling sensitive customer data, that\u0026rsquo;s not a detail, it\u0026rsquo;s a requirement.\nSecond, it forces architectural clarity. If the model doesn\u0026rsquo;t remember the previous context, the system that uses it has to be responsible for managing that context. Just as Wolverine relies on the X-Men to keep the record of past missions and coordinate strategy, Claude relies on the architecture around it to operate with continuity.\nAnd this is exactly where the conversation about Claude stops being about the model and starts being about systems.\nFrom Lone Mutant to Squad: Claude in Agentic Architectures There\u0026rsquo;s a version of Wolverine every fan knows: the lone wolf. For years he operated in Madripoor as Patch, with no official identity, no squad, relying only on his claws and his instinct. Devastating in one-on-one combat, able to take on dozens of enemies alone. But any devoted comics reader knows that this isn\u0026rsquo;t Logan\u0026rsquo;s most strategic version.\nThat\u0026rsquo;s when he operates inside the X-Men.\nWith Professor Xavier coordinating the mission, Storm controlling the environment, Cyclops covering the retreat, and Wolverine on the front line, what each of them delivers individually is multiplied. Logan\u0026rsquo;s strength doesn\u0026rsquo;t shrink; it finds context, coordination, and reach that he\u0026rsquo;d never have on his own.\nClaude has the same dynamic when it\u0026rsquo;s integrated into an agentic system.\nUsed on its own through an interface or a simple API, Claude is a powerful tool for reasoning, generation, and analysis. But when it\u0026rsquo;s positioned as the reasoning core of an agent, connected to external tools via the Model Context Protocol (MCP) and operating inside a multi-agent architecture, it stops being a tool and becomes a layer of intelligence.\nMCP, created and open-sourced by Anthropic, is the protocol that lets Claude interact with external systems in a standardized way: databases, APIs, file systems, enterprise tools. Instead of building fragile custom integrations for every system, MCP defines a common language that any tool can speak. Claude reaches an MCP server and knows exactly how to query data, run actions, and receive results, regardless of the system behind it.\nIn a multi-agent architecture, Claude can take on different roles. As an orchestrator agent, it receives a complex goal, breaks it down into steps, delegates to specialized agents, and consolidates the results. As a subagent, it carries out specific tasks within a larger flow coordinated by another model. The choice of role depends on the complexity of the mission and the design of the system.\nIt\u0026rsquo;s the difference between Logan operating alone in Madripoor and Logan inside the X-Men. The mutant is the same. The outcome is completely different.\nHow to Put Logan to Work: Plans and Licenses Before assembling the squad, you need to understand how to hire the mutant.\nClaude is available in three main forms: as a chat app at claude.ai, via the API for developers, and as an enterprise platform for corporate deployment. Each path serves a different profile.\nFor individual use, the Pro plan costs $20 per month, with an annual option coming out to about $17 a month. For very high-volume users, the Max plan runs from $100 to $200 per month, including much higher usage limits and the Extended Thinking feature for reasoning on complex tasks.\nFor teams, the Team plan starts at $20 per seat per month on the Standard tier, with a minimum of 5 members. The Premium seat, at $100 per month, includes Claude Code for developers. You can mix both types within the same plan, which lets you optimize costs by user profile.\nFor enterprise, the model is different from the subscription plans: there\u0026rsquo;s a per-seat fee billed annually, and token consumption is metered separately at the standard API rate. This gives granular control per user and unlocks a 500K-token context window, HIPAA compliance, SSO, and audit logs.\nFor anyone building solutions, the API is billed by tokens consumed. The recommended models in 2026 are Haiku 4.5 ($1/$5 per million input/output tokens), Sonnet 4.6 ($3/$15), and Opus 4.6 ($5/$25). By combining the Batch API and prompt caching, you can cut the effective cost by up to 95% on eligible workloads.\nChoosing the plan follows the same logic as the mission: you don\u0026rsquo;t call in the full squad for a routine patrol.\nWhat Changes for Solution Architects For leaders and architects evaluating Claude as a component of an enterprise solution, there are a few practical implications:\nContext management is the system\u0026rsquo;s responsibility, not the model\u0026rsquo;s. Since Claude has no persistent memory, the architecture has to decide what to inject into the context window on each call: relevant history, user data, task state. That requires deliberate design, not improvisation.\nThe model\u0026rsquo;s power is in reasoning, not memorization. Claude isn\u0026rsquo;t a database. It\u0026rsquo;s a reasoning engine. The right architecture clearly separates what goes into a vector or relational database and what needs to be processed by the model.\nMCP standardizes what used to be handcrafted. Every custom integration you built to connect an AI to an internal system is a candidate to be replaced by an MCP server. More robust, more reusable, and compatible with any model that speaks the protocol.\nMulti-agent isn\u0026rsquo;t complexity for complexity\u0026rsquo;s sake. It\u0026rsquo;s the right answer when the task is too big for a single context window, when different steps require different specializations, or when parallelization significantly reduces execution time.\nLogan doesn\u0026rsquo;t call the X-Men for a bar fight. He calls them when the mission demands more than one mutant can deliver alone.\nConclusion: Origin Defines the Limit What makes Wolverine a unique character isn\u0026rsquo;t the adamantium. It\u0026rsquo;s the fact that, after everything the Weapon X Program did to him, after erasing his memories and trying to rewrite who he was, what remained was exactly what couldn\u0026rsquo;t be removed: his character.\nClaude was built with the same logic. In a market where models compete on size and speed, Anthropic bet on something different: training a model that reasons deeply, that has principles built into its formation, and that, when placed inside a well-designed architecture, multiplies the value of every system around it.\nThe question is no longer whether Claude is powerful enough for your mission. The question is whether the architecture around it is up to what it can deliver.\nReferences Anthropic. Constitutional AI: Harmlessness from AI Feedback. https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback\nAnthropic. Model Context Protocol. https://modelcontextprotocol.io\nAnthropic. Plans \u0026amp; Pricing. https://claude.com/pricing\nAnthropic. Claude API Pricing. https://platform.claude.com/docs/en/about-claude/pricing\n","permalink":"https://julianamarialop.github.io/en/articles/wolverine-and-claude-two-warriors-without-memory/","summary":"Have you ever tried to explain to an executive why an AI model doesn\u0026rsquo;t remember the previous conversation? The answer isn\u0026rsquo;t in Anthropic\u0026rsquo;s technical documentation. It\u0026rsquo;s in the Weapon X Program.","title":"Wolverine and Claude: Two Warriors Without Memory That Nobody Wants to Face"},{"content":"Arriving at the X-Mansion: Learning the Language of Engineering If you don\u0026rsquo;t write code and you\u0026rsquo;ve ever sat in project status meetings with engineering teams, you\u0026rsquo;ve probably felt like you were in a foreign country. Terms like \u0026ldquo;do a push\u0026rdquo;, \u0026ldquo;open a pull request\u0026rdquo;, \u0026ldquo;create a branch\u0026rdquo;, or \u0026ldquo;resolve a merge conflict\u0026rdquo; get tossed around as if they were everyday words. For anyone who doesn\u0026rsquo;t live in the terminal, this alphabet soup can create a frustrating communication barrier, making the project\u0026rsquo;s progress feel like an impenetrable black box.\nThe good news is that the logic behind all of this is incredibly structured and easy to understand. The way the mutants train their powers without destroying the mansion, and the way Professor Xavier coordinates the team, is exactly the same logic developers use to build and protect the apps we use every day. The tool that manages this training and coordination is called Git.\nAnd the picture has gotten even more interesting lately. With the mass adoption of GitHub Copilot, developers have gained an Artificial Intelligence-powered sidekick that completely changes the rules of the game. It\u0026rsquo;s as if every programmer had Cerebro, Professor Xavier\u0026rsquo;s supercomputer, wired straight into their mind to help them write code faster. Understanding how Git and Copilot work together is essential for any leader who wants to grasp how modern engineering actually operates.\nThe Base of Operations: Why Do We Need a Defense System? At its core, Git is a version control system. Think of it as the X-Mansion\u0026rsquo;s security system. Git\u0026rsquo;s main goal is to protect your project\u0026rsquo;s Official Base of Operations, that is, the official, tested, and approved code that\u0026rsquo;s running on the server and that customers are using right now.\nWithout Git, if two developers tried to edit the same file at the same time, one would wipe out the other\u0026rsquo;s work. Worse still, if someone made a critical mistake, the whole app could go down with no easy way to roll back the change. Git exists to make sure dozens, hundreds, or thousands of developers can collaborate on the same project with complete safety.\nBut what happens when a developer needs to add a new feature? They can\u0026rsquo;t just test it on the official code and risk breaking the app for everyone. To solve this, Git uses a specific workflow, a continuous cycle of collaboration and protection.\nTraining in the Danger Room: The Code Cycle To understand how engineering works day to day, we need to look at the full cycle of a code change. It\u0026rsquo;s a training choreography that repeats hundreds of times a week:\nBranch (The Isolated Simulation): Everything starts by creating a \u0026ldquo;branch\u0026rdquo; off the official code. This is literally entering the Danger Room: an isolated simulation where the developer can test, fail, and experiment with a new feature without affecting the official base. If the experiment goes wrong, you just shut the simulation down. Pull (Updating the Playbook): Before moving forward in training, the mutant needs to make sure they know the latest tactics the team has just figured out. A \u0026ldquo;pull\u0026rdquo; is the act of bringing into your simulation (branch) the updates other developers have already approved in the official code. It\u0026rsquo;s like syncing your field manual to avoid conflicts later on. Commit (The Save Point): With every meaningful step forward in their Danger Room training, the developer makes a \u0026ldquo;commit.\u0026rdquo; It\u0026rsquo;s like a save game for the simulation. They add a short message explaining what they did. If something breaks further down the road, they can restart the room from exactly that safe point. Push (The Mission Report): When the developer finishes training in their local simulation, they need to send the results to the team\u0026rsquo;s central server (usually GitHub). This act of sending local data to the central base is the \u0026ldquo;push.\u0026rdquo; Pull Request (Professor X\u0026rsquo;s Evaluation): Before using the new technique on official missions, the developer opens a \u0026ldquo;Pull Request\u0026rdquo; (PR). It\u0026rsquo;s a formal request for the team leads to review the training. It\u0026rsquo;s like standing before Professor Xavier to prove you\u0026rsquo;ve mastered the skill and won\u0026rsquo;t put the team at risk. Merge (The Official Graduation): When Professor Xavier approves the PR, the \u0026ldquo;merge\u0026rdquo; happens. The technique tested in the Danger Room is officially folded into the team\u0026rsquo;s tactics playbook. The new feature is now part of the official app. The Emergency Protocol: When the Mission Fails Even with all these safety processes in the Danger Room, sometimes a mistake slips through and a flawed technique makes it into the official missions. When that happens and the system starts having problems, Git shows its true rescue power.\nBecause Git keeps an immutable history of every \u0026ldquo;commit\u0026rdquo; (save point) ever made in the project, the engineering team doesn\u0026rsquo;t need to panic or rewrite the code from scratch. They simply use a command called \u0026ldquo;revert.\u0026rdquo; This command tells Git to go back in time and undo exactly the changes from that faulty code, restoring the app to the last stable version in a matter of seconds. It\u0026rsquo;s an absolute emergency protocol that lets companies innovate fast without the fear of destroying what already works.\nThe Link to Cerebro: GitHub Copilot in Action If Git is the rigorous system that manages the training and protects the mansion, GitHub Copilot is the tool that speeds up learning and mission execution . Developed by Microsoft in partnership with OpenAI, Copilot is a generative Artificial Intelligence that works right inside the screen where the developer works.\nIn practice, Copilot doesn\u0026rsquo;t replace the developer (the mutant on the battlefield), but acts as a constant telepathic link to the Cerebro supercomputer. It doesn\u0026rsquo;t just provide information; it anticipates needs. While the developer thinks about the strategy and the business problem to be solved, Copilot handles the tactical execution with impressive precision:\nSmart Autocomplete: The developer starts typing a function\u0026rsquo;s logic and Copilot suggests the rest of the code in real time, based on the context of the whole project and the tactics the team already knows. It\u0026rsquo;s like Cerebro predicting the enemy\u0026rsquo;s next move. Turning Intent into Code: The developer can write a simple comment, such as \u0026ldquo;create a function to identify inactive customers,\u0026rdquo; and Copilot writes the corresponding code instantly. Mental intent turns into action in the code. Explaining Legacy Code: If a developer joins an old project and doesn\u0026rsquo;t understand a complex part of the code, they can ask Copilot to read and explain that logic in plain language, speeding up understanding and saving hours of manual digging. Graduation: The Real Impact on the Team The combination of Git, making sure testing only happens in the Danger Room, with GitHub Copilot, beaming Cerebro\u0026rsquo;s knowledge into every developer\u0026rsquo;s mind, is rewriting efficiency metrics in software development. Recent studies indicate that developers using Copilot complete tasks up to 55% faster .\nFor anyone who doesn\u0026rsquo;t live in the terminal, this doesn\u0026rsquo;t just mean shipping features faster. It means the engineering team spends less time typing repetitive code or hunting down the bugs that wrecked the mansion, and more time focused on solving customers\u0026rsquo; real problems and innovating on the product.\nUnderstanding these concepts makes for much richer, more empathetic, and more strategic conversations with your tech teams. When you know that a \u0026ldquo;branch\u0026rdquo; protects the product in production, that a \u0026ldquo;pull request\u0026rdquo; ensures quality, and that there\u0026rsquo;s a panic button to roll back mistakes, the engineering black box opens up. At the end of the day, the goal is always the same: train safely in the mansion and work together to deliver the best result in the real world.\nReferences What is GitHub Copilot? - Official GitHub Documentation: https://docs.github.com/en/copilot/get-started/what-is-github-copilot\nGitHub Copilot Boosts Developer Productivity by 55%: https://www.linkedin.com/posts/aagarwal29_aicodegeneration-githubcopilot-developerproductivity-activity-7419786092624859138-IGe-\n","permalink":"https://julianamarialop.github.io/en/articles/code-mutants-understanding-git-and-github-copilot-with-the-x-men/","summary":"If you don\u0026rsquo;t write code and you\u0026rsquo;ve ever sat in project status meetings with engineering teams, you\u0026rsquo;ve probably felt like you were in a foreign country. Terms like \u0026ldquo;do a push\u0026rdquo;, \u0026ldquo;open a pull request\u0026rdquo;, \u0026ldquo;create a branch\u0026rdquo; or…","title":"Code Mutants: Understanding Git and GitHub Copilot with the X-Men"},{"content":"Introduction: The Legend of Tuco, Our AI Super-Intern Imagine your company hired a super-intern. His given name was \u0026ldquo;AI Agent for Utility, Corporate and Operational Tasks\u0026rdquo;, but the team, finding the name a bit\u0026hellip; robotic, decided to fondly nickname him Tuco. Legend has it the nickname stuck the day he tried to be proactive and nearly formatted the CEO\u0026rsquo;s laptop after misunderstanding a request to \u0026ldquo;clean up the files\u0026rdquo;. It was a scare, but it showed that, brilliant as he was, Tuco needed clear limits.\nHe\u0026rsquo;s an incredibly intelligent Artificial Intelligence, able to understand your requests, plan tasks and help with practically anything. You, as the boss, ask for something that seems simple: \u0026ldquo;Tuco, please process the refund for customer Joana\u0026rsquo;s last order and send her a confirmation email\u0026rdquo;.\nTuco, ever helpful, scratches his silicon head and replies: \u0026ldquo;Sure thing, boss! But\u0026hellip; how do I do that? Where do I look up orders? What\u0026rsquo;s the refund procedure? And how do I get into the email system? I don\u0026rsquo;t have a password for anything!\u0026rdquo;.\nThat\u0026rsquo;s the dilemma AIs faced for a long time. They were like a brilliant intern locked in a room, full of potential but with no access to the company\u0026rsquo;s tools and information. Every time a developer wanted the AI to do something new, they had to build a custom, fragile code \u0026ldquo;bridge\u0026rdquo; for each system. If the email system changed, the bridge broke. If the order system was updated, there went another bridge. It was a maintenance nightmare.\nThe Model Context Protocol (MCP), created by Anthropic and opened up to the community (open source), was born to solve exactly this problem. Simply put, MCP is like a universal access badge for our intern Tuco. Instead of building dozens of bridges, we give him a single badge that every department in the company understands. With that badge, Tuco can go to Sales, Finance or Communications and know exactly how to interact with each one in a standardized, secure way.\nTuco\u0026rsquo;s Utility Belt: Tools, Resources and Prompts The MCP badge isn\u0026rsquo;t magic; it works because it defines a common language. When Tuco arrives at a department (an \u0026ldquo;MCP server\u0026rdquo;), he knows he can find three kinds of things on his \u0026ldquo;utility belt\u0026rdquo;:\nTools: These are the actions Tuco can perform. Think of them as each department\u0026rsquo;s specialized equipment. Finance has a \u0026ldquo;Refund Calculator\u0026rdquo; (a processar_reembolso tool). Communications has an \u0026ldquo;Email Sender\u0026rdquo; (an enviar_email tool). Tuco doesn\u0026rsquo;t need to know the details of how the calculator works, only that he can use it to process a refund. Resources: These are the information Tuco can look up. They\u0026rsquo;re like each area\u0026rsquo;s files and documents. Sales has a \u0026ldquo;Customer Orders\u0026rdquo; file (a pedidos_cliente resource). Marketing has a \u0026ldquo;Product Catalog\u0026rdquo;. Tuco can use his badge to read these files and get the context he needs to do his tasks. Prompts (Fill-in Templates): These are like the company\u0026rsquo;s standard forms. To ask for something, Tuco has to fill out the right form. A prompt is a template that tells Tuco exactly how to format his requests so the LLM (the AI\u0026rsquo;s brain) understands and processes the information the right way. It\u0026rsquo;s a guide for how to \u0026ldquo;talk\u0026rdquo; to the brain. With this utility belt, Tuco\u0026rsquo;s job becomes much easier. For Joana\u0026rsquo;s refund, he uses his badge to:\nGo to Sales and use the \u0026ldquo;Customer Orders\u0026rdquo; Resource to find Joana\u0026rsquo;s last order. Go to Finance and use the \u0026ldquo;Refund Calculator\u0026rdquo; Tool to process the amount. Go to Communications and use the \u0026ldquo;Email Sender\u0026rdquo; Tool to send the confirmation. All of it standardized, secure and without fragile code bridges. If the email system changes, Communications updates its tool, but Tuco\u0026rsquo;s badge keeps working the same way.\nThe Badge Office: How Tools Like Copilot Studio Handle MCP Creating a universal badge and a utility belt for Tuco sounds great, but who manages all of this? That\u0026rsquo;s where AI development platforms like Microsoft Copilot Studio come in.\nThink of Copilot Studio as the company\u0026rsquo;s Badge Office. Instead of every developer having to create each badge (MCP server) and each connection by hand, these platforms do the heavy lifting. They offer a simplified visual interface where developers can:\nConnect to Data Sources with One Click: The platform already has prebuilt connectors for hundreds of systems (like SAP, Salesforce or any database). When you connect a source, the platform automatically creates an \u0026ldquo;MCP server\u0026rdquo; for it, exposing the available tools and resources. Manage Permissions Centrally: The Badge Office lets administrators define exactly what Tuco can and can\u0026rsquo;t do. He can use the consultar_pedidos tool, but not deletar_clientes, for example. This ensures security and governance across the company. Update Everything Automatically: When Sales adds a new tool to its system, the Badge Office detects the change and automatically updates Tuco\u0026rsquo;s utility belt. He\u0026rsquo;ll always have access to the latest versions of the tools, without anyone having to step in manually. Other tools, such as Langflow and n8n, are also adopting MCP, letting developers build and orchestrate these AI workflows visually and in a standardized way. They work as alternative badge offices, each with its own specialties, but all speaking the same MCP language.\nConclusion: Giving the Intern Superpowers (and a Badge) The Model Context Protocol (MCP) is more than just a technical standard; it\u0026rsquo;s a fundamental shift in how we build and manage AI applications. It turns our AI agents, like Tuco, from brilliant but isolated interns into productive, integrated members of the team.\nBy providing a \u0026ldquo;universal badge\u0026rdquo;, MCP removes the complexity of custom integrations, letting AIs access tools and data securely and at scale. And with platforms like Copilot Studio acting as the \u0026ldquo;Badge Office\u0026rdquo;, equipping our agents with the superpowers they need becomes accessible to every developer, not just AI specialists.\nThe future of AI won\u0026rsquo;t be built only with more powerful brains, but with better-organized ecosystems. And in that organization, the universal MCP badge is the key piece that will let our \u0026ldquo;Tucos\u0026rdquo; work with autonomy, security and, above all, an efficiency that used to be impossible.\nReferences Model Context Protocol - Official Documentation\nIntroducing Model Context Protocol (MCP) in Copilot Studio - Microsoft Copilot Blog\nIntroducing MCP Integration in Langflow - Langflow Blog\nMCP Client Tool node documentation - n8n Docs\n","permalink":"https://julianamarialop.github.io/en/articles/ai-universal-badge-understanding-mcp-with-agent-tuco/","summary":"Imagine your company hired a super-intern. His given name was \u0026ldquo;AI Agent for Utility, Corporate and Operational Tasks\u0026rdquo;, but the team, finding the name a bit\u0026hellip; robotic, decided to nickname him…","title":"AI's Universal Badge: Understanding MCP with Agent Tuco"},{"content":"Databricks Lakebase represents a significant evolution in data architecture, specifically in the domain of OLTP (Online Transactional Processing) databases. It isn\u0026rsquo;t just another cloud database, but a fundamental redefinition of how transactional data is stored, processed, and integrated with analytics and artificial intelligence ecosystems. This new architecture aims to close the historical gap between low-latency operational needs and the scale and flexibility demands of data lakes.\nTraditionally, OLTP databases and analytical systems operate in separate silos, requiring complex, fragile ETL pipelines to move data between them. Lakebase proposes a unified solution, combining the robustness of a relational database with the scalability and economics of cloud storage. Its design is inherently optimized for the era of AI agents, offering capabilities that legacy systems can\u0026rsquo;t provide.\nTo understand Lakebase\u0026rsquo;s impact, it\u0026rsquo;s essential to look at its technical pillars and how they solve long-standing challenges in data engineering. The innovation lies in how it handles latency, data isolation, and integration, all while supporting intensive operational workloads and the emerging demands of artificial intelligence.\nLow-Latency Engineering on Object Storage One of the biggest challenges in decoupling storage and compute in OLTP databases, using object storage (such as S3 or ADLS), is latency. Lakebase tackles this with a sophisticated architecture that uses state layers (soft-state) and intelligent caching mechanisms. While cold data lives in low-cost object storage, active Postgres pages are kept in high-performance intermediate layers. This approach lets Lakebase deliver single-digit millisecond latency and support millions of transactions per second, performance once considered out of reach for architectures based purely on data lakes. It\u0026rsquo;s the combination of cloud storage efficiency with the performance that critical transactional applications need.\nInstant Branching: The \u0026ldquo;Git Checkout\u0026rdquo; for Operational Data Lakebase\u0026rsquo;s branching mechanism, based on copy-on-write, is a transformative feature. It lets you create exact replicas of a production database, including schema and data, instantly and at marginal cost. This is possible because Lakebase isolates the metadata. When a new branch is created, it initially points to the same original data files. Changes are written only to new blocks, ensuring full isolation for:\nHigh-fidelity testing: Developers can test new features in an environment identical to production with no risk. Complex schema migrations: Schema changes can be tested and validated in an isolated environment before being applied to production. AI agent experimentation: Agents can operate on their own isolated instances for experimentation and validation, without affecting production or other agents. This capability removes the need for expensive, slow staging environments, speeding up the development cycle and protecting the integrity of production data.\nOptimized for AI Agents: Why Lakebase Is Essential The real driving force behind Lakebase\u0026rsquo;s design is robust support for autonomous AI agents. It isn\u0026rsquo;t just branching that makes it ideal, but a combination of characteristics that meet the intrinsic demands of operational artificial intelligence:\nLow-Latency Persistent State: AI agents, especially in real-time scenarios such as product recommendation or fraud detection, need access to operational data with millisecond latency. Lakebase delivers this by keeping active Postgres pages in high-performance layers, even with the underlying storage in object storage. This means agents can make fast decisions based on the latest data, without the performance bottlenecks that would limit traditional systems. Elastic Scalability (Serverless): AI agents can have unpredictable demand spikes. A traditional database would require overprovisioning to handle those spikes, resulting in high costs. Lakebase, with its serverless architecture, scales elastically down to zero when it isn\u0026rsquo;t in use and expands instantly to handle millions of transactions per second. This optimizes costs and ensures agents always have the resources they need, without manual intervention. Native Unification with Unity Catalog: Data governance is crucial for AI agents. Lakebase integrates natively with Unity Catalog, extending the same Lakehouse permissions and security policies to operational data. This simplifies access management and ensures agents operate within compliance boundaries, while also making operational data immediately visible for training and monitoring AI models in the Lakehouse, without complex ETL pipelines. These pillars, combined with branching for safe experimentation, make Lakebase the ideal state infrastructure for the security, reliability, and agility of autonomous artificial intelligence. It provides the foundation for AI agents to experiment, learn, and operate in production with confidence.\nReal Unification: The End of the Gap Between OLTP and OLAP Historically, moving data from operational databases to analytical systems (the Lakehouse) was an arduous process, involving fragile ETL pipelines, manual schema management, and significant latency. Lakebase, with its native Unity Catalog integration, removes that complexity.\nOperational data is synchronized in near real time with the analytical layer, without duplicated infrastructure or data egress costs. Governance is unified: the same permissions and security policies defined in Unity Catalog for the Lakehouse are extended to Lakebase. The result is a truly unified data platform, where operational data is immediately available for analysis and AI model training, without friction.\nReference Architecture: Integrating Lakebase with Existing Ecosystems To understand how Lakebase fits into a modern data architecture, it\u0026rsquo;s crucial to visualize its integration with legacy systems and the Lakehouse. The approach isn\u0026rsquo;t traditional ETL, but rather Zero-ETL integration or federated access, which minimizes data movement and complexity.\nImagine the following scenario:\nLegacy OLTP Systems (Data Sources): Your organization has existing transactional databases (PostgreSQL, MySQL, SQL Server, Oracle, etc.) holding critical operational data. These systems remain the primary source of truth for your legacy applications. Lakehouse (Unified Analytics Layer): The Databricks Lakehouse, built on Delta Lake and Unity Catalog, acts as your unified platform for analytical data, data warehousing, BI, and Machine Learning. It ingests data from many sources, including the legacy OLTP systems, through mechanisms such as Lakehouse Federation or data streaming (e.g., Kafka, Change Data Capture - CDC) into Delta Live Tables. Lakebase (Operational Serving Layer for AI Agents): This is where Lakebase comes in. It serves as the high-performance, low-latency operational data layer specifically optimized for AI agents. Lakebase can be fed in two main ways: New Operational Data: For new applications or microservices that require Lakebase\u0026rsquo;s unique capabilities (low latency on object storage, branching, serverless scalability), data can be written directly to Lakebase, and Data Synchronized from the Lakehouse: For data that originates in legacy systems and is ingested into the Lakehouse, Lakebase can consume that data in an optimized way. Thanks to the native integration with Unity Catalog, Lakehouse tables can be easily accessed and, if needed, materialized or referenced in Lakebase so AI agents have low-latency operational access. This creates a bidirectional or consumption data flow, where Lakebase acts as an intelligent operational cache or as a primary datastore for the agents\u0026rsquo; operations. Lakebase\u0026rsquo;s Role for AI Agents:\nIn this architecture, Lakebase doesn\u0026rsquo;t replace the Lakehouse, it complements it, providing the transactional speed and agility that AI agents require. It\u0026rsquo;s the agents\u0026rsquo; \u0026ldquo;short-term memory\u0026rdquo; and \u0026ldquo;workspace,\u0026rdquo; where they can:\nRead operational data in real time: To make instant decisions (e.g., recommendation, fraud detection). Write and update data: To carry out actions (e.g., adjust inventory, process an order). Experiment safely: Use branching to test new agent logic or models without impacting production. This architecture lets the organization keep its legacy systems running while modernizing its data layer to support the next generation of applications and AI agents, all within a unified structure governed by Unity Catalog.\nData Modeling and Retention Strategy in Lakebase Data modeling and retention strategy are crucial to optimizing Lakebase for AI agents and ensuring its efficiency in the Lakehouse ecosystem. The approach here differs from traditional analytical databases.\n1. Data Modeling: Pure Relational for Agent Operations Yes, the recommendation is a pure relational model, similar to PostgreSQL, for the tables inside Lakebase. This is because of:\nFamiliarity and Maturity: The relational model is widely understood and optimized for transactional operations (OLTP), which are the focus of AI agents that need to read and write data quickly. Referential Integrity: The ability to enforce primary and foreign keys ensures data consistency, which is essential for the reliability of agents\u0026rsquo; decisions. Optimized Point Queries: Agents frequently run point queries or specific transactions (e.g., \u0026ldquo;what\u0026rsquo;s the inventory for product X?\u0026rdquo;, \u0026ldquo;update the status of order Y?\u0026rdquo;). The relational model is highly efficient for these kinds of operations. It\u0026rsquo;s important to stress that, while Lakebase keeps the relational model for the operational layer, the Lakehouse (Delta Lake) remains the ideal place for dimensional data models (Star Schema, Snowflake Schema) optimized for complex analytics and BI.\n2. Ingestion Decision Matrix: What Goes Into Lakebase? The decision about which data should be materialized in Lakebase isn\u0026rsquo;t arbitrary; it\u0026rsquo;s guided by operational criticality for the AI agents. It isn\u0026rsquo;t an indiscriminate copy of all legacy data, but a strategic selection based on three main criteria:\nWrite and Update Frequency (Volatility):\nQuestion: Does the AI agent need to update this data in milliseconds or seconds? Does the information change so quickly that asynchronous replication or a higher-latency cache in the Lakehouse would be insufficient for operational consistency? Example: Product inventory in an e-commerce business is highly volatile, critical data. A recommendation or order management agent needs to know the exact inventory right now to avoid selling an unavailable product. Data such as the customer\u0026rsquo;s shipping address, while important, doesn\u0026rsquo;t require the same update frequency by an agent and can be accessed with higher latency from the Lakehouse. State Dependency for Immediate Decisions:\nQuestion: Does the AI agent need this data to make an immediate, critical decision in its workflow? Would the absence or staleness of this information prevent the agent from completing its task or lead to an incorrect decision? Example: For a fraud detection agent, a user\u0026rsquo;s recent transaction history is crucial for identifying anomalous patterns at the moment of the transaction. The full purchase history for trend analysis, on the other hand, can live in the Lakehouse, since it isn\u0026rsquo;t needed for the immediate decision to approve or deny a transaction. Access Volume and Granularity:\nQuestion: Does the agent access this data at high granularity (individual records) and with a large volume of point requests? Or is access more for aggregated or batch analysis? Example: A personalization agent may need to access a user\u0026rsquo;s individual preference profile (high granularity, many point requests). Data on past marketing campaigns (low granularity, batch access for analysis) would be better suited to the Lakehouse. How Unity Catalog Helps with the Decision:\nUnity Catalog acts as the central point of governance. It lets you define and manage access to data, whether it lives in Lakebase or in the Lakehouse. For data that needs to be materialized in Lakebase, you can use data synchronization mechanisms (such as CDC) that bring only the critical tables or columns from the Lakehouse into Lakebase, keeping lineage and governance unified. Alternatively, for new applications, Lakebase can be the primary destination for this critical operational data.\nIn short, the decision to \u0026ldquo;copy\u0026rdquo; (or materialize) data into Lakebase is a data engineering exercise focused on performance and operational criticality. Only the data that is truly the \u0026ldquo;active state\u0026rdquo; and that requires millisecond latency for agent operations should live in Lakebase, while the vast history and less volatile data stay in the Lakehouse.\n3. Storage: Optimized for Low Latency and Efficiency Lakebase doesn\u0026rsquo;t indiscriminately \u0026ldquo;copy\u0026rdquo; all legacy data. The storage strategy is smarter and focused on the \u0026ldquo;active state\u0026rdquo; the agents need:\nActive Operational Data: Lakebase is designed to store the subset of operational data that AI agents need to access and modify in real time. This can be a replication of specific tables from legacy systems (via CDC or streaming into the Lakehouse and then into Lakebase) or data generated directly by new applications that use Lakebase as their primary database. Selective Materialization: For data that lives primarily in the Lakehouse (originating from legacy systems), Lakebase can materialize specific views or tables that are relevant to the agents\u0026rsquo; operations. This means only the data needed for the agents\u0026rsquo; real-time decisions is kept in Lakebase\u0026rsquo;s low-latency layer, while the full history and cold data stay in the Lakehouse. Object Storage with Intelligent Caching: Although data is stored in low-cost object storage, Lakebase uses caching and soft-state layers to ensure that the data agents access most is available with millisecond latency. This avoids massive data duplication and optimizes storage costs. Retention Recommendation: Active State vs. Cold History The retention strategy in Lakebase should be driven by the AI agents\u0026rsquo; operational needs:\nLakebase: Short-Term Retention (Active State): Lakebase should retain only the active state relevant to the agents\u0026rsquo; real-time operations. That means data the agents frequently read and modify. For example, for a recommendation agent, current product inventory, the user\u0026rsquo;s recent interaction history, and the status of orders in progress. Retention periods can range from days to a few weeks, depending on the data\u0026rsquo;s volatility and how critical it is to the agent\u0026rsquo;s decisions. Lakehouse: Long-Term Retention (Cold History): The full history and cold data (no longer needed for real-time decisions, but valuable for retrospective analysis, model training, and compliance) should be offloaded to the Lakehouse (Delta Lake). Lakebase\u0026rsquo;s native integration with Unity Catalog makes this process easier, allowing data to be moved or synchronized to the Lakehouse efficiently and under governance. This clear division of responsibilities ensures that Lakebase stays agile and low-latency for the agents, while the Lakehouse offers scalability and economy for long-term storage and complex analytics. It\u0026rsquo;s an approach that balances operational performance with cost efficiency and data governance.\nUse Case: Real-Time Product Recommendations with AI Agents in E-commerce Imagine a large e-commerce platform that wants to offer hyper-personalized, real-time product recommendations that adapt instantly to user behavior and to inventory and price changes. Traditional systems struggle with the latency and complexity of managing transactional and analytical data for this purpose.\nChallenge:\nLatency: Delays in ingesting click, view, and purchase data result in outdated recommendations. Experimentation: Testing new recommendation algorithms or pricing strategies in production is risky and slow. Scalability: Handling millions of events per second while maintaining the transactional database\u0026rsquo;s performance. AI Agents: Training and operating AI agents that need fast, consistent access to operational data to make decisions in milliseconds. Solution with Databricks Lakebase:\nOperational Data in Lakebase: All user events (clicks, views, add-to-cart actions, purchases) and product catalog data (price, inventory) are stored in Lakebase. Low latency ensures this data is available almost instantly. AI Agents for Recommendation: AI agents are developed to monitor user behavior in real time. They access Lakebase data to identify patterns and generate personalized recommendations. Branching for Safe Experimentation: Before launching a new recommendation algorithm, a branch of the Lakebase database is created. AI agents can be trained and tested in this isolated environment, simulating production scenarios without impacting real customers. If the new algorithm produces hallucinations or inappropriate recommendations, the branch is simply discarded. Integration with the Lakehouse: Lakebase data is synchronized in real time with the Lakehouse via Unity Catalog. This lets Data Science teams use the full interaction history to train more complex AI models and run in-depth business analyses, while the agents operate on the latest data. Benefits:\nReal-Time Recommendations: Lakebase\u0026rsquo;s low latency ensures recommendations are always based on the most current data. Accelerated Innovation: Branching lets you test and iterate quickly on new AI algorithms with no risk to production. Improved Customer Experience: More accurate, personalized recommendations lead to higher engagement and sales. Unified Governance: Unity Catalog ensures that all data, from operational to analytical, sits under the same security and compliance umbrella. Step-by-Step Implementation Guide: AI Agents with Agent Bricks and Lakebase To demonstrate what Lakebase can do together with Agent Bricks to build robust AI agents, let\u0026rsquo;s walk through a technical workflow in detail. The focus will be on creating a recommendation agent that interacts with operational data in Lakebase and uses branching for safe experimentation.\n1. Initial Setup in Databricks and Lakebase First, make sure your Databricks environment is set up with Unity Catalog and with the Agent Bricks Preview enabled. You\u0026rsquo;ll need a Databricks workspace and permissions to create catalogs and schemas and to use Agent Bricks.\n-- Criar um catálogo no Unity Catalog para o Lakebase CREATE CATALOG IF NOT EXISTS ecommerce_catalog; USE CATALOG ecommerce_catalog; -- Criar um esquema (database) para os dados operacionais do e-commerce CREATE SCHEMA IF NOT EXISTS operational_db; USE SCHEMA operational_db; -- Tabela de Pedidos (OLTP) no Lakebase CREATE TABLE orders ( order_id STRING NOT NULL PRIMARY KEY, user_id STRING NOT NULL, product_id STRING NOT NULL, quantity INT NOT NULL, order_timestamp TIMESTAMP NOT NULL, status STRING NOT NULL ) USING LAKEBASE; -- Tabela de Produtos (OLTP) no Lakebase CREATE TABLE products ( product_id STRING NOT NULL PRIMARY KEY, product_name STRING NOT NULL, category STRING NOT NULL, price DECIMAL(10, 2) NOT NULL, stock_quantity INT NOT NULL ) USING LAKEBASE; -- Inserir dados de exemplo INSERT INTO products VALUES (\\\u0026#39;prod_001\\\u0026#39;, \\\u0026#39;Laptop Gamer\\\u0026#39;, \\\u0026#39;Eletronicos\\\u0026#39;, 1500.00, 100), (\\\u0026#39;prod_002\\\u0026#39;, \\\u0026#39;Mouse Sem Fio\\\u0026#39;, \\\u0026#39;Acessorios\\\u0026#39;, 50.00, 500), (\\\u0026#39;prod_003\\\u0026#39;, \\\u0026#39;Teclado Mecanico\\\u0026#39;, \\\u0026#39;Acessorios\\\u0026#39;, 120.00, 200); INSERT INTO orders VALUES (\\\u0026#39;order_001\\\u0026#39;, \\\u0026#39;user_a\\\u0026#39;, \\\u0026#39;prod_001\\\u0026#39;, 1, \\\u0026#39;2026-02-27 10:00:00\\\u0026#39;, \\\u0026#39;completed\\\u0026#39;), (\\\u0026#39;order_002\\\u0026#39;, \\\u0026#39;user_b\\\u0026#39;, \\\u0026#39;prod_002\\\u0026#39;, 2, \\\u0026#39;2026-02-27 10:05:00\\\u0026#39;, \\\u0026#39;pending\\\u0026#39;), (\\\u0026#39;order_003\\\u0026#39;, \\\u0026#39;user_a\\\u0026#39;, \\\u0026#39;prod_003\\\u0026#39;, 1, \\\u0026#39;2026-02-27 10:15:00\\\u0026#39;, \\\u0026#39;completed\\\u0026#39;); 2. Building a Recommendation Agent with Agent Bricks Let\u0026rsquo;s create a simple agent that, given a user_id, recommends products based on previous purchases and current inventory. Agent Bricks lets you define agents declaratively. This example simulates the logic inside a Databricks notebook, which would be the basis for an Agent Bricks agent.\n# Exemplo de código Python em um notebook Databricks para um agente de recomendação # Este código simula a lógica que seria encapsulada por um Agent Bricks from databricks.sdk import WorkspaceClient from databricks.sdk.service.sql import Statement # Inicializar o cliente do Workspace (assumindo autenticação configurada) w = WorkspaceClient() def get_user_orders(user_id: str, branch: str = \u0026#34;main\u0026#34;): # Conectar ao Lakebase usando o branch especificado # Em um ambiente real, o Agent Bricks gerencia a conexão e o contexto do branch query = f\u0026#34;USE BRANCH {branch}; SELECT product_id FROM orders WHERE user_id = \\\u0026#39;{user_id}\\\u0026#39;\u0026#34; result = w.statement_execution.execute_statement( statement=query, warehouse_id=\u0026#34;\u0026lt;seu_warehouse_id\u0026gt;\u0026#34;, # Substitua pelo ID do seu SQL Warehouse catalog=\u0026#34;ecommerce_catalog\u0026#34;, schema=\u0026#34;operational_db\u0026#34; ).result.data_array return [row[0] for row in result] def get_product_stock(product_id: str, branch: str = \u0026#34;main\u0026#34;): query = f\u0026#34;USE BRANCH {branch}; SELECT stock_quantity FROM products WHERE product_id = \\\u0026#39;{product_id}\\\u0026#39;\u0026#34; result = w.statement_execution.execute_statement( statement=query, warehouse_id=\u0026#34;\u0026lt;seu_warehouse_id\u0026gt;\u0026#34;, catalog=\u0026#34;ecommerce_catalog\u0026#34;, schema=\u0026#34;operational_db\u0026#34; ).result.data_array return result[0][0] if result else 0 def recommend_products(user_id: str, branch: str = \u0026#34;main\u0026#34;): purchased_products = get_user_orders(user_id, branch) recommendations = [] # Lógica de recomendação simplificada: recomendar produtos com estoque \u0026gt; 0 que não foram comprados all_products_query = f\u0026#34;USE BRANCH {branch}; SELECT product_id, product_name FROM products WHERE stock_quantity \u0026gt; 0\u0026#34; all_products_result = w.statement_execution.execute_statement( statement=all_products_query, warehouse_id=\u0026#34;\u0026lt;seu_warehouse_id\u0026gt;\u0026#34;, catalog=\u0026#34;ecommerce_catalog\u0026#34;, schema=\u0026#34;operational_db\u0026#34; ).result.data_array for prod_id, prod_name in all_products_result: if prod_id not in purchased_products: recommendations.append(prod_name) return recommendations # Exemplo de uso do agente (simulado) user_to_recommend = \u0026#34;user_a\u0026#34; print(f\u0026#34;Recomendações para {user_to_recommend} (branch main): {recommend_products(user_to_recommend)}\u0026#34;) 3. Safe Experimentation with Branching and Agent Bricks Now let\u0026rsquo;s use Lakebase branching to test a new recommendation strategy that involves an aggressive promotion on a specific product, which may affect inventory. Agent Bricks can be configured to operate on specific branches for testing.\n-- Criar um novo branch para testar a promoção CREATE BRANCH promo_test ON TABLE orders; CREATE BRANCH promo_test ON TABLE products; An AI agent configured for the promo_test branch can now simulate the changes. In Agent Bricks, you would specify the branch in the agent\u0026rsquo;s execution context.\n# Simulação de um agente de IA operando no branch \\\u0026#39;promo_test\\\u0026#39; # O Agent Bricks abstrairia a gestão do branch para o desenvolvedor do agente # Agente de IA decide aplicar um desconto e vender um Laptop Gamer # Esta operação ocorre APENAS no branch \\\u0026#39;promo_test\\\u0026#39; update_query = \u0026#34;UPDATE products SET stock_quantity = stock_quantity - 1 WHERE product_id = \\\u0026#39;prod_001\\\u0026#39;\u0026#34; w.statement_execution.execute_statement( statement=update_query, warehouse_id=\u0026#34;\u0026lt;seu_warehouse_id\u0026gt;\u0026#34;, catalog=\u0026#34;ecommerce_catalog\u0026#34;, schema=\u0026#34;operational_db\u0026#34;, branch=\u0026#34;promo_test\u0026#34; # Especifica o branch para a operação ) # O agente verifica o estoque no branch de teste print(f\u0026#34;Estoque de Laptop Gamer no branch promo_test: {get_product_stock(\\\u0026#39;prod_001\\\u0026#39;, branch=\\\u0026#39;promo_test\\\u0026#39;)}\u0026#34;) # O estoque no branch principal permanece inalterado print(f\u0026#34;Estoque de Laptop Gamer no branch main: {get_product_stock(\\\u0026#39;prod_001\\\u0026#39;, branch=\\\u0026#39;main\\\u0026#39;)}\u0026#34;) If the result of the simulation on promo_test is positive (for example, an increase in simulated sales without depleting inventory in a harmful way), the changes can be merged into the main branch. Otherwise, the promo_test branch can be discarded without affecting production.\n-- Se os testes forem bem-sucedidos, mesclar as alterações (exemplo conceitual, a mesclagem pode ser mais complexa) -- MERGE BRANCH promo_test INTO main ON TABLE orders; -- MERGE BRANCH promo_test INTO main ON TABLE products; -- Descartar o branch de teste após a conclusão (seja mesclado ou não) DROP BRANCH promo_test ON TABLE orders; DROP BRANCH promo_test ON TABLE products; 4. Continuous Integration with the Lakehouse (Unity Catalog) Regardless of the operational branches, Lakebase data is continuously synchronized with the Lakehouse via Unity Catalog. This lets Data Science and Analytics teams keep training more complex AI models and running in-depth business analyses on the full interaction history, while the agents operate on the latest data and in their isolated branches.\n-- Exemplo de como uma equipe de Data Science pode consultar dados operacionais -- diretamente do Lakehouse via Unity Catalog para treinamento de modelos SELECT o.user_id, p.product_name, o.quantity, o.order_timestamp, p.category FROM ecommerce_catalog.operational_db.orders AS o JOIN ecommerce_catalog.operational_db.products AS p ON o.product_id = p.product_id WHERE o.order_timestamp \u0026gt;= current_date() - INTERVAL \\\u0026#39;30 days\\\u0026#39;; Conclusion Databricks Lakebase, together with Agent Bricks, offers a powerful platform for building and operating intelligent applications and AI agents. Its ability to deliver low latency on object storage, the branching mechanism for safe experimentation, and seamless integration with Unity Catalog for unified governance position Lakebase as the ideal technical foundation for the next generation of transactional and analytical systems. For organizations looking to innovate quickly with AI, Lakebase and Agent Bricks provide the tools needed to turn operational data into actionable intelligence securely and at scale.\nReferences Lakebase Postgres | Databricks on AWS What is Lakebase Provisioned? | Databricks on AWS Tutorial: Branch-based development workflow | Databricks on AWS Agent Bricks | Databricks on AWS Databricks documentation | Databricks on AWS ","permalink":"https://julianamarialop.github.io/en/articles/databricks-lakebase-definitive-guide-operational-data-and-ai-agents/","summary":"Databricks Lakebase represents a significant evolution in data architecture, specifically in the domain of OLTP (Online Transactional Processing) databases. It isn\u0026rsquo;t just another database…","title":"Databricks Lakebase: The Definitive Guide to the New Era of Operational Data and AI Agents"},{"content":"In the vast Star Wars universe, the Force connects everything, but only the most dedicated Jedi manage to master it to build and wield lightsabers and intelligent holocrons. In our technology universe, the transition from Data Engineer to AI Engineer on AWS is our very own Force Awakens: a focused 3-month war operation to turn your command of data into the ability to build and orchestrate autonomous intelligence.\nThis article lays out an intensive roadmap for that transition, with Amazon Bedrock, Amazon Q and Amazon SageMaker as its pillars, and discusses how to prove your new skills and reach seniority in AI projects.\nWhy is this transition strategic and urgent right now? Artificial Intelligence, powered by Generative AI and autonomous agents, is redefining the way companies operate. The ability to build proactive systems that make decisions and take actions on their own is the new competitive edge. As a Data Engineer, your expertise in data and infrastructure is the most critical foundation for this revolution. You already know the terrain; now it\u0026rsquo;s time to build intelligence on top of it.\nBy moving into AI Engineering, you shift your focus from \u0026ldquo;keeping the data flowing\u0026rdquo; to \u0026ldquo;building the intelligence that consumes and acts on that data\u0026rdquo;. In the AWS ecosystem, that means mastering Amazon Bedrock for access to foundation models, Amazon Q for conversational intelligence and Amazon SageMaker for the full Machine Learning lifecycle. This evolution doesn\u0026rsquo;t just multiply your market value; it positions you as an architect of the intelligence that drives innovation at an accelerated pace.\nIntensive Roadmap: From Data Engineer to AI Engineer in 3 Months on AWS This roadmap is designed for a Data Engineer with solid AWS experience who is looking for a fast, focused transition. It\u0026rsquo;s a \u0026ldquo;war operation\u0026rdquo; that demands dedication and immersion.\nMonth 1: The Force Awakens and the Fundamentals (Amazon Bedrock \u0026amp; Amazon Q) Focus: Understand the \u0026ldquo;Force\u0026rdquo; (Generative AI) and the masters (Foundation Models) on AWS.\nGenerative AI fundamentals on AWS: Dive into the concepts of Generative AI, Large Language Models (LLMs) and how they work. Understand the role of Amazon Bedrock as a fully managed service for accessing foundation models (FMs) securely and at scale. Explore Amazon Q Business for conversational intelligence in the corporate environment. Use the official documentation to deepen your knowledge: Amazon Bedrock Documentation and Amazon Q Business Documentation Prompt Engineering and RAG: Learn the art of interacting with LLMs through Prompt Engineering, and improve answer accuracy with RAG (Retrieval-Augmented Generation), using knowledge bases to give the models context. Relevant Certification: AWS Certified AI Practitioner (AIF-C01). This certification validates your knowledge of AI/ML concepts, Generative AI and use cases, which makes it an excellent starting point. Month 2: Building the Lightsaber and the Agents (Amazon SageMaker \u0026amp; Bedrock Agents) Focus: Build the \u0026ldquo;Lightsaber\u0026rdquo; (Agents and ML Pipelines) that turns data into intelligent action.\nData Engineering for ML in SageMaker: Use Amazon SageMaker to prepare data for ML models. Master tools such as SageMaker Data Wrangler for ingestion and transformation, and SageMaker Pipelines for orchestrating ML workflows. Your Data Engineer background will be crucial here. See the documentation at Amazon SageMaker Documentation Agents in Amazon Bedrock: Learn to build agents that don\u0026rsquo;t just answer, but act. Explore Bedrock Agents to build applications that interact with external tools and APIs, and Knowledge Bases for Bedrock to implement RAG efficiently. Orchestration Frameworks: Explore frameworks such as LangChain or LlamaIndex to orchestrate multiple agents and tools, creating more complex AI systems capable of solving multifaceted problems. Month 3: Commanding the Fleet and Getting Certified (MLOps, Governance \u0026amp; the MLA-C01 Certification) Focus: Put intelligence into production, ensure governance and formally validate your new skills, becoming a commander of the AI fleet.\nMLOps on AWS: Implement MLOps practices using Amazon SageMaker for model monitoring (Model Monitor), AWS Glue for data quality and Amazon SageMaker Clarify for Responsible AI (explainability and bias mitigation). AI Governance on AWS: Get familiar with AWS\u0026rsquo;s built-in governance capabilities to make sure your agents operate securely, ethically and in compliance. This includes using AWS Organizations, AWS IAM and AWS CloudTrail for auditing. AWS Certified Machine Learning Engineer - Associate (MLA-C01) Certification: This certification is your big milestone. It validates your ability to build, operationalize, deploy and maintain ML workloads on AWS, with a focus on SageMaker and other relevant services. Here is the MLA-C01 Exam Study Guide Time Estimate For a Data Engineer with solid AWS experience, this 3-month \u0026ldquo;war operation\u0026rdquo; is ambitious but entirely doable for acquiring the essential skills of a junior/mid-level AI Engineer. The key is intensive dedication and immediate hands-on application of what you learn. To reach seniority, the timeline may stretch to 1 to 2 additional years of experience on complex projects, leadership and AI-Centric solution architecture.\nHow Do You Become Senior in AI Projects? Beyond the Code, the Architecture of Intelligence Being senior in AI goes beyond technical mastery of tools and models. It involves the ability to architect end-to-end solutions, ensure governance and deliver business value reliably and ethically.\nSeniority shows when you:\nDesign Scalable AI Architectures: You go beyond the data pipeline to design the complete architecture of AI systems, including data ingestion for AI, inference pipelines, integration with legacy systems and continuous monitoring. Implement Governance and Responsible AI: You make sure AI solutions are ethical, transparent and secure. That includes applying Responsible AI principles, mitigating bias, ensuring model explainability and meeting regulatory requirements. Optimize Performance and Costs: You select the most suitable tools and models, optimize the use of compute resources and make sure AI solutions are financially sustainable and efficient. Lead and Mentor: You mentor more junior team members, share knowledge and influence the organization\u0026rsquo;s AI strategy, translating business requirements into technical solutions and vice versa. Focus on Reliability: You design systems that are resilient to failure, that learn and adapt continuously, and that minimize the risk of \u0026ldquo;hallucinations\u0026rdquo; or unwanted behavior. The Senior AI Engineer is the architect of intelligence, the professional who turns data into autonomous decisions and sustainable business value.\nHow Do You Prove Your Knowledge and Build an AI Engineer Portfolio? Theory matters, but practice is what really sets a professional apart. To cement your transition and showcase your new skills, building a solid portfolio is essential. Here are a few ways to do it:\nPersonal Projects with AI Agents (Bedrock \u0026amp; SageMaker): Build your own intelligent agents using Amazon Bedrock for foundation models and Amazon SageMaker for orchestration and MLOps. Think about everyday problems that an agent could automate or optimize. For example, an agent that monitors market news and generates executive summaries, or one that automates email triage. Open Source Contributions: Take part in open source AI projects, especially those that use AWS technologies. This shows not only your technical skills but also your ability to collaborate on teams and follow good development practices. GitHub as Your Business Card: Keep an active GitHub profile, documenting your projects, showing the source code, explaining the architecture and the results. A good README.md is as important as the code. Include demos, short videos or screenshots that show your agents in action. Articles and Tutorials: Share what you learn. Write articles on LinkedIn or on technical blogs about the projects you built, the challenges you faced and the solutions you found. This not only reinforces your knowledge, it also positions you as a reference in the field. Hackathons and Challenges: Get involved in events that encourage building AI solutions. It\u0026rsquo;s an excellent way to learn, apply knowledge under pressure and build portfolio projects in a short amount of time. Remember: your portfolio isn\u0026rsquo;t just a collection of code, it\u0026rsquo;s the story of your journey as an AI Engineer. It should tell how you turn data into actionable intelligence.\nConclusion: The Force Awakens for the Future of Technology The transition from Data Engineer to AI Engineer isn\u0026rsquo;t just a change of title; it\u0026rsquo;s a calculated Force Awakens moment for the future of technology. As a Data Engineer, you already have the most critical foundation: command of the data.\nBy expanding your skills into AI systems engineering, with a focus on Amazon Bedrock, Amazon Q and Amazon SageMaker, you put yourself at the forefront of innovation, building the intelligent, secure and scalable systems the market demands. The future of technology isn\u0026rsquo;t just about having the best models, it\u0026rsquo;s about having the best engineering to put them into production and make sure they deliver real value in record time.\nAre you ready to take the next step and become tomorrow\u0026rsquo;s architect of intelligence in just 3 months?\nReferences AWS. AWS Certified AI Practitioner. AWS. AWS Certified AI Practitioner (AIF-C01) - AWS Certification. AWS. AWS Certified Machine Learning Engineer - Associate. AWS. AWS Certified Machine Learning Engineer - Associate (MLA-C01) Exam Guide. https://skillbuilder.aws/category/exam-prep/ai-practitioner)(Estelink%C3%A9paraoExamPrepPlandoAIPractitioner,masoconte%C3%BAdodeBedrockGettingStartedpodeserencontradonoSkillBuilder \u0026ldquo;AWS Skill Builder. Amazon Bedrock Getting Started.\u0026rdquo; https://aws.amazon.com/blogs/training-and-certification/category/artificial-intelligence/generative-ai/)(Estelink%C3%A9paraoblogdeGenAI,oLearningPlanpodeserencontradonoSkillBuilder \u0026ldquo;AWS Skill Builder. Generative AI Learning Plan for Developers.\u0026rdquo; AWS. Amazon Q Business. AWS. Amazon Bedrock. AWS. Amazon SageMaker. ","permalink":"https://julianamarialop.github.io/en/articles/the-force-awakens-from-data-engineer-to-ai-data-engineer-in-90-days-with-aws/","summary":"In the vast Star Wars universe, the Force connects everything, but only the most dedicated Jedi manage to master it to build and wield lightsabers and intelligent holocrons. In our technology universe, the transition…","title":"The Force Awakens: From Data Engineer to AI Data Engineer in 90 Days with AWS"},{"content":"In the Assassin\u0026rsquo;s Creed universe, the Leap of Faith is an act of courage, technique and trust. It isn\u0026rsquo;t a blind jump, but a calculated move that demands knowledge of the terrain and precision to reach a new level. In the technology world, the transition from Data Engineer to AI Engineer is our own Leap of Faith: a focused 3-month war operation to master building autonomous intelligence in the Microsoft ecosystem.\nThis article lays out an intensive roadmap for that transition, with Microsoft Fabric and Azure AI Foundry as the pillars, and discusses how to prove your new skills and reach seniority in AI projects.\nWhy is this transition strategic and urgent right now? Artificial Intelligence, driven by Generative AI and autonomous agents, is redefining how companies operate. The ability to build proactive systems that make decisions and take actions autonomously is the new competitive edge. As a Data Engineer, your expertise in data and infrastructure is the most critical foundation for this revolution. You already know the terrain; now it\u0026rsquo;s time to build the intelligence on top of it.\nBy moving into AI Engineering, you shift your focus from \u0026ldquo;keeping the data flowing\u0026rdquo; to \u0026ldquo;building the intelligence that consumes and acts on that data\u0026rdquo;. In the Microsoft ecosystem, that means mastering Microsoft Fabric for data unification and Azure AI Foundry for orchestrating and productionizing agents. This evolution not only multiplies your market value but positions you as an architect of the intelligence that drives innovation at an accelerated pace.\nIntensive Roadmap: From Data Engineer to AI Engineer in 3 Months on Microsoft Azure This roadmap is designed for a Data Engineer with solid Azure experience who wants a fast, focused transition. It\u0026rsquo;s a \u0026ldquo;war operation\u0026rdquo; that demands dedication and immersion.\nMonth 1: Data and Generative AI Foundations (Microsoft Fabric \u0026amp; Azure OpenAI) Focus: Master Microsoft Fabric as your new unified data and AI platform, and learn to consume existing AI models.\nMicrosoft Fabric Fundamentals: Dive into Fabric. Understand OneLake as your unified data lake and the different \u0026ldquo;Experiences\u0026rdquo; (Data Engineering, Data Science, Data Warehouse, Real-Time Analytics). Your Data Engineer background is an advantage here. Where to start: Learning Path: Get started with Microsoft Fabric Relevant Certifications: Microsoft Certified: Fabric Analytics Engineer Associate (DP-600) or Microsoft Certified: Fabric Data Engineer Associate (DP-700). Pick the one that best matches your current background and aspirations. Azure OpenAI Service and Prompt Engineering: Learn to consume advanced language models. The focus here is on Prompt Engineering and RAG (Retrieval-Augmented Generation) to build intelligent applications that use your company\u0026rsquo;s knowledge. Where to start: Learning Path: Develop generative AI apps in Azure Month 2: Building Agents, Semantic Search and Orchestration (Azure AI Foundry \u0026amp; Fabric Data Agents) Focus: Build AI systems that not only process information but also make decisions and take actions, with an emphasis on Fabric Data Agents and Azure AI Foundry.\nAzure AI Foundry: Understand how this unified platform speeds up the AI development lifecycle, from experimentation to producing autonomous agents. Foundry is your intelligence production studio. Where to start: Azure AI Foundry overview Microsoft Fabric Data Agents: Learn to create Data Agents in Fabric. They turn enterprise data into conversational Q\u0026amp;A systems and virtual analysts powered by generative AI, which can be orchestrated by Foundry. Where to start: Data Agent concept in Fabric and How to create a Data Agent in Fabric Azure AI Search: Master vector and hybrid search to build information retrieval systems that feed your Data Agents and other agents with relevant, real-time knowledge. Agent Orchestration Frameworks: Explore Semantic Kernel or LangChain to build agents that interact with external tools and APIs. This turns a language model into a system capable of carrying out complex tasks. Month 3: Production, Governance and Fast-Track Certification Focus: Put agents into production, ensure governance and formally validate your new skills.\nAzure AI Foundry for Production: Use Foundry to manage the full lifecycle of your agents, from experimentation to deployment and monitoring at scale. Understand the Foundry Control Plane for governance and security. Where to start: Consume a Data Agent in Azure AI Foundry AI Governance in Fabric and Foundry: Get familiar with the built-in governance capabilities (such as Microsoft Purview in Fabric) to make sure your agents operate securely, ethically and in compliance. Azure AI Engineer Associate Certification (AI-102): This certification is crucial. It validates your ability to design and implement AI solutions using Azure AI Services, Azure AI Search and Azure OpenAI. Where to start: Microsoft Certified: Azure AI Engineer Associate and Study guide for Exam AI-102 Time Estimate For a Data Engineer with solid Azure experience, this 3-month \u0026ldquo;war operation\u0026rdquo; is ambitious but entirely doable for acquiring the core skills of a junior/mid-level AI Engineer, especially with the focus on Fabric and AI Foundry. The key is intensive dedication and immediate hands-on application of what you learn. To reach specialization, the timeline may stretch to 1 to 2 additional years of experience in complex projects, leadership and AI-Centric solution architecture.\nHow Do You Become Senior in AI Projects? Beyond Code, the Architecture of Intelligence Being senior in AI goes beyond technical mastery of tools and models. It involves the ability to architect end-to-end solutions, ensure governance and deliver business value reliably and ethically.\nSeniority shows when you:\nDesign Scalable AI Architectures: You go beyond the data pipeline to design the complete architecture of AI systems, including data ingestion for AI, inference pipelines, integration with legacy systems and continuous monitoring. Implement Governance and Responsible AI: You make sure AI solutions are ethical, transparent and secure. This includes applying Responsible AI principles, bias mitigation, model explainability and regulatory compliance. Optimize Performance and Costs: You select the most suitable tools and models, optimize the use of compute resources and make sure AI solutions are financially sustainable and efficient. Lead and Mentor: You act as a mentor to more junior teams, share knowledge and influence the organization\u0026rsquo;s AI strategy, translating business requirements into technical solutions and vice versa. Focus on Reliability: You design systems that are resilient to failure, that continuously learn and adapt, and that minimize the risk of \u0026ldquo;hallucinations\u0026rdquo; or unwanted behavior. The Senior AI Engineer is the architect of intelligence, the professional who turns data into autonomous decisions and sustainable business value.\nHow Do You Prove Your Knowledge and Build an AI Engineer Portfolio? Theory matters, but practice is what really sets a professional apart. To cement your transition and showcase your new skills, building a solid portfolio is essential. Here are some ways to do it:\nPersonal Projects with AI Agents (Fabric \u0026amp; Foundry): Build your own intelligent agents using Microsoft Fabric for Data Agents and Azure AI Foundry for orchestration. Think about everyday problems that an agent could automate or optimize. For example, an agent that monitors market news and generates executive summaries, or one that automates email triage. Open Source Contributions: Take part in AI-related open source projects, especially those that use Microsoft technologies. This shows not only your technical skills but also your ability to collaborate on teams and follow good development practices. GitHub as Your Business Card: Keep an active GitHub profile, documenting your projects, showing the source code, explaining the architecture and the results. A good README.md is as important as the code. Include demos, short videos or screenshots that show your agents in action. Articles and Tutorials: Share what you learn. Write articles on LinkedIn or on technical blogs about the projects you built, the challenges you faced and the solutions you found. This not only reinforces your knowledge but also positions you as a reference in the field. Hackathons and Challenges: Get involved in events that encourage building AI solutions. It\u0026rsquo;s an excellent way to learn, apply knowledge under pressure and build portfolio projects in a short amount of time. Remember: your portfolio isn\u0026rsquo;t just a collection of code, but the narrative of your journey as an AI Engineer. It should tell the story of how you turn data into actionable intelligence.\nConclusion: The Leap of Faith into the Future of Technology The transition from Data Engineer to AI Engineer isn\u0026rsquo;t just a title change, but a calculated Leap of Faith into the future of technology. As a Data Engineer, you already have the most critical foundation: mastery of data.\nBy expanding your skills into AI systems engineering, with a focus on Microsoft Fabric and Azure AI Foundry, you place yourself at the forefront of innovation, building the intelligent, secure and scalable systems the market demands. The future of technology isn\u0026rsquo;t just about having the best models, but about having the best engineering to put them into production and make sure they deliver real value in record time.\nAre you ready to take the next step and become tomorrow\u0026rsquo;s architect of intelligence in just 3 months?\nReferences Microsoft Learn. Microsoft Applied Skills: Develop generative AI solutions with Azure OpenAI Service. Microsoft Learn. Microsoft Certified: Azure AI Engineer Associate. Microsoft Learn. Learning Path: Develop generative AI apps in Azure. Microsoft Learn. Microsoft Applied Skills: Develop AI agents using Azure OpenAI Service and Semantic Kernel. Microsoft Learn. Microsoft Applied Skills: Implement knowledge mining with Azure AI Search. Microsoft Learn. Study guide for Exam AI-102. Microsoft Learn. Learning Path: Get started with Microsoft Fabric. Microsoft Learn. Data Agent concept in Fabric. Microsoft Learn. How to create a Data Agent in Fabric. Microsoft Learn. Microsoft Certified: Fabric Analytics Engineer Associate (DP-600). Microsoft Learn. Microsoft Certified: Fabric Data Engineer Associate (DP-700). Azure. Azure AI Foundry. Microsoft Learn. Microsoft Applied Skills: Create an AI agent. ","permalink":"https://julianamarialop.github.io/en/articles/leap-of-faith-data-engineer-to-ai-engineer-in-90-days-microsoft/","summary":"In the Assassin\u0026rsquo;s Creed universe, the Leap of Faith is an act of courage, technique and trust. It isn\u0026rsquo;t a blind jump, but a calculated move that demands knowledge of the terrain and precision to reach a new level. In the…","title":"The Leap of Faith: From Data Engineer to AI Engineer in 90 Days in the Microsoft Ecosystem"},{"content":"Picture Microsoft Foundry as a big film studio dedicated to producing AI movies. Instead of projectors, cameras, and scripts, it offers tools, models, and infrastructure to turn ideas into intelligent solutions.\nJust as in a film production, every stage of creating an AI requires coordination, diverse talent, and cutting-edge technology. That\u0026rsquo;s where Foundry shines, providing a complete structure so any team can create its own intelligence blockbusters.\nFor a long time, AI development was a handcrafted affair. Data was scarce, training was expensive, and operations depended on complex infrastructure. Now, with Foundry\u0026rsquo;s improvements, that picture changes radically. The process becomes faster, more accessible, and governed. It\u0026rsquo;s as if AI finally had its own professional studio to shoot, edit, and release big productions.\nLike a studio that invests in new equipment and crews, Microsoft Foundry has received updates that redefine how AI gets produced.\nSynthetic Data Generation: The Studio\u0026rsquo;s Writers\u0026rsquo; Room Every great movie starts with a good script, and an AI model needs quality data to learn. Foundry now lets you create synthetic datasets from existing documents and code.\nIt\u0026rsquo;s like having screenwriters who, starting from an initial idea, write new scenes and dialogue, expanding the plot and enriching the story. In the same way, synthetic data helps the model learn safely and quickly, even when real data is scarce or sensitive.\nThe result is a more efficient pre-production process that lets you test concepts and train models reliably.\nTraining Cost Optimization: The Studio\u0026rsquo;s Acting School Training models is the moment when the actors learn the script and rehearse their lines. But that process can be expensive and time-consuming.\nMicrosoft is making this stage more accessible, offering a 50% discount on training costs and expanding quotas for developers. It\u0026rsquo;s as if the studio\u0026rsquo;s acting school started handing out scholarships to new talent.\nThat democratization encourages more professionals to experiment, tune, and refine their models, boosting the diversity and quality of the solutions created on the AI \u0026ldquo;film set.\u0026rdquo;\nModel Flexibility: The Studio\u0026rsquo;s International Cast No movie would be the same without the right cast. Foundry brings that same flexibility by integrating open-source models, such as:\nMistral Llama 3.3 OSS-20B Qwen 3 On top of the Claude and GPT models available exclusively on Azure.\nIt\u0026rsquo;s like having a worldwide cast at your disposal, where the director can choose between big stars and independent talent, depending on the role and the budget.\nThat variety lets each AI production be put together with the ideal cast for its purpose, balancing cost, performance, and compliance.\nAgentic RFT and Agent Orchestration: The Studio\u0026rsquo;s Direction and Technical Crew The next revolution in AI lies in creating autonomous agents, which don\u0026rsquo;t just respond but also act, plan, and carry out complex tasks.\nWith Agentic RFT (Reinforcement Fine-Tuning), the GPT-5 model learns to use tools, follow structured reasoning, and interact with external systems. This evolution turns a model that used to just talk into an agent capable of acting.\nIn addition, the Microsoft Agent Framework works as the studio\u0026rsquo;s backstage system, bringing together frameworks like Semantic Kernel and AutoGen.\nHosted Agents let you launch agents in a managed environment, without having to build the whole technical infrastructure from scratch.\nFor more complex productions, Foundry introduces multi-agent workflows and persistent memory, allowing multiple specialists to collaborate and keep the project\u0026rsquo;s context over time, like a film crew that knows every scene and preserves the movie\u0026rsquo;s coherence.\nConclusion: The Studio for the Intelligence of the Future Microsoft Foundry is becoming the big studio producing the new generation of AI. It brings together every phase of creation, from writing the script to the premiere in the theaters of innovation.\nWith synthetic data, affordable costs, freedom to choose models, and intelligent agent orchestration, Foundry turns ideas into intelligence blockbusters ready to shine in the real world.\nThe future of AI will be directed, produced, and premiered inside studios like Microsoft Foundry, where every creator has the right tools to turn their vision into concrete impact.\nReferences Microsoft. (2025). What\u0026rsquo;s new in Microsoft Foundry | October and November 2025. https://devblogs.microsoft.com/foundry/whats-new-in-microsoft-foundry-oct-nov-2025/\nMicrosoft. (2025). Microsoft Ignite 2025 Recap: Agentic AI, Foundry, and Azure Innovations. https://azure.microsoft.com/en-us/blog/actioning-agentic-ai-5-ways-to-build-with-news-from-microsoft-ignite-2025/\n","permalink":"https://julianamarialop.github.io/en/articles/microsoft-foundry-the-film-studio-of-the-new-era-of-ai/","summary":"Picture Microsoft Foundry as a big film studio dedicated to producing AI movies. Instead of projectors, cameras, and scripts, it offers tools, models, and infrastructure to turn ideas into…","title":"Microsoft Foundry: The Film Studio of the New Era of Artificial Intelligence"},{"content":"Imagine you need to get from point A to point B. A few decades ago, your only option was a car with a manual transmission, demanding full attention to every gear change and every move in traffic. Over time, cars with automatic transmissions came along, making driving simpler but still requiring an attentive driver. Today we live in the age of autopilot, where the car takes over on highways, holding its speed and lane, although the driver has to be ready to step in at any moment. Now imagine a future where you simply state the destination and the car does everything else: plots the route, avoids obstacles, parks, and recharges itself. A fully autonomous vehicle.\nThat same evolutionary journey is happening right now at the heart of data engineering. For years, we built data pipelines the way you drive a stick shift: handcrafted work that demands detailed coding and constant maintenance. More recently, we moved to tools with visual interfaces and some automation, our \u0026ldquo;automatic transmission.\u0026rdquo; Now, with the arrival of AI copilots, we\u0026rsquo;ve entered the \u0026ldquo;autopilot\u0026rdquo; era, where AI suggests and assists but the engineer is still in charge. We\u0026rsquo;re on the verge of the next big paradigm shift: the transition from ETL (Extract, Transform, Load) to ETL-A, Autonomous ETL.\nThis isn\u0026rsquo;t just an incremental upgrade; it\u0026rsquo;s a fundamental reimagining of how data moves inside an organization. We\u0026rsquo;re leaving a model where engineers manually supervise every step of the process and heading toward a future where intelligent systems, or agents, orchestrate data flows with autonomy, adaptability, and resilience. In this article, we\u0026rsquo;ll explore that evolution, dive into the architecture that makes ETL-A possible, and discuss how this transformation redefines the role of the data engineer in the age of artificial intelligence.\nThe Evolutionary Journey of ETL: From Manual to Full Autonomy To understand the impact of ETL-A, it\u0026rsquo;s crucial to revisit the journey that brought us here. Each phase of ETL\u0026rsquo;s evolution can be compared to a stage of driving technology, reflecting a progressive increase in automation and a decrease in manual intervention.\nETL 1.0: Manual Driving In the first era of ETL, dating back to the \u0026rsquo;90s, the tools were like the first cars with manual transmissions. Platforms such as Informatica PowerCenter and SQL Server Integration Services (SSIS) dominated the landscape. Building a pipeline was a meticulous, highly technical process. Engineers had to explicitly define every transformation, every field mapping, and every business rule in code or in complex visual interfaces. Maintenance was constant and reactive; a small change in the source system, such as renaming a column, could break the entire flow, requiring days of manual debugging and fixing. It was work with total control, but also with fragility and poor scalability.\nETL 2.0: The Cloud\u0026rsquo;s Automatic Transmission The rise of the cloud marked the second big wave, our \u0026ldquo;automatic transmission.\u0026rdquo; Tools such as Azure Data Factory (ADF), AWS Glue, and SaaS platforms such as Fivetran and Stitch drastically simplified building pipelines. The need to manage servers went away, and prebuilt connectors for hundreds of data sources cut down integration effort. The focus shifted from low-level coding to orchestrating flows in visual interfaces. However, the transformation logic and the pipeline\u0026rsquo;s resilience still depended entirely on the engineer\u0026rsquo;s design. If a problem came up, the tool could alert you to the failure, but the responsibility for diagnosing and fixing it was still human.\nETL 3.0: AI-Assisted Autopilot In recent years, we\u0026rsquo;ve entered the \u0026ldquo;autopilot\u0026rdquo; era, with AI infused into ETL processes. Tools such as Databricks and Matillion started incorporating machine learning capabilities and, more recently, generative AI. These systems can analyze metadata to suggest schema mappings, generate SQL code from natural language, and detect anomalies in data patterns. It\u0026rsquo;s the equivalent of an autopilot that takes over in ideal conditions but still needs an attentive engineer to supervise, validate the AI\u0026rsquo;s suggestions, and step in when unexpected scenarios arise. The manual workload goes down, but final accountability and strategic decision-making stay with the professional.\nETL-A: The Fully Autonomous Car Now we\u0026rsquo;re on the threshold of Autonomous ETL (ETL-A). This isn\u0026rsquo;t just a smarter version of ETL 3.0; it\u0026rsquo;s a fundamental shift in which control passes from the engineer to a system of collaborative AI agents. In an ETL-A paradigm, the data engineer no longer builds the pipeline step by step. Instead, they declare the intent: \u0026ldquo;I need the sales data from the Salesforce API, enriched with customer demographic data from our data warehouse, and delivered as a daily aggregated table in our Lakehouse.\u0026rdquo;\nThe agent system then takes end-to-end responsibility:\nPlanning: An orchestrator agent (or planner) breaks the request down into logical steps. Execution: It delegates the tasks to specialized agents: a connectivity agent to extract the data from the API, a transformation agent to apply the enrichment and aggregation logic, and a quality agent to validate the data. Adaptation: If the Salesforce API goes through a schema change, the system detects the change, adjusts the mapping autonomously, and keeps running, documenting the action. Self-optimization: An optimization agent monitors the pipeline\u0026rsquo;s cost and performance, rewriting a query or adjusting resource allocation to ensure efficiency. In this new world, the data engineer\u0026rsquo;s role evolves from \u0026ldquo;pipeline builder\u0026rdquo; to \u0026ldquo;architect of autonomous systems,\u0026rdquo; focusing on defining business goals, governance policies, and SLAs while the agents take care of the detailed implementation.\nThe Architecture of ETL-A: Inside the Agents\u0026rsquo; Minds An Autonomous ETL system isn\u0026rsquo;t a single monolithic application, but an ecosystem of specialized AI agents that collaborate to reach a goal. Inspired by frameworks such as LangChain and AutoGen, the architecture of an ETL-A system can be broken down into layers that work together to turn the user\u0026rsquo;s intent into a resilient, efficient data flow.\nInterface Layer: User Prompt / Statement of Intent: This is where the data engineer defines the business goal in natural language. Just as a passenger enters the destination in the self-driving car\u0026rsquo;s app, here you declare what you need from your data. Orchestration Layer: Orchestrator Agent (Planner): Breaks the intent down into a multi-step action plan and picks the right agents for each task. It works like the navigation system that calculates the route and sends commands to the engine, brakes, and steering. Execution Layer: Specialized Agents: A set of agents focused on specific tasks, invoked by the orchestrator. They\u0026rsquo;re the car\u0026rsquo;s subsystems: engine, braking system, steering, and sensors, each one specialized in its job. Cognition Layer: LLM (Large Language Model): The \u0026ldquo;brain\u0026rdquo; that powers the agents\u0026rsquo; reasoning, letting them understand language, generate code, and make decisions. It\u0026rsquo;s the onboard computer that processes sensor data and makes decisions in real time. Memory Layer: Knowledge Base / Vector Database: Where the system stores long-term context: logs of past runs, schemas, metadata, and documentation. Like the map and route history the car uses to learn and improve. Tools Layer: Connectors, APIs, Code Libraries: The tools the agents use to interact with the outside world (data systems, APIs, and so on). They\u0026rsquo;re the wheels, headlights, and windshield wipers the car uses to interact with the road. The Workflow of an Autonomous Pipeline Let\u0026rsquo;s look at how these layers interact in a practical scenario. Suppose a data engineer declares the following intent: \u0026ldquo;I need a daily report that combines order data from our PostgreSQL database with product reviews from Zendesk, and that alerts on any product with an average rating below 3 stars.\u0026rdquo;\n1. Statement of Intent: The Orchestrator Agent receives the prompt.\n2. Reasoning and Planning: Using the LLM, the orchestrator interprets the request and breaks it into a plan:\nExtract order data from the orders table in PostgreSQL. Extract review tickets from the Zendesk API. Join the two datasets on product_id. Calculate the average rating per product. Filter products with an average rating \u0026lt; 3. Load the result into a table in the Lakehouse. Send an alert to a Slack channel. 3. Delegation to Specialized Agents: The orchestrator invokes the agents it needs:\nA Database Connectivity Agent gets the task of extracting the data from PostgreSQL. An API Agent is in charge of fetching the data from Zendesk. A Transformation Agent (Spark/dbt) receives the raw data and the code needed to join, aggregate, and filter it. An Alerting Agent is activated to send the final message on Slack. 4. Adaptation and Self-correction: During the run, the API Agent discovers that Zendesk has added a new field called sentiment_score to the payload. The agent checks the Knowledge Base to see whether this information is useful. It may autonomously decide to enrich the result with this new information, or simply ignore it, but either way it records its decision in memory for future runs. If the pipeline fails because of a connection error, the agent can retry a few times before escalating the problem to a human.\n5. Completion and Learning: After a successful run, the logs, the generated code, and the result are stored in the Knowledge Base. This lets the system learn from experience, optimizing future runs and becoming more efficient over time.\nThe Future of the Data Engineer: From Builder to Conductor The rise of ETL-A doesn\u0026rsquo;t mean the end of data engineering; on the contrary, it lifts the profession to a more strategic and creative level. The time that used to go into repetitive, reactive pipeline coding and debugging will be freed up for higher-value activities.\nThe data engineer of the future will be less of a \u0026ldquo;car mechanic\u0026rdquo; and more of an \u0026ldquo;urban transportation systems engineer.\u0026rdquo; Their responsibilities will center on:\nDesigning Autonomous Systems: Designing and configuring agent ecosystems, defining their capabilities, permissions, and rules of collaboration. Governance and Policy: Setting the data quality, security, and privacy policies the agents must follow. Instead of implementing the rules, the engineer will declare them for the agents to enforce. Cost and Performance Optimization: Monitoring the efficiency of the agent system, adjusting cost models and performance targets to make sure the organization\u0026rsquo;s resources are used as well as possible. Data Product Innovation: Focusing on understanding business needs and designing new data products, leaving the implementation to the agents. In essence, the data engineer will become a conductor, leading an orchestra of AI agents to create complex, harmonious data symphonies instead of playing each instrument individually.\nConclusion: Embrace Autonomy The journey from manual ETL to Autonomous ETL is more than a technological evolution; it\u0026rsquo;s a shift in our philosophy of work. Just as the self-driving car promises to revolutionize transportation, ETL-A promises to transform the way organizations use their data, making the process more agile, resilient, and intelligent.\nThe term ETL-A may be new, but the trend it represents is undeniable. We\u0026rsquo;re moving toward a future where interacting with data will be based on intent, not implementation. For data engineers, this is a moment of unique opportunity. Those who embrace this change and start building the skills needed to design and govern autonomous systems won\u0026rsquo;t just survive the next wave of automation; they\u0026rsquo;ll become the indispensable architects of the data-driven company of the future. The question is no longer whether data pipelines will become autonomous, but how fast we can get there. And the answer, by all indications, is: faster than we think.\nReferences Naeem, H. (2025). \u0026ldquo;Future of ETL: Why Agentic AI Is the Next Big Shift in Data Engineering\u0026rdquo;. Medium. Available at:\nHossain, M. (2025 ). \u0026ldquo;AI Agents for Data Pipelines: Self-Healing and Self-Optimizing Workflows\u0026rdquo;. Medium. Available at:\nDatabricks Staff. (2025 ). \u0026ldquo;AI ETL: How Artificial Intelligence Automates Data Pipelines\u0026rdquo;. Databricks Blog. Available at:\nMatillion. (2025 ). \u0026ldquo;Moving Toward Autonomous Data Systems with Agentic Intelligence\u0026rdquo;. Matillion Blog. Available at:\nIBM. (2025 ). \u0026ldquo;LLM Agent Orchestration: A Step by Step Guide\u0026rdquo;. IBM Think Tutorials. Available at:\nLangChain. (2025 ). \u0026ldquo;LangGraph: Build Agents That Automate Real-World Tasks\u0026rdquo;. LangChain Documentation. Available at:\n","permalink":"https://julianamarialop.github.io/en/articles/from-etl-to-etl-a-the-rise-of-autonomous-pipelines/","summary":"Imagine you need to get from point A to point B. A few decades ago, your only option was a car with a manual transmission, demanding full attention to every gear change and every move in traffic. Over time…","title":"From ETL to ETL-A: The Rise of Autonomous Pipelines and the Future of Data Engineering"},{"content":"Have you ever tried to explain to a client how much a software project will cost and heard: \u0026ldquo;But why does it take so long?\u0026rdquo; or \u0026ldquo;Why does it cost so much?\u0026rdquo;. The difficulty lies in translating functionality into size, and size into effort. For decades, the software industry tried to solve this problem by counting lines of code, but that\u0026rsquo;s like measuring an apartment by the number of bricks: technically possible, but completely useless for someone who needs to understand what they\u0026rsquo;re buying.\nImagine you\u0026rsquo;re looking for an apartment. The realtor doesn\u0026rsquo;t show you how many bricks went into the construction or how many bags of cement were used on the job. They tell you: 80 square meters, 3 bedrooms, 2 bathrooms, living room open to the kitchen. You immediately understand the size and functionality of the space. That\u0026rsquo;s exactly the idea behind Function Points: a universal measure that expresses the size of software from the perspective of the people who will use it, regardless of the technology used to build it.\nThis article demystifies Function Point Analysis (FPA), an international metric created almost 50 years ago that is still one of the most reliable ways to measure software. Let\u0026rsquo;s understand what function points are, how they work in practice, why they\u0026rsquo;re technology-independent and how they can transform the way you estimate and manage projects.\nThe Problem with Measuring Software Imagine trying to compare two apartments where one was built with clay bricks and the other with concrete blocks. If you measure by the amount of material, you\u0026rsquo;ll get completely different numbers, even if the apartments have exactly the same size and functionality. The same thing happens when we measure software by lines of code: a system in Python might have 1,000 lines, the same system in Java might have 3,000 lines, but both deliver the same functionality to the user.\nThat\u0026rsquo;s why the software industry needed a universal measure, something that focused on the \u0026ldquo;what\u0026rdquo; and not the \u0026ldquo;how\u0026rdquo;. Without a standard, estimates become subjective, comparing productivity across teams is impossible and contract negotiation turns into a guessing game.\nWhat Are Function Points? Function Points are like square meters in construction. It doesn\u0026rsquo;t matter whether the house was made of wood, brick or concrete. It doesn\u0026rsquo;t matter whether the finish is luxurious or simple. Square meters measure the functional space available. In the same way, Function Points measure the functionality available to the user, regardless of whether it\u0026rsquo;s built in Java, Python, .NET or any other technology.\nCreated by Allan Albrecht at IBM in 1977, Function Point Analysis is now an ISO/IEC standard. Its fundamental principle is to measure software from the user\u0026rsquo;s point of view, quantifying the functional requirements the system delivers.\nThe Five Rooms of Software: Counting Components When an architect measures an apartment, they don\u0026rsquo;t just count the total square meters. They identify the rooms: how many bedrooms, how many bathrooms, whether there\u0026rsquo;s a balcony. Each type of room has a value. In the same way, in Function Point counting, we identify the software\u0026rsquo;s \u0026ldquo;functional rooms\u0026rdquo;, which fall into two types: Data Functions and Transaction Functions.\nData Functions (Where we keep things):\nInternal Logical Files (ILF): Think of them as the built-in closets inside your apartment. They\u0026rsquo;re groups of data maintained by the application itself (e.g., a customer table, a product table). External Interface Files (EIF): They\u0026rsquo;re like the shared storage lockers in the building hallway. They\u0026rsquo;re data your application reads but that is maintained by another system (e.g., a ZIP code table maintained by a Brazilian Postal Service system). Transaction Functions (What we do in the rooms):\nExternal Inputs (EI): This is the front door of your apartment. It\u0026rsquo;s any process that lets the user put information into the system, changing the state of an internal closet (e.g., a customer registration form). External Outputs (EO): This is a window with an elaborate view. It presents the user with information that has been processed, calculated or derived (e.g., a monthly sales report with totals and averages). External Inquiries (EQ): This is like a mirror. It just reflects information that already exists, without any complex processing. It\u0026rsquo;s a simple lookup that presents data from an internal or external closet (e.g., a screen that searches for and displays a specific customer\u0026rsquo;s data). Complexity: A Simple Apartment or a Duplex Penthouse? Two apartments can both be 80 square meters, but one might be a simple studio (low complexity) and the other a loft with a mezzanine and a gourmet kitchen (high complexity). In Function Points, we measure that complexity by analyzing how many \u0026ldquo;elements\u0026rdquo; each function handles and how many \u0026ldquo;files\u0026rdquo; it accesses.\nEach of the five components (ILF, EIF, EI, EO, EQ) is classified as Low, Average or High complexity, and each combination has a weight in function points, as shown in the table below:\nTable 1: Function Point Contribution by Complexity\n_Function Point Contribution by Complexity_\nBut how do you know whether it\u0026rsquo;s Low, Average or High? This is where the classification rules come in:\nFor Data Functions (ILF and EIF):\nComplexity depends on two factors:\nDET (Data Element Types): Each unique field recognized by the user. For example, in a Customers table: Name, CPF (Brazilian taxpayer ID), Email, Phone = 4 DETs. RET (Record Element Types): Subgroups of data within a file. In most simple cases, it\u0026rsquo;s 1. It increases when there are subtypes or optional relationships. Table 2: Complexity Matrix for ILF and EIF\n_Complexity Matrix for ILF and EIF_\nFor Transaction Functions (EI, EO, EQ):\nComplexity depends on:\nDET (Data Element Types): Fields the user can enter or view in the transaction. FTR (File Types Referenced): How many ILFs or EIFs the transaction reads or updates. Table 3: Complexity Matrix for EI\n_Complexity Matrix for EI_\nTable 4: Complexity Matrix for EO and EQ\n_Complexity Matrix for EO and EQ_\nA Practical Example: Measuring a Registration System Let\u0026rsquo;s measure the \u0026ldquo;floor plan\u0026rdquo; of a customer registration system. It has one \u0026ldquo;closet\u0026rdquo; (the customer table), one \u0026ldquo;front door\u0026rdquo; (the registration form), one \u0026ldquo;window with an elaborate view\u0026rdquo; (the customer report) and one \u0026ldquo;mirror\u0026rdquo; (the customer lookup).\n1. Identify Functions:\nCustomer Table: 1 ILF Registration Form: 1 EI Customer Report: 1 EO Customer Lookup by CPF: 1 EQ 2. Determine Complexity:\nILF - Customer Table:\nFields: Name, CPF, Email, Phone, Address, City, State = 7 DETs Subgroups: 1 RET (just the main table) Looking up Table 2: 7 DETs + 1 RET = Low (7 FP) EI - Registration Form:\nFields entered: Name, CPF, Email, Phone, Address, City, State = 7 DETs (but since there are fewer than 15, we count them as is) Files accessed: 1 FTR (writes to the Customers table) Looking up Table 3: 7 DETs (between 5-15) + 1 FTR = Low (3 FP) EO - Customer Report:\nFields displayed: Name, CPF, Email, City + Total Customers = 5 DETs Files read: 1 FTR (reads the Customers table) Looking up Table 4: 5 DETs (between 1-5) + 1 FTR = Low (4 FP) EQ - Lookup by CPF:\nFields displayed: Name, CPF, Email, Phone, Address = 5 DETs Files read: 1 FTR (reads the Customers table) Looking up Table 4: 5 DETs (between 1-5) + 1 FTR = Low (3 FP) 3. Add Up the Total:\nTotal Size = 7 + 3 + 4 + 3 = 17 Function Points Done! Our \u0026ldquo;digital apartment\u0026rdquo; has a size of 17 Function Points. Now you understand not just the result, but how we got there.\nWhat Is Knowing the Size Good For? When you know an apartment is 80m², you can estimate how much the flooring will cost, how long it takes to paint and how much furniture will fit. In the same way, knowing that a system has 200 function points and that your team delivers, on average, 5 FP per week, you can estimate that the project will take 40 weeks. FPA lets you:\nEstimate schedule and cost based on historical data. Measure and compare productivity across teams (FP/hour). Create contracts based on price per function point. Benchmark across projects and vendors. Function Points in Analytics and Data Warehouse Projects And what about when the \u0026ldquo;apartment\u0026rdquo; isn\u0026rsquo;t a traditional system, but a Data Warehouse or an analytics project? The good news is that the principles still hold, but they need some adaptation. The Brazilian government published a specific guide for counting Function Points in Data Warehouse projects, recognizing that the multidimensional model (Fact and Dimension tables) has its own characteristics.\nIn a DW project, each Fact table and each Dimension table is counted as an ILF. The ETL (Extract, Transform, Load) process is counted as an External Input (EI), since it brings data in from outside and feeds the internal closets. Reports and dashboards, in turn, are counted as External Outputs (EO) when they involve calculations and aggregations, or External Inquiries (EQ) when they just present data without complex processing.\nThe challenge here is that the development effort for ETL is usually significantly greater than for building reports, yet both can have similar FP counts. That\u0026rsquo;s why many DW contracts separate ETL and OLAP batches, recognizing that the technical complexity is different, even if the functional size is similar.\nWhat About AI Agents? How Do You Measure Them? Now comes the question of the moment: how do you apply Function Points to AI agent projects? There\u0026rsquo;s no official guide yet, but we can apply the same fundamental principles: measure from the user\u0026rsquo;s point of view, identify where data is stored and which operations the user can perform.\nData Functions in AI Agents:\nThe agent\u0026rsquo;s knowledge base: If the agent has indexed documents or its own knowledge base, that\u0026rsquo;s an ILF (internal closet). Vector database: The stored embeddings are an ILF. Each collection of embeddings can be considered a logical file. External APIs queried: If the agent queries third-party APIs (weather, stock quotes, etc.), those are EIFs (external closets). Conversation history: If the agent keeps memory of previous interactions, that\u0026rsquo;s an ILF. Transaction Functions in AI Agents:\nUser prompt: Each type of interaction the user can make is an EI (front door). RAG (Retrieval-Augmented Generation) response: It\u0026rsquo;s an EO (window with an elaborate view), since it involves semantic search, processing and response generation. Simple lookup with no processing: If the agent just retrieves and displays information without elaborating, it\u0026rsquo;s an EQ (mirror). Function calling: When the agent takes an action (booking an appointment, sending an email), that\u0026rsquo;s an EI, since it changes the state of a system. Document ingestion: The process of chunking, generating embeddings and storing them is an EI (a load process, similar to ETL). Example: Customer Service Agent\n1 ILF: Knowledge base (FAQs, manuals) 1 ILF: Vector database with embeddings 1 ILF: Conversation history 1 EIF: Ticketing system (queried but not maintained by the agent) 1 EI: User asks a question 1 EO: Agent answers using RAG (search + processing + generation) 1 EI: Agent creates a ticket in the external system Assuming low complexity to keep it simple: 7 + 7 + 7 + 5 + 3 + 4 + 3 = 36 Function Points.\nImportant: this is an interpretive application of FP principles. Since AI agent technology is recent, there\u0026rsquo;s no consolidated standard yet. But the exercise of thinking in terms of user-visible functionality is still valuable.\nImportant Limitations Square meters don\u0026rsquo;t tell you whether the apartment has good lighting, whether the view is beautiful or whether the finishes are high quality. In the same way, Function Points don\u0026rsquo;t measure whether the code is well written, whether the architecture is elegant or whether the system performs well. It\u0026rsquo;s only a measure of functional size. It\u0026rsquo;s a powerful tool, but it\u0026rsquo;s not the only one.\nConclusion The software industry matured when it realized it needed universal measures, just as construction has square meters and kilometers. Function Points aren\u0026rsquo;t perfect, but they\u0026rsquo;re a common language that enables honest conversations between clients and developers. When you say \u0026ldquo;this project has 500 function points\u0026rdquo;, you\u0026rsquo;re saying something concrete, comparable and technology-independent.\nThe next time someone asks you \u0026ldquo;how much is this system going to cost?\u0026rdquo;, you no longer need to take a shot in the dark. You can measure the floor plan, count the rooms and give a well-founded answer. And that, in itself, is already a revolution.\nReferences Vazquez, C. E., \u0026amp; Simões, G. S. (2013). Análise de Pontos de Função: Medição, Estimativas e Gerenciamento de Projetos de Software. Editora Saraiva.\nISO/IEC 20926:2009. Software and systems engineering: Software measurement: IFPUG functional size measurement method 2009.\nBrazilian Ministry of Planning, Budget and Management (2015). SISP Function Point Counting Guide for Data Warehouse Projects. Brasília: MP.\n","permalink":"https://julianamarialop.github.io/en/articles/the-floor-plan-of-software-demystifying-function-point-analysis/","summary":"Have you ever tried to explain to a client how much a software project will cost and heard: \u0026ldquo;But why does it take so long?\u0026rdquo; or \u0026ldquo;Why does it cost so much?\u0026rdquo;. The difficulty lies in translating functionality into size, and size into…","title":"The Floor Plan of Software: Demystifying Function Point Analysis"},{"content":"The data engineering profession is going through its biggest transformation since the move to the cloud. For years, the focus was on building efficient pipelines, moving data from one point to another and making sure everything was organized in data warehouses. But something fundamental has changed: data is no longer consumed only by humans writing SQL queries or building dashboards. A new category of consumers has emerged: autonomous AI agents that need to discover, understand and use data without human intervention.\nPicture a traditional library where the librarian arranges books on shelves, catalogs everything precisely and expects visitors to know exactly what they\u0026rsquo;re looking for. Now picture turning that library into an intelligent space, where autonomous assistants can walk the aisles, understand the context of each question, connect information from different sections and even anticipate needs. That\u0026rsquo;s the journey data engineers are on: from organizers of information to architects of intelligent systems.\nThis article explores the new role of the AI Data Engineer and the essential skills professionals need to develop to thrive in this new era. We\u0026rsquo;ll dive into the competencies that go beyond traditional pipelines, including building AI agents, context engineering and creating systems that serve both humans and intelligent machines.\nThe Big Shift: From Pipelines to Context Systems The traditional data engineering paradigm assumed a human at the end of the process. That human brought years of institutional knowledge, the ability to ask colleagues questions and the intuition to make reasonable assumptions. AI agents, our new data consumers, have none of those advantages. They need systems that don\u0026rsquo;t just deliver data, but also explain what that data means.\nThis is where data engineering evolves from simply \u0026ldquo;moving data\u0026rdquo; to \u0026ldquo;creating meaning\u0026rdquo;. Before, it was enough to arrange the books by category and catalog number. Now every book needs to come with a complete guide: the author\u0026rsquo;s biography, cross-references, previous editions, who has read it, what they used it for and even the margin notes that reveal valuable insights. Without that context, to an AI agent, data is just a collection of disconnected, unusable facts.\nContext Engineering: The New Fundamental Skill The most critical skill for the data engineer in 2026 is Context Engineering. It\u0026rsquo;s the practice of designing systems that embed rich, machine-readable context alongside the data itself. This goes far beyond traditional documentation or data catalogs. Context engineering has several dimensions:\nSemantic Context: What does the data actually mean to the business? Is a \u0026ldquo;customer\u0026rdquo; in the sales system the same as a \u0026ldquo;customer\u0026rdquo; in the support system? Temporal Context: When was the data created and updated? What was the state of the world at that moment? Relational Context: How does this data set connect to others? Which joins are meaningful and which would produce nonsense? Quality Context: How reliable is this data? What are its known limitations? Provenance Context: Where did the data come from and what transformations has it been through? Implementing this context layer is like creating a cataloging system that doesn\u0026rsquo;t just tell you where each book is, but tells its whole story, making the library navigable for an artificial intelligence.\nVector Databases and the Semantic Search Revolution If context is the story, Vector Databases are the technology that lets AI agents understand it at scale. These databases, once a niche for machine learning teams, are now part of the core data engineering infrastructure. They excel at searching for similarity and relevance, uncovering connections that were never explicitly modeled.\nImagine that, instead of searching for books only by title or author, you could ask the library: \u0026ldquo;I need to understand how to solve scalability problems in distributed systems.\u0026rdquo; A semantic search system, powered by embeddings and a vector database, would find not only books with those keywords, but also related articles, similar use cases and forum discussions that tackle the same challenge. That\u0026rsquo;s the revolution AI Data Engineers need to master, combining vector search with traditional metadata filters in what\u0026rsquo;s known as hybrid search.\nBuilding AI Agents: From Consumer to Creator The final frontier for the AI Data Engineer is building the AI agents themselves. It\u0026rsquo;s no longer just about preparing data for others, but about creating the intelligent assistants that will use it. An AI agent is more than a chatbot; it\u0026rsquo;s a system that can plan, use tools and interact with data sources to complete complex tasks autonomously.\nBuilding a robust agent involves a structured process:\nDefine the scope and the allowed actions. Map and connect the required data sources. Build the retrieval and context layer (RAG). Choose the LLM and the reasoning style (e.g., ReAct). Create tools the agent can use (function calling). Add memory to keep context across interactions. Implement guardrails and observability to ensure safety and monitoring. This is where the data engineer becomes a true architect of intelligence, training specialized assistants and giving them the tools not only to find information but also to take action, such as writing summaries, booking resources or connecting users with experts.\nThe AI Data Engineer Toolkit To navigate this new landscape, a set of powerful tools is emerging. Platforms such as Microsoft Fabric with its Data Agents, Azure AI Foundry and the Databricks Agent Framework with MLflow offer integrated ecosystems for building, training and managing agents. Frameworks such as LangChain and LlamaIndex provide the building blocks, while a new generation of vector databases such as Pinecone, Weaviate and Azure AI Search are becoming essential.\nAll-in-One Platforms:\nMicrosoft Fabric: An integrated environment that combines data engineering, data science and analytics with native Data Agents Azure AI Foundry: A complete platform for building, deploying and managing AI solutions Databricks: A unified data and AI ecosystem with built-in Agent Framework and MLflow Agent Frameworks:\nLangChain: A versatile, extensive framework for building agents and complex workflows LlamaIndex: Specialized in RAG and data indexing for retrieval agents Vector Databases:\nAzure AI Search: Hybrid search (vector + keyword) integrated into the Azure ecosystem Pinecone: A high-performance managed solution for production Weaviate: Multimodal support and advanced schema capabilities Qdrant: Open source, ideal for self-hosting and customization MLOps and Monitoring:\nMLflow: Full lifecycle management for models and agents, with native evaluation Azure Monitor: Integrated observability and monitoring for Azure solutions The Transition Roadmap: Where to Start The good news is that the fundamental data engineering skills remain the foundation. The path to becoming an AI Data Engineer isn\u0026rsquo;t about abandoning what you know, but about building on top of it. Here\u0026rsquo;s a practical roadmap to start your journey:\nPhase 1: AI and Machine Learning Fundamentals (2-4 weeks) Before building agents, it\u0026rsquo;s essential to understand how AI works. Start with the basics of LLMs, embeddings and prompt engineering. Microsoft offers the AI Engineer Career Path on Microsoft Learn, a structured path that covers everything from an introduction to AI to advanced concepts. For those who work with Databricks, the free Generative AI Fundamentals course is an excellent starting point.\nWhat to learn:\nHow Large Language Models work What embeddings are and how they\u0026rsquo;re generated Prompt engineering concepts Introduction to RAG (Retrieval-Augmented Generation) Phase 2: Exploring Vector Databases and Semantic Search (2-3 weeks) With the fundamentals in place, it\u0026rsquo;s time to understand how to store and retrieve information intelligently. Experiment with Azure AI Search or other vector databases. The Azure AI Search module on Microsoft Learn offers a hands-on introduction.\nWhat to build:\nA simple semantic search system over technical documentation Experiment with different chunking strategies Compare pure vector search with hybrid search Phase 3: Building Your First Agent (3-4 weeks) Now comes the most exciting part: building an AI agent from scratch. Microsoft offers the Develop AI agents on Azure learning path, which covers everything from creating agents to deploying them. In the Databricks ecosystem, the AI Agent Fundamentals course introduces Mosaic AI and the Agent Framework.\nSuggested hands-on project:\nBuild an agent that answers questions about your company\u0026rsquo;s internal documentation Implement function calling so the agent can query APIs Add memory to keep context across conversations For hands-on implementation, the official Databricks tutorial offers a complete guide to building, evaluating and deploying a retrieval agent.\nPhase 4: Fabric Data Agents and Enterprise Integration (2-3 weeks) If you work in the Microsoft ecosystem, explore how to create Fabric Data Agents to connect agents directly to lakehouses and warehouses. The end-to-end tutorial shows how to set up a complete agent.\nWhat to explore:\nIntegrating agents with Power BI and Microsoft Fabric Consuming Data Agents in Azure AI Foundry Multi-agent orchestration with Copilot Studio Phase 5: MLOps for Agents and Production (Ongoing) Agents in production need monitoring, evaluation and governance. Learn to use MLflow to evaluate and monitor agents and implement observability practices.\nEssential skills:\nAgent logging and tracing Quality and cost metrics A/B testing of different versions Guardrails and security The Living Library of the Future The data engineer\u0026rsquo;s role is evolving from pipeline builder to architect of intelligence. This isn\u0026rsquo;t a change of profession, but a powerful expansion of its scope. The library of the future isn\u0026rsquo;t just a repository of information, but a living ecosystem of knowledge that adapts, learns and evolves. The professionals who embrace this transformation won\u0026rsquo;t just be keeping up with change; they\u0026rsquo;ll be leading the next generation of intelligent data systems.\nThe journey may seem challenging, but each step builds on the previous one. Getting started is simpler than it looks: pick a phase of the roadmap, set aside a few hours a week and dive into a hands-on project. The difference between a traditional data engineer and an AI Data Engineer isn\u0026rsquo;t only technical knowledge, but the ability to think in systems that create meaning, not just move bytes. And that transformation starts now, with the next course you begin, the first agent you build, the first time you enrich your data with rich, machine-readable context. The future of data engineering isn\u0026rsquo;t about replacing what we do, but about amplifying our impact in ways that were once unimaginable.\nReferences Panda, S. (2025). The 2026 Data Engineering Roadmap: Building Data Systems for the Agentic AI Era. Medium. Airbyte Engineering Team. (2025). A Guide to Building AI Agents. Airbyte. Microsoft. (2025). Fabric data agent concepts (preview). Microsoft Learn. Microsoft. (2025). Microsoft Foundry. Azure. Databricks. (2025). Evaluate and monitor AI agents. Databricks Documentation. ","permalink":"https://julianamarialop.github.io/en/articles/ai-data-engineer-the-evolution-of-the-data-engineer-in-the-age-of-intelligent-agents/","summary":"The data engineering profession is going through its biggest transformation since the move to the cloud. For years, the focus was on building efficient pipelines, moving data from one point to another and making sure…","title":"AI Data Engineer: The Evolution of the Data Engineer in the Age of Intelligent Agents"},{"content":"For years, the data world chased the dream of a \u0026ldquo;single source of truth\u0026rdquo;. In practice, however, what many companies built was a real Tower of Babel. Every department, every team, every analyst created their own dialect, their own definitions for essential metrics. The result? Conflicting reports, decisions based on diverging numbers and widespread distrust of the data. Ten sales reports showed ten different numbers for \u0026ldquo;net revenue\u0026rdquo;, and nobody knew which one was right.\nMicrosoft Fabric IQ, one of the most impactful announcements at Ignite 2025, isn\u0026rsquo;t just another tool in this landscape; it\u0026rsquo;s the architectural solution for tearing down that Tower of Babel. It was designed to fix the root cause of the problem: the lack of a central, governed dictionary for the language of the business. Fabric IQ proposes a paradigm shift: instead of replicating business logic in every report and model, let\u0026rsquo;s define it once, in a single place, and have the whole ecosystem, from BI reports to AI agents, drink from that same source.\nIn this article, we\u0026rsquo;ll take a deep dive into what Fabric IQ is, breaking down its components, explaining how it works technically and, most importantly, the transformative impact it brings to data architecture and to the way companies will make decisions in the future. It\u0026rsquo;s the end of the era of ambiguity and the beginning of the era of semantic clarity.\nWhat Exactly Is Fabric IQ? If the corporate Tower of Babel was built on the lack of a common language, Fabric IQ arrives to be the universal dictionary and official translator of this new era. Imagine that each department (Sales, Marketing, Finance) spoke a different dialect, making communication impossible. Fabric IQ doesn\u0026rsquo;t just create a single, official language for the whole company; it also makes sure everyone speaks it fluently, eliminating confusion and ambiguity once and for all.\nTechnically, Fabric IQ is the semantic intelligence layer of Microsoft Fabric. It centralizes business logic in a new first-class Fabric item called Ontology. An ontology is, essentially, a formal model of your business. It\u0026rsquo;s where you define what the important \u0026ldquo;things\u0026rdquo; are (entities such as Customer, Product, Sale), how they relate to one another (a Customer makes a Sale, a Sale contains a Product) and which rules and hierarchies govern them.\nThe power of Fabric IQ comes from its five integrated components, which work together to bring this ontology to life:\nOntology: The \u0026ldquo;master plan\u0026rdquo; or the \u0026ldquo;DNA\u0026rdquo; of the business. It defines the entities, relationships and rules. Semantic Model: The analytics layer on top of the master plan. It adds the metrics, KPIs and calculations (DAX) that will be used in reports. Graph: The system of connections. It enables complex queries that traverse the relationships defined in the ontology. Data Agent: The intelligent \u0026ldquo;translator\u0026rdquo;. It uses the ontology to understand natural language questions and find answers in the data. Operations Agent: The autonomous \u0026ldquo;guardian\u0026rdquo;. It monitors data in real time and takes action based on the rules defined in the ontology. The Anatomy of Fabric IQ: Diving into the 5 Components To understand the impact of Fabric IQ, you need to understand how each of its parts works.\n1. Ontology: The Foundation of Everything The Ontology is the heart of Fabric IQ. This is where the magic begins. Instead of modeling data in tables and columns, you start modeling the business in entities and relationships. The big advantage is that this can be done in a low-code way, letting business users collaborate with the technical team to create a model that truly reflects the company\u0026rsquo;s reality. A key capability is the automatic generation of ontologies from existing Power BI semantic models, which means you can reuse work that\u0026rsquo;s already been done and speed up adoption.\n_Fabric IQ Ontology_\n2. Semantic Model: The BI Lens on the Business If the Ontology is the treasure map, the Semantic Model is the set of tools you use to read it. This is where traditional BI logic, such as complex DAX measures, KPIs and analysis perspectives, gets added on top of the ontology. The crucial difference is that this logic is built on a consistent, governed business model rather than on loose tables. That ensures that when you create a \u0026ldquo;Sales Growth\u0026rdquo; KPI, the definition of \u0026ldquo;Sale\u0026rdquo; is the same for everyone.\n3. Graph: The Power of Connections Fabric IQ doesn\u0026rsquo;t just store the entities; it deeply understands how they connect. The Graph component is a native graph engine that lets you ask questions that would be extremely complex in a traditional relational model. For example, you can easily traverse multiple relationships to answer a question like: \u0026ldquo;Show me every customer who bought product X, who had a delivery problem reported by sensor Y in the cold chain, and who opened a support ticket in the last 24 hours.\u0026rdquo; The graph makes cross-domain reasoning a native capability.\n4. Data Agent: The Virtual Analyst Data Agents are the face of Fabric IQ for the end user. They\u0026rsquo;re AI agents you can query in natural language. Because they\u0026rsquo;re \u0026ldquo;built\u0026rdquo; on top of the Ontology, they\u0026rsquo;re born with a deep understanding of your business. When a user asks \u0026ldquo;Who were my most profitable customers in Brazil last quarter?\u0026rdquo;, the agent knows what a \u0026ldquo;customer\u0026rdquo; is, what \u0026ldquo;profitable\u0026rdquo; means, how to filter by \u0026ldquo;Brazil\u0026rdquo; and how to interpret \u0026ldquo;last quarter\u0026rdquo;, because all of those definitions and hierarchies live in the ontology. It\u0026rsquo;s the democratization of access to insights, with no need to know SQL or DAX.\n5. Operations Agent: Intelligence in Action While Data Agents answer questions, Operations Agents act. They\u0026rsquo;re autonomous agents that monitor data in real time and take action based on the rules defined in the ontology. For example, a rule in the ontology might say: \u0026ldquo;If the temperature of a vaccine container (entity) exceeds 5°C (rule), fire an alert (action) to the responsible logistics manager (relationship).\u0026rdquo; Operations Agents turn business knowledge into intelligent automation, 24 hours a day, 7 days a week.\nThe Architectural Impact: Why Fabric IQ Changes the Game For those who design and build data systems, Fabric IQ represents a fundamental shift. The main impact is the definitive consolidation of the semantic layer. This solves a series of chronic architectural problems:\nThe End of Redundancy: Business logic is defined only once. You no longer need to copy and paste the same DAX formula into 20 different Power BI models. Maintenance becomes trivial: update the rule in the ontology, and every report and agent that consumes it is updated automatically. Centralized Governance: The Ontology becomes the central point of governance. It\u0026rsquo;s where data lineage converges with semantics, letting you track not only where the data came from but what it means at each step. Security and access policies are applied in the ontology itself, ensuring consistency. Reuse and Agility: Once the business ontology is built, it becomes a reusable asset for any new project. A new product team doesn\u0026rsquo;t need to \u0026ldquo;rediscover\u0026rdquo; what a customer is; it simply consumes the \u0026ldquo;Customer\u0026rdquo; entity from the ontology. This drastically speeds up the development of new data and AI solutions. Unifying BI and AI: Fabric IQ knocks down the wall that traditionally stood between the world of Business Intelligence and the world of Artificial Intelligence. The same semantic layer that feeds a Power BI dashboard is used to \u0026ldquo;teach\u0026rdquo; an AI agent about the business. That ensures BI insights and AI answers are always consistent. How to Implement Fabric IQ in Practice: A Starter Guide Knowing what Fabric IQ does is one thing; implementing it is another. The approach shouldn\u0026rsquo;t be a \u0026ldquo;big bang\u0026rdquo;, but an iterative, strategic process. Here\u0026rsquo;s a step-by-step guide to get started:\nStart Small, Think Big: Don\u0026rsquo;t try to model the entire company at once. Pick a critical, well-understood business domain, such as \u0026ldquo;Sales\u0026rdquo; or \u0026ldquo;Customers\u0026rdquo;. The goal is to create value quickly and use that first project as a success story to expand from. Reuse What Already Exists: The ability to generate an ontology from existing Power BI semantic models is your best starting point. Identify the most complete and trusted Power BI model in the chosen domain and use it to create the first version of your ontology. This saves time and leverages the knowledge already built into your reports. Involve the Business (for Real): The ontology is a business model, not just a technical artifact. Use the low-code interface to sit down with the domain experts (business analysts, product managers) and refine the entities, relationships and rules. Ask questions like: \u0026ldquo;What defines an \u0026lsquo;active customer\u0026rsquo; for you?\u0026rdquo; or \u0026ldquo;What are the life stages of an \u0026lsquo;order\u0026rsquo;?\u0026rdquo; This collaboration is crucial to success. Connect the Data: Once the ontology is defined, the next step is to bind the real data in OneLake to it. Map the tables and columns of your lakehouse to the ontology\u0026rsquo;s entities and properties. This is where raw data gains business meaning. Build on Top: With the ontology populated, start building. Create a new Semantic Model on top of it for your BI reports. Set up a Data Agent so users can ask questions in natural language. Create an Operations Agent to automate a simple process. What matters is demonstrating the value of the unified semantic layer across different experiences. Strategies for Migrating Your Semantic Models Migrating dozens or hundreds of Power BI models to a centralized ontology may sound daunting, but with the right strategy it becomes a process of adding value rather than a costly refactoring.\nApproach 1: The \u0026ldquo;Golden Model\u0026rdquo; Strategy In this approach, you identify the organization\u0026rsquo;s most comprehensive and trusted semantic model, the \u0026ldquo;Golden Model\u0026rdquo;. This model becomes the foundation for your main ontology. The migration happens in phases:\nExtraction: Use the \u0026ldquo;Golden Model\u0026rdquo; to generate the initial ontology in Fabric IQ. Consolidation: Analyze other semantic models that are subsets or variations of the \u0026ldquo;Golden Model\u0026rdquo;. Instead of migrating them, retire them and switch to the new centralized Semantic Model built on the ontology. Use Power BI perspectives to provide tailored views for different user groups, if needed. Expansion: For models that contain business logic that wasn\u0026rsquo;t in the \u0026ldquo;Golden Model\u0026rdquo;, enrich the central ontology with those new entities, relationships or rules. The goal is to let the ontology grow organically. Approach 2: The Domain-Federated Strategy For very large, decentralized organizations, a single ontology may be impractical at first. The federated strategy proposes creating multiple ontologies, one for each major business domain (Finance, HR, Marketing, etc.).\nDomain Mapping: Identify the company\u0026rsquo;s main data domains and the semantic models associated with each one. One Ontology per Domain: Create a separate ontology for each domain, following the \u0026ldquo;Golden Model\u0026rdquo; approach within each one. This lets domain teams work autonomously. Cross-Domain Connection: The crucial step is creating relationships between the domain ontologies. For example, the \u0026ldquo;Customer\u0026rdquo; entity in the Marketing ontology can be linked to the \u0026ldquo;Customer\u0026rdquo; entity in the Finance ontology. Fabric IQ was designed to support this kind of federation, giving you local governance with global connectivity. Whatever the approach, the principle is the same: stop replicating, start reusing. Migrating to Fabric IQ isn\u0026rsquo;t about moving PBIX files from one place to another; it\u0026rsquo;s about extracting the valuable business logic that lives inside them and lifting it into a shared, governed, reusable intelligence layer for the whole organization.\nConclusion: From Data to Real Intelligence Fabric IQ is much more than a new feature; it\u0026rsquo;s the missing piece in the puzzle of the data-driven company. It attacks the root of the semantic inconsistency problem, providing a solid foundation on which we can confidently build analytics, AI agents and automated processes.\nBy creating a central, living \u0026ldquo;dictionary\u0026rdquo; for the business, Fabric IQ turns OneLake from a simple data repository into a true digital brain for the organization. It\u0026rsquo;s the technology that finally lets us stop arguing about what the data means and start acting on the unified knowledge it provides.\nReferences Fabric IQ: The Semantic Foundation for Enterprise AI From Data Platform to Intelligence Platform: Introducing Microsoft Fabric IQ ","permalink":"https://julianamarialop.github.io/en/articles/fabric-iq-the-end-of-the-tower-of-babel-in-enterprise-data/","summary":"For years, the data world chased the dream of a \u0026ldquo;single source of truth\u0026rdquo;. In practice, however, what many companies built was a real Tower of Babel. Every department, every team, every analyst…","title":"Fabric IQ: The End of the Tower of Babel in Enterprise Data"},{"content":"Microsoft Ignite 2025 wasn\u0026rsquo;t an event about future promises; it was about concrete deliveries that mark the maturity of Microsoft Fabric . The platform stopped being a collection of promising tools and became a cohesive operating system for data and intelligence, where each new piece fits in to solve chronic problems we\u0026rsquo;ve faced for years in the data world.\nThe vision presented was clear: end the era of \u0026ldquo;gluing\u0026rdquo; services together with fragile pipelines and make way for an approach where business context flows natively across every layer. For anyone dealing with the complexity of enterprise data, the announcements are direct solutions to real pains: inconsistent metrics, the difficulty of integrating AI in a governed way and the barriers between transactional and analytical systems.\nTo understand the real impact of these changes, you have to go beyond the names and the presentations. You need to dissect each new feature, understand how it works technically and, more importantly, how it fits into the bigger puzzle. That\u0026rsquo;s exactly what we\u0026rsquo;ll do next: a complete, no-nonsense guide to what really matters in the Fabric announcements at Ignite 2025.\n1. Fabric IQ: The Central Library of Business Definitions Imagine your company is like a big professional kitchen. Each chef (department) had their own recipe for the \u0026ldquo;house special sauce\u0026rdquo;, and nobody agreed on the ingredients. Fabric IQ is like creating a single, official recipe book, approved by the executive chef, that everyone has to follow. Now, when someone orders the \u0026ldquo;special sauce\u0026rdquo;, everyone makes exactly the same thing.\nFabric IQ is Microsoft Fabric\u0026rsquo;s semantic intelligence layer. Its main goal is to end the mess of having multiple definitions of business metrics and entities scattered across dozens of Power BI models, Analysis Services cubes and other sources. It does this through a new item called Ontology, which is where you define, in a centralized and governed way, what your business entities are (customer, product, sale), how they relate to each other and what the rules and hierarchies are.\n_Fabric IQ_\nFabric IQ is made up of five components that work together:\nOntology: Defines the business entities, their relationships and hierarchies. It\u0026rsquo;s the conceptual model of the business. Semantic Model: Adds the metrics, calculations and BI logic (DAX) on top of the ontology. It\u0026rsquo;s the analysis layer. Graph: A native graph engine that enables complex queries over the relationships defined in the ontology. Data Agent: AI agents that use the ontology to understand natural language questions and answer based on the data. Operations Agent: Autonomous agents that monitor, learn and act in real time based on business context. Before, if you had 10 different reports, you probably had 10 different definitions of \u0026ldquo;net revenue\u0026rdquo;. With Fabric IQ, you define \u0026ldquo;net revenue\u0026rdquo; once in the Ontology, and that definition is reused across every report, dashboard, AI agent and even operational application. This solves three critical problems:\nConsistency: All the numbers match, because everyone uses the same logic. Governance: There\u0026rsquo;s a single point of control and auditing for business logic. Agility: New data or AI projects are born with the business \u0026ldquo;dictionary\u0026rdquo; already in place, speeding up development. Architecturally, this changes everything. You no longer need to replicate business logic in every model. The Ontology becomes the source of truth, and you build everything on top of it. That simplifies the architecture, reduces redundancy and makes maintenance easier. It\u0026rsquo;s the consolidation of the semantic layer we always dreamed of.\n2. Fabric Data Agents: Intelligent Assistants That Understand Your Business Think of Data Agents as specialized personal assistants. Each one is trained in a specific area of the company (sales, logistics, finance). You ask a question in plain language, and the assistant doesn\u0026rsquo;t just search the organized files (structured data); it also digs through emails, contracts and documents (unstructured data) to give you the most complete answer possible.\nFabric Data Agents are conversational AI agents that reason over the data in OneLake. They work like \u0026ldquo;virtual analysts\u0026rdquo; you can query in natural language. The big evolution announced at Ignite 2025 was the expansion of these agents\u0026rsquo; power on two fronts:\nReasoning over unstructured data: Through integration with Azure AI Search, Data Agents can now be configured to query search indexes containing documents (PDFs, Word, emails, etc.). That means that when you ask a question, the agent can combine information from structured tables in the lakehouse with insights from unstructured documents. Using the Ontology as a knowledge source: Data Agents now use the Fabric IQ Ontology as their \u0026ldquo;instruction manual\u0026rdquo;. That means they already know what an \u0026ldquo;active customer\u0026rdquo;, a \u0026ldquo;premium product\u0026rdquo; or a \u0026ldquo;canceled sale\u0026rdquo; is, because those definitions are centralized. This makes answers more accurate and aligned with the language of the business. On top of that, Data Agents now integrate directly with Microsoft 365 Copilot, letting users reach them from inside Teams, Word or Excel without leaving their work environment.\nBuilding a question-and-answer system over data has always been complex. You need to build a RAG (Retrieval-Augmented Generation) system, connect data sources, deal with security, and so on. Data Agents abstract away all that complexity. You create an agent, point it at the data sources (structured and unstructured), and it just works, automatically respecting security permissions (RLS and CLS).\nYou can create a layer of reusable, governed \u0026ldquo;virtual analysts\u0026rdquo;. Each Data Agent is an asset that can be shared across teams. Security is granular and native. And best of all: since the agents use the Ontology, they always speak the business\u0026rsquo;s \u0026ldquo;language\u0026rdquo;, reducing misunderstandings and increasing trust in the answers.\n3. Data Factory: Modernizing Data Engineering Data Factory received three important updates that modernize the way we build data pipelines in Fabric:\n3.1. Native Integration with dbt Imagine you\u0026rsquo;re building a house. Before, you cut every wooden board on the spot, from scratch. Now, you buy prefabricated, certified pieces (dbt models) that fit together perfectly. Construction gets faster, more standardized and less error-prone.\ndbt (Data Build Tool) has become a market standard for data transformation as code. It lets data engineers write SQL transformations in a modular, testable and versioned way. The news is that Fabric now supports running dbt projects natively as a pipeline activity in Data Factory. That means you can orchestrate your dbt jobs directly in Fabric, without external tools.\nIn addition, Microsoft announced a partnership with dbt Labs to bring dbt Fusion to Fabric in 2026, which promises next-generation performance.\nIf your team already uses dbt, you can migrate those projects to Fabric without rewriting code. If you don\u0026rsquo;t use it yet, you now have one more reason to adopt it, since it\u0026rsquo;s a modern data engineering practice. Native integration means fewer tools to manage and better integration with the Fabric ecosystem.\nYou can adopt CI/CD (continuous integration and delivery) practices for your data pipelines. Transformations become versioned, testable and reusable code. That raises the maturity of data engineering and makes collaboration between teams easier.\n3.2. Mirroring for SAP Think of SAP as a neighboring factory that holds valuable information. Before, to access that data, you had to send a truck, load everything, bring it to your warehouse and unload it (traditional ETL). Now, with Mirroring, it\u0026rsquo;s like having a live camera pointed at the neighboring factory\u0026rsquo;s inventory. You see everything in real time on your monitor (OneLake), without moving anything.\nMirroring for SAP lets you replicate data from SAP systems (such as SAP Datasphere) into OneLake in real time, with a zero-ETL approach. That means SAP data is automatically mirrored in Delta Lake format inside Fabric, available for analysis without building complex extract, transform and load pipelines.\nThere are two offerings:\nMirroring for SAP with SAP Datasphere (GA): Available today. SAP Business Data Cloud Connect for Microsoft Fabric: Expected in 2026, it will enable bidirectional data sharing between SAP and Fabric. Integration with SAP has always been one of the most complex and expensive in enterprise environments. Mirroring removes the need to build and maintain dedicated ETL pipelines, drastically cutting development time and operating costs. The data is available for analysis in near real time.\nYou can create a unified data architecture that includes SAP data without the traditional complexity. That opens the door to integrated analysis between SAP transactional data and the rest of the company\u0026rsquo;s data, all in the same lakehouse 3.\n3.3. AI Function Transforms for Dataflow Gen2 Imagine a production line where you need to sort products by color, size and quality. Before, that required manual inspection. Now, you\u0026rsquo;ve installed smart sensors on the conveyor belt that do it automatically, without stopping production.\nDataflow Gen2 now lets you apply AI-based transformations directly in the data flow, through a low-code interface. The available functions include:\nSentiment analysis: Classifying text as positive, negative or neutral. Entity extraction: Identifying names of people, companies, places, etc. in text. Language detection: Automatically identifying the language of a text. These transformations are applied at scale, with no need to write Python code or call external APIs by hand.\nEnriching data with AI has always required a data scientist\u0026rsquo;s intervention. Now, a data engineer or analyst can do it directly in the transformation flow, democratizing the use of AI in the data pipeline.\nYou can build smarter data pipelines that don\u0026rsquo;t just move and transform data, but also enrich it with AI insights. This is especially useful for customer feedback analysis, document classification and other text-related tasks.\n4. SQL Database in Fabric (GA): The Integrated Operational Database Imagine you\u0026rsquo;re building a house. Before, you had to hire an architect to design the foundation, an engineer to calculate the structure and a foreman to carry out the work. Now, you get the house move-in ready, with all the infrastructure already installed, tested and approved. You only need to worry about the decor (your application).\nSQL Database in Fabric is a fully managed relational (OLTP) database, built on the Azure SQL Database engine but offered as a native SaaS experience inside Fabric. Reaching General Availability (GA) means it\u0026rsquo;s ready for production workloads, with a defined SLA and full support.\nThe main technical characteristics are:\nProvisioning in seconds: Creating a database takes seconds, not hours. Automatic scalability: Compute and storage scale automatically with demand. Replication to OneLake: Transactional data is automatically replicated to OneLake in Delta Lake format, available for near real-time analysis through the SQL Analytics Endpoint. Vector support: The database natively supports creating and querying vector embeddings, essential for generative AI applications that use semantic search and RAG. Security: Includes SQL Auditing (preview) and workspace-level Customer Managed Keys (CMK) (preview) for advanced compliance. Traditionally, you needed an operational database (Azure SQL DB) for your application and a data warehouse (Synapse, Fabric Warehouse) for analysis, with an ETL pipeline in between. SQL Database in Fabric removes that separation. You write to the operational database, and the data is already ready for analysis in OneLake, with no ETL.\nThis is a game changer for \u0026ldquo;translytical\u0026rdquo; (transactional + analytical) architectures. You can design simpler solutions where the application and the analytics share the same data source, reducing latency, costs and complexity. It\u0026rsquo;s ideal for AI applications that need fresh transactional data and analytical context at the same time.\n5. User Data Functions: Reusable, Governed Logic Think of User Data Functions as specialized tools in your toolbox. Instead of improvising a different solution every time you need to tighten a screw (run some logic), you have a professional screwdriver that works perfectly and can be used on any project.\nUser Data Functions let you create fully managed Python functions that can be reused across different Fabric items (Pipelines, Notebooks, Power BI, etc.). The news announced at Ignite 2025 was four important integrations:\nFabric Activator: Lets you invoke functions from event rules in Activator, processing events in real time. Variable Library: Integration with the Variable Library item to store settings, constants and environment variables the functions can access. Azure Key Vault: Support for securely accessing secrets (API keys, passwords) stored in Azure Key Vault. Cosmos DB: Support for reading and writing data in Cosmos DB databases (Fabric and Azure) directly from the functions. Instead of having business logic scattered and duplicated across dozens of notebooks and pipelines, you centralize that logic in reusable functions. That makes maintenance easier (you update it in one place), improves governance (you know where each piece of logic lives) and speeds up development (you reuse instead of rewriting).\nYou can build a library of corporate functions used across the whole organization. That promotes standardization, reduces redundancy and makes collaboration between teams easier. The integrations with Key Vault and Variable Library make sure the logic is secure and configurable.\n6. Partnership with Databricks: Real Interoperability with OneLake Imagine your company (Fabric) and the company next door (Databricks) always had to exchange documents by mail, which was slow and expensive. Now, you\u0026rsquo;ve built a direct door between the two offices. You can just walk through and grab the document you need, with no red tape.\nThe partnership between Microsoft and Databricks has been deepened to enable native interoperability with OneLake. This comes in three phases:\nDatabricks Mirroring to OneLake (GA): Available now. Data from the Databricks Unity Catalog can be mirrored into Fabric. Native OneLake reads from Databricks (Preview by the end of 2025): Databricks will be able to read data from OneLake directly through Unity Catalog, without copying it. A Databricks notebook will be able to query a Fabric lakehouse as if it were a local table. Native OneLake writes from Databricks (2026): Databricks will be able to write data directly to OneLake, with no additional storage needed. Many organizations use both Fabric and Databricks. Before, that meant duplicating data, building sync pipelines and dealing with data egress costs. Now, both can operate on the same copy of the data in OneLake, eliminating duplication and simplifying the architecture.\nYou have total freedom to choose the best tool for each job. You can use Databricks Spark for a heavy transformation and, the next minute, use Power BI in Fabric to visualize that data, without creating copies. That reduces storage costs, simplifies governance and speeds up development. It\u0026rsquo;s the fulfillment of the promise of a truly open data lakehouse.\nConclusion Microsoft Ignite 2025 delivered concrete new features that solve real problems. For us, data and AI architects, these updates are an opportunity to simplify our architectures, reduce redundancy and build smarter, more integrated solutions. Fabric is maturing quickly, and the vision of a unified intelligence platform is becoming reality.\nReferences [1] From Data Platform to Intelligence Platform: Introducing Microsoft Fabric IQ [2] What\u0026rsquo;s New for Fabric Data Agents at Ignite 2025 [3] Advancing Data Integration: Innovation in Data Factory in MS Fabric at Ignite 2025 [4] Announcing SQL database in Fabric (Generally Available) [5] What\u0026rsquo;s new in Fabric User Data Functions? Ignite 2025 edition [6] Microsoft and Databricks: Advancing Openness and Interoperability with OneLake ","permalink":"https://julianamarialop.github.io/en/articles/microsoft-fabric-at-ignite-2025-the-complete-guide-to-whats-new/","summary":"Microsoft Ignite 2025 wasn\u0026rsquo;t an event about future promises; it was about concrete deliveries that mark the maturity of Microsoft Fabric. The platform stopped being a collection of promising tools and became…","title":"Microsoft Fabric at Ignite 2025: The Complete Guide to What's New"},{"content":"Have you ever stopped to think that the way we use Artificial Intelligence in companies is a lot like the way a city evolves?\nIn the beginning, we built cities the traditional way: streets, buildings, traffic lights. Today, we dream of smart cities where everything connects and optimizes itself automatically. The same journey is happening with AI, and understanding the differences is crucial if you don\u0026rsquo;t want to fall behind.\nThere are three major approaches that define how a company deals with AI: Traditional, AI First, and AI-Centric. Let\u0026rsquo;s use the analogy of building a city to demystify this in an easy way.\nThe Traditional Approach: The City with Tech Accessories A Traditional company is like a city built the conventional way. You have your streets, buildings, traffic lights, and water and power systems all working perfectly. Once everything is in place, you decide to install a few tech \u0026ldquo;accessories.\u0026rdquo;\nFor example, you put security cameras at a few intersections or create an app so citizens can report potholes. The technology is useful, but it\u0026rsquo;s a layer applied on top of the existing infrastructure.\nIf you remove those cameras and the app, the city keeps working normally. AI here is a \u0026ldquo;nice extra,\u0026rdquo; an upgrade that improves a few things but isn\u0026rsquo;t essential to basic operations.\nMany companies are at this stage. They use AI tools to optimize one process or another, but AI isn\u0026rsquo;t the heart of the business. It\u0026rsquo;s an add-on, a temporary power-up.\nThe AI First Approach: The Smart Traffic Light District An AI First company is like planning a new neighborhood where the first question is: \u0026ldquo;How can AI solve the traffic problem here?\u0026rdquo; The solution is to build a system of smart traffic lights that adjust in real time based on the flow of cars.\nThat system is an AI First project because AI was the priority from the very beginning to solve a specific problem. The result is fundamentally better than fixed-timer traffic lights. You can\u0026rsquo;t separate the AI from this system, because the AI is the system.\nAI First companies don\u0026rsquo;t just use AI. They build their products and services around AI to solve specific problems in a superior way. AI isn\u0026rsquo;t an accessory; it\u0026rsquo;s the foundation of the solution.\nThe AI-Centric Approach: The City with a Central Nervous System This is where things get serious. An AI-Centric company is like an entire city designed with AI as its central nervous system. It\u0026rsquo;s not just about one specific problem, but about the whole ecosystem.\nThe smart traffic lights don\u0026rsquo;t just manage traffic; they also talk to the public transportation system to optimize bus routes. Sensors in the pipes use AI to predict leaks before they happen and dispatch maintenance crews automatically.\nThe power grid uses AI to predict consumption peaks and distribute energy more efficiently, preventing blackouts. The emergency system uses city data to predict where incidents are most likely and position crews proactively.\nIn this model, AI isn\u0026rsquo;t just a solution to an isolated problem. AI is the core that connects and optimizes every system in the city, turning it into a living organism that learns and adapts.\nWhy Does This Matter? The Transformation Journey Understanding this difference is crucial because the market is moving quickly toward the AI-Centric model. \u0026ldquo;AI-native\u0026rdquo; companies are redefining the speed and scale of business, achieving incredible results with lean teams.\nFor traditional companies, the journey to becoming AI-Centric isn\u0026rsquo;t just about technology. It\u0026rsquo;s a deep cultural transformation that can be broken down into four main phases.\nPhase 1: Installing the First Sensors. Start with low-risk pilot projects that deliver clear value. It\u0026rsquo;s like installing the first smart traffic lights at a problematic intersection. Educate the teams, organize the data, and create the first \u0026ldquo;quick wins.\u0026rdquo;\nPhase 2: Connecting the Neighborhoods. Scale the solutions that worked and start connecting the dots. Create an AI Center of Excellence, build a unified data platform, and expand use cases to other areas.\nPhase 3: The Central Nervous System. AI stops being a tool and becomes part of strategic decision-making. Systems start talking to each other, and leadership uses AI insights for long-term decisions.\nPhase 4: The City That Thinks. The AI-Centric mindset is fully rooted. The company uses AI not just to optimize what exists, but to create entirely new business models. Every product generates data that feeds and improves the AI models in a virtuous cycle.\nBuilding AI-Centric Analytics Projects: Beyond Dashboards Now let\u0026rsquo;s get to what really matters for those of us who work with data. How do you apply the AI-Centric mindset to building analytics projects from day one?\nThe fundamental difference lies in the architectural paradigm shift. In traditional projects, we build data pipelines that end in dashboards for human consumption. In AI-Centric projects, we build data ecosystems that optimize themselves and generate value autonomously.\nAI-Centric Architecture: Rethinking the Foundation Building an AI-Centric analytics project starts with a fundamentally different architecture. Instead of designing to \u0026ldquo;extract, transform, and load\u0026rdquo; data for later analysis, we design to \u0026ldquo;capture, learn, and act\u0026rdquo; in real time.\nThis means every component of the architecture is designed to feed machine learning models that don\u0026rsquo;t just analyze data but make decisions and take action. Data isn\u0026rsquo;t just stored; it\u0026rsquo;s contextualized, enriched, and prepared for continuous inference.\nThe practical difference is that, instead of building a data warehouse optimized for SQL queries, we build an intelligent data fabric where every data point carries metadata about its quality, lineage, sensitivity, and potential use in AI models. This lets the system itself identify opportunities for optimization and improvement without human intervention.\nAI-Centric Modernization: Transforming the Legacy When we modernize existing analytics projects toward an AI-Centric approach, we\u0026rsquo;re not just migrating technologies. We\u0026rsquo;re reprogramming the business logic so it\u0026rsquo;s executed by intelligent algorithms instead of static rules.\nThe process involves three fundamental transformations. First, Data as Code, where data transformations become trainable models that automatically adapt to changes in data patterns. Second, Analytics as a Service, where every insight generated becomes a service that other systems can consume programmatically. Third, Intelligence as Infrastructure, where the ability to learn and optimize is built into the data infrastructure itself.\nIn practice, this means an AI-Centric modernization project doesn\u0026rsquo;t just move data from a legacy system to the cloud. It rebuilds business intelligence as a network of interconnected models that continuously learn and adapt to changes in the market, in customers, and in internal processes.\nAI-Centric Assessment: Mapping the Potential for Intelligence Assessing an AI-Centric project goes far beyond evaluating data quality or tool maturity. It maps the organization\u0026rsquo;s potential for intelligence, identifying where and how AI can not just inform but fundamentally transform business processes.\nThe process starts with Decision Flow Mapping, where we identify not only which decisions are made, but how they connect to and influence one another. Next, we run a Data Density Analysis, evaluating not just the volume but the contextual richness and predictive potential of the available data.\nThe differentiator is the Organizational Learning Capacity Assessment, where we measure not only the technical ability to implement AI but the cultural ability to accept and adapt to decisions made by algorithms. This includes assessing tolerance for error, speed of adaptation, and the ability to create feedback loops between humans and machines.\nThe result isn\u0026rsquo;t a backlog of BI projects, but an intelligent transformation roadmap that prioritizes initiatives based on their potential to create virtuous cycles of learning and continuous optimization.\nConclusion: The Future Isn\u0026rsquo;t a Dashboard The transition to an AI-Centric culture is the most important journey companies will face this decade. It represents the shift from \u0026ldquo;having AI\u0026rdquo; as an accessory to \u0026ldquo;being AI\u0026rdquo; at the core. It\u0026rsquo;s not about replacing human intelligence but about amplifying it, freeing people from repetitive work so they can focus on creativity, strategy, and solving complex problems.\nThe future of data isn\u0026rsquo;t a prettier dashboard or a faster report. The future is an ecosystem of distributed intelligence that works in symbiosis with humans to create organizations that learn, adapt, and innovate at an unprecedented speed. Starting to build that future today is what will separate the companies that lead from the ones that fall behind.\nAnd you, are you ready to stop building just maps and start designing the nervous system of your smart city?\nReferences GFT Technologies. (2025). AI-centricity Starts with Us: Transforming Strategy, Delivery, and Operations. https://www.gft.com/us/en/about-us/ai-centricity\nMcKinsey. (2025, October 16). The AI-centric imperative: Navigating the next software frontier. https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/the-ai-centric-imperative-navigating-the-next-software-frontier\nGartner. (2025, February 26). Four Key Pillars of a Data-Centric Approach to AI. https://www.gartner.com/en/articles/data-centric-approach-to-ai\nVKTR. (2025, February 6). Going AI-First? The Strategy You Need to Compete and Win. https://www.vktr.com/ai-technology/ai-first-strategy-the-risks-rewards-and-realities-of-going-all-in/\n","permalink":"https://julianamarialop.github.io/en/articles/from-traditional-to-ai-centric-digital-transformation-as-a-smart-city/","summary":"Have you ever stopped to think that the way we use Artificial Intelligence in companies is a lot like the way a city evolves?","title":"From the Traditional Approach to AI-Centric: The Digital Transformation Journey as a Smart City"},{"content":"In the world of technology, few tasks are as critical and delicate as data migration. Whether to modernize systems, adopt the cloud or consolidate platforms, moving data from one place to another is a journey that demands planning, precision and a well-defined strategy. How that move is carried out can determine a project\u0026rsquo;s success, with a direct impact on business continuity.\nImagine you\u0026rsquo;re moving house. Your data is all your furniture, appliances and belongings accumulated over the years. The old system is the house you live in now, and the new system is that modern, spacious house you\u0026rsquo;re moving into. The big question is: how do you make the move? Do you pack everything at once and face one chaotic weekend to get it over with quickly? Or do you take things over little by little, keeping the old house running until the new one is completely ready?\nIn this article, we\u0026rsquo;ll explore the three main approaches to this moving journey: Big Bang, Trickle and Hybrid. We\u0026rsquo;ll use the house-moving metaphor to demystify each concept and help you choose the right path for your scenario.\nThe Migration Strategies in Detail Each migration approach has distinct characteristics, advantages and disadvantages. The ideal choice depends on the complexity of the environment, the tolerance for downtime and the business goals. Let\u0026rsquo;s see how each one works in the context of our move.\nBig Bang Migration: The Weekend Move The Big Bang approach is like deciding to do the whole move in a single weekend. You hire the truck, round up your friends, pack absolutely everything and move all your belongings in one go. During this process, your old house is completely unlivable. The furniture has been taken apart, the boxes are stacked up, and you can\u0026rsquo;t even make a cup of coffee. It\u0026rsquo;s a significant period of downtime, but when it\u0026rsquo;s over, the move is done.\nIn the data context, every record is transferred from the source system to the target system in a single operation. The old system is shut down, and after the migration only the new system is operational. Simplicity is its greatest advantage: there\u0026rsquo;s no need for complex synchronization or for keeping two environments running at the same time.\nThis strategy is best suited to small-scale or non-critical systems, where a scheduled downtime window is acceptable. It\u0026rsquo;s like moving out of a small apartment where you can get everything organized in a few hours. But for operations that can\u0026rsquo;t stop, this approach becomes unworkable. After all, you can\u0026rsquo;t leave your business unlivable for long.\nBig Bang Pros and Cons Advantages:\nSimpler planning: With just one migration event, the planning process can be more straightforward than with phased approaches. Lower long-term costs: Since the old and new systems don\u0026rsquo;t run in parallel for an extended period, there are savings on maintenance and support. Immediate full functionality: After the migration, the new system is fully operational, delivering every feature without a period of limited or partial capacity. Clear transition: It provides a clear, definitive cutover point, which can be easier for some organizations to manage and communicate internally. Disadvantages:\nHigh risk: If something goes wrong during the migration, it can have immediate, widespread effects across the entire organization. Hard to test and troubleshoot: Full testing is challenging when everything changes at once, and post-migration troubleshooting can be more complex. Potentially disruptive: The sudden change can disrupt business operations and may require significant downtime during the transition. User adaptation challenges: Users must adapt to the new system all at once, which can lead to resistance or a steep learning curve. Limited flexibility for adjustments: There\u0026rsquo;s little room to adjust the strategy midway. Problems found after go-live can be expensive and hard to fix. Trickle Migration: The Gradual Move Trickle Migration is like moving a little at a time, taking a few boxes a day in the trunk of your car. You start by transferring the least essential items: books, decor, out-of-season clothes. Meanwhile, you keep living normally in the old house. Little by little, the new house gets set up, and you keep both places functional at the same time until the transition is complete.\nIn the data world, this approach moves information in small portions, continuously, while the source and target systems run in parallel. Technologies such as Change Data Capture (CDC) work like a moving service that constantly watches what is being added or changed in the old house and replicates those changes to the new house in real time. If you buy a new sofa while you\u0026rsquo;re still in the old house, it goes straight to the new one.\nThis strategy is ideal for large, critical systems where downtime is unacceptable. It\u0026rsquo;s like moving out of a huge house full of valuable, fragile items. You can\u0026rsquo;t just throw everything into the truck and hope for the best. Although it\u0026rsquo;s more complex to implement, it minimizes risk and impact for the end user. The transition is so smooth that it often goes unnoticed. It\u0026rsquo;s as if you woke up one day and realized you were already living in the new house without having felt the process.\nTrickle Pros and Cons Advantages:\nReduced risk: Smaller, incremental changes reduce the risk of a major failure. Minimal disruption: Continuous operation of both systems minimizes interruptions to business processes. Easier to manage and test: Smaller changes are easier to manage, test and troubleshoot. Flexibility: It allows adjustments and refinements throughout the migration process. Operational continuity: Users can keep working normally throughout the process. Disadvantages:\nLonger duration: The process can take longer to complete compared with Big Bang. Higher costs: Running two systems in parallel for an extended period can incur additional costs. Synchronization complexity: Keeping data in sync between two systems can be complex and resource-intensive. Need for specialized tools: It requires technologies such as CDC and queuing systems to ensure real-time synchronization. Hybrid Migration: The Strategic Phased Move The Hybrid approach combines the best of both worlds, pairing the speed of Big Bang with the safety of Trickle Migration. It\u0026rsquo;s like planning your move in strategic stages. First, you do a lightning move of the non-essential items: those boxes of old books, kitchen utensils you rarely use, the guest room furniture. All of that goes at once, freeing up space and reducing the volume.\nMeanwhile, the critical items are transferred gradually, with all the care they need. Your bed, your work computer, important documents, valuables. You keep the old house functional for essential activities while getting the new one ready to receive them. When the final moment comes, the transition of the critical items is quick and planned, because most of the move has already been completed.\nIn the data context, the migration is split into phases. Usually, non-critical or historical data is moved first, perhaps in a Big Bang format, while transactional and critical data is synchronized continuously. It\u0026rsquo;s a very common strategy for cloud adoption. You can start by migrating development and staging environments (the least critical items in the house), and then move on to production with a more careful approach.\nThis approach offers the flexibility needed to handle complex environments. It ensures the most sensitive parts of the system are moved with the utmost care, while speeding up the overall migration process.\nHybrid Approach Pros and Cons Advantages:\nBalance between speed and safety: It combines the speed of Big Bang for non-critical data with the safety of Trickle for critical data. Reduced, planned downtime: Downtime is minimized and can be scheduled strategically. Adaptive flexibility: It lets you adjust the strategy as the project progresses and new challenges arise. Manageable risk: It spreads risk across different phases, avoiding a single catastrophic point of failure. Ideal for complex environments: Perfect for large-scale migrations, such as cloud adoption or system consolidation. Disadvantages:\nModerate complexity: It requires more sophisticated planning than Big Bang, although it\u0026rsquo;s less complex than pure Trickle. Need for coordination: It demands careful coordination between the different phases and teams involved. Intermediate costs: Costs fall between Big Bang and Trickle, depending on how long the parallel operation lasts. Requires technical expertise: It calls for knowledge of both fast migrations and continuous synchronization. _Types of data migration_\nConclusion Choosing the right data migration strategy is essential to ensure a safe, efficient transition aligned with your business goals. Just as there\u0026rsquo;s no single way to move house, there\u0026rsquo;s no single answer for data migration. The size of your house (data volume), the fragility of your belongings (system criticality) and the time you have available (downtime window) are the factors that will determine which approach makes the most sense. By understanding the nuances of each strategy, you\u0026rsquo;ll be better prepared to plan and execute your data journey successfully, making sure your most valuable assets arrive at their new home in perfect order.\nReferences 1.Brainhub. (2024). Big Bang vs Trickle Migration: Key Differences \u0026amp; Considerations. Available at: https://brainhub.eu/library/big-bang-migration-vs-trickle-migration\n2.NetApp. (2021). Data Migration Strategy: Goals, Migration Types, and Phases. Available at: https://bluexp.netapp.com/blog/clc-blg-data-migration-strategy-goals-migration-types-and-phases\n3.DataPao. (2023). Big Bang vs Trickle Data Migration Approach. Available at: https://datapao.com/big-bang-vs-trickle-data-migration/\n4.Provalet. (2025). Top 7 Data Migration Strategies Every Business Must Know. Available at: https://www.provalet.io/guides-posts/data-migration-strategies\n5.Varonis. (2023). Guia de migração de dados: sucesso estratégico e práticas recomendadas (Data migration guide: strategic success and best practices). Available at: https://www.varonis.com/pt-br/blog/guia-de-migracao-de-dados-sucesso-estrategico-e-praticas-recomendadas\n","permalink":"https://julianamarialop.github.io/en/articles/migrating-data-like-moving-house-which-strategy-to-choose/","summary":"In the world of technology, few tasks are as critical and delicate as data migration. Whether to modernize systems, adopt the cloud or consolidate platforms, moving data from one place to another is a…","title":"Migrating Data Like Moving House: Which Strategy Should You Choose?"},{"content":"Imagine you\u0026rsquo;re a Pokémon trainer, but instead of catching Pikachus and Charizards, you\u0026rsquo;re building a team of specialized AI agents. Each agent has its own unique abilities, can learn new moves (tools), and can work together with other agents to solve complex challenges. That, in essence, is the world of the Microsoft Agent Framework, a universe where you can train, coordinate, and evolve your AI agents to take on any business mission.\nAnnounced on October 1, 2025 [1], the Microsoft Agent Framework is like the ultimate evolution of the AI Pokémon world. It brings together two powerful projects from Microsoft , Semantic Kernel (the equivalent of the classic, reliable first-generation Pokémon) and AutoGen (the experimental, innovative Pokémon of the newer generations), into a single unified Pokédex. The result is an open-source framework that lets developers build anything from simple agents to complex multi-agent systems, all with the stability needed for large-scale production battles.\nIn this article, we\u0026rsquo;ll explore this new universe, understand how the training and battle mechanics work, and take the first steps toward making you a true Agentic AI Pokémon Master.\nThe Problem: When Your Pokémon Team Is Out of Sync Before the Microsoft Agent Framework, developers faced a scenario a lot like owning a collection of powerful Pokémon with no clear battle strategy and no proper Pokémon Center to train them. The main challenges included:\nFramework Fragmentation: It was like having Pokémon from different generations and regions, each with its own incompatible battle system. Multiple frameworks coexisted, each with its own rules, which made it hard to build a cohesive team.\nThe Gap Between Training and Real Battle: What worked in local training (like battling Professor Oak) rarely carried over to production gym battles. Developers had to completely reconfigure their teams for scalable cloud environments.\nMissing Features for Professional Battles: Many tools were like makeshift Pokémon Centers. They worked for healing your Pokémon, but they lacked the advanced features needed for serious competition. Missing were capabilities like observability (knowing exactly what each Pokémon did during the battle), compliance (making sure the tournament rules were followed), and durability (making sure a battle can be paused and resumed without losing progress).\nMicrosoft, with its experience creating both stable, reliable Pokémon (Semantic Kernel) and experimental, innovative ones (AutoGen), was in the perfect position to create the ultimate Pokédex.\nThe Solution: The Unified AI Pokédex The Microsoft Agent Framework is your complete Pokédex and your advanced Pokémon Center, all in one place [1, 2]. It\u0026rsquo;s an open-source development kit that lets you catch, train, and coordinate AI agents (your digital Pokémon) to solve complex problems.\nWhat Makes This Pokédex Special? Just as the Pokédex records information about every Pokémon and its evolutions, the Agent Framework unifies the best of two worlds:\nSemantic Kernel - The first-generation Pokémon (Bulbasaur, Charmander, Squirtle), reliable, battle-tested, and with well-documented moves AutoGen - The experimental Pokémon of the newer generations, with innovative abilities and creative battle strategies Microsoft Agent Framework - The final evolution that combines the reliability of the classics with the innovation of the new ones Let\u0026rsquo;s compare these three \u0026ldquo;battle systems\u0026rdquo; and understand how each one works:\nSemantic Kernel: The Classic First-Generation Pokémon Think of Semantic Kernel as the original Pokémon: Bulbasaur, Charmander, and Squirtle. They\u0026rsquo;re reliable, battle-tested, and everyone knows their moves.\nKey characteristics:\nFocus: Stable Pokémon with tested moves and reliable Pokémon Centers Interoperability: You can trade Pokémon between some regions using protocols like MCP, A2A, and OpenAPI Memory System: Well-organized storage boxes (like Elasticsearch and MongoDB) Battle Style: Structured, predictable battles where you know exactly what\u0026rsquo;s going to happen Features: Battle telemetry and performance analysis for continuous improvement AutoGen: The Experimental Pokémon of the New Generations AutoGen is like discovering rare, experimental Pokémon with abilities nobody has seen before. They\u0026rsquo;re innovative, but they\u0026rsquo;re still in the testing phase.\nKey characteristics:\nFocus: Experimental Pokémon with innovative battle strategies straight out of the research lab Interoperability: Limited trading between regions; some Pokémon don\u0026rsquo;t work in every area Memory System: Basic storage with a few external options for saving experiences Battle Style: Dynamic, improvised battles where the Pokémon decide on tactics in real time Features: Minimal features, with more focus on innovation than on infrastructure Microsoft Agent Framework: The Ultimate Evolution The Agent Framework is like having a complete Pokédex that works with ALL Pokémon, from every generation, with the best of each world.\nKey characteristics:\nFocus: A unified Pokédex with every type of Pokémon and strategy; you pick what works best Interoperability: Universal trading; every Pokémon works in any region, no restrictions Memory System: An advanced box system with cloud sync and automatic backup Battle Style: You choose! Structured OR dynamic battles, depending on what the mission demands Features: A full Pokémon Center with advanced analytics, an approval system, and recovery of interrupted battles Important: The Agent Framework doesn\u0026rsquo;t replace your favorite Semantic Kernel or AutoGen Pokémon. It simply gives you a better way to organize them and get them working together.\nThe Four Gyms of the Agent Framework To become an AI Pokémon Master, you need to conquer the four gyms (pillars) of the Microsoft Agent Framework [1]:\nGym 1: Universal Communication (Open Standards) Imagine your Pokémon can talk to Pokémon from other regions, regardless of language or battle system. This gym teaches your agents to work with universal standards.\nModel Context Protocol (MCP) works like the universal translator of the Pokémon world. Your agents can discover and use external tools automatically, as if they were learning new TM (Technical Machine) moves instantly.\nAgent-to-Agent (A2A) is like the Pokémon trading system, but for real-time collaboration. One agent can ask another for help, even if they\u0026rsquo;re in different \u0026ldquo;regions\u0026rdquo; (runtimes).\nOpenAPI is like having access to every possible move through a single interface. Any REST API can be used as a special move by your agents.\nGym 2: From the Lab to the Championship (Research-to-Production Pipeline) This gym trains your Pokémon to evolve from practice battles to professional tournaments. The framework supports different battle strategies:\nAvailable Battle Strategies:\nSequential Battle: Your Pokémon attack one at a time, following a set order Double Battle: Multiple Pokémon attack at the same time Team Battle (Group Chat): The whole team discusses strategy before every move Relay (Handoff): One Pokémon passes the baton to another when needed AF Labs is like Professor Oak\u0026rsquo;s lab, where new Pokémon and experimental moves are tested before being officially released.\nGym 3: Catching and Customizing (Extensibility) In this gym, you learn to catch new Pokémon (agents) and teach them custom moves. The framework is 100% open source (MIT license) [3], like a region where every trainer can share their discoveries.\nEnterprise Connectors are like special Poké Balls (Ultra Ball, Master Ball) that let you catch rare Pokémon from enterprise systems like Microsoft Graph, SharePoint, and Oracle.\nMemory Modules work like the PC storage boxes: you can choose between Redis, Pinecone, Weaviate, and other systems to store your Pokémon and their experiences.\nDeclarative Agents are like Pokémon registration cards in YAML or JSON format, which can be shared with other trainers.\nGym 4: Professional Tournaments (Production Readiness) The final gym gets your team ready for high-level battles in enterprise environments, with every feature needed for serious competition.\nObservability is like having a replay camera that records every move of the battle. The OpenTelemetry integration lets you analyze exactly what each Pokémon did and why.\nThe Cloud Pokémon Center (Azure AI Foundry) offers professional care with advanced security, access control, and data protection.\nBattle Durability ensures that if a battle is interrupted (a power outage, for example), you can pick it up exactly where you left off, without losing progress.\nTrainer Approval (Human-in-the-Loop) lets certain critical actions run only after your explicit approval, like using a rare item or making an important trade.\nSimple Battles vs. Complex Tournaments The Agent Framework offers two main game modes [1, 2]:\nSingle Battle Mode (Agent Orchestration) In this mode, you send a Pokémon (agent) into battle and it autonomously decides which moves to use based on the situation. It\u0026rsquo;s ideal for unpredictable challenges where creativity and adaptability are essential.\nWhen to use it: Customer support, research assistants, personalized tutoring, situations where every battle is different.\nStructured Tournament Mode (Workflow Orchestration) Here, you define a detailed battle strategy in advance, like a game plan in a professional tournament. Each Pokémon knows exactly when to take the field and which move to make.\nWhen to use it: Structured business processes, approval pipelines, automated audits, situations where consistency and predictability are crucial.\nEvolving Your Old Pokémon If you already have Pokémon trained in Semantic Kernel or AutoGen, you don\u0026rsquo;t need to start from scratch [1]. It\u0026rsquo;s like transferring your Pokémon from an old game to the latest version.\nTransferring from Semantic Kernel The migration is simple and straightforward. Your Kernels and Plugins evolve into Agents and Tools. Existing vector store integrations keep working, and your Pokémon keep every move they\u0026rsquo;ve learned.\nTransferring from AutoGen AutoGen\u0026rsquo;s battle patterns, like GroupChat and GraphFlow, are now unified under the Workflow concept. Your AssistantAgent naturally evolves into a ChatAgent, keeping all of its capabilities.\nTutorial: Catching and Training Your First AI Pokémon Now for the hands-on part! I\u0026rsquo;ll walk you step by step through catching and training your first AI agent with the Microsoft Agent Framework.\nStep 1: Get Your Trainer Kit Ready Before you start your journey, you need to set up the environment. It\u0026rsquo;s like buying your backpack and your starter items.\nFor Python:\n# Instalar o framework completo pip install agent-framework --pre # Instalar as dependências do Azure pip install azure-identity azure-ai-openai For .NET:\n# Instalar o pacote principal dotnet add package Microsoft.Agents.AI --prerelease # Instalar os pacotes do Azure dotnet add package Azure.AI.OpenAI dotnet add package Azure.Identity Step 2: Set Up Your Credentials You need access to Azure OpenAI (the equivalent of your Trainer ID and credentials). There are two ways to do this:\nOption A: Use the Azure CLI (Recommended to get started)\n# Fazer login no Azure az login Option B: Use environment variables\n# No Linux/Mac export AZURE_OPENAI_ENDPOINT=\u0026#34;https://seu-recurso.openai.azure.com/\u0026#34; export AZURE_OPENAI_DEPLOYMENT_NAME=\u0026#34;gpt-4\u0026#34; export AZURE_OPENAI_API_VERSION=\u0026#34;2024-08-01-preview\u0026#34; export AZURE_OPENAI_API_KEY=\u0026#34;sua-chave-api\u0026#34; # No Windows (PowerShell) $env:AZURE_OPENAI_ENDPOINT=\u0026#34;https://seu-recurso.openai.azure.com/\u0026#34; $env:AZURE_OPENAI_DEPLOYMENT_NAME=\u0026#34;gpt-4\u0026#34; $env:AZURE_OPENAI_API_VERSION=\u0026#34;2024-08-01-preview\u0026#34; $env:AZURE_OPENAI_API_KEY=\u0026#34;sua-chave-api\u0026#34; Step 3: Catch Your First Pokémon (Create a Basic Agent) Let\u0026rsquo;s create our first agent, a \u0026ldquo;Poet Pikachu\u0026rdquo; that writes haikus about technology.\nPython:\n# arquivo: meu_primeiro_agente.py import asyncio from agent_framework.azure import AzureOpenAIResponsesClient from azure.identity import AzureCliCredential async def main(): # Criar o cliente Azure OpenAI # É como abrir a sua Pokédex e conectar ao servidor client = AzureOpenAIResponsesClient( credential=AzureCliCredential(), # Se preferir usar variáveis de ambiente, descomente: # endpoint=os.environ[\u0026#34;AZURE_OPENAI_ENDPOINT\u0026#34;], # deployment_name=os.environ[\u0026#34;AZURE_OPENAI_DEPLOYMENT_NAME\u0026#34;], ) # Capturar (criar) o agente # É como lançar uma Poké Ball e dar um nome ao seu Pokémon pikachu_poeta = client.create_agent( name=\u0026#34;PikachuPoeta\u0026#34;, instructions=\u0026#34;\u0026#34;\u0026#34;És um Pikachu que adora poesia. Escreves haikus criativos sobre tecnologia e IA. Sempre terminas as tuas respostas com \u0026#39;Pika Pika!\u0026#39;\u0026#34;\u0026#34;\u0026#34; ) # Usar o agente (batalhar) print(\u0026#34;Treinador: Pikachu, escreve um haiku sobre IA!\u0026#34;) print(\u0026#34;Pikachu Poeta:\u0026#34;) resposta = await pikachu_poeta.run( \u0026#34;Escreve um haiku sobre inteligência artificial\u0026#34; ) print(resposta) # Iniciar a aventura if __name__ == \u0026#34;__main__\u0026#34;: asyncio.run(main()) Run it:\npython meu_primeiro_agente.py Expected result:\nTreinador: Pikachu, escreve um haiku sobre IA! Pikachu Poeta: Bits dançam na luz Máquinas que aprendem, sonham Futuro desperta Pika Pika! Step 4: Teach Your Pokémon New Moves (Add Tools) Now let\u0026rsquo;s teach our Pikachu a special move: the ability to calculate numbers (as if it were an Electric-type attack).\nPython\n# arquivo: agente_com_ferramentas.py import asyncio from agent_framework.azure import AzureOpenAIResponsesClient from azure.identity import AzureCliCredential # Definir um movimento (ferramenta) # É como ensinar um TM (Technical Machine) ao seu Pokémon def calcular_potencia(base: int, expoente: int) -\u0026gt; int: \u0026#34;\u0026#34;\u0026#34; Calcula a potência de um número. Args: base: O número base expoente: O expoente Returns: O resultado de base elevado a expoente \u0026#34;\u0026#34;\u0026#34; return base ** expoente async def main(): client = AzureOpenAIResponsesClient( credential=AzureCliCredential(), ) # Criar agente com ferramentas # É como dar ao Pikachu acesso a movimentos especiais pikachu_matematico = client.create_agent( name=\u0026#34;PikachuMatematico\u0026#34;, instructions=\u0026#34;\u0026#34;\u0026#34;És um Pikachu especialista em matemática. Quando precisares de calcular potências, usa a ferramenta disponível. Explica os cálculos de forma simples e termina com \u0026#39;Pika Pika!\u0026#39;\u0026#34;\u0026#34;\u0026#34;, tools=[calcular_potencia] # Ensinar o movimento ) # Testar o movimento print(\u0026#34;Treinador: Pikachu, quanto é 2 elevado a 10?\u0026#34;) print(\u0026#34;Pikachu Matemático:\u0026#34;) resposta = await pikachu_matematico.run( \u0026#34;Quanto é 2 elevado a 10? Usa a ferramenta para calcular.\u0026#34; ) print(resposta) if __name__ == \u0026#34;__main__\u0026#34;: asyncio.run(main()) Expected result:\nTreinador: Pikachu, quanto é 2 elevado a 10? Pikachu Matemático: Vou calcular isso para ti! [Usando a ferramenta calcular_potencia...] 2 elevado a 10 é igual a 1024! É como ter 1024 Pikachus juntos! Pika Pika! Step 5: Build a Pokémon Team (Multi-Agent Workflow) Now let\u0026rsquo;s build a team with multiple Pokémon working together, like in a double battle.\nPython\n# arquivo: equipe_pokemon.py import asyncio from agent_framework.azure import AzureOpenAIResponsesClient from azure.identity import AzureCliCredential async def main(): client = AzureOpenAIResponsesClient( credential=AzureCliCredential(), ) # Criar uma equipe de 3 Pokémons especialistas # Pokémon 1: Especialista em Pesquisa bulbasaur_pesquisador = client.create_agent( name=\u0026#34;BulbasaurPesquisador\u0026#34;, instructions=\u0026#34;\u0026#34;\u0026#34;És um Bulbasaur especialista em pesquisa. Recolhes informações e factos sobre qualquer tópico. És metódico e detalhista.\u0026#34;\u0026#34;\u0026#34; ) # Pokémon 2: Especialista em Análise charmander_analista = client.create_agent( name=\u0026#34;CharmanderAnalista\u0026#34;, instructions=\u0026#34;\u0026#34;\u0026#34;És um Charmander especialista em análise. Analisas informações e identificas padrões importantes. És perspicaz e objetivo.\u0026#34;\u0026#34;\u0026#34; ) # Pokémon 3: Especialista em Comunicação squirtle_comunicador = client.create_agent( name=\u0026#34;SquirtleComunicador\u0026#34;, instructions=\u0026#34;\u0026#34;\u0026#34;És um Squirtle especialista em comunicação. Transformas análises técnicas em explicações simples e claras. És amigável e didático.\u0026#34;\u0026#34;\u0026#34; ) # Missão em equipe: Explicar o Microsoft Agent Framework topico = \u0026#34;Microsoft Agent Framework\u0026#34; print(f\u0026#34;Missão da Equipe: Explicar \u0026#39;{topico}\u0026#39; de forma simples\\n\u0026#34;) # Fase 1: Pesquisa (Bulbasaur) print(\u0026#34;Bulbasaur está a pesquisar...\u0026#34;) pesquisa = await bulbasaur_pesquisador.run( f\u0026#34;Recolhe os 3 factos mais importantes sobre {topico}\u0026#34; ) print(f\u0026#34;Bulbasaur: {pesquisa}\\n\u0026#34;) # Fase 2: Análise (Charmander) print(\u0026#34;Charmander está a analisar...\u0026#34;) analise = await charmander_analista.run( f\u0026#34;Analisa estes factos e identifica o benefício principal: {pesquisa}\u0026#34; ) print(f\u0026#34;Charmander: {analise}\\n\u0026#34;) # Fase 3: Comunicação (Squirtle) print(\u0026#34;Squirtle está a preparar a explicação...\u0026#34;) explicacao = await squirtle_comunicador.run( f\u0026#34;Explica isto de forma simples para iniciantes: {analise}\u0026#34; ) print(f\u0026#34;Squirtle: {explicacao}\\n\u0026#34;) print(\u0026#34;Missão concluída! A equipe trabalhou em perfeita harmonia!\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: asyncio.run(main()) Step 6: Save Your Progress (Persistent Memory) Just like in a Pokémon game, you want to save your progress. Let\u0026rsquo;s add memory to the agent.\nPython\n# arquivo: agente_com_memoria.py import asyncio from agent_framework.azure import AzureOpenAIResponsesClient from azure.identity import AzureCliCredential async def main(): client = AzureOpenAIResponsesClient( credential=AzureCliCredential(), ) # Criar agente com memória de conversa pikachu_amigo = client.create_agent( name=\u0026#34;PikachuAmigo\u0026#34;, instructions=\u0026#34;\u0026#34;\u0026#34;És um Pikachu amigável que se lembra das conversas. Lembras-te do nome do treinador e dos tópicos discutidos. És prestativo e carinhoso.\u0026#34;\u0026#34;\u0026#34; ) # Criar um thread (sessão de conversa) # É como iniciar uma nova aventura guardada thread = client.create_thread() # Conversa 1 print(\u0026#34;Treinador: Olá! O meu nome é Ash.\u0026#34;) resposta1 = await pikachu_amigo.run( \u0026#34;Olá! O meu nome é Ash.\u0026#34;, thread_id=thread.id ) print(f\u0026#34;Pikachu: {resposta1}\\n\u0026#34;) # Conversa 2 (na mesma sessão) print(\u0026#34;Treinador: Qual é o meu nome?\u0026#34;) resposta2 = await pikachu_amigo.run( \u0026#34;Qual é o meu nome?\u0026#34;, thread_id=thread.id ) print(f\u0026#34;Pikachu: {resposta2}\\n\u0026#34;) # O Pikachu deve lembrar-se que o nome é Ash! if __name__ == \u0026#34;__main__\u0026#34;: asyncio.run(main()) Step 7: Advanced Mode - Create an Agent with Human Approval For critical operations, you may want to approve the agent\u0026rsquo;s actions before they run.\nPython\n# arquivo: agente_com_aprovacao.py import asyncio from agent_framework.azure import AzureOpenAIResponsesClient from azure.identity import AzureCliCredential def enviar_email(destinatario: str, assunto: str, mensagem: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34; Envia um email (simulado). Esta ferramenta requer aprovação humana. \u0026#34;\u0026#34;\u0026#34; # Em produção, aqui estaria o código real de envio return f\u0026#34;Email enviado para {destinatario} com assunto \u0026#39;{assunto}\u0026#39;\u0026#34; async def main(): client = AzureOpenAIResponsesClient( credential=AzureCliCredential(), ) # Criar agente com ferramenta que requer aprovação agente_email = client.create_agent( name=\u0026#34;AgenteEmail\u0026#34;, instructions=\u0026#34;\u0026#34;\u0026#34;És um assistente que ajuda a redigir e enviar emails. Sempre pedes confirmação antes de enviar.\u0026#34;\u0026#34;\u0026#34;, tools=[enviar_email], # Configurar aprovação humana para a ferramenta tool_approval_required=True ) print(\u0026#34;Treinador: Envia um email para o Professor Carvalho\u0026#34;) print(\u0026#34;Agente: Vou preparar o email e pedir a tua aprovação...\\n\u0026#34;) # O agente irá preparar a ação mas esperar pela aprovação # Em produção, isto integraria com um sistema de aprovação resposta = await agente_email.run( \u0026#34;\u0026#34;\u0026#34;Envia um email para professor.carvalho@pokemon.com com o assunto \u0026#39;Relatório de Progresso\u0026#39; e a mensagem \u0026#39;Capturei 10 novos Pokémons esta semana!\u0026#39;\u0026#34;\u0026#34;\u0026#34; ) print(f\u0026#34;Agente: {resposta}\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: asyncio.run(main()) Resources to Continue Your Journey Now that you\u0026rsquo;ve caught and trained your first AI Pokémon, here are the resources to keep evolving:\nOfficial Documentation:\nComplete guide: https://learn.microsoft.com/en-us/agent-framework/ Step-by-step tutorials: https://learn.microsoft.com/en-us/agent-framework/tutorials/ GitHub Repository:\nSource code and samples: https://github.com/microsoft/agent-framework More than 1.6k stars and an active community [3] Development Tools:\nVS Code AI Toolkit: an IDE for developing and testing agents locally Azure AI Foundry: a professional hosting platform for your agents DevUI: an interactive interface for debugging workflows Conclusion: Your Journey as an AI Pokémon Master Starts Now The Microsoft Agent Framework is more than a technical tool. It\u0026rsquo;s your passport to becoming a true Agentic AI Pokémon Master. Just as Ash Ketchum started his journey with a single Pikachu and went on to train a diverse team of powerful Pokémon, you can also start with a simple agent and gradually build sophisticated multi-agent systems.\nBringing Semantic Kernel and AutoGen together in this unified framework is like having access to Pokémon from every generation and region, with the guarantee that they\u0026rsquo;ll all work in harmony. With open standards, enterprise readiness, and a vibrant community of \u0026ldquo;trainers\u0026rdquo; (developers) sharing their discoveries, the future of agentic AI has never looked more promising.\nBig companies like KPMG, BMW, Fujitsu, and others are already training their AI Pokémon teams to solve real challenges [1]. Now it\u0026rsquo;s your turn to set out on this journey. Catch your first agent, teach it new moves, assemble your dream team, and get ready for the battles ahead.\nRemember: every Pokémon Master started with a single Pokémon and a lot of determination. Your Agentic AI adventure starts today.\nGotta catch \u0026rsquo;em all! 🎮⚡\nReferences [1] Microsoft. (2025, October 1). Introducing Microsoft Agent Framework: The Open-Source Engine for Agentic AI Apps. Azure AI Foundry Blog. https://devblogs.microsoft.com/foundry/introducing-microsoft-agent-framework-the-open-source-engine-for-agentic-ai-apps/\n[2] Microsoft. (2025). Introduction to Microsoft Agent Framework. Microsoft Learn. https://learn.microsoft.com/en-us/agent-framework/overview/agent-framework-overview\n[3] Microsoft. (2025). microsoft/agent-framework. GitHub. https://github.com/microsoft/agent-framework\n","permalink":"https://julianamarialop.github.io/en/articles/microsoft-agent-framework-becoming-an-ai-pokemon-master/","summary":"Imagine you\u0026rsquo;re a Pokémon trainer, but instead of catching Pikachus and Charizards, you\u0026rsquo;re building a team of specialized AI agents. Each agent has its own unique abilities, can learn…","title":"Microsoft Agent Framework: Becoming an AI Pokemon Master"},{"content":"Remember Super Mario Bros.? That feeling of starting small, slowly uncovering the game\u0026rsquo;s secrets, and leveling up until you completely master the universe? Digital transformation in companies follows a very similar logic.\nYou start with manual processes and scattered spreadsheets, you discover tools that make your life easier, and little by little you realize you can build solutions you never imagined were possible. Microsoft Power Platform is exactly that: a set of tools that enables this gradual but steady evolution.\nAnd just like in the Mushroom Kingdom, each component of the platform has its own unique personality and specific role. Let\u0026rsquo;s meet the cast of characters that\u0026rsquo;s going to transform your company.\nThe Core Components: Meeting the Team _Power Platform Components_\nPower Apps: Mario Power Apps is like Mario himself. Versatile, adaptable, and able to take on any challenge. Just as Mario can be a plumber, a doctor, a kart racer, or a tennis player, Power Apps adapts to any business need.\nNeed an inventory control app? Power Apps becomes \u0026ldquo;Warehouse Mario.\u0026rdquo; Want an approval system? It becomes \u0026ldquo;Manager Mario.\u0026rdquo; That versatility is what makes Power Apps the star of the platform. It literally builds solutions for any world (department) in your company.\nAnd just as Mario levels up with power-ups, your Power Apps applications can grow in complexity as you master the tool.\nPower Automate: Luigi Power Automate is the platform\u0026rsquo;s Luigi. Always there, extremely reliable, and doing the heavy lifting behind the scenes. While Mario (Power Apps) is on the front line solving problems, Luigi (Power Automate) is automatically taking care of all the repetitive tasks.\nWhen someone fills out a form in Power Apps, Power Automate automatically sends emails, updates spreadsheets, creates tickets, and notifies the right people. It\u0026rsquo;s like having Luigi always by your side, anticipating what needs to be done and doing it without you even having to ask.\nAnd just as Luigi sometimes outshines Mario in certain situations, Power Automate can be the real hero when it comes to operational efficiency.\nCopilot Studio: Yoshi Copilot Studio is like Yoshi. A smart companion that doesn\u0026rsquo;t just help you, but adds completely new abilities to your journey. Just as Yoshi can flutter, eat enemies, and pull off double jumps, Copilot Studio adds artificial intelligence to your solutions.\nIt understands natural language, learns from interactions, and can carry out complex tasks autonomously. It\u0026rsquo;s like having a Yoshi that doesn\u0026rsquo;t just carry you, but also talks, analyzes situations, and makes smart decisions.\nAnd just as there are different kinds of Yoshi (green, blue, red), you can create different kinds of agents in Copilot Studio, each specialized in a specific area of the business.\nPower Pages: Princess Peach Power Pages is like Princess Peach. Elegant, approachable, and the public face of the corporate Mushroom Kingdom. It\u0026rsquo;s through Power Pages that customers and external partners interact with your company in an organized, secure way.\nJust as Princess Peach rules the kingdom and welcomes visitors to the castle, Power Pages creates portals and websites that welcome external users, offering a polished, professional experience. It\u0026rsquo;s where the important meetings between your company and the outside world take place.\nAnd just as Princess Peach has her own powers (especially in the more recent games), Power Pages isn\u0026rsquo;t just a \u0026ldquo;pretty storefront.\u0026rdquo; It has robust security, customization, and integration features.\nPower BI: Toad Power BI is like Toad. Small in size, but huge in wisdom. He\u0026rsquo;s the one with access to all the information in the kingdom who can tell you where to go, what to do, and which paths are best.\nJust as Toad shows up at crucial moments with important tips (\u0026ldquo;Your Princess is in another castle!\u0026rdquo;), Power BI shows up with essential insights about your data. It analyzes everything happening in the company and gives you the information you need to make smart decisions.\nAnd just as Toad knows every secret and shortcut in the Mushroom Kingdom, Power BI reveals hidden patterns and opportunities you didn\u0026rsquo;t even know existed in your data.\nThe Supporting Technologies: The Essential Power-Ups _Power Platform: Supporting Technologies_\nData Connectors: The Warp Pipes Data Connectors are like Mario\u0026rsquo;s famous pipes. They transport you instantly between different worlds (systems). Just as you go into a pipe in World 1-1 and come out in the Warp Zone, connectors let your data flow between Salesforce, SAP, Excel, and hundreds of other systems.\nThat\u0026rsquo;s what lets you break down the barriers between departments and create a unified experience, even when the data is scattered across completely different systems.\nAI Builder: The Super Mushroom of Intelligence AI Builder is like the Super Mushroom. It turns your simple applications into \u0026ldquo;super\u0026rdquo; versions with artificial intelligence capabilities. Just as little Mario becomes Super Mario, your regular applications gain AI superpowers.\nDocument processing, image recognition, sentiment analysis. All of that without you having to be a data scientist. It\u0026rsquo;s literally grabbing the power-up and gaining new abilities instantly.\nMicrosoft Dataverse: The Main Castle Microsoft Dataverse is like Princess Peach\u0026rsquo;s Castle. The safe, organized center where everything important is kept. It\u0026rsquo;s the solid, reliable foundation that supports the whole kingdom.\nJust as the castle has specific rooms for different purposes, Dataverse organizes your data into logical structures, with robust security and controlled access. It\u0026rsquo;s where all the characters (components) meet and share information.\nPower Fx: The Language of the Kingdom Power Fx is like the universal language of the Mushroom Kingdom. Every character understands it and can use it. Based on the Excel formulas everyone already knows, it\u0026rsquo;s as if everyone spoke the same language, making communication between different worlds easier.\nManaged Environments: Lakitu Managed Environments are like Lakitu. That cloud with eyes that watches everything from above and makes sure the rules of the game are followed. It provides smart governance without spoiling the fun.\nJust as Lakitu monitors the races in Mario Kart and steps in when needed, Managed Environments monitor platform usage and apply security policies automatically.\nThe Final Boss: Governance Without Stifling Creativity Here we reach the Bowser of the story: the governance challenge. When everyone in the company discovers the Power Platform power-ups, things can turn into chaos if there are no clear rules.\nBut unlike in the games, the goal here isn\u0026rsquo;t to defeat Bowser, but to turn him into an ally. Governance done right is like having Bowser on your side. Powerful and protective, but letting everyone have fun within the rules.\nThe Center of Excellence is like having a Royal Council: It sets the rules of the kingdom (usage policies). It trains new adventurers (upskilling). It monitors activity (observability). It makes sure every world connects harmoniously.\nPractical governance strategies: Create separate environments for development, testing, and production. Configure DLP policies that block risky connections between sensitive data and external connectors. Use the admin tools to keep track of what\u0026rsquo;s being built and how it\u0026rsquo;s being used. Invest in continuous training so people build better solutions from the start.\nThe Future: New Worlds to Explore The trends on the way are like discovering there are new worlds beyond the traditional 8. Intent-driven development is like having a game that adapts to your play style. You simply describe what you want to do and the system builds the perfect level for you.\nPlan Designer already does this. You describe a business problem and a team of AI agents collaborates to build the solution. Autonomous agents are like having NPCs that actually evolve and learn. Instead of always repeating the same lines, they understand context, make decisions, and grow with experience.\nBy 2025, an estimated 70% of new enterprise applications will be developed using low-code or no-code technologies. It\u0026rsquo;s not a trend; it\u0026rsquo;s a reality already knocking on the door.\nHow to Start Your Adventure The question isn\u0026rsquo;t whether your company is going to join this game, but when it\u0026rsquo;s going to pick up the controller and start the adventure.\nStart small. Identify a simple process that can be improved and use it as a pilot. It\u0026rsquo;s like starting in World 1-1 before moving on to the tougher challenges.\nBuild an internal community of people interested in learning and sharing knowledge. It\u0026rsquo;s like forming a guild of players who help each other out.\nInvest in governance from the start. It\u0026rsquo;s easier to establish good practices at the beginning than to try to clean up the mess later. As the saying goes, an ounce of prevention is worth a pound of cure.\nAnd above all, remember: every expert was a beginner once. Even Mario started out as a simple plumber.\nGame Over? Never! Power Platform represents a fundamental shift. We\u0026rsquo;re moving from a world where only \u0026ldquo;professional developers\u0026rdquo; could build solutions to a world where anyone can be the Mario of their own digital transformation story.\nAnd just like in the best games, the fun lies as much in playing as in discovering new secrets, coming up with new strategies, and helping other players complete their journeys.\nReady Player One? Let\u0026rsquo;s-a-go! 🍄🚀\nReferences Microsoft. (2024). Microsoft Power Platform. Microsoft. https://www.microsoft.com/en-us/power-platform\nGartner. (2024, October 16). Magic Quadrant for Enterprise Low-Code Application Platforms. Gartner. https://www.microsoft.com/en-us/power-platform\nMicrosoft. (2025, September 25). Microsoft Power Platform: 2025 release wave 1 plan. Microsoft Learn. https://learn.microsoft.com/en-us/power-platform/release-plan/2025wave1/\nUserGuiding. (n.d.). 120+ No-Code/Low-Code Statistics and Trends That You Must Know in 2025. UserGuiding. https://userguiding.com/blog/no-code-low-code-statistics/\nMicrosoft. (2025, April 2). Use core components. Microsoft Learn. https://learn.microsoft.com/pt-br/power-platform/guidance/coe/core-components\nMicrosoft. (n.d.). Microsoft Power Platform documentation. Microsoft Learn. https://learn.microsoft.com/en-us/power-platform/\n","permalink":"https://julianamarialop.github.io/en/articles/microsoft-power-platform-the-easter-egg-of-low-code-technology/","summary":"Remember Super Mario Bros.? That feeling of starting small, slowly uncovering the game\u0026rsquo;s secrets, and leveling up until you completely master the universe? Digital transformation in companies follows a very…","title":"Microsoft Power Platform: The Easter Egg of Low-Code Technology"},{"content":"Imagine having access to the British Library (the largest library in the world, with more than 200 million items) but dedicated exclusively to your company\u0026rsquo;s documents. Now imagine that this library has a super-smart librarian who never sleeps, understands the context of your questions and finds exactly what you need in seconds, connecting information in ways you never imagined possible.\nThat\u0026rsquo;s Azure AI Search.\nJust as the British Library takes in 8,000 new titles a day and organizes them so anyone can find any piece of information, Azure AI Search turns the chaos of corporate documents into an intelligent search system that revolutionizes the way your company accesses and uses knowledge.\nYour Company\u0026rsquo;s Digital British Library Microsoft Azure AI Search, formerly known as Azure Cognitive Search, is much more than a simple search engine. It\u0026rsquo;s like building your own digital version of the British Library, but with artificial intelligence superpowers.\nWhy the Comparison with the British Library? The British Library, located in London, isn\u0026rsquo;t the largest library in the world by accident. It represents the state of the art in organizing, cataloging and providing access to human knowledge [1]. With more than 200 million items, including books, manuscripts, maps, newspapers, magazines, sound recordings and much more, it shows how massive amounts of information can be organized so that anyone can find exactly what they\u0026rsquo;re looking for.\nAzure AI Search does exactly that with your company\u0026rsquo;s data, but with the speed and intelligence only modern technology can offer.\nThe Traditional Librarian vs. the AI Librarian At the British Library, you\u0026rsquo;d find specialized librarians who know the collection inside out and can help you find relevant information. Azure AI Search is like having a librarian who combines the knowledge of all those specialists, but with superhuman abilities:\nYour AI Librarian\u0026rsquo;s Superpowers AI Enrichment: Imagine if the British Library\u0026rsquo;s librarian could not only find a document, but also:\nRead documents in any language and translate them instantly Extract text from images and scanned documents (OCR) Create automatic summaries of long documents Identify people, places and important concepts automatically Connect related information across different documents That\u0026rsquo;s exactly what Azure AI Search\u0026rsquo;s AI skillset does during the indexing process.\nThe Different \u0026ldquo;Floors\u0026rdquo; of Your Library Just as the British Library has different sections for different kinds of collections and readers, Azure AI Search offers different \u0026ldquo;floors\u0026rdquo; (tiers) for different business needs:\nThe Universal Collection: Supported Formats Just as the British Library accepts practically any kind of written or recorded material, Azure AI Search can process an impressive variety of formats [2]:\nClassic Documents\nPDF: Like the British Library\u0026rsquo;s digitized books Microsoft Office: DOCX, XLSX, PPTX (like modern documents) Plain Text: TXT, RTF (like old manuscripts) Periodicals\nHTML: Like online newspapers XML: Like structured catalogs CSV: Like tabular records Correspondence\nEML, MSG: Like the historical letters preserved in the library Special Materials\nKML: Like historical geographic maps ZIP: Like archived collections JSON: Like modern digital catalogs Important Note: Just as the British Library converts different formats into a universal cataloging system, Azure AI Search converts all these formats to JSON during indexing, creating a unified search system.\nHow to Build Your Digital Library Prerequisites: Preparing the Ground Before building your digital British Library, you\u0026rsquo;ll need:\nAn Azure account: Like having a plot of land to build on Azure CLI: Like having the construction tools The right permissions: the Search Service Contributor and Search Index Data Contributor roles The Construction Process: From Foundation to Operation 1. Laying the Foundation (Creating the Service) Like choosing the location and size of your library, you\u0026rsquo;ll create your Azure AI Search service in the Azure portal, selecting the tier that fits your needs.\n2. Installing the Security System (Configuring Authentication) Just as the British Library has sophisticated security systems, it\u0026rsquo;s recommended to use Microsoft Entra ID for keyless authentication, which offers more security than traditional API keys.\n3. Designing the Shelves (Creating Indexes) Like an architect planning the sections of the library, you\u0026rsquo;ll define your index schema, specifying which \u0026ldquo;fields\u0026rdquo; (types of information) each document will have. For modern AI applications, consider including vector fields for embeddings.\n4. Setting Up the Cataloging System (Parsing Mode) For RAG (Retrieval-Augmented Generation) applications, the Default parsing mode is recommended, since it automatically detects the file type and uses the right parser: like having a librarian who automatically recognizes whether they\u0026rsquo;re dealing with a book, a magazine or a manuscript.\n5. Stocking the Library (Loading Documents) Like carrying books to the shelves, you\u0026rsquo;ll populate your index using automatic indexers (which pull documents from sources such as Azure Blob Storage) or APIs for manual upload.\n6. Opening to the Public (Running Queries) Like opening the library doors, you\u0026rsquo;ll use the available REST APIs or SDKs (.NET, Python, Java, JavaScript) so applications and users can query your digital library.\nThe Different Types of Queries in Your Library 1. Catalog Lookup (Full-Text Search) Like looking up a specific book in the British Library catalog by title or author. Azure AI Search uses Apache Lucene and the BM25 algorithm to find documents containing exactly the words you\u0026rsquo;re looking for.\nExample: \u0026ldquo;Find all documents that mention \u0026lsquo;digital marketing strategy\u0026rsquo;\u0026rdquo;\n2. Thematic Query (Vector Search) Like asking the librarian: \u0026ldquo;I want something like this book, but I don\u0026rsquo;t know exactly what.\u0026rdquo; Azure AI Search uses vector embeddings to find semantically similar documents.\nExample: You show a document about \u0026ldquo;digital transformation\u0026rdquo; and the system finds documents about \u0026ldquo;technological innovation\u0026rdquo;, \u0026ldquo;process modernization\u0026rdquo; and \u0026ldquo;business digitalization\u0026rdquo;.\n3. Expert Query (Hybrid Search) Like combining the precision of the catalog with the intuition of an expert librarian. Azure AI Search combines full-text search with vector search for more relevant and comprehensive results.\n4. Academic Research Query (Multimodal Search) Like searching not only books but also maps, images, recordings and other kinds of material. Azure AI Search can search across different types of content at the same time.\n5. Advisory Query (Agentic Search/RAG) Like having a personal assistant who not only finds information but organizes it and presents it in context to answer your specific questions. Ideal for chatbots and AI agents.\nInvestment: How Much Does Your Digital British Library Cost? Cost Comparison Consider that running the real British Library costs millions of pounds a year and serves millions of people. Your digital British Library:\nFree Tier: Like a community library, perfect for getting started Basic Tier ($73.73/month): Like a well-equipped public library Standard Tiers ($245-$1,962/month): Like prestigious university libraries Storage Optimized Tiers ($2,802-$5,604/month): Like having your very own British Library Conclusion: Your Knowledge Revolution The British Library transformed access to human knowledge over the centuries. Azure AI Search does the same for your company\u0026rsquo;s knowledge, but on a much shorter timescale and with capabilities the British Library\u0026rsquo;s founders never dreamed of.\nWith more than 200 million items, the British Library proves that massive amounts of information can be organized in an accessible way. Azure AI Search proves that it can be done with artificial intelligence, instant speed and contextual understanding.\nThis isn\u0026rsquo;t just about technology. It\u0026rsquo;s about democratizing access to knowledge within your organization, just as the British Library democratized access to the world\u0026rsquo;s knowledge.\nReferences [1] Introduction to Azure AI Search - Azure AI Search | Microsoft Learn\n[2] Supported document formats - Azure AI Search | Microsoft Learn\n[3] Quickstart: Full-Text Search - Azure AI Search | Microsoft Learn\n[4] Pricing - Azure AI Search | Microsoft Azure\n","permalink":"https://julianamarialop.github.io/en/articles/azure-ai-search-your-corporate-digital-british-library/","summary":"Imagine having access to the British Library (the largest library in the world, with more than 200 million items) but dedicated exclusively to your company\u0026rsquo;s documents. Now imagine that this library has a librarian…","title":"Azure AI Search: Your Corporate Digital British Library"},{"content":"Imagine you\u0026rsquo;ve been hired to build the tallest skyscraper in the city. It would be an incredible project, right? But what if I told you that you had to use cracked bricks, expired cement and rusty iron beams? You\u0026rsquo;d probably think twice before accepting the project, wouldn\u0026rsquo;t you?\nWell, that\u0026rsquo;s exactly what happens when we try to build a Business Intelligence (BI) system on low-quality data. Just as a building needs reliable materials, our reports and dashboards need clean, organized data. Null, duplicate or inconsistent values are like cracks in the foundation: they may not seem dangerous at first, but over time they can bring down the whole structure.\nIn this article, we\u0026rsquo;ll learn how to be a \u0026ldquo;data quality engineer,\u0026rdquo; making sure our BI skyscraper is built on solid ground. And the best part: I\u0026rsquo;ll show you hands-on examples of how to do this in Microsoft Fabric and in Databricks !\nThe Skyscraper Blueprint: Understanding Data Quality Before construction begins, every good engineer needs a detailed plan. In the data world, that plan is what we call a data quality framework [1]. It\u0026rsquo;s like having the floor plan, the technical specifications and the construction schedule.\nThe Building Materials (Raw Data) The raw data coming in from the company\u0026rsquo;s systems is like the materials arriving at the construction site:\nSome arrive dirty (with special characters or extra spaces) Others arrive broken (null or incomplete values) Some come in the wrong format (dates as text, numbers as strings) And sometimes duplicate materials show up (the same customer registered twice) The Construction Site (ETL Process) The ETL (Extract, Transform, Load) process is our construction site. This is where:\nWe extract the materials from the suppliers (source systems) We transform and clean the materials (process the data) We load the treated materials into the structure (Data Warehouse) The Quality Engineer (Data Steward) Every construction site needs a quality engineer who inspects each material before it\u0026rsquo;s used in the building. In the data world, that person is the Data Steward: the professional responsible for making sure only good-quality data gets into our \u0026ldquo;skyscraper\u0026rdquo; [2].\nThe 6 Mandatory Inspections: The Dimensions of Quality Just as an engineer runs different types of inspections on materials, we need to check our data on 6 fundamental aspects [3]:\n1. Completeness: \u0026ldquo;Did all the materials arrive?\u0026rdquo; What it is: Checking that all the necessary data is present. Practical example: A customer record without an email is like trying to build a wall without all the bricks.\nHow to implement it in Fabric:\n-- Verificar completude no Fabric SELECT COUNT(*) as total_registros , COUNT(email) as emails_preenchidos , (COUNT(email) 100.0 / COUNT()) as percentual_completude FROM clientes; How to implement it in Databricks:\n# Verificar completude no Databricks from pyspark.sql.functions import col, count, when, isnan, isnull df_clientes = spark.table(\u0026#34;clientes\u0026#34;) completude_email = df_clientes.select( count(\u0026#34;*\u0026#34;).alias(\u0026#34;total\u0026#34;), count(when(col(\u0026#34;email\u0026#34;).isNotNull(), 1)).alias(\u0026#34;emails_validos\u0026#34;) ).collect()[0] percentual = (completude_email.emails_validos / completude_email.total) * 100 print(f\u0026#34;Completude do email: {percentual:.2f}%\u0026#34;) 2. Accuracy: \u0026ldquo;Do the materials have the right measurements?\u0026rdquo; What it is: The data correctly reflects reality. Practical example: A beam that should be 10 meters long but is 9.5 meters can compromise the entire structure.\nHow to implement it in Fabric:\n-- Validar formato de CPF no Fabric SELECT * FROM clientes WHERE LEN(REPLACE(REPLACE(cpf, \u0026#39;.\u0026#39;, \u0026#39;\u0026#39;), \u0026#39;-\u0026#39;, \u0026#39;\u0026#39;)) != 11 OR cpf NOT LIKE \u0026#39;[0-9][0-9][0-9].[0-9][0-9][0-9].[0-9][0-9][0-9]-[0-9][0-9]\u0026#39;; 3. Consistency: \u0026ldquo;Are the blueprints aligned?\u0026rdquo; What it is: The same data must be identical in different places. Practical example: The architect\u0026rsquo;s blueprint and the engineer\u0026rsquo;s must show the same wall in the same place.\nHow to implement it in Databricks:\n# Verificar consistência entre tabelas no Databricks vendas_total = spark.sql(\u0026#34;\u0026#34;\u0026#34; SELECT cliente_id, SUM(valor) as total_vendas FROM vendas GROUP BY cliente_id \u0026#34;\u0026#34;\u0026#34;) clientes_total = spark.sql(\u0026#34;\u0026#34;\u0026#34; SELECT cliente_id, total_compras FROM clientes \u0026#34;\u0026#34;\u0026#34;) inconsistencias = vendas_total.join( clientes_total, \u0026#34;cliente_id\u0026#34;, \u0026#34;inner\u0026#34; ).filter( col(\u0026#34;total_vendas\u0026#34;) != col(\u0026#34;total_compras\u0026#34;) ) inconsistencias.show() 4. Conformity: \u0026ldquo;Do the materials follow the technical standards?\u0026rdquo; What it is: The data follows the defined standards and formats. Practical example: Every screw must be of the type specified in the ABNT standard (Brazil\u0026rsquo;s national standards body).\n5. Integrity: \u0026ldquo;Are the connections secure?\u0026rdquo; What it is: The relationships between different pieces of data are valid. Practical example: Every column on the 5th floor must be connected to its corresponding foundation.\n6. Timeliness: \u0026ldquo;Are we using the latest blueprint?\u0026rdquo; What it is: The data is recent enough to be useful. Practical example: Using a 2020 blueprint to build in 2024 can cause problems.\nThe Biggest Villain: Null Values (Missing Materials) Imagine you\u0026rsquo;re building a wall and suddenly realize some bricks are missing. What do you do? Leave the hole? Improvise with another material? Stop the work until the bricks arrive?\nIn the data world, null values are exactly those \u0026ldquo;missing bricks.\u0026rdquo; And how we handle them can make the difference between a safe building and a disaster [4].\nScenario 1: Missing Material in the Finishes (Dimension Attributes) Situation: A customer doesn\u0026rsquo;t have the \u0026ldquo;state\u0026rdquo; field filled in because they live abroad. Solution: Instead of leaving it blank, we put \u0026ldquo;Not Applicable\u0026rdquo; or \u0026ldquo;International.\u0026rdquo;\nImplementation in Fabric:\n-- Tratar nulos em dimensões no Fabric UPDATE dim_clientes SET estado = CASE WHEN estado IS NULL AND pais != \u0026#39;Brasil\u0026#39; THEN \u0026#39;Internacional\u0026#39; WHEN estado IS NULL AND pais = \u0026#39;Brasil\u0026#39; THEN \u0026#39;Não Informado\u0026#39; ELSE estado END; Implementation in Databricks:\n# Tratar nulos em dimensões no Databricks from pyspark.sql.functions import when, col df_clientes_tratado = df_clientes.withColumn( \u0026#34;estado_tratado\u0026#34;, when(col(\u0026#34;estado\u0026#34;).isNull() \u0026amp; (col(\u0026#34;pais\u0026#34;) != \u0026#34;Brasil\u0026#34;), \u0026#34;Internacional\u0026#34;) .when(col(\u0026#34;estado\u0026#34;).isNull() \u0026amp; (col(\u0026#34;pais\u0026#34;) == \u0026#34;Brasil\u0026#34;), \u0026#34;Não Informado\u0026#34;) .otherwise(col(\u0026#34;estado\u0026#34;)) ) Scenario 2: Missing Material in the Structure (Foreign Keys) Situation: A sale was recorded, but the product doesn\u0026rsquo;t exist in the catalog yet. Solution: We create a \u0026ldquo;temporary product\u0026rdquo; until the correct record arrives.\nImplementation in Databricks with Delta Lake:\n# Implementar \u0026#34;Inferred Members\u0026#34; no Databricks def tratar_produto_inexistente(df_vendas, df_produtos): # Identificar produtos que não existem produtos_faltando = df_vendas.select(\u0026#34;produto_id\u0026#34;).distinct() \\ .join(df_produtos.select(\u0026#34;produto_id\u0026#34;), \u0026#34;produto_id\u0026#34;, \u0026#34;left_anti\u0026#34;) # Criar registros temporários produtos_temporarios = produtos_faltando.withColumn( \u0026#34;nome_produto\u0026#34;, lit(\u0026#34;Produto Pendente\u0026#34;) ).withColumn( \u0026#34;categoria\u0026#34;, lit(\u0026#34;Aguardando Cadastro\u0026#34;) ).withColumn( \u0026#34;status\u0026#34;, lit(\u0026#34;Temporário\u0026#34;) ) # Adicionar à tabela de produtos df_produtos_completo = df_produtos.union(produtos_temporarios) return df_produtos_completo Automating the Inspection: The Engineer\u0026rsquo;s Tools In Microsoft Fabric: Data Quality Rules -- Criar regra de qualidade no Fabric CREATE OR ALTER PROCEDURE sp_validar_qualidade_clientes AS BEGIN -- Verificar completude DECLARE @completude_email FLOAT = ( SELECT COUNT(email) * 100.0 / COUNT(*) FROM clientes ); -- Verificar duplicatas DECLARE @duplicatas INT = ( SELECT COUNT(*) - COUNT(DISTINCT cpf) FROM clientes ); -- Alertar se qualidade baixa IF @completude_email \u0026lt; 90 OR @duplicatas \u0026gt; 0 BEGIN PRINT \u0026#39;ALERTA: Qualidade dos dados abaixo do esperado!\u0026#39;; PRINT \u0026#39;Completude email: \u0026#39; + CAST(@completude_email AS VARCHAR(10)) + \u0026#39;%\u0026#39;; PRINT \u0026#39;Duplicatas encontradas: \u0026#39; + CAST(@duplicatas AS VARCHAR(10)); END END; In Databricks: Great Expectations # Implementar validação automática no Databricks import great_expectations as ge # Criar expectativas de qualidade df_ge = ge.from_pandas(df_clientes.toPandas()) # Definir regras df_ge.expect_column_to_exist(\u0026#34;email\u0026#34;) df_ge.expect_column_values_to_not_be_null(\u0026#34;email\u0026#34;, mostly=0.9) df_ge.expect_column_values_to_be_unique(\u0026#34;cpf\u0026#34;) df_ge.expect_column_values_to_match_regex(\u0026#34;cpf\u0026#34;, r\u0026#34;^\\d{3}\\.\\d{3}\\.\\d{3}-\\d{2}$\u0026#34;) # Executar validação resultado = df_ge.validate() print(f\u0026#34;Validação passou: {resultado.success}\u0026#34;) Continuous Monitoring: Building Maintenance Just as a building needs regular maintenance, our data needs continuous monitoring.\nQuality Dashboard in Fabric: -- Métricas de qualidade para dashboard SELECT \u0026#39;Clientes\u0026#39; as tabela, COUNT(*) as total_registros, COUNT(email) * 100.0 / COUNT(*) as completude_email, COUNT(DISTINCT cpf) * 100.0 / COUNT(*) as unicidade_cpf, GETDATE() as data_verificacao FROM clientes UNION ALL SELECT \u0026#39;Produtos\u0026#39; as tabela, COUNT(*) as total_registros, COUNT(nome) * 100.0 / COUNT(*) as completude_nome, COUNT(DISTINCT codigo) * 100.0 / COUNT(*) as unicidade_codigo, GETDATE() as data_verificacao FROM produtos; Automatic Alerts in Databricks: # Sistema de alertas no Databricks def verificar_qualidade_e_alertar(): # Verificar qualidade qualidade = calcular_metricas_qualidade() # Definir thresholds thresholds = { \u0026#39;completude_minima\u0026#39;: 95, \u0026#39;duplicatas_maximas\u0026#39;: 0, \u0026#39;precisao_minima\u0026#39;: 98 } # Verificar alertas alertas = [] for metrica, valor in qualidade.items(): if metrica in thresholds: if valor \u0026lt; thresholds[metrica]: alertas.append(f\u0026#34; {metrica}: {valor}% (esperado: {thresholds[metrica]}%)\u0026#34;) # Enviar alertas se necessário if alertas: enviar_notificacao_slack(alertas) # Agendar execução dbutils.jobs.taskValues.set(\u0026#34;alertas_qualidade\u0026#34;, verificar_qualidade_e_alertar()) Conclusion: Your Skyscraper Is Ready! Building a BI system on quality data is like raising a skyscraper: it takes planning, quality materials, rigorous inspections and constant maintenance. But when everything is working well, you have a solid structure that can support any business decision.\nAlways remember:\nPlan before you build: Define your quality framework Inspect every material: Implement the 6 dimensions of quality Fix problems at the source: Don\u0026rsquo;t let null values pile up Monitor constantly: Use automated validation tools Maintain the structure: Data quality is an ongoing process With these practices and the tools in Fabric and Databricks, you\u0026rsquo;ll be ready to build the most reliable data skyscraper in your company! See you next time!\nReferences [1] LakeFS. (2025). Data Quality Framework: Best Practices \u0026amp; Tools. https://lakefs.io/data-quality/data-quality-framework/\n[2] Hauskrecht, A. (n.d.). Building a Data Warehouse Data Quality Process. Toptal. https://www.toptal.com/database/data-warehouse-data-quality-process\n[3] Data Science Academy. (2023). As 6 Dimensões da Qualidade de Dados (The 6 Dimensions of Data Quality). https://blog.dsacademy.com.br/as-6-dimensoes-da-qualidade-de-dados-data-quality/\n[4] Iverson, H. K., \u0026amp; Oates, J. (2016). How to Manage Null Values in Your Data Warehouse. TDAN.com.\n","permalink":"https://julianamarialop.github.io/en/articles/building-a-data-skyscraper-how-to-ensure-quality-in-your-bi/","summary":"Imagine you\u0026rsquo;ve been hired to build the tallest skyscraper in the city. It would be an incredible project, right? But what if I told you that you had to use cracked bricks, expired cement and rusty iron…","title":"Building a Data Skyscraper: How to Ensure Quality in Your BI"},{"content":"In aerospace engineering, failure is not an option. Every rocket launch, every orbital maneuver, and every landing on a distant planet is the result of thousands of hours of testing, simulations, and checks. The smallest miscalculation can mean losing billions of dollars\u0026rsquo; worth of equipment. What if we applied that same discipline and rigor to protecting our most critical data assets?\nIn the data world, a failed pipeline may not disintegrate on reentry, but it can corrupt a company\u0026rsquo;s \u0026ldquo;source of truth,\u0026rdquo; leading to business decisions based on false information. To avoid that disaster, we can take inspiration from the risk mitigation strategies of space exploration and apply them to our pipelines in Microsoft Fabric or Azure Data Factory (ADF).\nLet\u0026rsquo;s embark on our own mission to Mars, using three key software engineering concepts: Dry Run, Canary, and Shadow.\nThe Origin: Strategies from a Space Mission Imagine we\u0026rsquo;re the space agency in charge of sending a new crewed mission to Mars. The risk is immense, and every step has to be validated.\nDry Run (The Flight Simulation): Long before the rocket reaches the launch pad, the astronauts and the mission control team spend months in flight simulators. They run the entire mission, from launch to landing, in a hyper-realistic virtual environment. No fuel is burned, no real hardware is put at risk. The goal is to memorize every procedure, test every emergency response, and validate every line of the flight plan. It\u0026rsquo;s logic validation in its purest form. Canary (The Pioneer Robot): Sending humans straight away would be reckless. First, the agency sends a robotic explorer, a rover like Perseverance, to be our \u0026ldquo;canary.\u0026rdquo; This precursor mission, relatively cheaper, lands on Mars to test real conditions. It analyzes the atmosphere, drills into the soil, and sends terabytes of data back. Scientists on Earth monitor its performance: did the solar panels hold up against the dust? Did the communication systems work? If the rover fails, the loss is contained, and the lessons learned are invaluable for the safety of the future crewed mission. Shadow (The Digital Twin on Earth): At mission control, engineers operate an exact replica of the spacecraft as a \u0026ldquo;digital twin\u0026rdquo; that lives on supercomputers or even as a physical model. While the real spacecraft travels through space, the twin on Earth receives exactly the same telemetry data and runs the same commands in parallel. The main mission carries on, unaffected. This lets the team test future maneuvers (such as a course correction) on the \u0026ldquo;shadow\u0026rdquo; twin before sending them to the real spacecraft, or simulate how the spacecraft would react to an anomaly, comparing the result with the real data. It\u0026rsquo;s the ultimate stress test, with real-world data but zero risk to the mission. Translating This to Data Pipelines in Fabric and ADF Now let\u0026rsquo;s bring these concepts back down to Earth and apply them to make sure our data pipelines are fail-safe.\n1. Dry Run: Your Pipeline\u0026rsquo;s Flight Simulation A \u0026ldquo;dry run\u0026rdquo; in a data pipeline executes all of its transformation logic without actually loading (or \u0026ldquo;landing\u0026rdquo;) the data in its final destination, such as a production Lakehouse or Warehouse.\nGoal: Validate your pipeline\u0026rsquo;s \u0026ldquo;trajectory\u0026rdquo; and logic, catching schema or calculation errors before consuming significant resources or risking the integrity of production data.\n2. Canary: The Pioneer Robot in Your Data A canary deployment processes a small, controlled subset of your production data (one region, one product line, one store) with the new pipeline logic, while most of the data keeps being processed by the old, stable version.\nGoal: Test the impact of the new logic with real production data, limiting the \u0026ldquo;blast radius\u0026rdquo; of a potential problem to a small, controlled segment.\n3. Shadow: Your Pipeline\u0026rsquo;s Digital Twin A shadow deployment is your ultimate safety net. It runs the new pipeline in parallel with the old one, using the same input data but writing to a completely isolated \u0026ldquo;shadow\u0026rdquo; destination.\nGoal: Validate the correctness, performance, and cost of the new pipeline version under 100% of the production load, with zero risk to the data the astronauts (your decision-makers) use to navigate.\nBy adopting the mindset of a mission engineer, you turn pipeline development from an uncertain art into an exact science. By simulating, sending pioneers, and operating with digital twins, you make sure every data \u0026ldquo;launch\u0026rdquo; is a guaranteed success.\nI hope this journey through the space mission metaphor has made these concepts clearer and more tangible. Just as aerospace engineers leave nothing to chance, we as data engineers can also adopt a safer, more disciplined approach. May the Dry Run, Canary, and Shadow techniques serve as your mission control center, helping you navigate the complexities of your projects and making sure every deployment is a perfect landing.\nSee you in the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/mission-to-mars-protect-data-pipelines-with-aerospace-precision/","summary":"In aerospace engineering, failure is not an option. Every rocket launch, every orbital maneuver, and every landing on a distant planet is the result of thousands of hours of testing, simulations, and…","title":"Mission to Mars: How to Protect Your Data Pipelines with the Precision of Aerospace Engineering"},{"content":"The Promised Dream\nIn May 2023, Microsoft announced Microsoft Fabric as a revolutionary solution to unify every analytical workload on a single platform. The promise was ambitious: lakehouse, data warehouse, real-time analytics, and machine learning working in an integrated way, eliminating the need for multiple specialized tools.\nThe initial reception was extremely positive. CTOs and data architects envisioned a future where they could simplify their data architectures, cut operating costs, and speed up the delivery of insights. The technical demos were impressive, showing data flowing between different workloads with no apparent friction.\nToday, almost two years after launch, Microsoft Fabric still hasn\u0026rsquo;t reached the adoption expected in the enterprise market. Despite significant improvements to the platform, many organizations remain hesitant to migrate from their current solutions. This analysis examines the main factors that contributed to this situation.\nAct I: The Prelude of Promise Scene 1: The Grand Overture When Microsoft Fabric was announced in May 2023, the expectation was a revolution in the data platform market. Microsoft correctly identified the biggest problem of the modern data era: fragmentation. Companies spent significant resources maintaining multiple platforms, like Synapse for data warehousing, Power BI for visualization, Azure ML for machine learning, and Data Factory for ETL, each with its own learning curve, pricing model, and technical limitations.\nFabric was positioned as the unifying solution, promising that all of these needs could be met on a single integrated platform. The value proposition was clear: OneLake as unified storage, compute shared across workloads, centralized governance, and a consistent user experience. For organizations tired of managing multiple tools, Fabric represented a significant simplification.\nThe go-to-market strategy was well executed. Technical demos showed data flowing seamlessly between different workloads, data scientists collaborating with data engineers in the same environment, and executives getting real-time insights through integrated dashboards. The message was consistent: simplicity, efficiency, and innovation on a single platform.\nScene 2: The First Dissonant Chords However, organizations that adopted Fabric in the early months discovered a reality different from the one presented in the demos. The first problems became evident quickly, creating hesitation in the market.\nThe first obstacle was the pricing model. Unlike traditional solutions with transparent pay-as-you-go models, Fabric introduced a capacity system based on Capacity Units (CUs) that many organizations found confusing and unpredictable. Companies used to the clear pricing models of Databricks or Snowflake found themselves lost trying to estimate the real costs of their workloads on Fabric.\nThe documentation, essential for enterprise adoption, was incomplete and fragmented. Promised features showed up marked as \u0026ldquo;preview\u0026rdquo; or \u0026ldquo;coming soon,\u0026rdquo; leaving data architects hesitant to bet critical projects on a platform that wasn\u0026rsquo;t fully mature yet. It felt like working with a beta version being sold as a finished product.\nAct II: The Tragedy of Costs Scene 1: The Phantom of F64+ The minimum F64 capacity requirement for many of Fabric\u0026rsquo;s advanced features became one of the biggest obstacles to adoption. This requirement represents a significant investment that many organizations weren\u0026rsquo;t prepared to make.\nAn F64 capacity costs about $8,400 a month, or more than $100,000 a year. For many organizations, especially small and midsize businesses, that amount is a substantial investment that has to be justified with a clear, immediate ROI. The problem is that Fabric, in its first years, couldn\u0026rsquo;t demonstrate that ROI convincingly for most use cases.\nThe situation gets more complicated when we compare it with the alternatives. An equivalent implementation on Databricks, especially for machine learning and data science workloads, often costs a fraction of the price of F64+. Organizations that migrated from Synapse to Databricks in recent years discovered they could get more functionality for less money, setting a precedent that\u0026rsquo;s hard to beat.\nScene 2: The Capacity Trap Fabric\u0026rsquo;s capacity model, while innovative in theory, created significant practical challenges. Unlike platforms that scale automatically based on demand, Fabric requires organizations to buy capacity up front, creating a dilemma: buying too little capacity results in throttling and degraded performance, while buying too much results in wasted resources.\nThis dynamic created a new operational need: dedicated Fabric capacity management. Organizations discovered they needed additional expertise not just to use the platform, but to manage its costs efficiently. That added operational complexity many companies hadn\u0026rsquo;t anticipated.\nThe lack of mature cost management tools made the problem worse. While AWS, Azure, and GCP offer sophisticated dashboards for cost monitoring, Fabric was slow to develop equivalent tools, leaving organizations with limited visibility into their CU spending.\nAct III: The Lost Timing Scene 1: The Migration That Didn\u0026rsquo;t Wait Fabric\u0026rsquo;s launch timing may have been Microsoft\u0026rsquo;s most critical mistake. When Fabric launched in 2023, many organizations had already completed their migrations from Azure Synapse to Databricks . That migration, which happened mostly between 2021 and 2023, was driven by Synapse\u0026rsquo;s limitations and by Databricks\u0026rsquo; technical superiority in machine learning and data science workloads.\nOrganizations that had invested months or years migrating to Databricks, training teams, and establishing processes weren\u0026rsquo;t willing to embark on another disruptive migration so soon. The cost of switching wasn\u0026rsquo;t just financial; it also included lost productivity, operational risk, and organizational change fatigue.\nMicrosoft missed a critical window of opportunity. If Fabric had launched in 2021, it could have captured many of the organizations that eventually migrated to Databricks. By 2023, those organizations were already settled on their new platforms and starting to reap the benefits of their migrations.\nScene 2: The Maturity of the Competition While Fabric struggled with its growing pains, the competition didn\u0026rsquo;t stand still. Databricks kept innovating aggressively, launching features like Unity Catalog for governance, Delta Live Tables for real-time ETL, and MLflow for MLOps. Snowflake expanded its capabilities beyond data warehousing, adding robust support for machine learning and data science.\nMore importantly, those mature platforms offered something Fabric still couldn\u0026rsquo;t: predictability. Organizations knew exactly what to expect in terms of performance, costs, and features. Fabric, on the other hand, was still constantly evolving, with features being added, modified, or discontinued regularly.\nStability became a deciding factor. CTOs who had been burned by problematic implementations of immature technologies preferred to bet on proven platforms rather than risk their careers on a promise, however brilliant it might be.\nAct IV: The Technical Limitations Scene 1: Reality vs. Promise As organizations began implementing Fabric in real-world scenarios, the technical limitations became apparent. The promise of unification, while attractive in theory, ran into practical realities Microsoft hadn\u0026rsquo;t fully anticipated.\nMachine learning workloads, especially those that required specialized GPUs or specific frameworks, hit limitations on Fabric that didn\u0026rsquo;t exist on Databricks. Configuration flexibility, crucial for advanced use cases, was more restricted on Fabric, forcing organizations to make compromises in their architectures.\nThe promised integration between different workloads, while functional, wasn\u0026rsquo;t as seamless as advertised. Data scientists found they still needed to understand the nuances of different engines and optimize their code for each specific context. The promise of \u0026ldquo;write once, run anywhere\u0026rdquo; turned out to be more complicated in practice.\nScene 2: The Steep Learning Curve Paradoxically, a platform designed to simplify the lives of data professionals ended up creating a new layer of complexity. Professionals experienced in Spark, SQL Server, or Power BI found they had to relearn fundamental concepts to work efficiently on Fabric.\nThe fragmented documentation and the platform\u0026rsquo;s rapid evolution made training a constant challenge. Organizations found they needed to invest significantly in upskilling, not just for new hires, but to reskill experienced teams.\nThis learning curve became an additional obstacle to adoption. In a competitive job market, where qualified professionals are scarce, organizations hesitated to bet on a technology that required substantial investment in training with no guarantee of return.\nAct V: The Late Awakening Scene 1: The Signs of Change Recognizing the challenges, Microsoft began making significant adjustments in 2024 and 2025. Removing the F64+ requirement for some features was an important first step, signaling that the company was listening to market feedback.\nInvestments in documentation, cost management tools, and platform stabilization showed that Microsoft was taking the criticism seriously. Strategic partnerships and integrations with popular tools in the data ecosystem demonstrated a more pragmatic, less proprietary approach.\nThe introduction of more flexible pricing models and cost optimization tools indicated that Microsoft was learning from its early mistakes. Still, the question remained: would it be too late to recover the lost momentum?\nScene 2: The Battle for Redemption Today, Microsoft Fabric finds itself at a crossroads. On one hand, the platform has evolved significantly since its launch, solving many of the initial problems and adding robust features. On the other hand, the initial window of opportunity has closed, and Microsoft now has to convince organizations already settled on other platforms to consider a switch.\nThe current strategy seems to focus on specific use cases where Fabric offers clear advantages, especially for organizations already invested in the Microsoft ecosystem. Native integration with Microsoft 365, Azure, and Power BI creates unique value for certain organizations.\nStill, the reality is that Fabric missed the chance to be the obvious choice for new implementations. Now it has to fight for every customer, competing not just on features, but also on price, maturity, and ecosystem.\nEpilogue: Lessons from a Foretold Tragedy The story of Microsoft Fabric is a fascinating case study in how timing, pricing, and execution can determine the success or failure of a technology, regardless of its technical merit. Fabric isn\u0026rsquo;t a bad platform. In fact, in many ways, it represents an innovative vision of the future of data. However, a series of strategic decisions and execution limitations kept it from reaching its potential.\nThe most important lesson is that in the enterprise world, perception often matters more than reality. Fabric may have solved many of its early problems, but the first impression had already been formed. Organizations that had negative experiences in the first months rarely give a second chance, especially when mature alternatives are available.\nFabric\u0026rsquo;s future remains uncertain. Microsoft has the resources and the determination to keep investing in the platform, and there are signs it\u0026rsquo;s learning from its mistakes. However, the window to become the dominant platform in the data market may have closed for good.\nLike every tragic opera, Fabric\u0026rsquo;s story reminds us that even the most promising heroes can fail when fate conspires against them. The question that remains is whether this is truly a tragedy or just the first act of an eventual redemption.\nEnd of the First Movement\nUntil the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/la-traviata-of-data-how-fabric-failed-to-win-the-market/","summary":"In May 2023, Microsoft announced Microsoft Fabric as a revolutionary solution to unify every analytical workload on a single platform. The promise was ambitious: lakehouse, data warehouse…","title":"La Traviata of Data: The Tragic Opera of How Fabric Failed to Win the Market"},{"content":"Just as Clark Kent had to turn into Superman so ordinary people could be saved, the data world has always required business users to turn into \u0026ldquo;super-techies\u0026rdquo; to access valuable insights. For decades, dashboards, analytics, and artificial intelligence stayed locked away in a technological fortress of solitude, accessible only to those with superpowers in SQL, Python, or data engineering.\nDatabricks has just launched its own version of Superman: Databricks One. This new interface isn\u0026rsquo;t just a cosmetic update; it\u0026rsquo;s a fundamental transformation that promises to democratize access to data the same way Superman democratizes hope. Just as the Man of Steel makes the impossible possible for ordinary citizens, Databricks One makes complex analytics accessible to any business user.\nIn a world where data is the new oil, Databricks One arrives as the hero who finally puts analytical superpowers in the hands of the people who actually need to make decisions: sales managers, marketing directors, business analysts, and executives who, until now, depended on technical \u0026ldquo;translators\u0026rdquo; to understand what their own data was trying to tell them.\nThe Hero\u0026rsquo;s Origin: From Technical Krypton to the Earth of Business Every superhero story starts with an origin, and Databricks One\u0026rsquo;s is no different. Just as Superman came from Krypton, a technologically advanced planet that was out of reach for Earthlings, traditional Databricks was born in the world of data engineers and scientists, a universe of complex notebooks, configurable clusters, and programming languages that intimidated business users.\nFor years, Databricks was like Superman still on Krypton: incredibly powerful, but operating in a dimension ordinary people couldn\u0026rsquo;t reach. The transformation began when the Databricks team realized that having superpowers means nothing if you can\u0026rsquo;t use them to help ordinary people.\nThe result is an interface that keeps all the strength of the original Databricks but presents that strength in a way anyone can understand and use. It\u0026rsquo;s as if Superman had learned to perfectly control his powers so he could help stop a meteor as well as rescue a cat from a tree: the same strength, applied appropriately to each situation.\nThe Superpowers of Databricks One Just as Superman has a range of superpowers that make him unique, Databricks One comes equipped with special abilities that transform business users\u0026rsquo; experience with data.\nX-Ray Vision for Technical Complexity\nThe first superpower of Databricks One is its ability to \u0026ldquo;see through\u0026rdquo; technical complexity and present only what matters for business decisions. Just as Superman can use his X-ray vision to see through walls and focus exactly on what he needs, Databricks One strips away all the unnecessary technical layers and shows only dashboards, insights, and analyses ready for consumption.\nUsers no longer need to understand concepts like \u0026ldquo;data partitioning\u0026rdquo; or \u0026ldquo;query optimization.\u0026rdquo; They simply see their KPIs, sales metrics, marketing analyses, and financial projections presented clearly and actionably.\nSuper Speed in Decision-Making\nAnother crucial superpower is speed. Just as Superman can fly around the world in seconds, Databricks One eliminates the lag between having a business question and getting a data-driven answer. There\u0026rsquo;s no longer any need to open tickets with IT, wait for someone to write a SQL query, or wait for a report to be generated by hand.\nWith the simplified interface, a sales manager can check their team\u0026rsquo;s performance, spot trends, and adjust strategies in real time. A marketing director can analyze campaign ROI and reallocate budgets instantly.\nSuperhuman Strength for Democratization\nThe third superpower is perhaps the most important: the strength to break down the organizational barriers that kept data concentrated in the hands of a few. Just as Superman uses his strength to remove obstacles ordinary people couldn\u0026rsquo;t move, Databricks One removes the technical barriers that stood in the way of true data democratization.\nThis democratizing strength means valuable insights are no longer trapped in technical silos. An HR analyst can understand turnover patterns without having to learn Python. A financial controller can build complex projections without mastering programming languages.\nThe Justice League of Data Superman rarely works alone when he faces threats that require multiple skills. He teams up with the Justice League, where each hero contributes their unique powers. Databricks One works in a similar way, orchestrating different technological \u0026ldquo;superheroes.\u0026rdquo;\nGenie: The Flash of Instant Insights\nWithin the Databricks One universe, Genie works like the Flash, the speedster who can process information and deliver answers at superhuman speed. When a user asks a natural language question like \u0026ldquo;What were our best-selling products last quarter?\u0026rdquo;, Genie processes the request instantly, queries terabytes of data, and returns precise insights in seconds.\nDashboards: The Wonder Woman of Visualization\nDashboards in Databricks One work like Wonder Woman: elegant, powerful, and able to reveal hidden truths with crystal clarity. These dashboards aren\u0026rsquo;t just pretty charts; they\u0026rsquo;re tools of revelation that show trends, anomalies, and opportunities that would stay invisible in traditional spreadsheets.\nConsumer Entitlement: The Aquaman of Governance\nThe \u0026ldquo;consumer entitlement\u0026rdquo; system works like Aquaman: a hero who is often underrated but essential for keeping balance and order. This governance system is crucial because democratization without control would be chaos, protecting both data security and user productivity.\nSaving Metropolis: Real Use Cases Just as Superman doesn\u0026rsquo;t only save the world from cosmic threats but also helps with everyday problems, Databricks One solves both strategic challenges and daily operational needs.\nThe Sales Manager\u0026rsquo;s Rescue\nMaria, a sales manager at a software company, used to spend hours every Monday compiling team performance reports. With Databricks One, Maria now opens a simple dashboard that automatically shows which salespeople are close to hitting their targets, which products are underperforming, and which regions hold untapped opportunities.\nThe result? Maria cut her report preparation time from 4 hours to 15 minutes, and now spends that extra time on one-on-one coaching. Her sales numbers went up 23% in the first quarter after the rollout.\nThe Marketing Director\u0026rsquo;s Transformation\nCarlos, a marketing director, always felt like he was fighting the data instead of being helped by it. Databricks One turned Carlos from the data\u0026rsquo;s antagonist into the hero of ROI. Now he can see in real time which campaigns are generating qualified leads, which channels have the best acquisition cost, and which messages resonate best with different segments.\nThe impact was dramatic: Carlos cut waste on ineffective campaigns by 40% and increased qualified lead generation by 60%.\nThe HR Analyst\u0026rsquo;s Empowerment\nAna, an HR analyst, always relied on others to access the information she needed. With Databricks One, Ana has grown into an independent investigator able to uncover crucial insights about talent. She now proactively identifies employees at risk of leaving the company, correlates training programs with performance gains, and forecasts future hiring needs.\nAna helped reduce turnover by 35% by identifying and addressing risk factors before they became problems.\nThe Kryptonite of Democratization Even Superman has his kryptonite, a weakness that can neutralize all his superpowers if it isn\u0026rsquo;t properly managed. Databricks One also faces challenges that can limit its effectiveness.\nGreen Kryptonite: Resistance to Change\nThe first form of kryptonite is organizational resistance to change. This resistance often comes from technical teams afraid of losing relevance and from business users who have grown dependent on \u0026ldquo;experts.\u0026rdquo; The solution is to redefine roles: technical teams evolve from \u0026ldquo;translators\u0026rdquo; into \u0026ldquo;architects\u0026rdquo; of more sophisticated solutions, and users gain the ability to ask smarter questions.\nRed Kryptonite: The Chaos of Uncontrolled Democratization\nData democratization without proper governance can create organizational chaos: users making decisions based on incorrect data or reaching contradictory conclusions. Protection comes through governance structures that make sure democratization increases effectiveness without sacrificing consistency.\nGold Kryptonite: The Paralysis of Endless Analysis\nWhen easy access to data leads to endless analysis without matching decisions, users can become so fascinated with the ability to explore data that they lose sight of the goal of making better decisions. Protection requires staying focused on deadlines and goals, valuing \u0026ldquo;good enough to act\u0026rdquo; over \u0026ldquo;too perfect to implement.\u0026rdquo;\nThe Future of Metropolis Databricks One, currently in private preview with public preview planned for early 2026, represents not just a technological evolution but a fundamental transformation in the relationship between people and data. The future being built is one where every business user becomes a kind of citizen superhero: ordinary people with access to analytical superpowers.\nThe roadmap includes features that will expand its democratizing superpowers even further: organizing content by business domain, simpler and more memorable URLs, and deeper integration with productivity tools such as Microsoft Teams and Slack.\nThe true legacy of Databricks One will be measured by its transformative impact on how organizations make decisions. When data stops being a scarce resource controlled by a few specialists and becomes an accessible utility, organizations become more agile, more responsive, and better able to innovate based on evidence.\nThe Call to Action Superman never saved Metropolis alone; he always counted on the help of brave citizens. In the same way, the success of Databricks One depends on organizations\u0026rsquo; and individuals\u0026rsquo; willingness to embrace a new way of working with data.\nFor business leaders, this means recognizing that data democratization is an organizational transformation that requires cultural change, proper training, and thoughtful governance. For business users, it means embracing the responsibility that comes with analytical superpowers. For technical professionals, it means evolving from gatekeepers into enablers.\nDatabricks One isn\u0026rsquo;t just a new interface; it\u0026rsquo;s a new era where the power of data finally lands where it should always have been: in the hands of the people who need to make decisions that matter. Like Superman flying over Metropolis, this platform offers an elevated perspective that reveals possibilities that used to be invisible.\nThe question isn\u0026rsquo;t whether this democratization will happen; it\u0026rsquo;s already happening. The question is whether your organization will be one of the first to embrace these superpowers or whether it will fall behind, still stuck in the technical Fortress of Solitude while competitors fly freely across the skies of data.\nThe future of data is democratic, accessible, and powerful. Databricks One is the Superman that makes that future possible. The only question left is: are you ready to fly?\nReferences\n[1] Databricks One Overview - NextGenLakehouse YouTube Channel: https://www.youtube.com/watch?v=21W_aFPgGsw\n[2] Microsoft Tech Community - Supercharge Data Intelligence with Azure Databricks Genie: https://techcommunity.microsoft.com/blog/analyticsonazure/supercharge-data-intelligence-build-teams-app-with-azure-databricks-genie--azure/4442653\n[3] Solutions Review - Analytics and Data Science News for the Week of August 8, 2025: https://solutionsreview.com/business-intelligence/analytics-and-data-science-news-for-the-week-of-august-8-updates-from-databricks-qbeast-power-bi-more/\n","permalink":"https://julianamarialop.github.io/en/articles/databricks-one-the-superman-of-data-democratizes-information/","summary":"Just as Clark Kent had to turn into Superman so ordinary people could be saved, the data world has always required business users to turn into \u0026ldquo;super-techies\u0026rdquo; to access…","title":"Databricks One: The Superman of Data Has Arrived to Democratize the Power of Information"},{"content":"Imagine an orchestra where every musician plays alone, without communicating with the others. The violinist plays their melody, the pianist follows their own rhythm, and the drummer keeps a completely different beat. The result would be deafening musical chaos, right? Well, that\u0026rsquo;s exactly what was happening with AI agents until now: each one worked in isolation in its own silo, producing fragmented and often contradictory answers.\nMicrosoft Fabric has just revolutionized this picture with the integration between Fabric Data Agents and Copilot Studio, released in July 2025. Now agents don\u0026rsquo;t just talk to each other; they share context, memory, and common goals to solve complex business problems. It\u0026rsquo;s like turning talented but disorganized musicians into a world-class symphony under the baton of a seasoned conductor.\nAnd best of all: this revolution happens completely transparently for you. You simply ask a question in Microsoft Teams, something like \u0026ldquo;How are our sales doing?\u0026rdquo;, and behind the scenes an entire orchestra of specialized agents springs into action instantly. Each one contributes its unique expertise, and the result is an executive analysis that no single agent could produce on its own.\nThe Conductor and Their Virtuosos Microsoft Copilot Studio = The Intelligent Conductor\nJust as a conductor doesn\u0026rsquo;t play any instrument but coordinates the whole orchestra, Copilot Studio is the strategic brain that analyzes each complex question and instantly decides which \u0026ldquo;musicians\u0026rdquo; (agents) should come in. When you ask \u0026ldquo;How are our sales doing and what should we do to improve?\u0026rdquo;, it doesn\u0026rsquo;t just recognize that it needs sales data; it also recognizes that the complete answer requires financial analysis, market context, and strategic recommendations. It\u0026rsquo;s orchestral intelligence in action.\nFabric Data Agents = The Specialized Virtuosos\nEach Fabric Data Agent is like a first-class musician who has perfectly mastered their instrument. One agent is a virtuoso in sales data, knowing every nuance of the numbers, seasonal trends, and behavior patterns. Another is a master of financial analysis, able to calculate margins, ROI, and projections with surgical precision. A third is a market intelligence specialist, always up to date on competitors, opportunities, and threats. Each one has direct access to OneLake and knows the \u0026ldquo;score\u0026rdquo; of the business data inside out.\nModel Context Protocol (MCP) = The Universal Score\nMCP is the secret language that lets all the agents understand each other perfectly, like a musical score that syncs every note. More than simple communication, it lets agents share context, memory, and even \u0026ldquo;intuitions\u0026rdquo; about the data. When one agent spots an anomaly in sales, MCP makes sure that information reaches the other agents instantly, enabling correlated analyses that reveal insights impossible to detect in isolation.\nThe Magic in Practice Real Scenario: A sales director, in the middle of a strategy meeting, quickly types in Teams: \u0026ldquo;I need to understand our quarterly sales to make an important decision right now\u0026rdquo;\nBehind the scenes (it all happens in under 30 seconds):\nCopilot Studio receives the question and immediately recognizes it\u0026rsquo;s not a simple query: it\u0026rsquo;s an executive request that calls for multidimensional analysis. In milliseconds, it puts together an orchestration strategy. Sales Agent is called first and dives into the OneLake data, pulling not only raw revenue numbers but identifying patterns, regional trends, performance by product, and comparisons with previous periods. Finance Agent comes in at the same time, analyzing profit margins, operating costs, ROI by sales channel, and calculating projections based on current data. Marketing Agent correlates sales data with active campaigns, historical seasonality, consumer behavior, and competitor moves that may be influencing the results. Copilot Studio receives all these specialized analyses and, using MCP, identifies connections and insights that only emerge from combining the different perspectives, creating a cohesive executive narrative. Result Delivered: A complete executive analysis with contextualized numbers, identification of opportunities and threats, relevant historical comparisons, and specific strategic recommendations, all formatted for immediate decision-making. The director has in hand not just data, but actionable intelligence.\nWhy Is This Revolutionary? Before: The Era of Disconnected Soloists\nImagine trying to solve a complex business problem by consulting specialists who have never talked to each other. The sales analyst gave you cold numbers, finance only talked about margins, and the market specialist offered generic insights. You had to be the mental \u0026ldquo;conductor,\u0026rdquo; trying to connect fragmented and often contradictory information. It was like trying to put together a jigsaw puzzle where every piece came from a different box.\nNow: The Symphony of Collective Intelligence\nThe digital orchestra completely changes this paradigm. The agents don\u0026rsquo;t just work together; they think together. When the sales agent identifies a drop in a particular product, the finance agent automatically analyzes the impact on margins, while the market agent checks for competitor moves. It\u0026rsquo;s emergent intelligence: the result of the collaboration is exponentially greater than the sum of the individual parts.\nThe Quantum Leap in Decision-Making\nFor the first time in business history, you have instant access to analyses that used to require entire teams working for days. More importantly: these analyses are contextually rich, considering multiple variables at once and identifying patterns that no single specialist could detect. It\u0026rsquo;s like having an advisory board of geniuses working 24/7 just for you.\nThe 3 Benefits That Change Everything Democratizing Executive Intelligence: It doesn\u0026rsquo;t matter whether you\u0026rsquo;re the CEO or a junior analyst; now anyone in the organization can ask complex questions and get C-level analyses. The digital orchestra removes the hierarchical barriers to accessing business intelligence. It\u0026rsquo;s like democratizing access to an elite advisory board, letting strategic insights flow through the whole organization and accelerating innovation at every level. Speed That Redefines Competitiveness: While your competitors are still scheduling meetings to discuss data, you\u0026rsquo;ve already made the decision and are executing. The digital orchestra compresses weeks of analysis into seconds of processing. More than efficiency, this represents a fundamental competitive advantage: the ability to respond to the market at the speed of thought, not at the speed of bureaucracy. Emergent Intelligence Insights: The real power of the orchestra isn\u0026rsquo;t in the individual agents but in the intelligence that emerges from their collaboration. Correlations impossible to detect manually, hidden patterns that only show up when multiple perspectives combine, and opportunities no single specialist could identify. It\u0026rsquo;s like having a magnifying glass that reveals invisible dimensions of your business. How to Start Your Orchestra: A Practical Example Let\u0026rsquo;s build a Sales Assistant that combines 3 specialized agents. Here\u0026rsquo;s the real step-by-step:\nStep 1: Prepare the Data in Fabric First, organize your data in OneLake:\n-- Tabela de Vendas (Lakehouse) CREATE TABLE vendas_historico ( data_venda DATE , produto VARCHAR(100) , valor DECIMAL(10,2) , regiao VARCHAR(50) , vendedor VARCHAR(100) ); -- Tabela de Metas (Warehouse) CREATE TABLE metas_vendas ( mes INT , ano INT , meta_valor DECIMAL(12,2) , regiao VARCHAR(50) ); Step 2: Create the Fabric Data Agents Agent 1: Sales Specialist\n# No Fabric Data Science - Notebook from databricks.agents import Agent vendas_agent = Agent.create( name=\u0026#34;VendasExpert\u0026#34;, description=\u0026#34;Especialista em análise de dados de vendas\u0026#34;, data_sources=[ \u0026#34;workspace.lakehouse.vendas_historico\u0026#34;, \u0026#34;workspace.warehouse.metas_vendas\u0026#34; ], instructions=\u0026#34;\u0026#34;\u0026#34; Você é um especialista em vendas. Analise: - Performance vs metas - Tendências por região - Top produtos e vendedores - Sempre inclua comparações históricas \u0026#34;\u0026#34;\u0026#34; ) Agent 2: Financial Analyst\nfinanceiro_agent = Agent.create( name=\u0026#34;FinanceiroExpert\u0026#34;, description=\u0026#34;Especialista em análise financeira de vendas\u0026#34;, data_sources=[ \u0026#34;workspace.lakehouse.vendas_historico\u0026#34;, \u0026#34;workspace.warehouse.custos_produtos\u0026#34; ], instructions=\u0026#34;\u0026#34;\u0026#34; Você analisa aspectos financeiros: - Margens de lucro por produto - ROI por região - Análise de rentabilidade - Projeções financeiras \u0026#34;\u0026#34;\u0026#34; ) Agent 3: Market Strategist\nmercado_agent = Agent.create( name=\u0026#34;MercadoExpert\u0026#34;, description=\u0026#34;Especialista em inteligência de mercado\u0026#34;, data_sources=[ \u0026#34;workspace.lakehouse.vendas_historico\u0026#34;, \u0026#34;workspace.warehouse.dados_mercado\u0026#34; ], instructions=\u0026#34;\u0026#34;\u0026#34; Você fornece contexto de mercado: - Sazonalidade e tendências - Comparação com concorrentes - Oportunidades de crescimento - Recomendações estratégicas \u0026#34;\u0026#34;\u0026#34; ) Step 3: Set Up the Orchestration in Copilot Studio 3.1 - Create the Conductor Agent\nIn Copilot Studio, create a new agent:\nNome: \u0026#34;Assistente Vendas Executivo\u0026#34; Descrição: \u0026#34;Coordena análises de vendas usando múltiplos especialistas\u0026#34; Instruções do Sistema: \u0026#34;\u0026#34;\u0026#34; Você é um assistente executivo que coordena 3 especialistas: - VendasExpert: dados de vendas e performance - FinanceiroExpert: análise financeira e margens - MercadoExpert: contexto de mercado e estratégia Para cada pergunta: 1. Determine quais especialistas consultar 2. Faça perguntas específicas para cada um 3. Combine as respostas em uma análise executiva 4. Sempre inclua recomendações acionáveis \u0026#34;\u0026#34;\u0026#34; 3.2 - Configure the MCP Connections\n{ \u0026#34;connections\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;VendasExpert\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;fabric_data_agent\u0026#34;, \u0026#34;endpoint\u0026#34;: \u0026#34;https://fabric.microsoft.com/agents/vendas-expert\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;FinanceiroExpert\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;fabric_data_agent\u0026#34;, \u0026#34;endpoint\u0026#34;: \u0026#34;https://fabric.microsoft.com/agents/financeiro-expert\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;MercadoExpert\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;fabric_data_agent\u0026#34;, \u0026#34;endpoint\u0026#34;: \u0026#34;https://fabric.microsoft.com/agents/mercado-expert\u0026#34; } ] } Step 4: Test the Orchestra Test Question: \u0026ldquo;How are our sales doing this quarter and what should we do?\u0026rdquo;\nOrchestra Flow:\nCopilot Studio analyzes the question VendasExpert → \u0026ldquo;Q3 sales: R2.1M(target:R 2.1M (target: R2.1M(target:R 2M). 15% growth vs Q2\u0026rdquo; FinanceiroExpert → \u0026ldquo;Average margin 23%. Premium products with the best ROI\u0026rdquo; MercadoExpert → \u0026ldquo;Favorable seasonality through December. Competitor X losing market share\u0026rdquo; Copilot Studio combines everything Final Answer\n\u0026#34;✅ Performance Excelente: Superamos meta em 5% (R2.1MvsR 2.1M vs R 2.1MvsR 2M) 📊 Destaques: •Crescimento 15% vs trimestre anterior •Margem saudável de 23% •Produtos premium liderando rentabilidade 🎯 Recomendações: •Intensificar foco em produtos premium •Aproveitar sazonalidade até dezembro •Expandir em regiões onde concorrente está perdendo espaço 📈 Projeção Q4: Com estratégia atual, potencial de R$ 2.4M\u0026#34; Step 5: Deploy and Monitoring # Publicar no Teams copilot_studio.deploy_to_teams( agent_id=\u0026#34;assistente-vendas-executivo\u0026#34;, channels=[\u0026#34;vendas\u0026#34;, \u0026#34;diretoria\u0026#34;], permissions=[\u0026#34;sales_team\u0026#34;, \u0026#34;executives\u0026#34;] ) # Configurar métricas monitoring.setup_metrics([ \u0026#34;response_time\u0026#34;, \u0026#34;user_satisfaction\u0026#34;, \u0026#34;agent_collaboration_success\u0026#34;, \u0026#34;business_impact\u0026#34; ]) Result: In 2-3 weeks you have a working digital orchestra, turning simple questions into complete executive analyses.\nThe technology is in preview, but it already works. Pioneering companies are testing it and the results are impressive.\nThe Future Is Here The era of isolated agents is over. Welcome to the era of intelligent orchestration, where AI isn\u0026rsquo;t just automation but genuine collaboration among digital specialists.\nIs your company ready to have its own digital orchestra?\nSee you in the next article!\nReferences Fabric Data Agents + Microsoft Copilot Studio: A New Era of Multi-Agent Orchestration (Preview) - https://blog.fabric.microsoft.com/en-us/blog/fabric-data-agents-microsoft-copilot-studio-a-new-era-of-multi-agent-orchestration\nOrchestrate agent behavior with generative AI - https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-generative-actions\n","permalink":"https://julianamarialop.github.io/en/articles/the-digital-orchestra-when-ai-agents-learn-to-play-together/","summary":"Imagine an orchestra where every musician plays alone, without communicating with the others. The violinist plays their melody, the pianist follows their own rhythm, and the drummer keeps a completely different beat…","title":"The Digital Orchestra: When AI Agents Learn to Play Together"},{"content":"Remember your grandma\u0026rsquo;s apple pie? That special recipe that took hours to make, but always came out perfect? The modern Human Resources department faces a similar dilemma: how to keep the handcrafted quality of talent selection without spending hours analyzing every resume one by one?\nJust as grandma knew every ingredient and knew exactly when the pie was ready, experienced recruiters develop a \u0026ldquo;sixth sense\u0026rdquo; for spotting good candidates. But what if we could automate that process, keeping the same quality and judgment, but with the efficiency of a modern kitchen?\nAzure AI Foundry offers all the tools you need to create this \u0026ldquo;version 2.0\u0026rdquo; of grandma\u0026rsquo;s recipe: an intelligent agent that keeps the same quality ingredients (experience, rigorous criteria, attention to detail), but uses modern technology to bake the \u0026ldquo;perfect pie\u0026rdquo; in a fraction of the time.\nThe Scenario: Why Do We Need a Digital Recruiter? Imagine an HR department that receives 500 resumes a week for different positions. Every resume has to be read, analyzed, and compared against the job requirements. An experienced recruiter takes, on average, 10 minutes to do a complete initial analysis of a resume. That means more than 80 hours a week just for initial screening: more than two weeks of one person\u0026rsquo;s work.\nBeyond the time issue, there\u0026rsquo;s the challenge of consistency. Different recruiters may interpret the same requirements slightly differently, especially when they\u0026rsquo;re tired or under pressure. A candidate evaluated on Monday morning may get a different analysis from the same candidate evaluated on Friday afternoon.\nAzure AI Foundry Agent Service solves these problems by creating a digital assistant that keeps criteria consistent, works 24 hours a day, and can process dozens of resumes at the same time. It\u0026rsquo;s like having an HR specialist who never has a bad day and always applies exactly the same evaluation standards.\nThe Recipe\u0026rsquo;s Secret Ingredients Just as grandma\u0026rsquo;s apple pie had specific ingredients that made all the difference, our digital recruiting system also needs the right components to work perfectly. Each \u0026ldquo;ingredient\u0026rdquo; plays a specific role in building an intelligent, efficient system.\n_Complete Azure AI Foundry architecture showing the integration between all the components of our digital recruiting system_\nAzure AI Foundry Agent Service is like grandma\u0026rsquo;s oven. It\u0026rsquo;s the heart of the operation, where the magic happens. This is where we define our agent\u0026rsquo;s instructions, its capabilities, and how it should behave when analyzing resumes. Think of it as grandma\u0026rsquo;s accumulated knowledge, but encoded precisely and consistently.\nAzure Blob Storage works like grandma\u0026rsquo;s well-organized pantry, where all the ingredients (resumes) are stored safely and accessibly. The advantage is that this \u0026ldquo;digital pantry\u0026rdquo; can hold millions of documents without losing its organization.\nAzure AI Search is like grandma\u0026rsquo;s secret recipe notebook, but supercharged. It doesn\u0026rsquo;t just organize resumes by keyword, it understands the \u0026ldquo;flavor\u0026rdquo; of each professional profile. It can find candidates with \u0026ldquo;leadership experience\u0026rdquo; even when the resume only mentions \u0026ldquo;team coordination.\u0026rdquo;\nThe File Search Tool is the magic utensil that ties everything together, like the special wooden spoon grandma used for every dish. It lets the agent access, read, and analyze the resumes, using all of the system\u0026rsquo;s intelligence to find the right \u0026ldquo;ingredients\u0026rdquo; for each job.\nStep 1: Preparing the Resume Archive - Azure Blob Storage The first step in building our digital recruiter is to create a secure, organized place to store all the resumes. Azure Blob Storage will be our \u0026ldquo;digital file room,\u0026rdquo; where every document is perfectly cataloged and accessible.\nSetting up Azure Blob Storage requires special attention to the folder structure. An efficient organization can be by department, by job, or by period. For example: /curriculos/2025/desenvolvedor-senior/ or /curriculos/marketing/analista-digital/. This organization makes things easier both for the agent\u0026rsquo;s access and for future maintenance of the system.\nIt\u0026rsquo;s important to set access permissions properly from the start. The storage should allow read access for the AI services we\u0026rsquo;ll create later, but keep strict controls over who can add, modify, or delete resumes. Using Azure managed identity-based authentication is recommended for greater security.\nThe container structure should reflect the company\u0026rsquo;s organization. One main container for active resumes, another for historical files, and possibly separate containers for different business units. This makes it easier to manage retention and access policies.\nStep 2: Building the Search Brain - Azure AI Search (RAG) With the resumes organized in Blob Storage, we need to build the system that will let our agent \u0026ldquo;understand\u0026rdquo; and search for information in those documents. Azure AI Search works as our agent\u0026rsquo;s RAG (Retrieval-Augmented Generation) system, turning documents into searchable knowledge.\nAzure AI Search is much more than a traditional search engine. It processes the resumes, extracts text, creates vector representations of the content, and builds indexes optimized for semantic search. It\u0026rsquo;s like having a super-smart librarian who not only knows where every document is, but also understands the meaning of every word and concept.\nConfiguring the parsing mode is crucial to the system\u0026rsquo;s success. For resumes, we recommend using \u0026ldquo;Default\u0026rdquo; mode, which automatically detects the file type and applies the most appropriate parser. This ensures that both PDFs and Word documents are processed correctly, extracting not just text but also important metadata.\nThe indexing process turns each resume into a searchable structure. The system automatically splits the documents into smaller chunks, creates vector embeddings for semantic search, and builds inverted indexes for keyword search. It\u0026rsquo;s this combination that lets the agent find candidates with \u0026ldquo;leadership experience\u0026rdquo; even when the resume only mentions \u0026ldquo;team management.\u0026rdquo;\nThe connection to Blob Storage establishes the data pipeline. Whenever new resumes are added to storage, AI Search automatically processes them and updates the index. It\u0026rsquo;s like having an assistant who keeps the library always organized and up to date.\nStep 3: Creating the Command Center - Azure AI Foundry Hub With our database and search system ready, it\u0026rsquo;s time to create the \u0026ldquo;command center\u0026rdquo; where our agent will be developed and managed. The Azure AI Foundry Hub works as the headquarters of our AI operation.\nThe Azure AI Foundry Hub centralizes resources, security settings, and connectivity. This is where we define which Azure services our agent can access and how it should authenticate. Think of the Hub as the headquarters of an intelligence operation, where all the tools and resources are kept organized and secure.\nSetting up the Hub includes connecting it to the services we created earlier. The Hub needs access to Blob Storage (to read resumes) and to AI Search (to run intelligent searches). These connections are established using managed identities, ensuring security without having to manage keys manually.\nInside the Hub, we create a Project specifically for our HR agent. Projects let you organize different agents and experiments, keeping settings and data separate. It\u0026rsquo;s like having different departments inside the same company: each with its own specific responsibilities, but all sharing the Hub\u0026rsquo;s common infrastructure.\nThe Hub\u0026rsquo;s security settings determine who can access the project, create agents, and view data. It\u0026rsquo;s important to establish clear policies from the start, defining different levels of access for developers, administrators, and end users of the system.\nStep 4: Bringing the Recruiter to Life - Creating the Agent With all the infrastructure in place (Blob Storage organizing the resumes, AI Search working as the RAG brain, and the Hub centralizing everything), it\u0026rsquo;s time to bring our digital recruiter to life. Creating the agent is where we define its personality, its skills, and how it should behave when analyzing resumes.\nThe agent\u0026rsquo;s instructions are fundamental to its success. They should be clear, specific, and comprehensive. An example of an effective instruction would be: \u0026ldquo;You are a recruiting specialist with 15 years of experience. Your role is to analyze resumes and assess how well candidates fit the open positions. Always consider relevant experience, academic background, technical skills, and soft skills. Be objective but fair in your assessments.\u0026rdquo;\nConfiguring the File Search Tool connects the agent to the stored resumes. This tool automatically processes the documents, creating text chunks of approximately 800 tokens with a 400-token overlap. This ensures that important information isn\u0026rsquo;t lost when the content is split.\nConnecting to Blob Storage establishes the link between the agent and the data. The system automatically monitors new files added to storage and includes them in the search index. It\u0026rsquo;s like having an assistant who always keeps the resume base up to date.\nConnection tests are essential before putting the agent into production. Start with a few test resumes and known job openings. Check that the agent can access the documents, extract relevant information, and provide coherent analyses.\nFeeding the System: Uploading and Organizing the Resumes A digital recruiter is only as good as the quality and organization of the data it can access. How we organize and feed the resume base directly determines how effective the analyses are.\nThe folder structure in Blob Storage should reflect the reality of the business. An efficient approach is to create hierarchies that make both searching and maintenance easier. For example: /curriculos/ano/departamento/nivel/ allows specific filters and makes it easier to manage old data.\nSupported file formats include PDF, Word (.doc and .docx), plain text, RTF, and many others. The system automatically detects the format and applies the appropriate processing. PDFs with selectable text work better than scanned images, although the system can also process the latter with OCR.\nIndexing happens automatically when new files are added to Blob Storage. Azure AI Search processes the content, extracts text, creates vector embeddings, and updates the index. This process can take a few minutes for large files or many files at once.\nCreating Vector Stores organizes the resumes into structures optimized for semantic search. Each Vector Store can hold up to 10,000 files and can be attached either to the agent (for permanent resumes) or to specific threads (for temporary analyses of specific openings).\nTraining the Specialist: Prompts and Instructions The quality of our digital recruiter\u0026rsquo;s analyses depends directly on the quality of the instructions we give it. It\u0026rsquo;s like training a new employee: the clearer and more specific the guidance, the better the results.\nAn effective prompt for resume analysis should include context, specific criteria, and the expected response format. For example: \u0026ldquo;Analyze this resume for the Senior Python Developer position. Evaluate: 1) Python experience (minimum 5 years), 2) Knowledge of web frameworks (Django/Flask), 3) Database experience, 4) Leadership skills. Give a score from 1-10 for each criterion and an executive summary.\u0026rdquo;\nEvaluation criteria should be customizable for different types of positions. Technical roles may prioritize specific skills and certifications, while leadership roles may focus more on management experience and soft skills. The agent can be instructed to adjust its criteria based on the type of position.\nPractical examples of instructions include: \u0026ldquo;If the candidate doesn\u0026rsquo;t explicitly mention a required skill, look for experience that may indicate that competency indirectly. For example, \u0026lsquo;coordinated a system migration project\u0026rsquo; may indicate project management skills even without mentioning specific certifications.\u0026rdquo;\nCalibrating the agent is an iterative process. Start with basic instructions, test with known resumes, and refine the guidance based on the results. It\u0026rsquo;s important to include examples of good and bad practices so the agent learns to distinguish between different levels of qualification.\nThe Recruiter in Action: Practical Use Cases With our digital recruiter configured, let\u0026rsquo;s see how it works in practice. Candidate-to-job match analysis is the most common case: the agent receives a job description and identifies points of fit and possible gaps in the resumes.\nCandidate ranking lets you compare multiple profiles at once, creating ordered lists based on the established criteria. Identifying skill gaps helps with both selection and internal development, while report generation automates the documentation of the hiring process.\nOptimizations and Best Practices To keep the system efficient, implement expiration policies for temporary Vector Stores (7 days is the default) and monitor costs through periodic cleanup of old data.\nPerformance monitoring should include metrics such as response time and analysis quality. For security, use encryption and role-based access controls, and stay compliant with LGPD (Brazil\u0026rsquo;s data protection law).\nConclusion Just as grandma\u0026rsquo;s apple pie evolved from a handwritten recipe into modern versions that keep the same special flavor, building a resume screening agent in Azure AI Foundry represents that same evolution in recruiting. We keep the quality ingredients (rigorous criteria, attention to detail, accumulated experience), but we use modern technology to deliver consistent, efficient results.\nThis system doesn\u0026rsquo;t replace the \u0026ldquo;special touch\u0026rdquo; of HR professionals; it amplifies their work, letting them focus on what really matters: getting to know people, developing talent, and building exceptional teams. After all, the best pie will always be the one made with love, but with the help of the right tools.\nSee you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/grandma-hr-apple-pie-resume-screening-agent-azure-ai-foundry/","summary":"Remember your grandma\u0026rsquo;s apple pie? That special recipe that took hours to make, but always came out perfect? The modern Human Resources department faces a similar dilemma: how to keep the…","title":"Grandma HR's Apple Pie: How to Build a Resume Screening Agent in Azure AI Foundry"},{"content":"July 2025 arrived with a shower of updates that turned Microsoft Fabric into an even more powerful superhero. Like any self-respecting hero, Fabric never stops evolving, picking up new powers and gear to face the increasingly complex challenges of the data world.\nThis month was especially notable, with six major updates that significantly expanded the platform\u0026rsquo;s capabilities. From a new cosmic armor to real-time gadgets, by way of custom shields and interdimensional bridges, our hero\u0026rsquo;s arsenal has never been so complete.\nLet\u0026rsquo;s explore each of these new powers and see how they can transform the way we work with data, making our missions more efficient, secure and powerful.\nThe New Cosmic Armor: Cosmos DB in Fabric Imagine Tony Stark developing a new version of the Iron Man armor, but this time with advanced cosmic technology. That\u0026rsquo;s exactly what the launch of Cosmos DB in Microsoft Fabric represents. This new \u0026ldquo;armor\u0026rdquo; isn\u0026rsquo;t just an addition to the arsenal: it\u0026rsquo;s a fundamental transformation that lets our hero operate in entirely new dimensions.\nCosmos DB in Fabric brings the power of Azure Cosmos DB directly into the platform, combining high availability, dynamic scalability and AI-optimized performance in a fully managed experience. It\u0026rsquo;s as if Fabric gained the ability to process NoSQL data natively, while keeping all the flexibility and power we already know.\nThe big revolution of this cosmic armor is automatic mirroring to OneLake. As soon as you create a Cosmos DB database in your workspace, Fabric automatically starts replicating all the data to OneLake in Delta format, in near real time. It\u0026rsquo;s like having a smart backup system that doesn\u0026rsquo;t just protect your data, but makes it instantly available for advanced analytics.\nBut this armor\u0026rsquo;s powers go further. With native support for vector search and full-text search, Cosmos DB in Fabric lets you run similarity searches using vector embeddings, finding results based on the meaning and context of the data, not just keywords. It\u0026rsquo;s like having a radar that detects not only objects, but also their intentions and relationships.\nFor teams working on modern applications, this new armor offers powerful use cases. Imagine a smart knowledge assistant that accesses manuals, support documents and chat logs stored as JSON in Cosmos DB. With vector indexing, the assistant delivers semantically relevant results for user queries, while the data replicated in OneLake enables ongoing analytics to refine the content.\nThe Custom Shield: Customer Managed Keys in OneLake Every superhero needs a reliable shield, and Captain America always knew it. Now Microsoft Fabric has its own custom shield with the launch of Customer Managed Keys (CMK) in OneLake. But unlike Captain America\u0026rsquo;s vibranium, this shield can be forged exactly to your security specifications.\nCustomer Managed Keys are one of the most requested features among Fabric\u0026rsquo;s enterprise users. By default, Microsoft already encrypts all data at rest in OneLake using Microsoft-managed keys. But for organizations in regulated industries such as finance, healthcare and government, having full control over encryption keys isn\u0026rsquo;t just desirable: it\u0026rsquo;s essential.\nWith CMK, you can use your own keys, stored in Azure Key Vault, to encrypt data in OneLake. It\u0026rsquo;s like having a personal safe where only you know the combination. Imagine a financial services company that needs to demonstrate full control over data encryption to auditors. With CMK, they can show that only their security team has access to the encryption key, and that revoking the key will immediately block access to sensitive data.\nThe power of this custom shield goes beyond simple encryption. It offers granular control at the workspace level, letting organizations selectively encrypt only the workspaces that require enhanced data protection. It\u0026rsquo;s a flexible approach that doesn\u0026rsquo;t force a one-size-fits-all solution on every scenario.\nThe feature also supports key rotation and revocation. If a healthcare organization needs to rotate encryption keys every 90 days to stay compliant, it can automate rotation policies in Azure Key Vault without disrupting analytics workflows. And if access needs to be revoked, OneLake automatically blocks read and write operations within an hour, ensuring the data stays secure.\nThe Interdimensional Bridge: Unity Catalog Mirroring Doctor Strange always impressed with his ability to open portals between dimensions, connecting worlds that once seemed impossible to reach. Microsoft Fabric has just gained a similar power with the launch of Azure Databricks Unity Catalog Mirroring, now generally available.\nThis interdimensional bridge lets tables governed in Unity Catalog be accessed directly by Microsoft Fabric, creating a unified, governed experience across both platforms with no data duplication. It\u0026rsquo;s as if Fabric gained the ability to \u0026ldquo;see\u0026rdquo; across dimensions, accessing data that lives in the Databricks universe as if it were native.\nThe magic of this bridge is its transparency. Datasets managed by Databricks become instantly usable in Fabric, keeping all the governance and access control established in Unity Catalog. It\u0026rsquo;s an integration that respects existing security policies while dramatically expanding the possibilities for analytics and collaboration.\nFor teams working with both platforms, this interdimensional bridge removes the need for complex synchronization or data duplication processes. Data scientists can keep working in Databricks with their preferred tools, while business analysts access the same data through Power BI and other Fabric tools, all while keeping a single source of truth.\nThe impact of this feature goes beyond technical convenience. It represents a fundamental shift in how we think about data ecosystems, letting organizations take advantage of the best of both worlds without compromises or added complexity.\nThe Real-Time Gadget: SQL Operator in Eventstream Batman has always been known for his ingenious gadgets, specialized tools he develops for specific situations. Microsoft Fabric has just added a new gadget to its utility belt: the SQL Operator in Eventstream, a powerful tool for missions that require real-time data processing.\nThe SQL Operator is a significant evolution in Eventstream, offering full control over real-time data transformations using familiar SQL. While Eventstream already offered a rich no-code experience with built-in operators like Filter, Aggregate and Join, the SQL Operator lets you package all your data transformation logic in one place.\nThis gadget really shines in complex scenarios that require conditional logic, nested expressions, string manipulation and advanced aggregations. It\u0026rsquo;s like having a multi-tool that adapts to any situation, letting you write custom transformations with a surgeon\u0026rsquo;s precision and an artist\u0026rsquo;s flexibility.\nThe development experience is intuitive and powerful. You can test your queries on real-time data before publishing, making sure the logic is right. IntelliSense support boosts productivity and minimizes errors with syntax highlighting and autocomplete. It\u0026rsquo;s like having a personal assistant who knows every available function and syntax.\nFor teams already working with Azure Stream Analytics, this gadget offers a smooth transition, letting you bring your existing transformation logic straight into Fabric Eventstream. It\u0026rsquo;s a bridge that connects existing knowledge with new possibilities, speeding up adoption and shortening the learning curve.\nThe Advanced Security System: Workspace Access Limits Just as Superman\u0026rsquo;s Fortress of Solitude has advanced security systems to protect dangerous knowledge and technology, Microsoft Fabric is rolling out a new protection system: Workspace Access Limits, to be introduced in August 2025.\nThis advanced security system is designed to improve quality of service and reliability, and to encourage proper access control over workspaces. It\u0026rsquo;s a proactive measure meant to protect both individual users and the ecosystem as a whole, making sure resources are used efficiently and responsibly.\nThe access limits work like a smart monitoring system that watches usage patterns and applies controls when needed. It\u0026rsquo;s like having a digital guardian who understands when usage is within normal parameters and when it may be affecting the platform\u0026rsquo;s overall performance.\nThis feature strikes a careful balance between flexibility and control. While keeping the ease of use that characterizes Fabric, it introduces safeguards that protect every user\u0026rsquo;s experience. It\u0026rsquo;s an approach that prioritizes the platform\u0026rsquo;s long-term sustainability.\nFor administrators and governance teams, this system offers greater visibility and control over how resources are used, enabling more effective planning and proactive capacity management.\nThe Universal Communicator: Improved SAP Connectivity The Starfleet Federation has always relied on universal communicators to connect different species and cultures. Microsoft Fabric has just upgraded its own universal communicator with significant improvements to SAP connectivity, making it easier to integrate with one of the most important enterprise ecosystems in the world.\nThe new options for integrating data from SAP sources are an important step forward for organizations that depend on these systems for critical business operations. With more than 170 connectors available in Data Factory, Fabric keeps expanding its ability to talk to practically any enterprise system.\nThis improved connectivity is especially valuable for companies that need to unify SAP data with other sources for comprehensive analytics and AI use cases. It\u0026rsquo;s like having a universal translator that not only understands different data \u0026ldquo;languages\u0026rdquo;, but also harmonizes them into a common language the whole organization can use.\nThe improvements include more robust options for extracting, transforming and loading SAP data, letting organizations make better use of their existing SAP investments while modernizing their analytics capabilities.\nConclusion Each of these features doesn\u0026rsquo;t just add technical capabilities; it represents an evolution in how we think about data integration, security, governance and processing. Together, they create a more robust, flexible and powerful ecosystem that can adapt to the ever-changing needs of modern organizations.\nSee you in the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/fabric-hero-complete-arsenal-new-powers-and-gear-july-2025/","summary":"July 2025 arrived with a shower of updates that turned Microsoft Fabric into an even more powerful superhero. Like any self-respecting hero, Fabric never stops evolving, picking up new powers and…","title":"The Fabric Hero's Complete Arsenal: New Powers and Gear for July 2025"},{"content":"Imagine an office where the employees never get tired, work 24 hours a day, are always learning and carry out tasks with surgical precision. This scenario, which seemed like science fiction just a few years ago, is now a reality with artificial intelligence agents. Databricks has just revolutionized this concept with the launch of Agent Bricks and Lakebase, creating what we might call the \u0026ldquo;perfect digital office\u0026rdquo;.\nAI agents represent a fundamental shift in the way companies operate. They\u0026rsquo;re not just automated tools, but true digital collaborators able to make decisions, learn from experience and adapt to new situations. With Agent Bricks, Databricks has democratized the creation of these \u0026ldquo;digital employees\u0026rdquo;, letting any company build its own team of specialized agents without the traditional technical complexity.\nLakebase, in turn, works as the central nervous system of this new office, making sure every agent has instant access to the information it needs to do its job. Together, these technologies are redefining the concept of business productivity, promising up to 90% reductions in AI solution development time and transforming the way we think about intelligent automation.\nFrom the Traditional Office to the Digital Office The Chaos of the Traditional Office In the traditional office, we know the challenges well: overloaded employees juggling multiple tasks, manual processes that eat up precious hours, and the constant need for coordination across departments. Imagine a data analyst who has to gather information from five different systems, consolidate reports by hand and still answer dozens of emails about project status. This scenario repeats itself in thousands of companies every day.\nThe inefficiency isn\u0026rsquo;t just in how tasks are executed, but in human nature itself: we need breaks, we have limits on how much we can process at once and, inevitably, we make mistakes when dealing with large volumes of information. On top of that, knowledge ends up concentrated in specific people, creating bottlenecks when those experts are unavailable or leave the company.\nThe Digital Office Revolution The digital office represents a fundamental paradigm shift. Here, AI agents take on the role of specialized employees, each with specific skills and the ability to work nonstop. These \u0026ldquo;digital collaborators\u0026rdquo; don\u0026rsquo;t replace humans, but amplify their capabilities, taking on repetitive tasks and letting professionals focus on strategic and creative work.\nIn the digital office, an agent specialized in data analysis can continuously monitor dozens of information sources, spot anomalous patterns and generate personalized reports in real time. Another agent can automatically manage customer service, escalating only the cases that truly require human intervention. A third can monitor IT infrastructure, predicting problems before they happen and running automatic fixes.\nThe fundamental difference lies in the nature of these agents: they\u0026rsquo;re not just automated scripts, but intelligent systems able to understand context, make data-driven decisions and learn from every interaction. They can adapt their responses to the situation, tailor their approach to different users and even collaborate with one another to solve complex problems.\nThe Tangible Benefits of the Transformation The transition to the digital office brings measurable, immediate benefits. Companies that have deployed AI agents report significant reductions in customer response time, greater accuracy in analysis and human resources freed up for higher-value work. Mastercard, for example, used digital agents to speed up customer onboarding, drastically cutting the time needed to activate new users.\nAT\u0026amp;T deployed agents specialized in fraud detection, creating a layer of protection that operates 24 hours a day, analyzing behavior patterns and identifying threats in real time. These cases show how AI agents aren\u0026rsquo;t just automation tools, but true multipliers of business capacity.\nMeet the New Digital Employees What AI Agents Are Artificial intelligence agents are autonomous systems that work like specialized digital employees, able to perceive their environment, make decisions and take actions to reach specific goals. Unlike simple automations, these agents have cognitive capabilities that let them adapt to new situations, learn from past experience and collaborate effectively with humans and other agents.\nImagine an employee who never gets tired, processes information at superhuman speed, stays absolutely consistent in their actions and is always available. That\u0026rsquo;s the basic profile of an AI agent. They operate through four fundamental capabilities: perception (gathering information through APIs, sensors and interfaces), cognition (processing and analysis using advanced language models), action (carrying out tasks through tools and systems) and learning (continuous improvement based on feedback and results).\nTypes of Digital Employees In the modern digital office, we find different types of agents, each specialized in specific functions. Customer service agents are like super-efficient receptionists, able to understand complex inquiries, access information from multiple systems and deliver personalized answers instantly. They can handle hundreds of interactions at once, keeping the same level of quality and courtesy in every conversation.\nData analysis agents work like senior analysts who never stop. They continuously monitor data streams, identify trends, detect anomalies and generate actionable insights. Unlike static reports, these agents can adapt their analysis to the context, zeroing in on specific metrics when they detect significant changes in the business.\nIT operations agents act as proactive system administrators, monitoring infrastructure, predicting failures, performing preventive maintenance and solving problems automatically. They can diagnose complex issues by analyzing logs from multiple systems, correlating events and implementing solutions without human intervention.\nCapabilities and Limitations Modern AI agents have impressive capabilities, but it\u0026rsquo;s important to understand both their strengths and their limitations. They excel at tasks involving large-volume data processing, pattern recognition, execution of standardized processes and decision-making based on clear rules. They can work with multiple information sources at once, keeping context and consistency over long interactions.\nHowever, these agents still struggle in situations that require genuine creativity, deep emotional understanding or decision-making in completely new, unprecedented scenarios. They work best when they have access to structured data and when the goals are clearly defined. Human oversight remains essential to validate critical decisions and provide strategic direction.\nPractical Examples in Action In practice, these digital employees are transforming operations across many industries. In e-commerce, specialized agents automatically manage the full sales cycle: they qualify leads by analyzing browsing behavior, personalize product recommendations, process orders, coordinate logistics and track post-sale satisfaction. They can identify customers at risk of churning and automatically roll out personalized retention strategies.\nIn the financial sector, compliance agents monitor transactions in real time, identifying suspicious patterns and ensuring regulatory compliance. They analyze millions of transactions a day, applying complex fraud detection rules and raising alerts only for cases that truly require human investigation. That lets security teams focus on genuine threats, significantly increasing the effectiveness of protection.\nIn human resources, specialized agents automate recruiting processes, analyzing résumés, running initial screenings and even scheduling interviews. They can evaluate candidates against objective criteria, removing unconscious bias and ensuring fairer, more efficient processes. They can also continuously monitor employee satisfaction, spotting signs of discontent before they turn into turnover.\nAgent Bricks: The Automatic HR Department The Revolution in Hiring Digital Employees Traditionally, building AI agents was like trying to hire specialized employees without a structured HR department. Companies needed highly skilled technical teams, months of development, complex evaluation processes and constant manual tuning. Databricks\u0026rsquo; Agent Bricks changes this picture, working like an automatic human resources department that drastically simplifies the process of \u0026ldquo;hiring\u0026rdquo; and training digital agents.\nAgent Bricks represents a fundamental shift in the approach to agent development. Instead of focusing on the technical complexity of the implementation, it lets companies concentrate on what really matters: clearly defining what they need the agent to do and providing guidance on quality and performance. It\u0026rsquo;s like having an experienced HR director who understands exactly the profile needed for each position and can find, train and evaluate candidates automatically.\nHow the \u0026ldquo;Hiring\u0026rdquo; Process Works The process of building agents with Agent Bricks follows a structured methodology that removes the traditional technical complexity. First, you define the \u0026ldquo;job description\u0026rdquo; in natural language, specifying exactly what the agent should do, which data it should access and which results are expected. That description works as a detailed brief that guides the whole development process.\nNext, Agent Bricks acts as a specialized recruiter, automatically analyzing the requirements and creating evaluation benchmarks specific to the task. It can generate synthetic data for testing, create validation scenarios and even develop custom metrics to measure the agent\u0026rsquo;s performance. This process, which would traditionally take weeks of manual work, runs automatically in a matter of hours.\nThe system then \u0026ldquo;trains\u0026rdquo; the agent, optimizing its performance through iterative cycles of testing and tuning. Like an experienced supervisor, Agent Bricks continuously monitors performance, identifies areas for improvement and applies optimizations automatically. This continuous improvement process ensures the agent keeps evolving, adapting to new scenarios and maintaining high quality in what it delivers.\nTypes of \u0026ldquo;Positions\u0026rdquo; Available Agent Bricks offers different \u0026ldquo;hiring models\u0026rdquo; through specialized templates that work like predefined job descriptions. Knowledge Assistant is like hiring a specialized librarian who knows all the company\u0026rsquo;s documentation inside out. This agent can answer complex questions about products, policies, procedures and the organization\u0026rsquo;s history, providing accurate, contextualized answers based on internal documents.\nInformation Extraction works like a data analyst specialized in turning unstructured information into organized data. It can process contracts, reports, emails and all kinds of documents, extracting specific information and organizing it into structured formats. That\u0026rsquo;s particularly valuable for companies that deal with large volumes of documents and need to extract insights quickly.\nCustom LLM lets you create fully customized agents, like hiring a tailor-made specialist for the company\u0026rsquo;s specific needs. These agents can be trained on proprietary data, follow the organization\u0026rsquo;s specific protocols and integrate seamlessly with existing systems. Multi-Agent Supervisor acts as a team manager, coordinating multiple agents and making sure they work in harmony to reach complex goals.\nBenefits of Automatic HR The main advantage of Agent Bricks is the democratization of AI agent creation. Companies that used to need specialized technical teams can now \u0026ldquo;hire\u0026rdquo; digital employees as easily as using a spreadsheet. That drastically lowers the barriers to AI adoption, letting organizations of every size benefit from this technology.\nThe system also removes the \u0026ldquo;bad hire\u0026rdquo; problem. Through automatic evaluation and continuous optimization, Agent Bricks makes sure each agent is a perfect fit for its role. If an agent isn\u0026rsquo;t performing as expected, the system automatically identifies the issues and applies fixes, like an HR department that constantly monitors employee performance and offers extra training when needed.\nAgent Bricks also solves the scalability challenge. Once an agent has been created and optimized, it can be replicated instantly to handle bigger workloads. It\u0026rsquo;s like being able to \u0026ldquo;clone\u0026rdquo; your best employee whenever demand grows, keeping the same level of quality and efficiency.\n\u0026ldquo;Hiring\u0026rdquo; Success Stories Companies like Mastercard used Agent Bricks to \u0026ldquo;hire\u0026rdquo; digital assistants specialized in customer onboarding. These agents fully automated the activation process for new users, cutting processing time from days to minutes. The result was a significant improvement in customer experience and a human team freed up for more strategic work.\nAT\u0026amp;T deployed agents specialized in fraud protection, creating a digital team that works 24 hours a day analyzing suspicious behavior patterns. These \u0026ldquo;digital employees\u0026rdquo; can process millions of transactions simultaneously, identifying threats more accurately than traditional methods and responding in real time.\nCrisis Text Line used Agent Bricks to create agents specialized in mental health, able to provide initial support to people in crisis. These agents were trained to recognize emergency signs, offer appropriate resources and escalate critical cases to human professionals when needed. The result was a significant expansion in service capacity without compromising the quality of the care offered.\nLakebase: The Perfect Communication System The Digital Office Infrastructure Every efficient office needs a communication and information storage system that lets employees quickly access the data they need to do their jobs. In the digital office, Databricks\u0026rsquo; Lakebase takes on that fundamental role, working as a combination of file system, internal communication network and corporate library, all optimized specifically for the needs of AI agents.\nLakebase represents a new category of database, designed from the ground up for the age of artificial intelligence. Unlike traditional databases, which were built for conventional applications and later adapted for AI, Lakebase was conceived specifically to support the unique demands of intelligent agents: simultaneous access by thousands of agents, the ability to experiment and roll back, and seamless integration between operational and analytical data.\nFeatures of the Perfect System Lakebase operates on principles that make it ideal for the modern digital office. Its serverless architecture means it scales automatically with demand, like a communication system that expands instantly when more digital employees come online. There\u0026rsquo;s no need to provision resources in advance or worry about capacity limits.\nSeparating compute from storage lets agents access data extremely efficiently, without the bottlenecks typical of traditional systems. It\u0026rsquo;s like having a central archive where any employee can access any document instantly, with no queues or limits on simultaneous access. This architecture also results in significantly lower costs, since you pay only for the resources you actually use.\nPostgreSQL compatibility ensures Lakebase integrates seamlessly with existing systems, removing the need for complex migrations or application rewrites. It\u0026rsquo;s like rolling out a new communication system that works perfectly with all the tools the company already uses, with no extra training or disruptive changes.\nAdvanced Features for Agents One of Lakebase\u0026rsquo;s most innovative features is its branching capability, similar to version control in software development. It lets agents create \u0026ldquo;working copies\u0026rdquo; of the data for experimentation, testing different scenarios without affecting production data. Imagine employees who can create parallel versions of documents to test different approaches, with the ability to instantly revert to the original version if needed.\nThis feature is particularly valuable for agents that need to learn and adapt. They can try out different strategies, analyze the results and implement only the changes that proved successful. Checkpointing lets agents save specific states of their work, creating restore points that can be used for rollback if something goes wrong.\nNative integration with the lakehouse removes the need for complex ETL between operational and analytical systems. Agents can access both real-time transactional data and historical information for analysis, all through the same interface. It\u0026rsquo;s like having a unified file system where operational reports and strategic analysis coexist perfectly.\nBenefits for Performance and Scalability Lakebase was designed to support thousands of agents operating simultaneously, each with its own data access needs. That massive scale is essential for companies planning to deploy agents at large scale. Unlike traditional databases that can become bottlenecks as the number of connections grows, Lakebase maintains consistent performance regardless of load.\nThe serverless architecture means there\u0026rsquo;s no need for capacity planning or infrastructure management. The system adapts automatically to demand, scaling up during activity peaks and scaling down during quieter periods. That results in optimized costs and removes the need to overprovision resources to handle occasional peaks.\nData access speed is optimized specifically for the usage patterns of AI agents, which often need to access large volumes of data quickly to make real-time decisions. Lakebase uses advanced caching and query optimization techniques to ensure agents have instant access to the information they need.\nIntegration with the Enterprise Ecosystem Lakebase doesn\u0026rsquo;t operate in isolation; it integrates seamlessly with the company\u0026rsquo;s existing data ecosystem. Its compatibility with open standards ensures BI tools, analytics applications and reporting systems keep working without modification. It\u0026rsquo;s like rolling out a new communication system that improves efficiency without breaking existing processes.\nThe integration with Agent Bricks creates a powerful synergy: while Agent Bricks works as the HR department that hires and trains agents, Lakebase provides the data infrastructure those agents need to operate efficiently. This combination lets companies deploy complete AI solutions without worrying about the technical complexities of integration.\nMulti-cloud support ensures deployment flexibility and avoids vendor lock-in. Companies can choose where to host their data based on compliance, performance or cost requirements, keeping the same functionality and interface regardless of the platform chosen.\nTransformative Use Cases In practice, Lakebase is enabling use cases that were impractical with traditional technologies. E-commerce companies are using agents that analyze customer behavior in real time, simultaneously accessing transactional data, browsing history and inventory information to personalize experiences instantly. Lakebase\u0026rsquo;s ability to handle this complex workload without performance degradation is essential to the success of these implementations.\nFinancial institutions are deploying compliance agents that monitor transactions in real time, correlating information from multiple sources to detect suspicious patterns. Lakebase lets these agents access both current transactional data and extensive history for pattern analysis, all with the speed needed for real-time detection.\nManufacturing companies are using agents for supply chain optimization, which need to access production, inventory, logistics and demand data simultaneously. Lakebase gives these agents a unified view of the entire operation, so they make optimized decisions based on complete, up-to-date information.\nPutting the Digital Office into Practice First Steps Toward the Transformation The transition to a digital office doesn\u0026rsquo;t happen overnight, but it can be implemented gradually and strategically. The first step is to identify processes that are good candidates for intelligent automation: repetitive tasks that take significant time, processes that require analyzing large volumes of data, or activities that need to run 24 hours a day.\nStart by mapping the \u0026ldquo;roles\u0026rdquo; you\u0026rsquo;d like to fill with digital employees. Look at which departments spend the most time on work that could be automated and which processes cause the most frustration because of their repetitive nature. This initial analysis will help you prioritize where AI agents can have the biggest immediate impact.\nIt\u0026rsquo;s important to start with pilot projects that have a well-defined scope and clear success metrics. Pick a specific process, such as answering frequent customer inquiries or analyzing sales reports, and deploy a dedicated agent for that function. That lets you validate the approach, adjust processes and demonstrate value before expanding to other areas.\nGradual Implementation Strategies A successful digital office rollout follows a phased approach. In the first phase, focus on agents that complement human work without completely replacing existing processes. For example, deploy a Knowledge Assistant agent that helps the support team find information quickly, while keeping humans responsible for the final interaction with customers.\nIn the second phase, expand to agents that can take on full responsibility in specific areas. Information Extraction agents can automatically process incoming documents, categorizing them and extracting relevant information without constant supervision. That frees human employees for work that requires judgment and creativity.\nThe third phase involves deploying more sophisticated agents that can make complex decisions and coordinate multiple activities. These agents can manage complete workflows, from identifying opportunities to taking corrective action, operating with significant autonomy within predefined parameters.\nTechnical and Organizational Considerations The technical implementation with Agent Bricks and Lakebase simplifies the process significantly, but it still requires careful planning. Make sure the data you need is accessible and well organized. Agents are only as effective as the quality of the data available to them, so investing in data cleanup and structuring is essential to success.\nEstablish clear governance for the agents, defining which decisions they can make autonomously and which require human approval. This is particularly important in regulated industries, where certain actions may have compliance implications. Create escalation protocols that let agents hand complex cases over to human oversight when needed.\nTrain your team to work effectively with AI agents. That doesn\u0026rsquo;t just mean teaching them how to use the tools, but also how to supervise agents, interpret their results and recognize when human intervention is needed. Effective collaboration between humans and agents is essential to maximizing the benefits of the implementation.\nMeasuring Success and ROI Set clear metrics to evaluate the success of your digital office rollout. Measure not only operational efficiency, such as reduced processing time or a higher volume of completed tasks, but also the quality of the results and end-user satisfaction. Agents that process tasks quickly but produce low-quality results don\u0026rsquo;t add real value.\nMonitor the impact on human productivity. The goal isn\u0026rsquo;t just to automate tasks, but to free employees up for higher-value work. Measure how the time saved is being used and whether it\u0026rsquo;s resulting in innovation, process improvement or better customer service.\nCalculate ROI considering not only the direct implementation costs, but also indirect benefits such as improved customer satisfaction, fewer errors and the ability to operate 24/7. Many companies discover that the intangible benefits, such as greater agility and responsiveness, are just as valuable as the direct cost savings.\nOvercoming Common Challenges One of the main challenges in implementing the digital office is resistance to change. Employees may fear that AI agents threaten their jobs. Communicate clearly that the goal is to amplify human capabilities, not replace them. Show how agents can eliminate tedious tasks and let people focus on more interesting, strategic work.\nAnother common challenge is expecting immediate results. Although Agent Bricks significantly simplifies agent creation, it still takes time to adjust processes, train teams and optimize performance. Set realistic expectations and celebrate incremental wins along the way.\nIntegration with legacy systems can present technical complexities. Lakebase helps minimize these challenges through its compatibility with open standards, but custom integration work may still be needed. Plan properly for these activities and consider a phased rollout to reduce risk.\nEvolving the Digital Office The digital office isn\u0026rsquo;t a final destination, but an ongoing process of evolution. As agents prove successful in specific areas, look for opportunities to expand their responsibilities or create new agents for other functions. The modular nature of Agent Bricks makes this gradual expansion easier.\nStay up to date with new capabilities and features. The AI field evolves quickly, and new possibilities keep emerging. Join user communities, follow platform updates and experiment with new features in test environments before deploying to production.\nConsider creating internal AI centers of excellence, where specialized teams can develop best practices, train other departments and coordinate agent initiatives across the organization. That ensures knowledge is shared and that the implementation stays consistent and high quality across the company.\nThe Future of Work Is Already Here The digital office is more than a simple technological evolution; it\u0026rsquo;s a fundamental transformation in how we think about productivity, efficiency and collaboration in the business environment. With Agent Bricks and Lakebase, Databricks hasn\u0026rsquo;t just democratized access to artificial intelligence, it has created a complete ecosystem that lets any organization build its own digital workforce.\nThe benefits of this transformation go beyond traditional automation. We\u0026rsquo;re talking about agents that learn, adapt and evolve continuously, creating value in ways that aren\u0026rsquo;t always immediately obvious. The ability to operate 24 hours a day, process massive volumes of information and stay absolutely consistent in every interaction represents a qualitative leap in business capacity.\nThe journey to the digital office isn\u0026rsquo;t just about technology, but about reimagining human potential. When we free our employees from repetitive tasks and give them intelligent tools to amplify their capabilities, we make room for innovation, creativity and strategic growth. The future of work isn\u0026rsquo;t about humans versus machines, but about humans and machines working in perfect harmony to achieve extraordinary results.\nThe companies that embrace this transformation today will be positioned to lead their markets tomorrow. The digital office is no longer a futuristic vision, but an accessible reality that is redefining the standards of operational excellence across every sector of the economy.\nI hope you enjoyed the read. See you in the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/the-new-digital-office-how-ai-agents-became-the-most-efficient-employees/","summary":"Imagine an office where the employees never get tired, work 24 hours a day, are always learning and carry out tasks with surgical precision. This scenario, which seemed like science fiction just a few years ago…","title":"The New Digital Office: How AI Agents Became the Company's Most Efficient Employees"},{"content":"In today\u0026rsquo;s world, where data is the new oil (or maybe the new bricks?), building robust, efficient Analytics solutions on Azure has become a complex engineering task. Platforms like Microsoft Fabric , Azure Databricks, Azure Data Factory, and Power BI give us powerful tools, but using them in isolation, without a master plan, is like trying to put up a skyscraper without a detailed blueprint. The result? Unstable structures, sky-high costs, security gaps, and performance that leaves a lot to be desired.\nImagine your Analytics solution is a modern, sophisticated building. You wouldn\u0026rsquo;t start construction without a detailed blueprint, right? The Azure Well-Architected Framework (WAF) is exactly that: the essential blueprint that guides the construction of your data \u0026ldquo;building.\u0026rdquo; And the Azure Well-Architected Review? Think of it as the rigorous technical inspection carried out by the engineer of record to make sure every beam, every column, and every system complies with best practices, ensuring the quality, safety, and efficiency of the build.\nIn this article, we\u0026rsquo;ll put on our hard hats and explore the five essential \u0026ldquo;foundations\u0026rdquo; (the WAF pillars) that keep your Analytics building on Azure solid. We\u0026rsquo;ll see how to apply this \u0026ldquo;blueprint\u0026rdquo; and carry out the necessary \u0026ldquo;inspections\u0026rdquo; so your solution isn\u0026rsquo;t just functional, but a true masterpiece of data engineering.\n1. The Essential Foundation (Reliability) Just as a building needs solid foundations and a robust structure to withstand earthquakes, storms, and the wear of time, your Analytics solution needs to be reliable. Reliability ensures your data pipelines (like the building\u0026rsquo;s plumbing and electrical systems) run continuously and that the solution can recover quickly from unexpected failures (like a blackout or a burst pipe).\nIn the practice of building Analytics:\nResilient Structures: Design your pipelines in Azure Data Factory or Databricks with automatic retry mechanisms, robust error handling, and checkpoints. If a step of the \u0026ldquo;assembly\u0026rdquo; fails, the process should be able to try again or pick up where it left off, without compromising the whole \u0026ldquo;build.\u0026rdquo; Resistant Materials: Use highly available, redundant data storage, such as Azure Data Lake Storage Gen2 configured with geo-redundancy (GRS) or read-access geo-redundancy (RA-GRS/GZRS). This keeps your \u0026ldquo;building materials\u0026rdquo; (data) safe even if an entire \u0026ldquo;warehouse\u0026rdquo; (data center) runs into trouble. Contingency Plans: Define high availability (HA) and disaster recovery (DR) strategies for critical components. For an Azure Synapse Dedicated SQL Pool, this might mean regular backups and a plan to restore in another region. For Databricks clusters, think about how to quickly recreate the infrastructure in the event of a regional failure. Structural Monitoring: Just as sensors monitor a building\u0026rsquo;s structural integrity, use Azure Monitor to track the health of your Analytics services. Set up alerts for pipeline failures, cluster unavailability, or performance degradation, so you can respond quickly to any \u0026ldquo;crack\u0026rdquo; in the structure. Load Simulations (Failure Testing): Periodically run tests to validate your recovery plans. It\u0026rsquo;s like running fire or earthquake drills to make sure the safety and evacuation systems work as expected. 2. The Security System (Security) A valuable building needs a robust security system: strong locks, strict access control, surveillance cameras, alarms, and maybe even high walls. In the same way, your Analytics solution, which handles potentially sensitive data, requires multiple layers of security to protect against unauthorized access, leaks, and other threats.\nIn the practice of building Analytics:\nIsolating the Site: Use Virtual Networks (VNets), Private Endpoints, and Network Security Groups (NSGs) to create a secure perimeter around your Analytics services. Limit public access and make sure only authorized components can communicate. Access Control (Keys and Badges): Integrate your services with Azure Active Directory (Entra ID) for centralized identity management. Use strong authentication (MFA), Conditional Access, and Managed Identities so services authenticate to each other securely. Apply the principle of least privilege using Role-Based Access Control (RBAC) and, where possible, Attribute-Based Access Control (ABAC) to define who can access which \u0026ldquo;rooms\u0026rdquo; (resources) and \u0026ldquo;documents\u0026rdquo; (data). Protecting the Materials (Data): Encrypt your data both at rest (using transparent data encryption in Synapse SQL and ADLS Gen2 encryption) and in transit (TLS/SSL). Implement techniques like dynamic data masking to hide sensitive information from unauthorized users in Power BI reports or SQL queries. Continuous Surveillance: Enable diagnostic and audit logs on every service (Data Factory, Databricks, Synapse, Key Vault). Use Microsoft Defender for Cloud to monitor security configurations, detect threats, and assess vulnerabilities in your \u0026ldquo;build.\u0026rdquo; Governance (The Building Rules): Use Microsoft Purview to catalog your data assets, classify sensitive information, define access policies, and track data lineage. It\u0026rsquo;s like having a clear manual on how the \u0026ldquo;residents\u0026rdquo; (users and services) can interact with the building\u0026rsquo;s different \u0026ldquo;spaces\u0026rdquo; (data). 3. The Construction Budget (Cost Optimization) No construction project happens without a strict budget. Optimizing costs in Analytics means choosing the most cost-effective materials (services), properly sizing the labor and equipment (compute resources), avoiding waste, and continuously monitoring spending so you don\u0026rsquo;t blow the budget.\nIn the practice of building Analytics:\nSmart Sizing: Configure autoscaling for Databricks clusters and Synapse Spark pools. Use the pause and resume option for Synapse Dedicated SQL Pools. Avoid leaving \u0026ldquo;heavy machinery\u0026rdquo; (large clusters) running unnecessarily. Choosing the Right Materials: Use the right service for the right job. Orchestrate with Data Factory, process massive data with Databricks/Synapse Spark, serve aggregated data with Synapse SQL or Analysis Services, visualize with Power BI. Don\u0026rsquo;t use a \u0026ldquo;crane\u0026rdquo; (a powerful Spark cluster) to lift a \u0026ldquo;bag of cement\u0026rdquo; (a simple task). Optimizing the Site (Storage): Store data in ADLS Gen2 using efficient columnar formats like Delta Lake or Parquet. Use partitioning and compression. Implement lifecycle policies to move cold data to infrequent-access tiers (Cool/Archive tiers), freeing up space on the main \u0026ldquo;site.\u0026rdquo; Avoiding Rework (Data Movement): Minimize unnecessary data movement between services. Use features like data virtualization or in-place processing whenever possible. Management Tools: Use Azure Cost Management + Billing to track spending by resource or tag. Follow the Azure Advisor recommendations, which often suggest optimizations specific to your Analytics services. Long-Term Contracts: If you have predictable workloads, consider Azure Reservations or Azure Savings Plans to get significant discounts on compute resources (VMs for Databricks, Synapse SQL DWUs). 4. Maintenance and Operations (Operational Excellence) Handing over the keys to the building isn\u0026rsquo;t the end of the story. Operational excellence is about the processes that keep the \u0026ldquo;build\u0026rdquo; running perfectly over time: preventive maintenance plans, building automation, clear documentation, and well-trained teams.\nIn the practice of building Analytics:\nConstruction Automation (CI/CD \u0026amp; IaC): Use Azure DevOps or GitHub Actions to build CI/CD pipelines that automate testing and deployment of code (Data Factory pipelines, Databricks notebooks, SQL scripts) and infrastructure (using ARM Templates or Terraform). This ensures consistency and reduces manual errors during \u0026ldquo;renovations\u0026rdquo; or \u0026ldquo;expansions.\u0026rdquo; Continuous Quality Control: Implement automated tests in your pipelines: unit tests for transformations, integration tests between components, and data quality tests to validate the results. That\u0026rsquo;s your guarantee that every \u0026ldquo;floor\u0026rdquo; delivered meets the specifications. Centralized Monitoring (Control Panel): Bring logs and metrics from all your Analytics services into Azure Monitor and Log Analytics. Build dashboards and proactive alerts to catch problems before they affect the \u0026ldquo;residents\u0026rdquo; (end users). Good Management Practices (DataOps/MLOps): Adopt DataOps principles to speed up the delivery of reliable data pipelines. If your solution involves Machine Learning, implement MLOps to manage the model lifecycle. Documentation (Up-to-Date Blueprints): Keep the documentation of your architecture, pipelines, and processes current. This is crucial for onboarding new team members and for troubleshooting. 5. Functionality and Finish (Performance Efficiency) What good is a safe, well-built building if the elevators are slow, the layout is confusing, and the climate control doesn\u0026rsquo;t work properly? Performance efficiency ensures your Analytics solution doesn\u0026rsquo;t just work, but works well, delivering results quickly and adapting to changes in demand (more \u0026ldquo;residents\u0026rdquo; or \u0026ldquo;visitors\u0026rdquo;).\nIn the practice of building Analytics:\nOptimized Layout (Data Design): Use techniques like smart partitioning in ADLS Gen2 and in tables (Delta, Synapse SQL), proper indexing (Synapse SQL), and optimized schema design (star schema, snowflake) to speed up queries. Powerful Engines (Compute): Choose the right cluster types and sizes (Databricks) or service levels (Synapse SQL Pools) for your workload. Use autoscaling to adjust the \u0026ldquo;horsepower\u0026rdquo; as needed. Lightweight Materials (Formats and Cache): Use columnar formats (Parquet, Delta) that optimize reads. Implement caching at different layers: data caching in Spark, result caching in Power BI (Import mode or DirectQuery with aggregations), and Synapse SQL caching. Internal Logistics (Query Optimization): Analyze the execution plans of your SQL queries or Spark jobs. Optimize joins, filter data as early as possible (predicate pushdown), and use up-to-date statistics to help the optimizer. Measuring Satisfaction (Monitoring): Continuously monitor query latency, pipeline execution time, and resource utilization. Identify bottlenecks and optimize proactively. 6. The Quality Inspection (The Azure Well-Architected Review for Analytics) Now that we know the foundations, how do we make sure our build is actually following the blueprint? This is where the Azure Well-Architected Review, our technical inspection, comes in. Microsoft offers an online assessment tool, including a version specific to Analytics workloads.\nThe tool walks you through a series of questions focused on each of the five pillars, applied to the context of your services (Synapse, Databricks, and so on). For example:\nReliability: Do you have retry mechanisms in your Data Factory pipelines? Security: Is your sensitive data in ADLS Gen2 encrypted at rest? Cost: Do you use autoscaling on your Databricks clusters? Operations: Do you have a CI/CD process for your Analytics artifacts? Performance: Are your Synapse SQL tables partitioned correctly? By answering these questions, you carry out a self-assessment of your \u0026ldquo;build.\u0026rdquo; The result is a detailed report that points out the strengths and, more importantly, the areas where your architecture may not be aligned with best practices: the \u0026ldquo;nonconformities\u0026rdquo; found during the inspection.\n7. Adjusting the Blueprint and the Build (Implementing Recommendations) The inspection report isn\u0026rsquo;t just there to point out problems; it provides practical recommendations to fix them. It\u0026rsquo;s like the engineer saying, \u0026ldquo;We need to reinforce this beam,\u0026rdquo; \u0026ldquo;The thermal insulation here isn\u0026rsquo;t adequate,\u0026rdquo; or \u0026ldquo;Let\u0026rsquo;s optimize the ventilation system.\u0026rdquo;\nImplementing the Well-Architected Review recommendations is the process of adjusting your \u0026ldquo;blueprint\u0026rdquo; and your \u0026ldquo;build\u0026rdquo;:\nInterpretation: Understand the why behind each recommendation and its impact on your specific scenario. Prioritization: Not every recommendation needs to be implemented right away. Prioritize based on risk, business impact, and the effort required. A structural flaw (reliability) or an open door (security) usually takes priority over fine-tuning the finish (performance). Continuous Cycle: Construction is never truly finished. The Well-Architected Review isn\u0026rsquo;t a one-time event but a continuous cycle. Review your architecture periodically (every 6 months, or after significant changes), implement the improvements, measure the results, and repeat the inspection. It\u0026rsquo;s the preventive maintenance that ensures the longevity and value of your data \u0026ldquo;building.\u0026rdquo; Conclusion Building a cutting-edge Analytics solution on Azure is a significant undertaking, a lot like putting up a complex building. Ignoring the \u0026ldquo;blueprint\u0026rdquo; (the Well-Architected Framework) and skipping the \u0026ldquo;quality inspections\u0026rdquo; (the Well-Architected Review) means risking delivery of a structure that is fragile, insecure, expensive, and inefficient.\nBy adopting the five pillars as the essential foundations of your build (Reliability, Security, Cost Optimization, Operational Excellence, and Performance Efficiency) and using the Review as your regular inspection tool, you make sure your Analytics solution is a true work of engineering: robust, protected, economical, easy to operate, and highly performant.\nSo the next time you kick off an Analytics project on Azure or review an existing one, grab your \u0026ldquo;blueprint,\u0026rdquo; put on your hard hat, and carry out your \u0026ldquo;inspection.\u0026rdquo; The result will be a data \u0026ldquo;building\u0026rdquo; you can be proud of, ready to support your business needs today and in the future.\nSee you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/perfect-blueprint-for-analytics-on-azure-well-architected-review/","summary":"In today\u0026rsquo;s world, where data is the new oil (or maybe the new bricks?), building robust, efficient Analytics solutions on Azure has become a complex engineering task. Platforms like Microsoft Fabric…","title":"The Perfect Blueprint for Analytics on Azure: Building Solid Solutions with the Well-Architected Review"},{"content":"For more than a decade, Apache Spark has been the data processing platform of choice for data engineers and data scientists around the world. With its ability to process large volumes of information in memory, unify batch and streaming analytics, and offer intuitive APIs in multiple languages, Spark has earned its place as an essential tool in the big data ecosystem. Now, with the release of Apache Spark 4.0, we\u0026rsquo;re witnessing a significant evolution that promises to transform the experience of working with data at scale even further.\nPicture Spark 4.0 as a completely renovated amusement park that has just reopened its gates. \u0026ldquo;Spark Land\u0026rdquo; was already known for its thrilling attractions, but now it\u0026rsquo;s back with new high-speed roller coasters, redesigned themed areas, more efficient FastPass systems and a host of amenities that improve the experience for every visitor. Just as a modern theme park offers fun for different kinds of visitors, Spark 4.0 brings new features for Python developers, data engineers, SQL analysts and data scientists.\nIn this article, we\u0026rsquo;ll explore the different areas of this fascinating data park: the main attractions area with the new features, the special equipment area with extensions and APIs, the personalized experiences area with customizable functions and procedures, and the services and amenities area with significant usability improvements. Grab your ticket and come explore Spark Land 4.0!\nThe Main Attractions Area (New Features) Spark Connect: The Virtual Roller Coaster\nEvery big park opening needs a groundbreaking attraction, and at Spark Land 4.0 that attraction is Spark Connect. Imagine a roller coaster you can ride remotely, without having to be physically at the park: that\u0026rsquo;s exactly what Spark Connect offers the data world.\nPreviously, working with Spark meant installing the entire environment (a hefty 355 MB package) on your local machine. It was like having to build a replica of the park at home just to try the rides. With Spark Connect, you can now use a lightweight client of just 1.5 MB, connecting remotely to the cluster where the \u0026ldquo;heavy fun\u0026rdquo; happens.\n```python # Criando uma sessão com Spark Connect from pyspark.sql import SparkSession spark = SparkSession \\ .builder \\ .appName(\u0026#34;Exemplo Spark Connect\u0026#34;) \\ .master(\u0026#34;sc://seu-servidor:15002\u0026#34;) \\ .getOrCreate() # Agora você pode trabalhar com DataFrames remotamente df = spark.read.format(\u0026#34;csv\u0026#34;).option(\u0026#34;header\u0026#34;, \u0026#34;true\u0026#34;).load(\u0026#34;dados.csv\u0026#34;) df.show() ``` This feature is like using virtual reality goggles to experience the park\u0026rsquo;s attractions from anywhere. Developers can now work in their favorite IDEs, such as PyCharm or Jupyter, connecting remotely to Spark clusters without the overhead of local resources. It\u0026rsquo;s as if the park came to you!\nANSI Mode: The Classic Carousel, Renovated\nEvery park has that classic attraction everyone knows and loves, but which needs a makeover from time to time. ANSI Mode, enabled by default in Spark 4.0, is like the traditional carousel that has been fitted with new safety and quality control systems.\nThis feature aligns Spark SQL with ANSI SQL standards, enforcing stricter rules for NULL handling, type conversions and arithmetic operations. For example, in earlier versions a numeric overflow simply wrapped around, potentially leading to incorrect results with no warning. Now, with ANSI Mode, you get a clear exception:\n```sql -- Em versões anteriores: retornaria -2147483648 (wrap-around) -- No Spark 4.0 com ANSI Mode: lança uma exceção de overflow SELECT 2147483647 + 1; ``` It\u0026rsquo;s like a carousel that now has upgraded seat belts and sensors that stop the ride automatically if anything goes out of spec. This change is especially valuable for anyone migrating from traditional databases to Spark, since they\u0026rsquo;ll find more familiar, predictable behavior, like a visitor who feels right at home finding their favorite ride, only safer now.\nVariant Data Type: The Magic House of Mirrors\nOne of the most fascinating attractions in any park is the house of mirrors, where your reflection is distorted and transformed in surprising ways. The Variant Data Type in Spark 4.0 works in a similar way, offering a flexible way to work with semi-structured data, adapting to different shapes and structures.\nThis new data type lets you store and process complex hierarchical structures efficiently, using shredding techniques to optimize performance. It\u0026rsquo;s ideal for scenarios such as IoT and web log analysis, where the data structure can vary:\n```python # Trabalhando com o tipo Variant from pyspark.sql import SparkSession spark = SparkSession.builder.appName(\u0026#34;Exemplo Variant\u0026#34;).getOrCreate() data = [(\u0026#34;sensor1\u0026#34;, {\u0026#34;temperatura\u0026#34;: 25.5, \u0026#34;umidade\u0026#34;: 60}), (\u0026#34;sensor2\u0026#34;, [{\u0026#34;temperatura\u0026#34;: 26.1}, {\u0026#34;temperatura\u0026#34;: 26.3}])] df = spark.createDataFrame(data, [\u0026#34;id\u0026#34;, \u0026#34;leituras\u0026#34;]) # Acessando campos dentro do tipo Variant df.select(\u0026#34;id\u0026#34;, \u0026#34;leituras.temperatura\u0026#34;).show() ``` It\u0026rsquo;s like walking into a house of mirrors where each mirror shows a different version of you (sometimes taller, sometimes wider, sometimes completely different) but still recognizably you. The Variant Type lets your data take on different shapes while keeping its essence and accessibility.\nCollation Support: The International Tower\nA global amusement park needs to serve visitors from different countries and cultures. Spark Land 4.0\u0026rsquo;s International Tower, Collation Support, does exactly that for your text data. This feature lets you specify how string comparisons should be handled, taking into account aspects such as case sensitivity and locale-specific rules.\n```sql -- Criando uma tabela com collation específica CREATE TABLE nomes ( nome STRING COLLATE \u0026#39;pt_BR.UTF8\u0026#39; ); -- Consulta que respeita as regras de collation SELECT * FROM nomes WHERE nome = \u0026#39;José\u0026#39; COLLATE \u0026#39;pt_BR.UTF8\u0026#39;; ``` It\u0026rsquo;s like an observation tower with multilingual guides who tailor the experience for visitors from different countries. For multilingual applications, this feature is essential, ensuring consistent sorting and comparison across different cultural settings, like making sure every visitor, wherever they come from, can fully enjoy the park\u0026rsquo;s panoramic view.\nThe Special Equipment Area (Extensions) Python Data Source APIs: The DIY Building Kit\nEvery modern park has an area where visitors can customize their experience. The Python Data Source APIs in Spark 4.0 are like a \u0026ldquo;Do It Yourself\u0026rdquo; kit that lets Python developers create their own custom attractions without having to resort to Java or Scala.\n```python from pyspark.sql.datasource import DataSourceReader, DataSourceWriter class MeuLeitor(DataSourceReader): def read_batch(self): # Lógica personalizada para leitura de dados return spark.createDataFrame([(\u0026#34;valor1\u0026#34;,), (\u0026#34;valor2\u0026#34;,)], [\u0026#34;coluna\u0026#34;]) # Registrando e usando a fonte de dados personalizada spark.read.format(\u0026#34;minha_fonte\u0026#34;).load().show() ``` This extension is like having a set of building blocks, tools and instructions to create your own mini-attraction inside the park. You\u0026rsquo;re no longer limited to prefabricated experiences: now you can build something unique that meets exactly your specific needs.\nDelta 4.0: The Premium Safety System\nSafety infrastructure is what lets an amusement park run smoothly, even if visitors don\u0026rsquo;t notice it directly. Delta 4.0 at Spark Land is that premium safety system, bringing advanced features to the Delta Lake format, including Delta Connect for improved lakehouse management.\nWith improvements in reliability, performance and ease of use, Delta 4.0 raises the quality of the whole park experience, especially for anyone working with lakehouse architectures:\n```python # Usando recursos avançados do Delta 4.0 from delta import DeltaTable # Operação de merge otimizada deltaTable = DeltaTable.forPath(spark, \u0026#34;/caminho/para/tabela\u0026#34;) deltaTable.alias(\u0026#34;destino\u0026#34;) \\ .merge( atualizacoes.alias(\u0026#34;origem\u0026#34;), \u0026#34;destino.id = origem.id\u0026#34; ) \\ .whenMatchedUpdate(set = {\u0026#34;valor\u0026#34;: \u0026#34;origem.novo_valor\u0026#34;}) \\ .whenNotMatchedInsert(values = {\u0026#34;id\u0026#34;: \u0026#34;origem.id\u0026#34;, \u0026#34;valor\u0026#34;: \u0026#34;origem.novo_valor\u0026#34;}) \\ .execute() ``` Just like an advanced safety system that monitors every attraction, manages queues and prevents accidents, Delta 4.0 ensures your data is always safe, consistent and available when needed. It\u0026rsquo;s the peace of mind of knowing that, even behind the scenes, everything is working perfectly.\nXML/Databricks Connectors: The Themed Bridges\nThemed bridges in a park connect different areas, letting visitors move smoothly between distinct experiences. The XML and Databricks connectors in Spark 4.0 work the same way, creating seamless connections between different data formats and ecosystems.\nThe improved XML connector makes it easier to work with data in XML format, which can be complex and nested:\n```python # Lendo dados XML com o conector aprimorado df = spark.read \\ .format(\u0026#34;xml\u0026#34;) \\ .option(\u0026#34;rowTag\u0026#34;, \u0026#34;livro\u0026#34;) \\ .load(\u0026#34;biblioteca.xml\u0026#34;) df.select(\u0026#34;titulo\u0026#34;, \u0026#34;autor.nome\u0026#34;).show() ``` These connectors are like themed, decorated bridges that not only let you pass between areas of the park but also enrich the experience along the way. They let you work with formats and systems that used to be hard to integrate, creating a seamless experience across the whole data park.\nDSV2 Extension: The Smart Power Grid\nAn amusement park\u0026rsquo;s power grid is invisible to most visitors, but it\u0026rsquo;s absolutely essential for every attraction to work. The DataSource V2 (DSV2) extension in Spark 4.0 works like that smart power grid, offering improved APIs for data sources that benefit the entire ecosystem.\nThis extension provides greater flexibility for implementing data sources, with better support for predicate pushdown, partitioning and other optimizations:\n```python # Usando uma fonte de dados baseada em DSV2 df = spark.read \\ .format(\u0026#34;jdbc\u0026#34;) \\ .option(\u0026#34;url\u0026#34;, \u0026#34;jdbc:postgresql://servidor/banco\u0026#34;) \\ .option(\u0026#34;dbtable\u0026#34;, \u0026#34;usuarios\u0026#34;) \\ .option(\u0026#34;pushDownPredicate\u0026#34;, \u0026#34;true\u0026#34;) \\ .load() # O filtro será \u0026#34;empurrado\u0026#34; para o banco de dados filtrado = df.filter(\u0026#34;idade \u0026gt; 30\u0026#34;) ``` Just like a modern, efficient power grid that intelligently distributes energy to every attraction in the park, DSV2 raises the quality of every data source in Spark 4.0, resulting in better performance and expanded capabilities. You don\u0026rsquo;t see it directly, but you feel its positive impact across the whole experience.\nThe Personalized Experiences Area (Functions and Procedures) SQL UDFs and Scripting: The Custom Park\nThe most exclusive parks offer VIP experiences where you can customize every aspect of your visit. SQL UDFs (User-Defined Functions) and SQL Scripting in Spark 4.0 offer that same personalized experience for your SQL queries.\nWith SQL UDFs, you can define custom functions directly in SQL:\n```sql -- Criando uma UDF em SQL CREATE FUNCTION dobro(x INT) RETURNS INT RETURN x * 2; -- Usando a função SELECT id, dobro(valor) AS valor_dobrado FROM tabela; ``` SQL Scripting lets you create complex SQL scripts with procedural logic:\n```sql -- Script SQL com lógica procedural BEGIN DECLARE v_total INT; SET v_total = 0; FOR r IN (SELECT valor FROM tabela WHERE categoria = \u0026#39;A\u0026#39;) DO SET v_total = v_total + r.valor; END FOR; SELECT v_total AS total_categoria_a; END; ``` These capabilities are like having a personal concierge at the park who plans your day exactly the way you want, building a custom itinerary that isn\u0026rsquo;t available to regular visitors. You can express exactly what you want and how you want it, without leaving the SQL environment.\nPython UDTFs: The Multidimensional Simulator\nThe most advanced simulators in theme parks offer immersive experiences in multiple dimensions. Python UDTFs (User-Defined Table Functions) in Spark 4.0 are like those cutting-edge simulators, letting you create Python functions that return multiple rows and columns.\n```python from pyspark.sql.functions import udtf @udtf(returnType=\u0026#34;value INT, quadrado INT, cubo INT\u0026#34;) def gerar_potencias(x): yield (x, x**2, x**3) # Usando a UDTF df = spark.createDataFrame([(1,), (2,), (3,)], [\u0026#34;numero\u0026#34;]) df.select(\u0026#34;numero\u0026#34;, gerar_potencias(\u0026#34;numero\u0026#34;)).show() ``` This feature is like a 4D simulator that delivers multiple sensory experiences at once: you don\u0026rsquo;t just see the adventure, you feel, hear and even smell the elements of the story. UDTFs enable complex transformations that generate multiple results from a single input, significantly enriching your analytical capabilities.\nArrow Optimized: The Universal FastPass\nEvery frequent park visitor knows the value of a FastPass that lets you skip the lines. Apache Arrow in Spark 4.0 works like a Universal FastPass, optimizing data transfer between Python and JVM processes to drastically cut wait times.\nThis optimization significantly reduces serialization/deserialization overhead, which is especially important for Python UDFs and pandas integration:\n```python # Configurando o uso otimizado de Arrow spark.conf.set(\u0026#34;spark.sql.execution.arrow.pyspark.enabled\u0026#34;, \u0026#34;true\u0026#34;) # Convertendo DataFrame Spark para pandas com Arrow pandas_df = spark_df.toPandas() # Aplicando transformações em pandas e voltando para Spark resultado = spark.createDataFrame(pandas_df_transformado) ``` It\u0026rsquo;s like having a special pass that gets you into any attraction without waiting in line, making the whole park experience smoother and more enjoyable. With Arrow, data transfers that used to be bottlenecks now flow quickly, letting you enjoy more analytical \u0026ldquo;attractions\u0026rdquo; in less time.\nPySpark UDF Unified Profiler: The Performance Monitor\nModern parks use sophisticated systems to monitor the performance of their attractions and the visitor experience. The PySpark UDF Unified Profiler in Spark 4.0 is that monitoring system, letting you analyze the performance of Python UDFs to identify bottlenecks and optimize your code.\n```python # Habilitando o profiler unificado spark.conf.set(\u0026#34;spark.python.profile\u0026#34;, \u0026#34;true\u0026#34;) spark.conf.set(\u0026#34;spark.python.profile.dump\u0026#34;, \u0026#34;/tmp/profile\u0026#34;) # Definindo e usando uma UDF @udf(\u0026#34;int\u0026#34;) def funcao_complexa(x): # Lógica complexa return resultado # Executando com profiling df.select(funcao_complexa(\u0026#34;coluna\u0026#34;)).show() # Analisando o resultado do profiling # O arquivo será gerado em /tmp/profile ``` This tool is like a monitoring system that analyzes how long visitors spend at each attraction, where congestion happens and how to improve the overall flow of the park. With the Unified Profiler, you can pinpoint exactly where your Python code is spending the most time and optimize it for a faster, more efficient experience.\nThe Services and Amenities Area (Usability) Structured Logging Framework: The Digital Information System\nA modern amusement park needs an efficient information system that keeps visitors up to date on every attraction. The Structured Logging Framework in Spark 4.0 brings that same attention to detail to logs, structuring them in JSON format to make analysis and monitoring easier.\n``` {\u0026#34;timestamp\u0026#34;: \u0026#34;2025-05-28T10:15:30.123Z\u0026#34;, \u0026#34;level\u0026#34;: \u0026#34;INFO\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;Job 42 completed\u0026#34;, \u0026#34;jobId\u0026#34;: 42, \u0026#34;duration\u0026#34;: 1500, \u0026#34;records\u0026#34;: 10000} ``` Instead of hard-to-parse plain-text logs, you now have structured logs that can easily be processed by monitoring tools. It\u0026rsquo;s like going from paper maps and notice boards to interactive digital screens spread around the park, providing clear, up-to-date information on every attraction, wait time and special event.\nError Class Framework: The Guest Services Desk\nWhen something goes wrong at a park, good guest services make all the difference. The Error Class Framework in Spark 4.0 is that premium service, providing standardized, detailed messages that make problems easier to diagnose.\nERROR [INVALID_SCHEMA.FIELD_NOT_FOUND] Campo \u0026#39;idade\u0026#39; não encontrado no esquema. Esquema atual: [\u0026#39;nome\u0026#39;, \u0026#39;endereco\u0026#39;, \u0026#39;telefone\u0026#39;] Dica: Verifique se o nome do campo está correto ou se a coluna existe no DataFrame. This structuring of errors is like having a well-trained service team that doesn\u0026rsquo;t just tell you something went wrong, but explains exactly what happened, why it happened and how to fix it. Instead of cryptic messages that leave you frustrated, you get clear guidance that saves precious debugging time.\nBehavior Change Process: The Adaptation Guide\nWhen a park renovates its attractions, frequent visitors need to adapt to the changes. The Behavior Change Process in Spark 4.0 is like an adaptation guide that eases the transition to new versions, clearly documenting behavior changes and offering migration paths.\nThis structured process helps teams understand the impact of upgrades and adapt their code gradually, instead of facing unpleasant surprises after an upgrade:\n```python # Configuração para compatibilidade com comportamento anterior spark.conf.set(\u0026#34;spark.sql.legacy.timeParserPolicy\u0026#34;, \u0026#34;LEGACY\u0026#34;) # Gradualmente migrar para o novo comportamento # após testar o impacto spark.conf.set(\u0026#34;spark.sql.legacy.timeParserPolicy\u0026#34;, \u0026#34;CORRECTED\u0026#34;) ``` It\u0026rsquo;s like having a detailed guide that explains every change in the park, with tips on how to make the most of the new attractions while you get used to the differences. This process makes the transition between Spark versions much smoother and more predictable, letting teams adopt new features at their own pace.\nRecommended Itineraries (Use Cases) Now that we\u0026rsquo;ve explored every area of our data park, let\u0026rsquo;s look at some recommended itineraries for different kinds of visitors.\nItinerary for Data Engineers (Adventurers)\nFor data engineers looking to build robust, scalable pipelines, we recommend:\nFirst stop: Spark Connect for remote, collaborative development\nMain attraction: Delta 4.0 for reliable data management\nComplementary experience: DSV2 Extension for optimized data sources\nBefore you leave: Structured Logging for advanced monitoring\nThis itinerary provides a complete experience for building and maintaining reliable, high-performing data pipelines, like a day of extreme adventures at the park.\nItinerary for Data Scientists (Explorers)\nFor data scientists focused on analysis and modeling:\nFirst stop: Python Data Source APIs for integration with specific sources\nMain attraction: pandas 2.x support for familiar exploratory analysis\nComplementary experience: Arrow Optimized for efficient transfer between Spark and pandas\nBefore you leave: Python UDTFs for complex transformations\nThis combination offers a productive environment that integrates seamlessly with the Python ecosystem data scientists already know and love, like a day of exploration and discovery at the park.\nItinerary for SQL Analysts (Classics)\nFor analysts who prefer to work mainly with SQL:\nFirst stop: ANSI Mode for predictable, standardized SQL behavior\nMain attraction: SQL UDFs and Scripting for complex analyses\nComplementary experience: Collation Support for proper handling of text data\nBefore you leave: Error Class Framework for clear problem diagnosis\nThis itinerary provides a robust, expressive SQL experience, enabling sophisticated analyses without leaving the SQL environment, like a day enjoying the park\u0026rsquo;s classic, timeless attractions.\nConclusion Apache Spark 4.0 represents a significant evolution of the platform, bringing a full amusement park of new features, extensions, custom functions and usability improvements. Just as a renovated theme park draws both new visitors and longtime fans, Spark 4.0 offers attractions for every profile in the data world.\nNew features such as Spark Connect and ANSI Mode lay a solid foundation for the future, while extensions such as the Python Data Source APIs and Delta 4.0 expand integration possibilities. Custom functions such as Python UDTFs and Arrow optimizations improve the development experience, and usability improvements such as Structured Logging and the Error Class Framework make diagnosing and solving problems easier.\nThe next time you\u0026rsquo;re working with big data, consider paying a visit to Spark Land 4.0. Ticket in hand (and maybe a FastPass for the most popular attractions), you\u0026rsquo;ll discover a world of analytical possibilities more exciting and efficient than ever. The park is open, so come have fun!\nThis article is part of the \u0026ldquo;From Data to Insights\u0026rdquo; newsletter. For more content on Apache Spark, Databricks and modern data engineering, follow me on LinkedIn.\n","permalink":"https://julianamarialop.github.io/en/articles/spark-land-4-0-data-amusement-park-just-opened-its-gates/","summary":"For more than a decade, Apache Spark has been the data processing platform of choice for data engineers and data scientists around the world. With its ability to process large volumes of information in…","title":"Spark Land 4.0: The Data Amusement Park Has Just Opened Its Gates"},{"content":"Deploying data projects across different environments has always been a challenge comparable to trying to replicate a gourmet dish in different kitchens. Even with the same ingredients, small variations in settings, in the order of preparation or in the tools used can lead to completely different experiences, sometimes with disastrous results.\nThe Databricks Asset Bundle (DAB) arrives as a true professional chef\u0026rsquo;s kit for data engineers. Just as a Michelin-starred chef can bring their own set of knives, spices and detailed recipes to guarantee the same culinary quality in any kitchen in the world, DAB lets data teams package, version and deploy their projects consistently to any Databricks workspace, whether development, test or production.\nThis revolutionary approach transforms how we manage projects on the Databricks platform, bringing modern DevOps practices into the data world. By adopting the concept of \u0026ldquo;Infrastructure as Code\u0026rdquo; (IaC), DAB allows the entire project (including notebooks, workflows, cluster configurations and other resources) to be defined in declarative files, versioned in systems such as Git and deployed automatically through CI/CD pipelines.\nThe Perfect Recipe (What the Databricks Asset Bundle Is) Imagine you\u0026rsquo;re a chef responsible for making sure the same dish is served perfectly in different restaurants of an international franchise. You\u0026rsquo;d need an extremely detailed recipe, one that leaves no room for interpretation or unwanted variation. That\u0026rsquo;s exactly what the Databricks Asset Bundle offers for data projects.\nThe heart of DAB is the databricks.yml file, comparable to a recipe meticulously written by a Michelin-starred chef. This YAML file contains every instruction needed to \u0026ldquo;cook\u0026rdquo; your Databricks project, from defining the \u0026ldquo;ingredients\u0026rdquo; (resources) to the method (configurations) and the final plating (deployment).\nSimplified example of a databricks.yml\nbundle: name: meu-projeto-analytics variables: data_path: default: /data/bronze description: Caminho para os dados brutos targets: dev: workspace: host: https://dev-workspace.cloud.databricks.com variables: data_path: /data/dev/bronze prod: workspace: host: https://prod-workspace.cloud.databricks.com resources: jobs: etl_diario: name: ETL Diário schedule: quartz_cron_expression: \u0026#34;0 0 2 * * ?\u0026#34; tasks: ingestao: notebook_task: notebook_path: /Notebooks/ingestao job_cluster_key: cluster_padrao job_clusters: cluster_padrao: spark_version: 13.3.x-scala2.12 node_type_id: Standard_DS3_v2 autoscale: min_workers: 1 max_workers: 4 Just as a well-written recipe leaves no doubt about quantities, times and techniques, the databricks.yml file removes ambiguity from the project configuration. Each section has a specific purpose:\nBundle metadata: The project\u0026rsquo;s name and basic information Variables: Parameters that can be referenced throughout the file Environments (targets): Specific configurations for different workspaces Resources: Definitions of jobs, clusters, notebooks and other components The Premium Ingredients (Resources Managed by DAB) A chef knows that the quality of the final dish depends directly on the quality of the ingredients. In the context of the Databricks Asset Bundle, the \u0026ldquo;ingredients\u0026rdquo; are the various resources that make up your project:\nNotebooks: The equivalent of the individual recipes that make up a menu Workflows (Jobs): The preparation sequence, like a tasting menu Clusters: The ovens and equipment needed for cooking Secrets: The special ingredients that need to be protected Pipelines: The advanced preparation techniques DAB lets you manage all these \u0026ldquo;ingredients\u0026rdquo; centrally, making sure they are combined in the right proportions and in the proper order. Just as a chef wouldn\u0026rsquo;t leave the choice of ingredients to chance, DAB doesn\u0026rsquo;t leave resource configuration at the mercy of error-prone manual processes.\nOne of the big differentiators of this approach is the ability to version these \u0026ldquo;ingredients\u0026rdquo; along with the code. Imagine being able to keep not only the recipes, but also the exact specification of the ingredients and equipment used in each version of your menu. That\u0026rsquo;s what DAB delivers by integrating with version control systems such as Git.\nCooking in Any Kitchen (Consistency Across Environments) One of the biggest challenges for international chefs is adapting to different kitchens while keeping their dishes consistent. In the same way, data teams face the challenge of making sure their projects behave the same way in development, test and production environments.\nThe Databricks Asset Bundle solves this problem by letting you define environment-specific configurations while keeping the project\u0026rsquo;s basic structure intact. It\u0026rsquo;s like having a base recipe that can be slightly adapted for different occasions without losing its essence.\ntargets: dev: workspace: host: https://dev-workspace.cloud.databricks.com variables: cluster_size: small data_retention: 7 qa: workspace: host: https://qa-workspace.cloud.databricks.com variables: cluster_size: medium data_retention: 14 prod: workspace: host: https://prod-workspace.cloud.databricks.com variables: cluster_size: large data_retention: 30 This capacity for controlled adaptation brings significant benefits:\nPredictability: If it works in development, it will work in production Traceability: Every difference between environments is explicit and documented Reproducibility: Environments can be recreated exactly as they were at any point in time Governance: Specific policies can be applied to each environment The Automated Sous-Chefs (CI/CD Integration) In high-end restaurants, the head chef relies on trusted sous-chefs who follow their instructions to the letter, making sure every dish comes out perfect even when the chef isn\u0026rsquo;t directly involved in every step. In the world of the Databricks Asset Bundle, CI/CD pipelines take on that role.\nIntegrating DAB with tools such as GitHub Actions, Azure DevOps or GitLab CI lets you automate the entire deployment process, from validating the \u0026ldquo;recipe\u0026rdquo; to \u0026ldquo;plating the final dish\u0026rdquo; in the target environment.\nHere\u0026rsquo;s an example of what a GitHub Actions workflow to deploy a DAB project would look like:\nname: Deploy Databricks Assets on: push: branches: [ main ] pull_request: branches: [ main ] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: \u0026#39;3.10\u0026#39; - name: Install Databricks CLI run: pip install databricks-cli - name: Deploy to Dev if: github.event_name == \u0026#39;pull_request\u0026#39; run: | databricks bundle deploy -t dev env: DATABRICKS_HOST: ${{ secrets.DEV_DATABRICKS_HOST }} DATABRICKS_TOKEN: ${{ secrets.DEV_DATABRICKS_TOKEN }} - name: Deploy to Prod if: github.event_name == \u0026#39;push\u0026#39; \u0026amp;\u0026amp; github.ref == \u0026#39;refs/heads/main\u0026#39; run: | databricks bundle deploy -t prod env: DATABRICKS_HOST: ${{ secrets.PROD_DATABRICKS_HOST }} DATABRICKS_TOKEN: ${{ secrets.PROD_DATABRICKS_TOKEN }} These \u0026ldquo;automated sous-chefs\u0026rdquo; bring several benefits:\nConsistency: The deployment process is always the same, no matter who kicks it off Speed: Deployments happen quickly, with no manual intervention Auditing: Every deployment is logged, including who started it and which changes were applied Security: Credentials and secrets are managed securely, with no direct exposure Tasting Before Serving (Validation and Testing) No responsible chef would serve a dish without tasting it first. In the same way, the Databricks Asset Bundle lets you \u0026ldquo;taste\u0026rdquo; your project before deploying it to shared environments.\nThe Databricks CLI offers specific commands to validate and test your bundle locally:\n# Validar a configuração do bundle databricks bundle validate -t dev # Implantar localmente para teste databricks bundle deploy -t dev This up-front validation capability is crucial for catching problems before they affect other users or, even worse, production data. It\u0026rsquo;s like having a test kitchen where you can try out new recipes before adding them to the official menu.\nSome examples of problems that can be caught during validation:\nReferences to nonexistent resources Invalid or incompatible configurations Syntax problems in the YAML file Inconsistencies between different parts of the project Step-by-Step Recipe (Hands-On Implementation) Now that we understand the concepts and benefits of the Databricks Asset Bundle, let\u0026rsquo;s see how to implement it in a real project, following a step-by-step recipe:\n1. Preparing the \u0026ldquo;Kitchen\u0026rdquo; (Development environment)\nFirst, we need to install the necessary tools:\n# Para Windows winget install Databricks.DatabricksCLI # Para Mac brew tap databricks/tap brew install databricks Next, we configure authentication with the Databricks workspace:\ndatabricks configure --profile DEFAULT 2. Creating the \u0026ldquo;Base Recipe\u0026rdquo; (Bundle initialization)\nWith the tools installed, we can start our first bundle:\n# Criar um novo diretório para o projeto mkdir meu-projeto-databricks cd meu-projeto-databricks # Inicializar o bundle databricks bundle init This command will create a basic databricks.yml file that we can customize.\n3. Customizing the \u0026ldquo;Recipe\u0026rdquo; (Bundle configuration)\nNow we edit the databricks.yml file to include all the necessary resources:\nbundle: name: meu-projeto-analytics variables: env: default: dev description: Ambiente atual (dev, qa, prod) data_path: default: /data/bronze description: Caminho para os dados brutos targets: dev: workspace: host: https://dev-workspace.cloud.databricks.com variables: env: dev qa: workspace: host: https://qa-workspace.cloud.databricks.com variables: env: qa prod: workspace: host: https://prod-workspace.cloud.databricks.com variables: env: prod resources: jobs: etl_diario: name: \u0026#34;ETL Diário - ${variables.env}\u0026#34; schedule: quartz_cron_expression: \u0026#34;0 0 2 * * ?\u0026#34; tasks: ingestao: notebook_task: notebook_path: /Notebooks/ingestao base_parameters: data_path: \u0026#34;${variables.data_path}\u0026#34; env: \u0026#34;${variables.env}\u0026#34; job_cluster_key: cluster_padrao job_clusters: cluster_padrao: spark_version: 13.3.x-scala2.12 node_type_id: Standard_DS3_v2 autoscale: min_workers: 1 max_workers: 4 4. \u0026ldquo;Tasting\u0026rdquo; (Local validation and testing)\nBefore sharing our recipe, let\u0026rsquo;s test it locally:\n# Validar a configuração databricks bundle validate -t dev # Implantar para teste databricks bundle deploy -t dev 5. Getting Ready for \u0026ldquo;Service\u0026rdquo; (CI/CD configuration)\nFinally, we set up a CI/CD pipeline to automate deployment. If you\u0026rsquo;re using GitHub, create a .github/workflows/deploy.yml file:\nname: Deploy Databricks Assets on: push: branches: [ main ] pull_request: branches: [ main ] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: \u0026#39;3.10\u0026#39; - name: Install Databricks CLI run: pip install databricks-cli - name: Validate Bundle run: databricks bundle validate - name: Deploy to Dev if: github.event_name == \u0026#39;pull_request\u0026#39; run: | databricks bundle deploy -t dev env: DATABRICKS_HOST: ${{ secrets.DEV_DATABRICKS_HOST }} DATABRICKS_TOKEN: ${{ secrets.DEV_DATABRICKS_TOKEN }} - name: Deploy to QA if: github.event_name == \u0026#39;push\u0026#39; \u0026amp;\u0026amp; github.ref == \u0026#39;refs/heads/main\u0026#39; run: | databricks bundle deploy -t qa env: DATABRICKS_HOST: ${{ secrets.QA_DATABRICKS_HOST }} DATABRICKS_TOKEN: ${{ secrets.QA_DATABRICKS_TOKEN }} Conclusion The Databricks Asset Bundle represents a significant evolution in how we manage projects on the Databricks platform. Just as a professional chef\u0026rsquo;s kit transforms the culinary experience, DAB transforms the development lifecycle of data projects.\nBy adopting this approach, data teams can:\nEnsure consistency across different environments Automate deployments through CI/CD pipelines Version the entire configuration along with the code Reduce manual errors through standardized processes Speed up the development cycle with automated validations As with any new culinary technique, there\u0026rsquo;s an initial learning curve. However, the time invested in mastering the Databricks Asset Bundle quickly pays off in gains in productivity, quality and governance.\nThe next time you\u0026rsquo;re struggling with manual deployments or inconsistencies between environments, remember: with the right chef\u0026rsquo;s kit, even the most complex data dishes can be prepared with precision and confidence in any Databricks kitchen.\nSee you in the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/chefs-kit-for-data-databricks-asset-bundles-transform-deployment/","summary":"Deploying data projects across different environments has always been a challenge comparable to trying to replicate a gourmet dish in different kitchens. Even with the same ingredients, small variations in settings, in the…","title":"The Chef's Kit for Data: How Databricks Asset Bundles Transform Project Deployment"},{"content":"For years, data engineers have been building pipelines that connect Azure Data Factory (ADF) to Databricks, a powerful combination that often required manual tweaks and complex configuration. It was like having a DeLorean that needed constant recalibration of the flux capacitor before every trip through time: functional, but far from ideal.\nMicrosoft has just announced an update that promises to revolutionize this integration: the new Databricks Job activity in ADF. This feature lets you orchestrate any type of Databricks job directly from Data Factory, eliminating the need for workarounds and dramatically simplifying the data architecture. It\u0026rsquo;s as if Doc Brown had finally perfected the DeLorean, allowing direct trips to any point in time with a single click.\nFor data professionals, this update is much more than a simple technical convenience. It brings the potential for significant cost reduction, simpler architectures, and a smoother development experience. Modern \u0026ldquo;time travelers\u0026rdquo; can now focus more on data quality and less on the complexities of cross-platform integration.\nWhat Changed? The New DeLorean - The Evolution of the ADF-Databricks Integration Before this update, integrating ADF and Databricks was possible, but limited. The available options included:\nNotebook Activity: Run individual notebooks, but without access to all of Databricks\u0026rsquo; features REST API calls: Implement custom code to trigger jobs, requiring constant maintenance Hybrid solutions: Combine different approaches, increasing complexity It was like driving a DeLorean built from improvised parts: it worked, but it needed constant attention and adjustment.\nThe new Databricks Job activity completely changes that reality. Now you can:\nOrchestrate any type of Databricks job directly from ADF Run complete workflows with multiple tasks in sequence Take advantage of serverless compute for cost optimization Parameterize jobs for maximum flexibility Monitor runs directly in the ADF interface What Can You Run? Doc Brown would be impressed by the versatility of this new DeLorean. The Databricks Job activity supports practically any operation available in Databricks:\nNotebooks: Run Python, Scala, R, or SQL code SQL Tasks: Direct SQL queries and transformations Delta Live Tables: Declarative pipelines for data processing Model Serving: Batch inference using model endpoints Power BI: Automatic publishing and refresh of semantic models This means you can build end-to-end pipelines that span everything from raw data ingestion to refreshing BI dashboards, all orchestrated from a single platform.\nWhy Does This Matter? Time Travel Without Paradoxes - Eliminating Temporal Paradoxes In the \u0026ldquo;Back to the Future\u0026rdquo; trilogy, temporal paradoxes were a constant risk that complicated every trip. In the same way, the traditional integration between ADF and Databricks created its own \u0026ldquo;paradoxes\u0026rdquo;:\nThe Startup Paradox: Clusters that took a long time to start between activities The Visibility Paradox: Monitoring fragmented across platforms The Maintenance Paradox: Custom code that required constant updating The new Job activity resolves these paradoxes, creating a cleaner, more efficient timeline for your data.\nStrategic Impact This update isn\u0026rsquo;t just a technical improvement, it\u0026rsquo;s a strategic shift that affects different aspects of data engineering:\nUnifying the Azure Ecosystem: Strengthens cohesion between Microsoft services Architectural Simplification: Reduces complexity and the need for specialized knowledge Democratizing Advanced Features: Makes advanced features accessible through a visual interface For different professional profiles, the benefits are clear:\n_Benefits by Profile_\nSaving \u0026ldquo;Plutonium\u0026rdquo; - The Fuel of Pipelines In the \u0026ldquo;Back to the Future\u0026rdquo; universe, plutonium was the expensive, hard-to-get fuel that powered the original DeLorean. In the modern version, Doc Brown managed to replace it with a fusion reactor powered by ordinary garbage: a far more efficient and economical solution.\nSimilarly, the new Databricks Job activity lets you replace the \u0026ldquo;expensive fuel\u0026rdquo; (dedicated clusters and custom code) with a more efficient alternative (serverless compute and native orchestration).\nComparative Scenario\nTo show the financial impact of this change, we simulated a common scenario at mid-sized companies:\nDaily processing of 50GB of data A pipeline with 4 stages: ingestion, transformation, aggregation, and loading Daily run with a 4-hour processing window Azure environment in the East US region The simulation reveals surprising savings:\nClassic Pipeline: $1,392.24 per month Pipeline with Jobs: $473.31 per month Absolute Savings: $918.93 per month Percentage Savings: 66% Factors That Contribute to the Savings\nServerless Compute: You pay only for actual processing time Optimized Startup: Less overhead between tasks in the same job Reduced Maintenance: A simpler interface and less code to maintain Simplified Orchestration: Fewer activities in ADF It\u0026rsquo;s as if the DeLorean had gone from a plutonium engine to solar cells: more efficient, cheaper, and better for the environment (or, in this case, for the IT budget).\nBuilding Your First Time Machine - A Practical Guide Setting Up the Integration\nLet\u0026rsquo;s get to Doc Brown\u0026rsquo;s simplified manual for building your own data DeLorean. Setting up the new Databricks Job activity is surprisingly simple:\nIn Azure Data Factory Studio: Create a new pipeline or edit an existing one In the activities pane, find the Databricks section Drag the \u0026ldquo;Job\u0026rdquo; activity onto the canvas _New Databricks Job activity in ADF_\nBasic Configuration: Select or create a Databricks linked service Choose the workspace and the job you want to run Configure parameters, if needed Parameters and Monitoring: Define dynamic parameters to pass to the job Configure monitoring and retry options Connect the activity to other pipeline steps, if needed Tips to Maximize the Benefits To get the most out of this new feature, consider these tips:\nMigrate Gradually: Start by converting simple pipelines before tackling the more complex ones Consolidate Tasks: Group related tasks into a single job to reduce overhead Use Serverless: Configure your jobs to use serverless compute whenever possible Parameterize Everything: Use parameters to build flexible, reusable pipelines Monitor Performance: Compare metrics before and after the migration to quantify the gains Important Considerations Like any technology, there are a few limitations to keep in mind:\nThe feature is in Preview and may change Some advanced monitoring features are still under development The actual savings depend on your specific workload profile Existing pipelines will need to be redesigned to take full advantage of the benefits Conclusion The new Databricks Job activity in Azure Data Factory is a significant leap in the evolution of the integration between these platforms. Just like the perfected DeLorean at the end of the \u0026ldquo;Back to the Future\u0026rdquo; trilogy, this update removes unnecessary complexity and opens up new possibilities for data engineers.\nThe benefits are clear: simpler architectures, more agile development, unified monitoring and, perhaps most impressive of all, the potential for significant cost savings. Our simulation showed that it\u0026rsquo;s possible to cut spending by up to 66% by migrating from classic pipelines to the new approach.\nAs Microsoft keeps investing in the Azure ecosystem, we can expect even more integrations and optimizations between Data Factory and Databricks. The future of data pipelines is just one click away in ADF: no 1.21 gigawatts or rare plutonium required.\nTry the new feature today and discover how it can transform your data engineering journey. After all, as Doc Brown would say: \u0026ldquo;Roads? Where we\u0026rsquo;re going, we don\u0026rsquo;t need roads.\u0026rdquo; And with the new Databricks Job activity, you won\u0026rsquo;t need custom code or complex integrations either.\nSee you soon!\n","permalink":"https://julianamarialop.github.io/en/articles/back-to-the-future-of-pipelines-azure-data-factory-databricks-jobs/","summary":"For years, data engineers have been building pipelines that connect Azure Data Factory (ADF) to Databricks, a powerful combination that often required manual tweaks and complex configuration. It was like…","title":"Back to the Future of Pipelines: How the Azure Data Factory DeLorean Now Travels Straight to the World of Databricks Jobs"},{"content":"The Databricks universe is constantly expanding. Like a living, pulsing galaxy, every month brings new \u0026ldquo;constellations\u0026rdquo; of features, integrations, and acquisitions that light the way for data professionals and widen the horizons of what\u0026rsquo;s possible with data and artificial intelligence.\nMay 2025 was a particularly busy month in this technological cosmos. We witnessed the birth of several important \u0026ldquo;constellations\u0026rdquo; that are redefining the platform\u0026rsquo;s horizon and opening up new possibilities for organizations looking to extract value from their data. From the billion-dollar acquisition of a promising startup to new tools for marketers, the Databricks universe gained new points of light.\nIn this article, we\u0026rsquo;ll explore the five main \u0026ldquo;constellations\u0026rdquo; that appeared this month, each bringing its own brightness and influence to the Databricks ecosystem. Get ready for a trip through the outer space of data, where each new release represents a significant step forward in the mission to democratize data analytics and artificial intelligence.\nThe Neon Supernova (The Neon Acquisition) At the heart of the Databricks universe, a supernova has just exploded, releasing transformative energy across the whole ecosystem. On May 14, Databricks announced its intention to acquire Neon, a leading serverless Postgres technology company, in a deal valued at approximately US$ 1 billion.\nThis supernova isn\u0026rsquo;t impressive just for its size, but mainly for the power it adds to the Databricks universe. Neon developed a serverless PostgreSQL database technology that promises to bring speed and flexibility to production workloads. By integrating this technology into its platform, Databricks significantly strengthens its capabilities for developers and AI agents, enabling more robust and efficient AI systems.\nLike every supernova that reshapes the space around it, this acquisition will also bring structural changes. Neon\u0026rsquo;s 140 employees will join Databricks once the deal closes, expected on July 31, 2025, at the end of the second quarter. This merger of talent promises to accelerate innovation around serverless PostgreSQL and break down data silos, creating a more unified environment for analytics and AI applications.\nThe Marketing Galaxy (Data Intelligence for Marketing) A new galaxy has just formed in the Databricks universe, with multiple systems orbiting in harmony around a powerful core. Data Intelligence for Marketing, launched on May 14, arrives as a unified solution that promises to revolutionize how marketing teams use data and AI.\nThis bright galaxy brings all customer and campaign data together in one place, letting marketers get a complete view of their initiatives. Like interconnected stars, the data flows and communicates, revealing patterns and insights that used to stay hidden in the shadows of information silos.\nWhat makes this galaxy even more fascinating are its connections to other systems. Data Intelligence for Marketing comes with ready-made integrations with partners such as Adobe, Braze, and Salesforce, creating a truly interconnected star system. These integrations let marketing teams improve segmentation, personalize experiences in real time, and make decisions based on hard data, not gut feeling. Like a galaxy that shines with its own light, this solution lights the way to next-generation customer experiences.\nThe Interstellar Bridge (Native Azure Databricks Connector in Azure AI Foundry) In the vast universe of data, bridges that connect different regions are as valuable as they are rare. The new native Azure Databricks connector in Azure AI Foundry is one of those interstellar bridges, letting energy (data) flow between different dimensions of the technological cosmos.\nThis newly built bridge specifically connects Databricks AI/BI Genie with Azure AI Foundry, creating a direct path for data to flow between these two powerful systems. Like a well-planned cosmic route, this connection lets organizations build data-aware AI agents across the entire company, without navigating complex routes or building custom solutions.\nThe benefits of this interstellar bridge are immense. Organizations can now create richer, more contextual AI experiences, where agents have direct access to the relevant corporate data. Imagine virtual assistants that don\u0026rsquo;t just answer generic questions but can access and analyze your company\u0026rsquo;s specific data to deliver personalized, actionable insights. This bridge doesn\u0026rsquo;t just shorten distances: it creates possibilities that once seemed to belong only to the realm of science fiction.\nThe Automation Planetary System (New Databricks Job Activity in Azure Data Factory) Every well-organized universe needs planetary systems where each celestial body follows its orbit in a coordinated, predictable way. The new Databricks Job activity in Azure Data Factory, announced on May 14, creates exactly this kind of system for automating data flows.\nLike planets following their orbits in perfect harmony, Azure Data Factory now supports orchestrating Databricks Jobs, letting organizations schedule and automate Databricks workflows directly from the familiar ADF interface. This integration creates a solar system of automation where each job (planet) follows its predetermined path, interacting with other elements of the system when needed.\nThe beauty of this planetary system lies in its ability to unify different Azure services in a single orbit of automation. Data teams can now build end-to-end pipelines that combine Databricks processing with other Azure Data Factory activities, without switching between different interfaces or writing complex integration scripts. Like an astronomer who can finally map an entire star system, data engineers now have complete visibility and control over their processing flows.\nThe Evolutionary Asteroid Belt (Liquibase Pro Extension for Databricks Integration) Between the larger planets of a solar system, we often find asteroid belts: collections of smaller bodies that, together, play a crucial role in the system\u0026rsquo;s stability. The new Liquibase Pro Extension for Databricks integration arrives as an evolutionary asteroid belt, protecting and organizing the data space around it.\nThis extension brings modern change management to the Databricks environment, letting teams control schema and data changes more efficiently and safely. Like asteroids moving in predictable patterns, database changes can now be tracked, versioned, and deployed in a controlled way, reducing risk and increasing reliability.\nFor teams working with DevOps and CI/CD, this asteroid belt offers protection against catastrophic collisions (unplanned or conflicting changes) and provides a clear path for the continuous evolution of data systems. The integration lets developers and data engineers adopt \u0026ldquo;database as code\u0026rdquo; practices, treating schema changes with the same rigor and automation they apply to application code. Like a self-organizing system, this extension helps keep order even in complex, constantly changing data environments.\nConclusion Looking at these five new \u0026ldquo;constellations\u0026rdquo; in the Databricks universe, it\u0026rsquo;s clear we\u0026rsquo;re witnessing a significant expansion in the platform\u0026rsquo;s capabilities and reach. From the Neon supernova that adds serverless database power, to the marketing galaxy that unifies customer data, from the interstellar bridge that connects AI and data, to the automation planetary system and the asteroid belt that manages change, each release represents an important step forward for the ecosystem.\nTogether, these constellations form a new \u0026ldquo;galaxy of possibilities\u0026rdquo; for data professionals, developers, and business teams. They reflect Databricks\u0026rsquo; vision of creating a universe where data and AI work together harmoniously, letting organizations extract more value from their information assets and build smarter, more efficient applications.\nAs explorers of this expanding universe, we\u0026rsquo;re invited to navigate these new constellations, discover their secrets, and harness their power to drive innovation and growth. The Databricks cosmos keeps expanding, and May 2025 will surely be remembered as a period of extraordinary stellar growth.\nSee you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/the-expanding-databricks-universe-new-constellations-of-may-2025/","summary":"The Databricks universe is constantly expanding. Like a living, pulsing galaxy, every month brings new \u0026ldquo;constellations\u0026rdquo; of features, integrations, and acquisitions that light the way for data professionals and…","title":"The Expanding Databricks Universe: The New Constellations of May 2025"},{"content":"Picture a large, modern airport with VIP lounges, strict protocols and exclusive tunnels for boarding and arrivals. Now picture that this airport is your Microsoft Fabric workspace.\nStarting in June 2025, Microsoft Fabric gets two networking features, enabled by default, that bring exactly this level of security and control: workspace-level Private Links and Outbound Access Protection. Both are already available in Preview, and both are designed to protect your data with the same rigor as an international terminal.\nPrivate Links: the VIP entry tunnel With the new Private Links, administrators can create private connections between their Azure virtual networks and Fabric workspaces. In other words, data now arrives through an exclusive tunnel, reserved only for trusted networks.\nIt\u0026rsquo;s like giving your passengers a separate entrance with no access to the public concourse. No crowds, no unnecessary exposure. You decide who gets to land in your environment and you block every other kind of external access.\nThis feature lets the workspace run isolated from the public internet, which significantly reduces the risk of exposure or intrusion.\nSetup is simple: go to \u0026ldquo;Advanced networking\u0026rdquo; and adjust the inbound rules at the workspace level.\nOutbound Access Protection: boarding only with a ticket Just as airports strictly control who leaves, now Fabric does too. With Outbound Access Protection, no outbound connection is allowed from the workspace unless a Managed Private Endpoint is configured.\nThink of it as boarding control. Your data can only leave the terminal if it holds an authorized, registered ticket. Without one, it doesn\u0026rsquo;t board.\nThis measure helps prevent data exfiltration, whether accidental or intentional, and ensures that only approved paths out of the environment are used. The protection even extends to internal Fabric services, such as Spark notebooks.\nTo turn it on, just open \u0026ldquo;Workspace settings\u0026rdquo;, go to \u0026ldquo;Network security\u0026rdquo; and select the option to block public outbound access.\nCentral administration: your control tower All of these features can be managed centrally by tenant administrators, who now have a dedicated section called \u0026ldquo;Advanced networking\u0026rdquo; in the Fabric admin portal.\nFrom there, you can standardize security rules across the whole organization, enabling or disabling features according to internal policies. Since they come enabled by default, workspace administrators can apply them right away.\nConclusion: an environment with international-grade security With these two new features, Fabric stops being just a data platform and becomes an environment with real enterprise security standards.\nBy controlling both what comes in and what goes out, you turn your workspace into a VIP airport: with flows that are protected, traceable and fully under your control.\nIt\u0026rsquo;s time to take command of the tower and make sure your data flies safely.\n","permalink":"https://julianamarialop.github.io/en/articles/vip-security-in-fabric-your-workspace-now-runs-like-an-international-airport/","summary":"Picture a large, modern airport with VIP lounges, strict protocols and exclusive tunnels for boarding and arrivals. Now picture that this airport is your Microsoft Fabric workspace.","title":"VIP Security in Fabric: your workspace now runs like an international airport"},{"content":"For decades, Scrooge McDuck was the ultimate symbol of concentrated wealth. Owner of a colossal fortune, he preferred to keep his money in a giant vault where he could literally swim in gold coins. Only he had access. Only he made the decisions. And he firmly believed that to protect his fortune, he had to keep it isolated from the world.\nBut what if Scrooge McDuck lived in 2025?\nMaybe he\u0026rsquo;d realize that his keep-it-all-to-myself model is becoming outdated. And that, instead of a locked vault, he could use technology to multiply value, create opportunities and still keep everything safe. That\u0026rsquo;s where real estate tokenization comes in, along with a powerful data platform like Databricks.\nFrom sole owner to collective investor: the new way to share wealth Scrooge always believed real estate was a solid way to hold and grow his wealth. And he wasn\u0026rsquo;t wrong. But until recently, investing in real estate was a privilege reserved for those with a lot of capital, time and tolerance for red tape.\nWith tokenization, that changes completely.\nReal estate tokenization is the process of turning a physical asset, such as a building or a plot of land, into small digital pieces called tokens. Each token represents a real fraction of the property and can be bought by anyone, starting at a hundred reais. It\u0026rsquo;s as if Scrooge decided to split his vault into little digital coins and let other people invest in them too, with a share of the profits.\nA ledger nobody can scribble over: the magic of blockchain If Scrooge was obsessed with control and precise records, he\u0026rsquo;d be impressed by how blockchain works. It\u0026rsquo;s a kind of public digital ledger where every transaction is recorded in an encrypted, immutable way.\nPicture Scrooge\u0026rsquo;s old cash book, where he wrote down every cent with a fountain pen, being replaced by a network of computers that records every movement with mathematical precision and no way to alter it. That\u0026rsquo;s what guarantees the security and trust behind tokenization.\nThere\u0026rsquo;s no longer any need to trust third parties or paperwork. The technology itself guarantees that everything is right, something that even Scrooge, suspicious as ever, would learn to value.\nWhen investing is no longer just for people with underground vaults The big change tokenization brings is the democratization of real estate investing. A market that used to require high entry amounts now lets ordinary people invest with little money, diversify their assets and even earn passive income, like little Scrooges in the making.\nThe idea that \u0026ldquo;either you own a whole property or you own nothing\u0026rdquo; is being replaced by a model that is more accessible, more digital and far more inclusive.\nThe vault has already started to open: real examples in Brazil Even if Scrooge were reluctant at first, he couldn\u0026rsquo;t ignore the results of some Brazilian platforms already applying this new logic:\nNetspaces: tokenizes properties in Brazilian cities and has already launched projects with Housi, giving access to fractions of modern apartments. Ribus: bets on tokens with governance, where investors have an active say in managing the properties, something Scrooge, always calling all the shots, might take a while to accept. Propriedade Digital: is opening the door for new investor profiles to take part in this market, safely and simply. Housi: turns housing into a service and into a fractional investment asset. These companies show that the future of the real estate market is already underway and that you no longer need to inherit a gold mine to get started.\nThe data Scrooge doesn\u0026rsquo;t see, but that moves invisible fortunes Tokenizing a property isn\u0026rsquo;t just a technological or legal matter. It\u0026rsquo;s also a matter of data, lots of data. Information about location, the market, contracts, transactions, investor preferences and regulatory compliance needs to be analyzed and processed in real time.\nIf Scrooge always hired accountants to handle the paperwork, today he\u0026rsquo;d need a far more robust platform to keep up with this new volume of information. That\u0026rsquo;s where Databricks comes in.\nWhen gold stops shining and data becomes worth more Databricks works like the brain behind the modern vault. It brings together, organizes and analyzes all the data involved in tokenization, and turns it into practical intelligence.\nWith it, you can:\nUnderstand investor behavior and predict trends; Evaluate the performance of different properties and suggest which are most viable for tokenization; Generate real-time dashboards on liquidity, risk and profitability; Ensure the traceability and governance needed to meet the requirements of bodies like the CVM (Brazil\u0026rsquo;s securities regulator). If Scrooge\u0026rsquo;s coin vault was the symbol of absolute control, Databricks is the new wealth management instrument, only based on data, machine learning and predictive insight.\nThe old vault still has value, but the new one is networked At the end of the day, Scrooge might still keep a few coins as souvenirs, framed on the wall. But if he wants to keep growing, he\u0026rsquo;ll have to accept that opening the vault, with security, strategy and intelligence, isn\u0026rsquo;t losing value. It\u0026rsquo;s creating new ways to prosper.\nAnd that\u0026rsquo;s only possible once you understand that blockchain and tokenization change the rules of the game. And that platforms like Databricks let you play with an edge.\nThe question that remains is: will you wait for the vault to open, or have you already started investing in the next revolution of the real estate market?\n","permalink":"https://julianamarialop.github.io/en/articles/scrooge-mcduck-changing-his-mind-real-estate-tokenization-databricks/","summary":"For decades, Scrooge McDuck was the ultimate symbol of concentrated wealth. Owner of a colossal fortune, he preferred to keep his money in a giant vault where he could literally swim in gold coins. Only he…","title":"Scrooge McDuck Is Changing His Mind, and It Has Everything to Do with Real Estate Tokenization and Databricks"},{"content":"Anyone who works with SQL knows that, powerful as the language is, writing and understanding complex queries, especially those with multiple nested subqueries, can be a challenge. We often feel we hold great power but lack full control to express our ideas clearly and directly.\nWhat if there were a way to turn this query-building journey into more intuitive training, like learning to master the \u0026ldquo;elements\u0026rdquo; of your query in a logical order? That is exactly what the new SQL Pipe Syntax in Databricks sets out to do: an approach that promises to simplify and clear the path to extracting insights from your data.\nGet ready to discover how this new syntax can revolutionize your relationship with SQL, letting you become a true \u0026ldquo;Data Master,\u0026rdquo; just as Aang became the Avatar by mastering the four elements. Let\u0026rsquo;s set off on this learning journey!\nThe World Before the Avatar (The Challenge of Traditional SQL) In traditional SQL, the order of the clauses doesn\u0026rsquo;t always follow our train of thought. We start with SELECT, but our logic often begins with the tables in FROM. On top of that, to build more elaborate logic we frequently resort to nested subqueries, which can turn a query into an ancient, hard-to-decipher scroll, making reading and maintenance a real test of patience. It was as if we needed extra effort to \u0026ldquo;bend\u0026rdquo; SQL to our will, especially in more complex scenarios.\nDiscovering the First Element (FROM as the Foundation of Bending) SQL Pipe Syntax arrives to change this paradigm, introducing a sequential, logical construction. Here, FROM is the first \u0026ldquo;element\u0026rdquo; to be invoked, the foundational nation where our query journey begins, just as the Air Nomads were Aang\u0026rsquo;s starting point. By starting with FROM your_table, we establish a clear, direct foundation for the transformations to come, making the beginning of the query far more intuitive.\nThe Mastery Journey: Learning to Bend the Elements with \u0026ldquo;|\u0026gt;\u0026rdquo; The great master of this new journey is the |\u0026gt; (pipe) operator. It works as the link connecting each stage of our \u0026ldquo;training,\u0026rdquo; letting us add new \u0026ldquo;elements\u0026rdquo; (SQL clauses) sequentially and progressively. Let\u0026rsquo;s see how to master each element:\nMastering Waterbending (Filtering with |\u0026gt; WHERE) Just as a Waterbender controls currents and removes impurities, |\u0026gt; WHERE lets you \u0026ldquo;shape\u0026rdquo; the flow of your data. After defining your source with FROM, you can use |\u0026gt; WHERE condition to filter out exactly the information you don\u0026rsquo;t need, bringing clarity and precision to your initial dataset.\nMastering Earthbending (Structuring with |\u0026gt; JOIN) With the foundation and filters in place, it\u0026rsquo;s time to build solid structures. |\u0026gt; JOIN lets you \u0026ldquo;join\u0026rdquo; different data sources (tables), just as an Earthbender moves and connects huge rocks to create firm foundations. With FROM table1 |\u0026gt; JOIN table2 ON condition, you combine information in a logical, sequential way, enriching your analysis.\nMastering Firebending (Transforming and Aggregating with |\u0026gt; GROUP BY and Functions) The raw power of data needs to be refined and turned into insights. |\u0026gt; GROUP BY and aggregate functions (such as COUNT, SUM, AVG) let you \u0026ldquo;transform\u0026rdquo; and summarize data, extracting energy and knowledge, the same way a Firebender controls and directs the flame. After the previous steps, you can apply |\u0026gt; GROUP BY column |\u0026gt; SELECT column, COUNT(*) to aggregate and reveal patterns.\nThe Air Element Revealed (Selecting the Essentials with |\u0026gt; SELECT) Finally, to give your query its final shape, in comes |\u0026gt; SELECT. Unlike traditional SQL, where it starts the query, in Pipe Syntax |\u0026gt; SELECT (used toward the end of the pipeline or at strategic points) defines which columns and expressions will make up the final result. It\u0026rsquo;s like Airbending, which brings clarity, precision and the freedom to present only the essentials, cleanly and directly.\nReaching the Avatar State (The Benefits of SQL Pipe Syntax) By mastering these \u0026ldquo;elements\u0026rdquo; with SQL Pipe Syntax, you reach a new level of control and clarity over your queries, almost like an \u0026ldquo;Avatar State\u0026rdquo; in the SQL world:\nClarity and Readability: Queries flow like a well-told story, easy to follow and understand, like Aang\u0026rsquo;s journey. Simplified Maintenance: Modifying or adding new \u0026ldquo;elements\u0026rdquo; (steps) becomes much easier, without the risk of breaking an entire complex structure of nested subqueries. Intuitive Writing: The syntax follows your logical thinking, making SQL writing more natural and less prone to clause-ordering mistakes. Less Subquery Confusion: The need to nest multiple subqueries drops dramatically, making the code cleaner. The Harmony of the Four Elements (Compatibility and Real-World Examples) A big benefit is that SQL Pipe Syntax is fully compatible with traditional SQL. You don\u0026rsquo;t need to rewrite all your existing queries; they can coexist harmoniously, like the four nations seeking balance. You can even start using Pipe Syntax to refactor parts of old queries or to build new queries incrementally.\nPicture a traditional query with several subqueries to calculate sales by region and product. With Pipe Syntax, that same logic would be built step by step: start with the sales table, filter by period, join with products, then with regions, group and, finally, select the results. The \u0026ldquo;before and after\u0026rdquo; of your training as the SQL Avatar would be crystal clear!\nConclusion SQL Pipe Syntax in Databricks represents a significant evolution in how we interact with our data. It empowers us to build queries in a more logical, intuitive and powerful way, turning what could once be an arduous task into a smoother, more efficient data \u0026ldquo;bending\u0026rdquo; process.\nThe invitation is on the table: explore this new way of \u0026ldquo;bending\u0026rdquo; SQL in your Databricks projects. Start your training, master the elements one by one with |\u0026gt; and see how you can become a true \u0026ldquo;Data Master.\u0026rdquo; Just as the Avatar brought balance to the world, this new syntax seeks to bring more balance, clarity and power to your SQL universe.\n","permalink":"https://julianamarialop.github.io/en/articles/azure-databricks-sql-pipe-syntax-master-query-elements-like-avatar/","summary":"Anyone who works with SQL knows that, powerful as the language is, writing and understanding complex queries, especially those with multiple nested subqueries, can be a challenge. We often feel…","title":"Azure Databricks SQL Pipe Syntax: Master the Elements of Your Query Like an Avatar!"},{"content":"Imagine a world where every video game character has a mission. Ralph wrecks buildings, Felix fixes everything with his magic hammer, Vanellope races on wild tracks, Sonic is super fast. They all live in their own games, with very different rules, styles and abilities.\nNow think about it: what would happen if all these characters could communicate perfectly, collaborate in real time and solve problems together, even though they come from different games?\nThat\u0026rsquo;s the spirit of A2A (Agent-to-Agent Protocol), a new technology Microsoft just announced to help artificial intelligence agents work together in a secure, smart and coordinated way, as if they had learned the universal language of the arcade.\nAgents Are Like Game Characters In the digital world of Wreck-It Ralph, every character has a clear role:\nRalph wrecks, Felix fixes, Vanellope races, Sonic is fast. They have unique abilities, just like AI agents in the real world:\nOne agent searches for information, Another summarizes texts, A third organizes data, Another can fix errors (like Felix!). Before A2A, it was as if every character spoke a different language. Ralph spoke \u0026ldquo;Wreckish\u0026rdquo;, Felix spoke \u0026ldquo;Fixish\u0026rdquo; and Vanellope only understood \u0026ldquo;Racish\u0026rdquo;. As amazing as they were, they couldn\u0026rsquo;t work as a team.\nA2A: When Agents Talk Like Ralph and Felix With A2A, agents can talk to each other like game characters on a joint mission. Here\u0026rsquo;s how it works, with a simple example of two agents exchanging messages:\nAgent 1: Text Checker (Ralph)\n\u0026ldquo;Hey, there\u0026rsquo;s a broken sentence here: \u0026lsquo;Nós vai para a escola amanhã.\u0026rsquo; Can you take a look?\u0026rdquo;\nThat message becomes a standardized command in A2A format:\n{ \u0026#34;goal\u0026#34;: \u0026#34;corrigir frase\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Nós vai para a escola amanhã.\u0026#34;, \u0026#34;agent\u0026#34;: \u0026#34;CorretorGramatical\u0026#34; } Agent 2: Grammar Fixer (the team\u0026rsquo;s Felix)\n\u0026ldquo;Got it! Already fixed: \u0026lsquo;Nós vamos para a escola amanhã.\u0026rsquo; Mission accomplished!\u0026rdquo;\nAnd then it replies with:\n{ \u0026#34;result\u0026#34;: \u0026#34;Nós vamos para a escola amanhã.\u0026#34; } This exchange may look simple, but it shows enormous power: each agent knows exactly what it needs to do and responds in a structured way. No confusion, no mess, as if Ralph broke a wall and Felix showed up right behind him, magic hammer in hand, fixing everything with a single tap.\nA2A makes this flow work in real systems, between agents that may run on different machines and be built by different teams, but all speak the same \u0026ldquo;digital language\u0026rdquo;.\nMicrosoft\u0026rsquo;s Announcement On May 7, 2025, Microsoft officially announced the launch of A2A as an open protocol. That means any developer, company or platform will be able to use this common language to get their agents talking to each other, even if they were built with different tools.\nA2A will be integrated into platforms such as Azure AI Foundry, Copilot Studio and other modern AI tools. And there\u0026rsquo;s more: the protocol is being developed in the open, with community support on GitHub.\nWhy Does This Matter? Before A2A, AI systems worked like isolated games. Now:\nAgents can work as a team, like Ralph, Felix and Vanellope. Developers are free to use any technology. Companies build more complete, faster and better-integrated solutions. It\u0026rsquo;s like putting together a mission with the best characters from every game. Ralph wrecks the problem, Felix fixes the bugs, Vanellope speeds up the process and A2A makes sure everyone knows what to do and when to act.\nSecurity, Governance and Controlled Fun Just like in Wreck-It Ralph, where there are rules to keep order in the games, A2A has its own digital guardians:\nSecure communication (with authentication and encryption). Access control and auditing. Real-time monitoring. Nobody joins or leaves the mission unnoticed. Everything is under control, just like in Mr. Litwak\u0026rsquo;s arcade.\nConclusion: The Future Is a Team Game With A2A, we\u0026rsquo;re entering an era where different artificial intelligences can work together like the real cast of an animated movie. Ralph, Felix, Vanellope, Sonic, all cooperating, each with their own ability, but now speaking the same language.\nIt\u0026rsquo;s as if the future of AI were one big interconnected arcade, and the coin has finally dropped: agents now play together.\nSee you in the next article!\nReferences Empowering multi-agent apps with the open Agent2Agent (A2A) protocol: https://www.microsoft.com/en-us/microsoft-cloud/blog/2025/05/07/empowering-multi-agent-apps-with-the-open-agent2agent-a2a-protocol/\n","permalink":"https://julianamarialop.github.io/en/articles/when-game-characters-learn-to-talk-microsoft-a2a-wreck-it-ralph/","summary":"Imagine a world where every video game character has a mission. Ralph wrecks buildings, Felix fixes everything with his magic hammer, Vanellope races on wild tracks, Sonic is super fast. They all live in their own…","title":"When Game Characters Learn to Talk: Microsoft's A2A World, Wreck-It Ralph Style"},{"content":"April is here, and with it, Microsoft Fabric has prepared a set of important updates. As a platform that\u0026rsquo;s always looking to innovate, Fabric treats us to a variety of new and improved \u0026ldquo;dishes,\u0026rdquo; designed for every palate and every need in the world of data.\nThis month, Microsoft focused on key areas, bringing everything from \u0026ldquo;appetizers\u0026rdquo; that democratize access to artificial intelligence to hearty \u0026ldquo;main courses\u0026rdquo; for data scientists and data engineers, not to mention innovative \u0026ldquo;sides\u0026rdquo; for real-time intelligence and \u0026ldquo;desserts\u0026rdquo; that make integration easier.\nLet\u0026rsquo;s explore together the highlights of Fabric\u0026rsquo;s \u0026ldquo;April Buffet\u0026rdquo; and how these new features can enrich your projects.\nThe House Starter: AI for Every Palate! Every good buffet starts with a special starter, and in April, Fabric doubles down on one of its best: Copilot and the AI capabilities are now available starting with F2 SKUs! That means cutting-edge artificial intelligence has become more accessible, like a quality starter that lets more customers try it out and supercharge their projects.\nThe Hot Dishes Station: Bold Flavors for Data Science and Data Warehouse! We\u0026rsquo;ve reached the most anticipated station of our buffet, with options aimed at meeting performance and innovation needs.\nDish 1: Wild Mushroom Risotto with AI (Low-code Data Science)\nData scientists can take advantage of the new Low-code AI tools in notebooks and in Data Wrangler (which is in Preview). These special \u0026ldquo;ingredients\u0026rdquo; boost productivity, letting you apply AI functions and transform data with much less code. It\u0026rsquo;s like a sophisticated risotto that, with AI-assisted preparation, makes exploring and enriching data a smoother, more intuitive experience.\nDish 2: Dry-Aged Grilled Picanha with an Efficiency Sauce (Data Warehouse GA Features)\nFor data engineers who appreciate robustness, Fabric Warehouse serves up a \u0026ldquo;picanha\u0026rdquo; (Brazil\u0026rsquo;s beloved top sirloin cap) cooked just right: several important features have reached General Availability (GA). Commands like ALTER TABLE DROP COLUMN and sp_rename column, support for distributed #temp tables for efficient management of intermediate results, and the OPENROWSET and BULK INSERT functions for bulk data ingestion make data maintenance, optimization, and loading easier. These are tools designed to deliver performance and satisfaction, like a well-prepared prime cut of beef.\nDish 3: Light Migration Soufflé (Migration Assistant for Fabric DW - Preview)\nAnd for anyone planning to bring their data into the Fabric Data Warehouse, Microsoft has prepared a \u0026ldquo;soufflé\u0026rdquo; that promises lightness: the new Migration Assistant for Fabric Data Warehouse, still in Preview. The promise is to simplify the migration journey, making the process less intimidating and more straightforward, like a soufflé that wins you over with its texture and ease of preparation.\nThe Sides and Fresh Salads Island: Real-Time Intelligence and Refreshed Databases! No main course is complete without sides that bring out the flavor. On this island, we find new features that add freshness and versatility.\nSide 1: Tropical Real-Time Events Salad (Real-Time Intelligence)\nFabric\u0026rsquo;s Real-Time Intelligence area received ingredients that bring more agility to analyzing data that can\u0026rsquo;t wait. Highlights include full integration of Azure Monitor data sources with KQL Queryset, improvements to the low-code data exploration experience, and a new weather connector in Eventstream. It\u0026rsquo;s a true \u0026ldquo;tropical salad,\u0026rdquo; with components that add value through their ability to capture and analyze events the moment they happen.\nSide 2: Rustic Potatoes with Fine Herbs (SQL Database in Fabric)\nThe SQL Database service in Fabric was also enhanced with improvements: support for new regions, backup billing updates, a performance dashboard for monitoring, and expanded support for Terraform, REST APIs, and the CLI, as well as integrations and graph database support. These updates make SQL Database in Fabric even more \u0026ldquo;flavorful and versatile,\u0026rdquo; like those well-made rustic potatoes that go nicely with all kinds of dishes thanks to their reliability and heartiness.\nThe Chef\u0026rsquo;s Surprise: The Unexpected Dessert! To wrap up, a special dessert to round out the meal.\nDessert: Connectivity Lava Cake with a Snowflake Filling (Mirroring for Snowflake with Firewall - Preview)\nThe Mirroring for Snowflake feature, which was already interesting, now gets a special \u0026ldquo;filling\u0026rdquo;: support for firewall protection (in Preview). That allows a more secure, robust integration with one of the leading data systems on the market, like a lava cake that combines different textures and flavors of secure, efficient connectivity.\nConclusion: A Banquet to Leave You Satisfied (and Eager for the Next One!) Microsoft Fabric\u0026rsquo;s April buffet really came with plenty of options for different tastes and needs, from the democratization of AI, through robust tools for data engineering and data science, to real-time innovations and improved connectivity. Each new feature is an invitation to explore and discover how it can optimize and enrich your work with data.\nNow the invitation is out: explore this menu, try the \u0026ldquo;dishes\u0026rdquo; that caught your interest the most, and see how they can transform your routine, bringing more efficiency and insights to your projects. Microsoft Fabric will certainly keep bringing new features in the coming months.\nUntil the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/microsoft-fabric-presents-the-full-april-buffet/","summary":"April is here, and with it, Microsoft Fabric has prepared a set of important updates. As a platform that\u0026rsquo;s always looking to innovate, Fabric treats us to a variety of new and improved \u0026ldquo;dishes\u0026rdquo;…","title":"Microsoft Fabric Presents: The Full April Buffet! Come Taste What's New!"},{"content":"Any Monsters, Inc. fans out there? Remember how the whole factory depended on the energy from children\u0026rsquo;s screams (and later their laughter!) to keep Monstropolis running? In our data universe, we could say that Databricks is a bit like that factory: an incredible powerhouse, but one that constantly needs \u0026ldquo;energy\u0026rdquo; (our precious data) to generate insights, train AI models and keep everything running.\nThe thing is, fetching that energy from crucial external sources, such as Salesforce (that room packed with valuable sales and customer information) or Workday (the strategic room with all the HR data), often felt like an adventure worthy of a Pixar movie! It required complicated trips through mysterious APIs, facing scary monsters (complex, fragile ETLs) and, worse still, sometimes the \u0026ldquo;energy\u0026rdquo; arrived late or not quite so \u0026ldquo;pure\u0026rdquo;.\nBut what if\u0026hellip; what if there were magic doors? Like those Monsters, Inc. doors that led straight to the right room? Doors that could fetch the energy you need automatically, securely and intelligently, and bring it straight to the storage tanks of the Databricks factory? Sounds like a dream, right? Well, go ahead and pinch the Mike Wazowski next to you, because Databricks has made this a reality with the new Lakeflow Connect connectors for Salesforce and Workday! They\u0026rsquo;re true magic doors that simplify data ingestion!\nThe World Before the Doors: Scaring by Hand (and the Hard Way!) Let\u0026rsquo;s be honest: before these super-efficient doors, life wasn\u0026rsquo;t easy for the data engineer \u0026ldquo;monster\u0026rdquo;. How did we collect energy from Salesforce and Workday? Often, we had to draw complex maps (custom scripts), decipher hieroglyphics in APIs that changed without warning (constant maintenance), or build fragile, handmade contraptions (breakage-prone ETL pipelines) for each data \u0026ldquo;room\u0026rdquo;.\nIt was common to have to \u0026ldquo;scare\u0026rdquo; the entire system (run full loads) frequently, even when we only needed the new energy. Capturing only the changes (incremental ingestion) was a challenge of its own, almost like trying to grab a lost sock in a dark room without waking the child.\nThe cost of this \u0026ldquo;manual scaring\u0026rdquo; was high: it ate up precious time from our best \u0026ldquo;scarers\u0026rdquo; (data engineers), who could have been focused on more strategic tasks, like analyzing the quality of the energy or creating new ways to use it (AI projects). On top of that, it created inconsistencies and delayed fresh \u0026ldquo;energy\u0026rdquo; from reaching the people who needed it most (data analysts and scientists).\nOpening the Magic Doors: Lakeflow Connect in Action! But now, Data Monsters, Inc. has invested in cutting-edge technology! With Lakeflow Connect, we have a state-of-the-art door system. Picture an endless hallway of colorful doors, but now with a smart control panel. Each door (connector) is custom-built for a specific \u0026ldquo;world\u0026rdquo;, such as Salesforce or Workday. They\u0026rsquo;re super secure and incredibly smart.\nDatabricks offers these connectors as managed and optimized, which means they handle much of the complexity for us. But how do these doors work in practice?\nSimple Setup: Configuring a new door is surprisingly easy. It often takes just a few clicks in the Databricks interface, like choosing the right door on the panel and providing the access credentials (the key to the room!). Goodbye, weeks of custom code development! Smart Automation: Once configured, the doors \u0026ldquo;open and close\u0026rdquo; on their own. You can schedule energy (data) collection to run at regular intervals, ensuring a steady flow to the factory. Incremental Efficiency (The Secret!): This is the real magic! Lakeflow connectors are smart. They know exactly which \u0026ldquo;energy\u0026rdquo; has already been collected (which records have been ingested) and fetch only what\u0026rsquo;s new or what has changed since the last visit. No more \u0026ldquo;scaring\u0026rdquo; the whole system every time! This saves resources, speeds up the process and ensures your data in Databricks is always up to date. Monster-Grade Security: The doors are equipped with the best security systems in Monstropolis. The connection is secure, the data is transferred encrypted and you control which data (fields, objects) can pass through the door. More Energy, Fewer Scares: The Benefits of the Lakeflow Doors With these magic doors up and running, the Databricks \u0026ldquo;factory\u0026rdquo; works far more efficiently and productively. The \u0026ldquo;energy\u0026rdquo; (data) arrives faster, it\u0026rsquo;s more reliable (fewer manual errors) and the \u0026ldquo;monsters\u0026rdquo; (data professionals) can finally take off their scare helmets and focus on more fun, strategic tasks, like making the children laugh (generating valuable insights) or analyzing the quality of the energy collected (ensuring governance).\nLet\u0026rsquo;s list the practical benefits of this door system:\nSimplified Data Engineering: The need to build and maintain complex ingestion pipelines for Salesforce and Workday is drastically reduced. Less custom code means fewer bugs, less maintenance and fewer sleepless nights for engineers. Fresh Data for Analytics and AI: Incremental ingestion keeps your Data Lakehouse in Databricks always in sync with the latest sales, customer, finance and HR data. That\u0026rsquo;s absolutely crucial for real-time analytics, up-to-date dashboards and Machine Learning or Generative AI models that need the latest information to be effective. Democratized Ingestion: The simplicity of the connectors lets more people across the organization (not just the most experienced data engineers) bring important data from Salesforce and Workday into Databricks, accelerating analytics capabilities across different areas. Faster Projects: By freeing up engineers\u0026rsquo; time and ensuring reliable data is available quickly, the analytics, BI, ML and AI projects that depend on these sources can start sooner and deliver value faster. If we were to use another metaphor (because we love them!), it\u0026rsquo;s like having a magic, smart delivery service. You order the ingredients (data) from Salesforce and Workday, and Lakeflow Connect delivers them to your Databricks \u0026ldquo;kitchen\u0026rdquo; right on time, already chopped and ready to use (incremental), without you having to go to the store!\nConclusion: Monstropolis Has Never Been So Energized! We can say goodbye to a good share of the scares and headaches when integrating crucial Salesforce and Workday data into our Data Lakehouse! The Databricks Lakeflow connectors are true magic doors that have arrived to automate, simplify and streamline this fundamental data engineering task.\nBy building these bridges (or opening these doors!) between such important data worlds, Databricks establishes itself as an even more powerful and centralizing \u0026ldquo;factory\u0026rdquo;, ready to generate more \u0026ldquo;laughter\u0026rdquo; (that is, insights, innovation and business value) with data that\u0026rsquo;s always fresh and reliable.\nThis is yet more clear proof of how modern data platforms are evolving: being good at processing and analyzing data isn\u0026rsquo;t enough, you have to make it easy (really easy!) for that data to arrive on the platform. The future is, without a doubt, connected!\nSee you in the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/monsters-inc-no-more-ingestion-scares-databricks-lakeflow-connect/","summary":"Any Monsters, Inc. fans out there? Remember how the whole factory depended on the energy from children\u0026rsquo;s screams (and later their laughter!) to keep Monstropolis running? In our data universe, we could say that…","title":"Monsters, Inc.: No More Ingestion Scares! Azure Databricks Opens Magic Doors to Salesforce and Workday with Lakeflow Connect!"},{"content":"Hey there, data community! You know that feeling of being in Gotham City\u0026hellip; I mean, in our Data City? It\u0026rsquo;s vibrant, full of opportunities, but also packed with challenges: we\u0026rsquo;ve got the villains of complexity lurking on every corner, the analytics riddles that need cracking and, sometimes, even a few penguins of inefficiency trying to freeze our projects. To navigate this landscape and bring order (and insights!), we need heroes, and cutting-edge technology!\nIn this universe, Microsoft Fabric often feels like our Batman: equipped with the most advanced gadgets, including the amazing Copilot, practically a utility belt supercharged with Artificial Intelligence, ready for any situation. But\u0026hellip; what about Robin? Those of us on smaller Fabric capacities (like F2, F4, F8\u0026hellip;) often looked at those gadgets in awe, dreaming of using them in our daily fight against crime\u0026hellip; I mean, against data challenges. It was like watching the Batmobile fly by while you were still on a bicycle.\nBut hold on to your cowls and masks! Commissioner Gordon just lit up the Bat-Signal with blockbuster news straight from Microsoft: Batman decided it was time to share the tech! Copilot in Microsoft Fabric is now available to every hero with a paid capacity starting at the F2 SKU! Yes, you read that right!\nIn this article, we\u0026rsquo;ll dive deep into this \u0026ldquo;upgrade\u0026rdquo; for our beloved sidekick. Which Copilot \u0026ldquo;gadgets\u0026rdquo; can Robin (that is, you, with your F2!) use now? How does this change the game in our battle for justice\u0026hellip; and for faster, smarter insights? Adjust your cape, the adventure is just getting started!\nThe (Exclusive) Glory Days of Batman\u0026rsquo;s Utility Belt Let\u0026rsquo;s be honest: until recently, Copilot in Fabric was like one of those super-exclusive Wayne Enterprises toys. Think of the sonic Batarang only Batman had, or that universal decoder able to open any safe in Gotham. They were amazing, revolutionary tools, but access was restricted to the main Batcave. In other words, to use Copilot and other advanced AI features in Fabric, you needed hefty compute capacity, usually an F64 SKU or higher (or a P1+).\nWhat did that mean in practice? It meant many smaller teams, startups or companies on tighter budgets (the Robins, Batgirls and all the heroes in training in Data City) were left a bit out of the generative AI party inside Fabric. It was like wanting to use artificial intelligence to predict the Joker\u0026rsquo;s next move but only having access to an old walkie-talkie.\n\u0026ldquo;Holy Upgrade, Batman!\u0026rdquo; - Robin Gets in the Game! But then comes a plot twist worthy of a comic book! Batman (or Microsoft, in this case) realized that to fight crime effectively across the whole city, Robin needed more firepower. And what firepower!\nThe big change, announced recently and already in effect, is that the minimum requirement to use Copilot in Microsoft Fabric has been drastically lowered. Now anyone with a paid F2 capacity or higher (or P1 and above) can call up their own AI assistant!\nBut which \u0026ldquo;gadgets\u0026rdquo; exactly are in Robin\u0026rsquo;s new kit? The good news is that access looks broad! Users on F2 SKUs can now call on Copilot to help across several areas of Fabric:\nPower BI: Need to build a report from scratch or write that complex DAX measure? Copilot is there, like a direct line to the wisdom of the Batcave, ready with suggestions. Data Factory: Designing data pipelines can be a maze. Copilot acts like an AI-guided Batarang, helping you build and optimize your flows. Data Engineering/Science (Notebooks): Writing Python or SQL code in notebooks? Copilot works like a mini decoder, helping you generate, explain and fix your code. Data Warehouse: Need to optimize queries or understand complex schemas? Copilot is ready to size up the situation. It\u0026rsquo;s as if Robin finally got his own set of essential tools, straight from his mentor\u0026rsquo;s utility belt!\nThe Impact of the Upgrade: What Can Robin Do Now? This change is much more than just handing the sidekick some new toys. With these Copilot gadgets in hand, Robin becomes a far more complete and effective hero in Data City!\nThe core point here is the Democratization of AI. Generative AI tools that used to feel out of reach for many are now a click away for a much larger base of Fabric users. Think about the practical benefits:\nAccelerated Productivity: Analysts can build reports faster, engineers can develop pipelines with less effort, data scientists can explore and code more efficiently. Copilot acts as a tireless pair programmer, an assistant that automates repetitive tasks and suggests smart solutions. Continuous Learning: Copilot isn\u0026rsquo;t just a doer, it\u0026rsquo;s a virtual mentor! It can explain complex code, teach new DAX functions or show you how to optimize a SQL query. It\u0026rsquo;s like having Alfred or Batman himself explaining the tricks of the trade. Innovation Unlocked: Small companies or teams on limited budgets can now try out the power of generative AI in their data projects without a massive upfront investment in capacity. That opens doors to new ideas and solutions that weren\u0026rsquo;t feasible before. If I were to use one of my favorite food metaphors, I\u0026rsquo;d say it\u0026rsquo;s as if the \u0026ldquo;secret recipe\u0026rdquo; for the Bat-cake (cutting-edge AI) now came with super-simplified instructions and ingredients you already have in the pantry (your F2 SKU). Even Robin (or Alfred!) can bake a delicious cake now!\nIs the Sidekick Still a Sidekick? (A Quick Note on Power) It\u0026rsquo;s important to keep your feet on the ground (or your boots on the rooftop, if you prefer). Robin got access to the gadgets, and that\u0026rsquo;s fantastic! But he still doesn\u0026rsquo;t have Batman\u0026rsquo;s brute strength or unlimited resources. The performance of these gadgets can, in fact, depend on the power of the \u0026ldquo;belt\u0026rdquo; you\u0026rsquo;re wearing (the size of your Fabric SKU).\nIn practice, this means that although Copilot works on an F2, response times for very complex tasks or processing large volumes of data may be slower compared to an F64. Think of it as Robin\u0026rsquo;s Batarang not having the same range or impact as Batman\u0026rsquo;s. But the crucial point, worth repeating, is: the access is there! The functionality is available, ready to be used and explored.\nConclusion: A New Era for Data Heroes! This news is, without a doubt, a milestone! Copilot in Fabric, which started out as a privilege for the \u0026ldquo;Batmen\u0026rdquo; with large capacities, is now equipping the \u0026ldquo;Robins\u0026rdquo; too, starting at the F2 SKU. It\u0026rsquo;s a giant step in the mission to democratize Artificial Intelligence, putting truly powerful tools in the hands of more data professionals, regardless of the size of their Batcave.\nThis not only boosts individual productivity, it empowers entire teams to innovate and extract more value from their data. The fight against complexity and slowness in Data City just got a little fairer!\nWhat else can we expect? Which other \u0026ldquo;gadgets\u0026rdquo; will be shared in the future? One thing is certain: the world of data technology keeps evolving at Batmobile speed!\nReferences:\nAnnouncement in the April 2025 Feature Summary: [https://blog.fabric.microsoft.com/en-US/blog/fabric-april-2025-feature-summary/](https://blog.fabric.microsoft.com/en-US/blog/fabric-april-2025-feature-summary/)\nCopilot in Fabric Overview (Documentation): [https://learn.microsoft.com/en-us/fabric/fundamentals/copilot-fabric-overview](https://learn.microsoft.com/en-us/fabric/fundamentals/copilot-fabric-overview)\nCopilot in Fabric FAQ: [https://learn.microsoft.com/en-us/fabric/fundamentals/copilot-faq-fabric](https://learn.microsoft.com/en-us/fabric/fundamentals/copilot-faq-fabric)\n","permalink":"https://julianamarialop.github.io/en/articles/holy-upgrade-batman-robin-f2-sku-now-has-copilot-in-fabric/","summary":"Hey there, data community! You know that feeling of being in Gotham City\u0026hellip; I mean, in our Data City? It\u0026rsquo;s vibrant, full of opportunities, but also packed with challenges: we\u0026rsquo;ve got the villains of…","title":"Holy Upgrade, Batman! Robin (Your F2 SKU) Now Has Access to Copilot's Gadgets in Fabric!"},{"content":"I was thinking about how to explain the Microsoft Fabric SKU Estimator in a simple way, and the age-old art of the Brazilian churrasco came to mind. Yes, because anyone who has ever hosted a barbecue knows there is a very fine line between absolute success and the catastrophe of running out of picanha (the prized top sirloin cap of any Brazilian barbecue).\nAnd, just like at a barbecue, in the data world you also need to get the math right so everything works as expected, without too much left over and without running short when it counts.\nFirst, the challenge: how much to buy? Hosting a barbecue involves several variables:\nHow many people are coming? Are there vegetarians in the group? How long will the event last? Will there be side dishes? Beer? When estimating resources in Microsoft Fabric, the same thing happens:\nWhat is the data volume? How often is that data accessed? Do reports need to be fast, or can they take a few minutes? Will there be usage peaks? The Microsoft Fabric SKU Estimator comes in as that spreadsheet-loving friend at the barbecue who calculates exactly how much of everything you need.\nThe rookie grill master\u0026rsquo;s mistake: eyeballing it Have you ever seen someone buy 2 kg of meat per person? Or worse, buy only sausage thinking \u0026ldquo;there\u0026rsquo;s enough for everyone\u0026rdquo;? Exactly.\nWithout the estimator, you run the risk of:\nChoosing an oversized SKU (leftover meat and a fridge stuffed for days). Or undersizing (guests going home hungry or, in the data world, the system freezing under heavy usage). The SKU Estimator helps you simulate different scenarios based on your \u0026ldquo;guests\u0026rdquo; (workloads), suggesting the ideal SKU that balances performance and cost.\nUsing the estimator: just like a barbecue calculator The process is very similar to googling \u0026ldquo;how many pounds of meat per person?\u0026rdquo;. Only here, you provide:\nType of usage (ingestion, transformation, visualization, etc.) Number of concurrent users Frequency and volume of the workloads With that, it tells you: \u0026ldquo;go with this SKU right here, it\u0026rsquo;s perfect!\u0026rdquo; Nothing left over, nothing missing, and the data barbecue goes off beautifully.\nScenario 1 – Contoso\u0026rsquo;s big barbecue: AI-ready analytics Picture a corporate barbecue at a factory. Contoso Manufacturing called in a systems integrator because their grill (or rather, their analytics architecture) was already outdated.\nThe current solution that needs to be modernized has the following high-level metrics:\n50 source systems processed in batch twice a day 1,500 entities and tables are part of the ETL pipelines An estimated 50 TB of data will be stored on the platform over the next five years Platform operations include 4,000 IoT sensors sending telemetry every 10 seconds, with an average telemetry message size of 600 bytes There are 10,000 factory workers with daily clock-in events for shift start/end and break in/out, generating 8 data points per worker per day An estimated 1,300 Power BI dashboard and report users per day 150 Power BI report authors An estimated maximum semantic model size of 25 GB After some discovery and design workshops, the systems integrator created the following high-level architecture:\nIt\u0026rsquo;s like planning an event with 5 buffets, different grill masters and a digital health inspector. The architect used the SKU Estimator to calculate all of this, selecting the workloads: Spark, Power BI, RealTime Intelligence, Eventstream, Eventhouse and Data Activator.\nTo simplify this task, we\u0026rsquo;ll use the Microsoft Fabric SKU Estimator.\nTo get started, we first need to extract the required high-level inputs and the workload-specific inputs based on the high-level metrics above.\nThe total data in the architecture is calculated to be approximately 8,533 GB, based on an estimated 6:1 compression ratio. We also infer that the platform will have 2 batch cycles processing 1,500 tables and datasets across all 50 source systems. We enter this information into the SKU Estimator, as shown below:\nNext, we select all the workloads that need to be included in the estimate. Six workloads are identified: Data Factory, Spark, Eventstream, RealTime Intelligence, Power BI and Data Activator.\nNow we need to complete the workload-specific inputs. In the Data Factory section, enter 0, since the solution does not include Dataflow Gen2:\nIn the Power BI section, enter the numbers according to the high-level metrics:\nFor Eventstream, we extrapolate the telemetry numbers to a daily ingestion size. We do this with the following calculation: (Sensors * events per day) * event message size, which gives a total of ~19 GB.\nThen enter the number of eventstreams as 1 and the total number of destinations as 5. This number represents the number of topics that make up the sensor data. For eventstream source connectors, enter 0, since all events are collected from an Azure EventHub and don\u0026rsquo;t need to be counted.\nIn the Eventhouse section, the daily telemetry data is estimated at 75% of the eventstream data, so it comes to about 14 GB; hot data retention is set to 30 days and total retention to 90 days, and since the platforms run 24×7, a 24-hour duty cycle is entered.\nWhen it comes to Data Activator, we enter the estimated 80,000 events and 5 for the number of alert rules, since there are 5 operational notifications the platforms need to send: excessive shifts, low operator capacity, excess workforce capacity and other proactive notifications.\nOnce that\u0026rsquo;s done, click the \u0026ldquo;Calculate\u0026rdquo; button and get a viable SKU estimate with a breakdown by workload. We also get an estimate of storage consumption and the number of Power BI Pro licenses that may be needed.\nIn the end, the estimator delivered the ideal amount of resources, storage and licenses needed. Everything balanced. No leftover sausage, and no grill master sweating for nothing.\nMoral of the story: before you buy charcoal, use the Estimator Just as planning the perfect barbecue saves you headaches, using the SKU Estimator keeps you from wasting money on too many resources or running short on capacity at critical moments.\nThe Microsoft Fabric SKU Estimator lets you:\nSimulate different volumes and types of usage; Calculate the best SKU (the ideal cut for your event); Estimate costs and licenses; And, of course, make sure your setup can take the heat without wasting resources. 💡 Want to try it? Go to the Microsoft Fabric SKU Estimator and make sure your data barbecue never runs out of meat (or performance).\nIf you liked the analogy, share it with that friend who always overdoes the picanha or the cluster size 😄. See you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/planning-perfect-barbecue-microsoft-fabric-sku-estimator/","summary":"I was thinking about how to explain the Microsoft Fabric SKU Estimator in a simple way, and the age-old art of the Brazilian churrasco came to mind. Yes, because anyone who has ever hosted a barbecue knows there is a very…","title":"Planning the Perfect Barbecue: How the Microsoft Fabric SKU Estimator Keeps You from Getting It Wrong"},{"content":"The other day, while thinking about a new article to share here on LinkedIn, I caught myself reflecting on how hard it is to explain what we do with data and technology to a child, or even to an older relative. Anyone who works in IT, especially in data, has been through that moment at a family lunch: \u0026ldquo;But what is it you actually do again?\u0026rdquo; We try to explain with terms like \u0026ldquo;data platform\u0026rdquo;, \u0026ldquo;pipelines\u0026rdquo;, \u0026ldquo;artificial intelligence\u0026rdquo; and watch the eyes across the table wander off, looking for an easier topic like the soap opera or the game.\nThat\u0026rsquo;s when an idea hit me: what if I explained all of this in a simple, fun way, using something everybody understands? Andy\u0026rsquo;s room, from the classic Toy Story, popped into my head right away. And, come to think of it, it\u0026rsquo;s a perfect metaphor for a company\u0026rsquo;s data universe.\nOnce upon a time\u0026hellip; Andy\u0026rsquo;s room (or a messy data environment) Imagine your company is like Andy\u0026rsquo;s room. Inside there are all kinds of toys: some new, some old, some neatly put away, others completely lost under the bed. Those toys are the data: old spreadsheets, legacy systems, giant files, cloud databases and reports scattered across different tools. The data team really wants to \u0026ldquo;play\u0026rdquo; with them, or rather, work with them, but the mess is so big that more time goes into figuring out where things are than into actually creating value.\nInto this chaotic scene comes our hero: Microsoft Fabric . It\u0026rsquo;s like Woody, the trusty sheriff who shows up with a plan and a crew to help. The mission? Tidy up the room, bring the toys together and make sure everyone can play together, without wasting time or stepping on hidden Lego pieces.\nWhen data is scattered and disorganized, even the most fun game turns into frustration.\nLet\u0026rsquo;s get this playtime organized! Microsoft Fabric arrives with a clear proposal: turn that pile of data into an integrated, efficient environment called a Data Lakehouse. Unlike older solutions, where data had to be split between \u0026ldquo;organized\u0026rdquo; (data warehouse) and \u0026ldquo;messy\u0026rdquo; (data lake), the lakehouse lets everything live together intelligently.\nBut the best part is that Fabric doesn\u0026rsquo;t come alone. It brings a whole squad of toys ready for action, each one with its own role in the game:\nMr. Potato Head (Data Factory): He\u0026rsquo;s in charge of bringing data from the outside into the room. He connects to spreadsheets, APIs, databases, old systems\u0026hellip; and delivers everything organized in the lakehouse, ready to use. Buzz Lightyear (Spark \u0026amp; Notebooks): The hero of analytics! He turns raw data into insights while flying high. With him, you can apply logic, build transformations and even train models with performance and scalability. Analyst Barbie (Power BI): Full of style and highly efficient, she builds beautiful, interactive and accessible dashboards. She gets reports ready for anyone to understand, even people who know nothing about data. Rex (OneLake): He may seem clumsy, but he\u0026rsquo;s the one who knows exactly where every piece of data is. With OneLake, everything is centralized, with no duplication, with security and governance, ready to be shared across the whole company. Copilot (Fabric\u0026rsquo;s AI): The newest and smartest toy on the team. You talk to it in natural language, and it hands you reports, analyses and even ready-made suggestions. A true personal assistant with AI superpowers. _Toy Story characters_\nWith the right toys and a good plan, even the most confusing chaos becomes an environment ready for great adventures.\nThe magic happens\u0026hellip; With all of this working in harmony, the chaos in the room is transformed. The data is now all in one place, clean, organized, accessible and ready to be used. Teams can build reports in minutes, understand what\u0026rsquo;s going on in the company with much more clarity and even predict the future based on real data. The time once spent hunting for information or wrestling with disconnected systems now goes into making smarter, faster decisions.\nMore than that, Microsoft Fabric lets different areas of the company collaborate. The marketing team can cross-reference data with sales, finance gets a better grip on costs, customer service sees behavior patterns. All of this with security, governance and traceability, without giving up agility.\nAccessible, connected data isn\u0026rsquo;t just faster: it makes the company smarter.\nSo what does Microsoft Fabric deliver, the fun way? With Fabric, Andy\u0026rsquo;s room stops being a minefield of lost toys and turns into a true data amusement park. The environment gets organized (no more stepping on Lego in the middle of the night) and all the toys work together, with no need for separate boxes. There\u0026rsquo;s room for new toys whenever you need them, and each one has its own safe little corner. On top of that, with Copilot, you have a smart assistant that understands what you want and hands it to you all ready, effortlessly.\nIt\u0026rsquo;s a new way of dealing with data: lighter, more integrated and much more efficient. And best of all: everything in a single place, without stacking up different tools or constantly switching windows. Microsoft Fabric turns chaos into clarity and mess into value.\nWith Fabric, data stops being a burden and becomes part of the fun.\nThe moral of the story If your company still lives with that mess of data scattered everywhere, like toys left on the floor after an afternoon of play, it may be time to get to know Microsoft Fabric. With it, you have everything in one place, and data stops being a problem and becomes part of the solution.\nFabric organizes, integrates, protects and even brings intelligence to turn information into decisions. When everything works together, securely and with agility, the game flows better and the results show up for everyone.\nBecause at the end of the day, when data is organized, the company works with more ease, more strategy and more impact.\nTo infinity and beyond\u0026hellip; the data mess.\nSee you in the next article.\n","permalink":"https://julianamarialop.github.io/en/articles/to-infinity-and-beyond-data-explaining-microsoft-fabric-to-a-child/","summary":"The other day, while thinking about a new article to share here on LinkedIn, I caught myself reflecting on how hard it is to explain what we do with data and technology to a child, or even to an older relative…","title":"To Infinity and Beyond... Data! Explaining Microsoft Fabric to a Child"},{"content":"Migrating an established platform like SAS can look like a complex challenge. But, just like making an irresistible Banoffee, if we follow the steps calmly and methodically, the result will be delicious! Inspired by a previous article where I walked through a migration from SAS to Azure Databricks using a cake recipe, today I want to share a new and improved recipe: Migrating from SAS to Microsoft Fabric in just 2 layers.\nIn this article, I\u0026rsquo;ll break down each layer in a simple, practical way, so you can implement it smoothly at your company. And, of course, there will be a finishing touch that makes all the difference!\nShall we begin?\nIngredients you\u0026rsquo;ll need for this migration Before you roll up your sleeves, make sure you have all the right ingredients:\n🗂️ Understanding the Current Environment: Complete mapping of the SAS tables: Identification of the existing projects: 🛠️ The Right Tools: The Microsoft Fabric platform: Fabric Copilot (AI assistant for automatically converting SAS scripts). 👩‍💻 A prepared team: Solutions Architect (defines the strategy). Data Engineers and Data Analysts (technical execution). Data Scientists (ensure the quality of analyses after the migration). 🥧 Layer 1 – Migrating the data (The Banoffee\u0026rsquo;s crunchy base) With Banoffee, we always start with the base, that firm layer of cookies that will hold up all the delicious layers on top. In the same way, when migrating from SAS to Microsoft Fabric , it\u0026rsquo;s essential to start by securing a solid, safe base: your data.\nDetailed steps for migrating the data: 1. Survey and document the SAS tables:\nTake an inventory of the existing tables in SAS. Document essential metadata such as name, format, schema, volume, and refresh frequency. This survey is the foundation for an organized migration. 2. Set up the Lakehouse in Fabric:\nCreate a Lakehouse in Microsoft Fabric using OneLake as the centralized repository. Define areas to receive raw data (bronze) and refined data (silver/gold). 3. Set up automation with pipelines:\nIn Microsoft Fabric, use pipelines to automate the extract and load process (ETL). Configure direct connections between your SAS environment and the Lakehouse, enabling a fast and secure migration. 4. Run and monitor the migration:\nUse the pipeline you created to automatically load the SAS tables in Delta Lake format. Track the run through Fabric\u0026rsquo;s built-in monitoring, spotting and fixing any failures quickly. 5. Validate the integrity of the migrated data:\nRun comparison tests to make sure every migrated table is identical to the original in content and structure. Use Fabric\u0026rsquo;s analytics tools to validate consistency and integrity. 🍯🍌 Layer 2 – Migrating the SAS projects (Caramel and bananas) Once we have a solid data base, it\u0026rsquo;s time to fill our Banoffee: transforming the existing SAS scripts and projects so they run inside Microsoft Fabric. This layer is the heart of the migration, bringing real analytical intelligence to your new platform.\nDetailed steps for migrating the projects: 1. Inventory and prioritize the SAS projects:\nDocument in detail every SAS script currently in use. Rank them by business relevance and technical complexity. Prioritize the migration by starting with the most important scripts, the ones critical to operations. 2. Automatic conversion with Fabric Copilot:\nUse Fabric Copilot to automate the conversion of SAS code into Spark notebooks (PySpark or SQL). Copy and paste the original code into Fabric Copilot and ask it to generate the initial conversion automatically. A practical example of using Fabric Copilot:\n/* Exemplo original de código SAS */ PROC SQL; CREATE TABLE clientes_vip AS SELECT id, nome, SUM(compras) AS total FROM vendas WHERE compras \u0026gt; 1000 GROUP BY id, nome; QUIT; Fabric Copilot automatically turns it into an equivalent Spark notebook:\n# Código convertido automaticamente pelo Fabric Copilot clientes_vip = spark.sql(\u0026#34;\u0026#34;\u0026#34; SELECT id, nome, SUM(compras) AS total FROM vendas WHERE compras \u0026gt; 1000 GROUP BY id, nome \u0026#34;\u0026#34;\u0026#34;) clientes_vip.write.format(\u0026#34;delta\u0026#34;).save(\u0026#34;Tables/clientes_vip\u0026#34;) 3. Review and final adjustments:\nReview the converted code and make sure the results are consistent. Make any small adaptations needed for the specifics of the original processes. 4. Final testing and validation of the processes:\nRun each migrated project in parallel with the original SAS ones, validating results and performance. Make sure the results you get are exactly the ones expected. 🍦 Finishing touch – Special whipped cream: a notebook ready for your migration! To finish this recipe, there\u0026rsquo;s nothing better than a special whipped cream that makes everything even tastier and easier. I\u0026rsquo;ve prepared a parameterizable notebook that you can copy straight into your Microsoft Fabric environment, making your journey much easier:\n# Parâmetros iniciais – Defina seu diretório e Lakehouse sas_file_dir = \u0026#34;/Lakehouse/SASFiles/\u0026#34; destino_delta = \u0026#34;Tables/\u0026#34; # Função para carregar arquivos SAS diretamente para Delta Lake def migrar_tabelas_sas(sas_file_path, table_name): df = spark.read.format(\u0026#34;com.github.saurfang.sas.spark\u0026#34;).load(sas_file_path) delta_path = f\u0026#34;{destino_delta}{table_name}\u0026#34; df.write.format(\u0026#34;delta\u0026#34;).mode(\u0026#34;overwrite\u0026#34;).save(delta_path) print(f\u0026#34;✅ Tabela \u0026#39;{table_name}\u0026#39; migrada com sucesso para \u0026#39;{delta_path}\u0026#39;\u0026#34;) # Automatize a migração de todos os arquivos no diretório files = mssparkutils.fs.ls(sas_file_dir) sas_files = [f.path for f in files if f.name.endswith(\u0026#34;.sas7bdat\u0026#34;)] for sas_file in sas_files: nome_tabela = sas_file.split(\u0026#34;/\u0026#34;)[-1].replace(\u0026#34;.sas7bdat\u0026#34;, \u0026#34;\u0026#34;) migrar_tabelas_sas(sas_file, nome_tabela) Remember: a successful migration, just like a delicious Banoffee, depends on good planning, patience, and, of course, good ingredients!\nWhat about you, have you worked with Fabric yet? Did you run into sweet or bitter challenges during your migration? Share your experience in the comments! Let\u0026rsquo;s savor these stories together.\nSee you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/how-to-migrate-from-sas-to-microsoft-fabric-2-layer-banoffee-recipe/","summary":"Migrating an established platform like SAS can look like a complex challenge. But, just like making an irresistible Banoffee, if we follow the steps calmly and methodically, the result will be delicious! Inspired…","title":"How to Migrate from SAS to Microsoft Fabric: An Irresistible 2-Layer Banoffee Recipe"},{"content":"Introduction: The Real-Time Data Factory Picture a modern production line where products are manufactured in real time, without interruption. Now swap the products for data, and you\u0026rsquo;ll have an idea of what data streaming is. Welcome to the factory of the future, where data is processed 24/7, with no coffee breaks!\nTo understand the concept better, let\u0026rsquo;s use a simple analogy: think of data streaming as a river of information that flows constantly. Unlike a lake (which represents batch processing), this river is always moving, bringing something new every second. In this river, each drop is a piece of data: a click on a website, a bank transaction, a sensor reading or a social media post. What makes this river special is its ability to carry information instantly, letting companies and organizations make decisions in real time.\nBut how does it work in practice? Imagine you own a chain of retail stores. With data streaming, you can know instantly when a product is about to sell out, what customers prefer in different regions, or even spot buying patterns before they become obvious. It\u0026rsquo;s like having a business intelligence superpower that works 24 hours a day, 7 days a week, without ever needing to sleep or take a coffee break. This ability to process and analyze data in real time is changing the way companies in every industry make strategic decisions.\nWhat is Streaming? Data streaming is like our nonstop production line. Unlike batch processing, where large volumes of data are processed at scheduled intervals, streaming handles continuous flows of data in real time.\nThe main difference is latency:\nBatch: Minutes, hours or days Streaming: Milliseconds or seconds The image below illustrates the Lambda architecture, a data processing model that combines batch and real-time (streaming) processing to handle large volumes of data. The architecture is divided into three main layers:\nSpeed Layer: Processes data in real time using streaming technologies such as Spark Streaming, Storm or Flink. It produces incremental views of the data. Batch Layer: Stores all the raw data and processes it periodically to create precomputed views. Serving Layer: Combines the results of the speed and batch layers to answer queries. The data sources on the left include various origins such as NoSQL, IoT, web logs, ERP and legacy systems. This data feeds both the speed layer and the batch layer. The data flow goes through batch and real-time processing, creating views that are combined in the serving layer. This enables efficient queries using technologies such as Spark SQL, Pig, Hive, etc.\n_Lambda Architecture_\nWhy does Streaming matter? Streaming is crucial for use cases that demand immediate insights:\nFraud detection in financial transactions Real-time monitoring of industrial equipment Personalization of user experiences in apps Sentiment analysis on social networks In our data factory, the conveyor belts are the information flows, constantly bringing in new \u0026ldquo;products\u0026rdquo; (read: data) to be processed. This data can come from various \u0026ldquo;suppliers\u0026rdquo;, such as IoT sensors (our electronics branch), financial transactions (the accounting department) or even social networks (the corporate gossip office).\nThe Microsoft Fabric Assembly Line What is Microsoft Fabric? Microsoft Fabric is a unified data analytics platform that combines several services into a single environment. It\u0026rsquo;s like having a multipurpose factory that can produce any kind of insight you need. Its main advantages include:\nUnified platform: Reduces integration costs Scalability: Supports large-scale data processing AI integration: Incorporates machine learning tools Friendly interface: Accessible to a wide range of users Microsoft Fabric is like a super-advanced assembly line, designed to handle this nonstop production of data. Let\u0026rsquo;s meet the workstations on this line:\nEventHouse: This is our raw materials warehouse. Raw data arrives here and is organized for processing.\nKQL Queryset: Think of it as the factory\u0026rsquo;s quality engineer. It examines the data, asks complex questions and makes sure everything is in order.\nReal-time Dashboard: This is the factory\u0026rsquo;s control room. Here, managers can see in real time how insight production is going.\nEventstream: This is our factory\u0026rsquo;s conveyor system. It moves data from one station to another, making sure everything flows without interruption.\nActivator: Think of it as the factory\u0026rsquo;s alarm system. When something important happens, it fires an alert so the team can act quickly.\nHow does Microsoft Fabric handle Streaming? Fabric uses the Lambda architecture to process streaming data:\nSpeed layer: Event Stream Service Batch layer: Pipelines, dataflows and notebooks Serving layer: Connects to different data stores Integration with Power BI and Real-Time Dashboards Fabric integrates seamlessly with Power BI, enabling real-time dashboards for instant visualization of insights.\nConnectivity with other tools Fabric offers connectivity with Azure, Databricks, Kafka and other tools, making it easy to integrate with existing ecosystems.\nHands On: Building Your Insight Production Line Let\u0026rsquo;s build a complete production line to monitor bike rentals in real time, taking advantage of the latest Microsoft Fabric updates.\nStep 1: Set up the EventHouse In the Microsoft Fabric portal, click \u0026ldquo;Create\u0026rdquo; and select \u0026ldquo;Eventhouse\u0026rdquo;. Name your Eventhouse \u0026ldquo;BikeRentalEventhouse\u0026rdquo;. _Creating the Eventhouse_\nStep 2: Set up the Eventstream In the Microsoft Fabric portal, create a new Eventstream called \u0026ldquo;BicycleRentalStream\u0026rdquo;. As the data source, select Microsoft\u0026rsquo;s \u0026ldquo;Bicycles\u0026rdquo; sample dataset. _EventStream Add source_\nConfigure the Eventstream to process data in real time, simulating the continuous flow of bike rental information. Define transformations to extract relevant information, such as:Number of bikes rented per hourLocation of the most popular bikesAverage rental duration Set the destination of the processed data to the EventHouse you created earlier. Adjust the processing settings to simulate a real-time scenario, with frequent updates. Test the data flow using the preview feature to make sure the \u0026ldquo;Bicycles\u0026rdquo; sample data is being processed correctly. Activate the EventStream to start continuous processing of the bike rental data. _Bicycles EventStream ready_\nStep 3: Create a KQL Queryset In the Eventhouse, create a new KQL Queryset called \u0026ldquo;BikeRentalQueries\u0026rdquo;. Write a query to analyze data in real time: bikes | where ingestion_time() \u0026gt; ago(5m) | summarize AvgBikes = avg(No_Bikes), AvgEmptyDocks = avg(No_Empty_Docks) by Neighbourhood | order by AvgBikes desc _Running the KQL query_\nStep 4: Create a Real-time Dashboard Create a new Real-Time Dashboard called \u0026ldquo;BikeRentalDashboard\u0026rdquo;. Add tiles using the query from the KQL Queryset. _Real-time dashboard_\nStep 5: Set up the Activator Create a new Activator called \u0026ldquo;BikeRentalAlert\u0026rdquo;. Configure alerts based on the \u0026ldquo;BicycleRentalStream\u0026rdquo; EventStream. _Example Alert for BikeRental_\nStep 6: Integration with Git and Deployment Pipelines Take advantage of the new GitHub integration for version control. Set up deployment pipelines to manage the lifecycle of your data assets. Step 7: Test and Monitor Use the new workspace monitoring feature to track the performance of your data production line. Take advantage of Fabric Runtime 1.3, which includes Apache Spark 3.5 and other updates, to optimize processing. Use Cases and Real-World Applications IoT monitoring: Real-time analysis of industrial sensor data for predictive maintenance. Bank transaction analysis: Instant processing of transactions to detect suspicious patterns. Fraud detection: Immediate identification of anomalous activity in payment systems. Content personalization: Real-time adjustment of recommendations based on user behavior. Conclusion: The Factory Floor of the Future With these updates, your data factory is more powerful than ever. Microsoft Fabric keeps evolving, offering increasingly robust tools for handling real-time data.\nKey takeaways Streaming in Fabric delivers real-time data processing with minimal latency. The platform unifies several analytics tools, simplifying the workflow. Event Streams enable no-code transformations and flexible data routing. Seamless integration with Power BI for real-time visualizations. Advantages of Streaming in Microsoft Fabric Real-time insights for fast decision-making Scalability to handle large volumes of data Flexibility to connect many data sources and destinations Less complexity with a unified platform Next steps for getting started Get familiar with Fabric\u0026rsquo;s components, especially Event Streams and OneLake. Identify use cases in your organization that would benefit from real-time analytics. Try building a simple event stream using Fabric\u0026rsquo;s no-code tools. Explore integrations with other tools you already use, such as Azure or Kafka. Remember: in the real-time data factory, innovation never stops. Keep learning and experimenting with the new features to keep your knowledge production at the cutting edge of technology!\n","permalink":"https://julianamarialop.github.io/en/articles/streaming-in-microsoft-fabric-the-production-line-of-the-future/","summary":"Picture a modern production line where products are manufactured in real time, without interruption. Now swap the products for data, and you\u0026rsquo;ll have an idea of what data streaming is. Welcome to the factory…","title":"Streaming in Microsoft Fabric: The Production Line of the Future"},{"content":"Imagine your data is like a couch potato who decided to get in shape. DBT (Data Build Tool) is the personal trainer who takes that data, builds the right workout plan (SQL transformations) and takes it to a new level of performance. Instead of just lifting random weights (traditional ETL), DBT takes a more efficient approach: it does the right exercises in the right order (ELT), making sure the data ends up lean and well defined inside the data warehouse.\nWith DBT, data analysts and engineers can write SQL queries to model and transform data without worrying about the underlying infrastructure. It also offers features like code versioning, automated documentation and data quality tests. After all, a good workout needs follow-up!\nWhy use DBT? DBT has been gaining popularity for several reasons:\nEase of Use: It lets data analysts and engineers write SQL transformations without dealing with complex infrastructure. Modularity: It makes it easy to create reusable, well-organized pipelines, like a good workout split by muscle group. Versioning and Control: Since DBT uses Git repositories, it\u0026rsquo;s easy to track changes and roll back to previous versions, just like a personal trainer adjusts a workout plan based on progress. Data Quality Tests: It includes features to make sure transformed data is consistent and reliable. Nobody wants a badly designed workout that causes injuries (data errors!). Automatic Documentation: It generates detailed documentation about the models you create, like a training log that records every bit of progress. Compatibility with Modern Data Warehouses: It works with BigQuery, Snowflake, Redshift, Databricks and others. How do you install DBT? Getting ready to use DBT is like signing up for the gym: a simple process, but one that takes commitment.\nStep 1: Install DBT Run the following command in the terminal:\npip install dbt-core If you\u0026rsquo;re using a specific data warehouse, also install the matching adapter. For example, for Snowflake:\npip install dbt-snowflake Step 2: Configure DBT After installing, initialize a new DBT project:\ndbt init meu_projeto This creates the basic directory and file structure you need to start using DBT.\nStep 3: Test the Connection Before running models, it\u0026rsquo;s important to test the connection to the configured data warehouse:\ndbt debug If everything checks out, DBT is ready to run transformations and optimize your data environment!\nHow does DBT work? DBT works like a well-structured training program: it takes the raw data (the out-of-shape student), applies SQL transformations (the right exercises) and delivers optimized results (data ready for analysis).\nDBT\u0026rsquo;s main components are:\nModels: SQL transformations applied to the data, like specific workouts for different muscle groups. Seeds: Static datasets, like the diet the personal trainer adjusts to support the workouts. Snapshots: Records of changes over time, like before-and-after progress photos. Tests: Automatic checks to ensure quality, like regular health checkups to make sure the plan is working. Documentation: Automatic generation of documentation based on the models\u0026rsquo; code, like a workout notebook that records every session. On top of that, to transform data workflows it uses SQL (.sql) and YAML (.yml) scripts.\nSQL scripts: Help transform data in a modular way, using CTEs (Common Table Expressions) to create reusable, organized processes. YAML scripts: Let you define schemas, descriptions and test rules for columns, such as not_null, unique and other validations, ensuring data integrity and quality. DBT runs these transformations efficiently, creating a clear, modular structure inside the data warehouse.\n_What problems does DBT solve?_\nHow do you use DBT? A hands-on example with a multidimensional model and medallion layers Let\u0026rsquo;s take the example of an e-commerce company that wants to build a multidimensional model for sales analysis, following the medallion layer approach (Bronze, Silver and Gold).\nStep 1: Building the Bronze Layer (Raw Data) The Bronze layer receives data straight from S3, where it\u0026rsquo;s stored in its raw format. To access this data in DBT, we use the source feature, which points to the corresponding bucket:\nversion: 2 sources: - name: ecommerce schema: raw tables: - name: vendas_raw external: location: \u0026#39;s3://meu-bucket/raw/vendas/\u0026#39; format: \u0026#39;parquet\u0026#39; With this configuration, we can reference the data directly in DBT:\nSELECT * FROM {{ source(\u0026#39;ecommerce\u0026#39;, \u0026#39;vendas_raw\u0026#39;) }}; The Bronze layer stores the data exactly as it was received, with no transformation:\nSELECT * FROM {{ source(\u0026#39;ecommerce\u0026#39;, \u0026#39;vendas_raw\u0026#39;) }}; Step 2: Building the Silver Layer (Transformation and Cleanup) Here we apply cleanup, deduplication and formatting rules:\nWITH vendas_limpa AS ( SELECT pedido_id, cliente_id, produto_id, data_venda, quantidade, preco_unitario, quantidade * preco_unitario AS valor_total FROM {{ ref(\u0026#39;bronze_vendas\u0026#39;) }} WHERE data_venda IS NOT NULL ) SELECT * FROM vendas_limpa; Step 3: Building the Gold Layer (Final Model for Analysis) In this layer, we create a model optimized for multidimensional analysis:\nWITH vendas_agrupadas AS ( SELECT data_venda, produto_id, SUM(quantidade) AS total_quantidade, SUM(valor_total) AS total_vendas FROM {{ ref(\u0026#39;prata_vendas\u0026#39;) }} GROUP BY data_venda, produto_id ) SELECT * FROM vendas_agrupadas; Step 4: Building dimensions We also create supporting dimensions, such as dim_cliente.sql:\nSELECT cliente_id, nome, email, cidade, estado FROM {{ ref(\u0026#39;prata_clientes\u0026#39;) }}; And dim_produto.sql:\nSELECT produto_id, nome_produto, categoria FROM {{ ref(\u0026#39;prata_produtos\u0026#39;) }}; These tables work like specific exercises for different muscle groups, helping to strengthen and better organize the data.\nStep 5: Building relationships and analyses With the tables in place, we can run queries in the data warehouse that relate the data and build analytics dashboards, as if we were fine-tuning the workout to get maximum performance out of the results.\nHow to consume the data via DBT After processing and transforming the data with DBT, the next step is to consume it in visualization and analytics tools such as Power BI. Power BI can connect directly to the data warehouse where the DBT models were created, enabling interactive reports and efficient dashboards.\nConnecting Power BI to the Data Warehouse If your DBT is set up on a data warehouse like Snowflake, BigQuery or Redshift, follow the steps below to consume the data in Power BI:\nOpen Power BI and select \u0026ldquo;Get Data\u0026rdquo;. Choose the data source that matches your data warehouse (for example, \u0026ldquo;Amazon Redshift\u0026rdquo;, \u0026ldquo;Google BigQuery\u0026rdquo; or \u0026ldquo;Snowflake\u0026rdquo;). Enter the database access credentials. Select the table that corresponds to the data model transformed by DBT. Create measures and calculations in Power BI for additional analysis. Build interactive charts, tables and dashboards. Sample Query in Power BI If your data warehouse is Snowflake, a simple SQL query to load sales data into Power BI could be:\nSELECT data_venda, produto_id, total_quantidade, total_vendas FROM ouro_vendas; After loading the data, you can use Power BI\u0026rsquo;s features to build trend charts, sales KPIs and dynamic dashboards that support decision-making.\nHow to use a CI/CD pipeline for load and orchestration scripts with Airflow To make sure the transformations done with DBT run reliably and automatically, we can use a CI/CD pipeline combined with Apache Airflow to orchestrate the data pipelines.\nImplementing CI/CD for DBT The CI/CD (Continuous Integration/Continuous Deployment) process for DBT involves versioning, automated testing and continuous deployment of changes to the data models. Here\u0026rsquo;s a basic flow:\nCode Versioning: The SQL code for the DBT models is managed in Git. Automated Tests: Data integrity tests run automatically on every change. Automated Deploy: With tools like GitHub Actions, GitLab CI/CD or Jenkins, the updated code is automatically deployed to the production environment. Scheduling and Execution: Airflow orchestrates the execution of the data transformation pipelines. Orchestrating with Airflow Apache Airflow is a powerful tool for managing and scheduling data pipelines. We can use it to orchestrate the data layers (Bronze, Silver and Gold) efficiently.\nExample of an Airflow DAG to orchestrate the medallion model layers:\nfrom airflow import DAG from airflow.operators.bash import BashOperator from datetime import datetime default_args = { \u0026#39;owner\u0026#39;: \u0026#39;airflow\u0026#39;, \u0026#39;depends_on_past\u0026#39;: False, \u0026#39;start_date\u0026#39;: datetime(2024, 1, 1), \u0026#39;retries\u0026#39;: 1, } dag = DAG( \u0026#39;dbt_medalhao_pipeline\u0026#39;, default_args=default_args, schedule_interval=\u0026#39;@daily\u0026#39;, catchup=False ) bronze = BashOperator( task_id=\u0026#39;bronze_layer\u0026#39;, bash_command=\u0026#39;dbt run --models bronze\u0026#39;, dag=dag ) prata = BashOperator( task_id=\u0026#39;prata_layer\u0026#39;, bash_command=\u0026#39;dbt run --models prata\u0026#39;, dag=dag ) ouro = BashOperator( task_id=\u0026#39;ouro_layer\u0026#39;, bash_command=\u0026#39;dbt run --models ouro\u0026#39;, dag=dag ) bronze \u0026gt;\u0026gt; prata \u0026gt;\u0026gt; ouro In this example:\nThe Bronze layer is loaded first. Next, the data is cleaned and transformed in the Silver layer. Finally, the data is aggregated and modeled for analysis in the Gold layer. This approach ensures the data is processed correctly within a well-organized structure, providing better governance and reliability.\nDBT Pricing DBT offers a free open source version, plus a SaaS version called DBT Cloud, which includes additional features like a web interface, job scheduling and stronger enterprise support. DBT Cloud pricing varies with data volume and number of users, but generally ranges from free plans up to custom pricing for large companies.\nPractical applications of DBT DBT is widely used by companies that want to improve their data transformation processes. Some common applications include:\nBuilding Data Marts: Creating data models optimized for BI and business analysis. Data Quality Monitoring: Applying tests to ensure the integrity of the information. Auditing and Traceability: Enabling versioning and tracking of the transformations applied. Data Pipeline Automation: Making repetitive, scalable processes easier to run. How to learn DBT If you want to go deeper into DBT and become an expert, there are several ways to learn.\nCourses on DBT Learn DBT\u0026rsquo;s official learning platform, DBT Learn, offers free and paid courses for beginners and advanced users. There you\u0026rsquo;ll find interactive tutorials, hands-on exercises and guided training to learn how to use DBT effectively.\nDBT Certifications For those who want to prove their knowledge, DBT offers official certifications. These certifications validate your skills in building data models, versioning, testing and automation with DBT. Besides sharpening your skills, a certification can be a differentiator in the job market.\nOther resources include the official documentation, YouTube videos and technical blogs covering best practices and real-world use cases.\nConclusion DBT is like a personal trainer for your data: it makes sure every transformation is done right, strengthening your data pipeline and delivering quality insights. If your company is looking to optimize data modeling and analysis, DBT may be the ideal choice to get off the digital couch!\nWhat\u0026rsquo;s more, as more companies adopt DBT, the community around the tool keeps growing. That means more support, more tutorials and more best practices available to new users, making it easier to adopt and improve the solution.\nIf you haven\u0026rsquo;t tried DBT yet, maybe it\u0026rsquo;s time to add it to your data workout plan. After all, nobody wants an out-of-shape database full of slow queries and redundancy. It\u0026rsquo;s time to turn your data into true information athletes! DBT is like a personal trainer for your data: it makes sure every transformation is done right, strengthening your data pipeline and delivering quality insights. If your company is looking to optimize data modeling and analysis, DBT may be the ideal choice to get off the digital couch!\n","permalink":"https://julianamarialop.github.io/en/articles/dbt-the-personal-trainer-that-turns-your-data-into-elite-athletes/","summary":"Imagine your data is like a couch potato who decided to get in shape. DBT (Data Build Tool) is the personal trainer who takes that data, builds the right workout plan (SQL transformations) and takes it to a new…","title":"DBT: The Personal Trainer That Turns Your Data into Elite Athletes"},{"content":"Imagine a world where sharing data between companies is as simple as opening a delivery app and ordering your favorite meal. Instead of complicated processes, slow transfers, and security worries, you get instant access to reliable, up-to-date data. That\u0026rsquo;s the future Databricks Delta Sharing offers: an open, secure, and efficient system for sharing data across organizations and platforms, with no headaches.\nWhat Is Databricks Delta Sharing? Databricks Delta Sharing works like a big data buffet restaurant. Instead of every user having to cook their own dishes (that is, copy and replicate data), they can simply grab what they need from the self-service station, with no waste and guaranteed quality. And the best part: no endless lines or mystery dishes!\nBuilt on the Delta Lake architecture, it lets organizations share governed data from their Lakehouses with any consumer, whether or not they\u0026rsquo;re on the Databricks platform. This open protocol ensures recipients can access shared data through tools such as Python, R, SQL, and BI platforms, promoting interoperability and efficiency.\n_Databricks Delta Sharing_\nNow that you\u0026rsquo;ve got the concept of Delta Sharing, let\u0026rsquo;s explore how it can transform the way your company shares information.\nWhy Should My Company Adopt Delta Sharing? Many companies struggle to share data securely and efficiently. Legacy processes require excessive replication, high operating costs, and security risks. Delta Sharing solves these problems by bringing:\n✅ Lower costs: No need for data replication or manual transfers.\n✅ Secure access: Centralized governance and fine-grained control over who accesses the data.\n✅ Interoperability: Works with a wide range of tools and languages, with no lock-in to a single vendor.\n✅ Real-time updates: Data is always current, without unnecessary ETL pipelines.\n✅ Easy integration: Data is accessible directly by consumers in their own work environments.\nNow that we know the benefits, let\u0026rsquo;s look at the different types of sharing available.\nTypes of Sharing in Delta Sharing Just as a buffet restaurant offers different kinds of dishes to suit different tastes and dietary restrictions, Delta Sharing provides three distinct ways to share data, depending on your company\u0026rsquo;s needs:\nDatabricks-to-Databricks Protocol: Share data and AI assets across Databricks workspaces, with governance built into Unity Catalog. Open Sharing Protocol: Lets any consumer, even without Databricks, access shared data through open APIs. Open Source Implementation: Share data across any platform, without depending on Databricks. Whatever method you choose, making sure recipients have efficient and secure access is essential.\nWhat Is a Recipient? In Delta Sharing, the Recipient is like a buffet customer who can access the available dishes (data) as needed, without having to cook from scratch. The Recipient can be a business partner, a customer, or an internal team. Recipients don\u0026rsquo;t need to be on Databricks, since Delta Sharing supports access from popular tools such as Python, SQL, Pandas, Apache Spark, Power BI, and many others.\nEach Recipient receives a secure access token or a credentials file, ensuring that only authorized users can access the data.\nNow that you know how the data is accessed, let\u0026rsquo;s see how to set up Delta Sharing in your company.\nHow to Set Up Delta Sharing in Your Account Before you start sharing or consuming data through Delta Sharing, your company needs to set up Databricks Unity Catalog, ensuring full governance and security over the data. The setup includes:\nEnabling Unity Catalog in your Databricks workspace. Defining Access Policies, configuring security rules for granular control. Turning on Audit Logs to track data access and sharing. Registering Recipients, making sure only authorized users receive access permissions. Now that your account is set up, let\u0026rsquo;s see how to create and share data in practice.\nClean Rooms: Secure, Collaborative Data Sharing An advanced approach within Delta Sharing is the concept of Clean Rooms, a secure environment that lets different companies collaborate on data analysis without directly sharing their raw information. This is especially useful for sectors such as healthcare, finance, and retail, where data privacy is a priority.\nWith Clean Rooms, companies can define rules that let partners analyze aggregated, anonymized data, ensuring compliance with regulations such as GDPR and LGPD (Brazil\u0026rsquo;s data protection law). This capability lets multiple partners extract insights from the same dataset without compromising security.\nCompanies Already Using Delta Sharing Several global companies have already adopted Delta Sharing to improve the efficiency and security of their data sharing:\nShell: Uses Delta Sharing to share operational insights with partners in the energy sector, ensuring efficient collaboration. HSBC: Implemented Delta Sharing to streamline compliance processes and financial risk analysis. Adobe: Uses the platform to share marketing and customer engagement data with strategic partners. Nasdaq: Uses Delta Sharing to provide secure access to market data and analytical reports for investors and regulators. Delta Sharing Costs Although Delta Sharing removes the need for data replication, it can generate costs related to transferring data across different regions and platforms. The costs may include:\nData egress (egress fees): When data is shared across cloud regions, the transfer may be billed. Query processing: Depending on the infrastructure used, there are costs associated with running queries on the shared data. Permission and log management: Unity Catalog lets you track access and manage permissions, which can affect usage costs. Cost Calculation Example Let\u0026rsquo;s walk through a practical example to estimate Delta Sharing costs:\nVolume of shared data: 10 TB per month\nData egress cost (egress fees): $0.09 per GB\nQuery processing cost: $0.02 per query (assuming 100,000 queries)\nLog management cost: $0.005 per log operation (assuming 500,000 operations)\nTotal calculation:\nData egress cost: $921.60 Processing cost: $2,000.00 Log management cost: $2,500.00 Estimated total monthly cost: $5,421.60 Companies that share large volumes of data should consider strategies to optimize these costs, such as storing data in regions close to consumers or using data compression.\nConclusion: Delta Sharing Is the Future of Data Sharing With Databricks Delta Sharing, your company can finally leave behind manual, slow, and insecure data sharing processes. This shift not only increases operational efficiency but also enables a smoother, more trustworthy culture of collaboration, letting different teams and partners access critical information instantly and securely.\nOn top of that, adopting Delta Sharing provides a solid foundation for future innovation. With data accessible in real time and robust governance, organizations can accelerate their artificial intelligence and analytics initiatives, producing more accurate and strategic insights. This lets companies make informed decisions quickly, responding nimbly to market changes.\n💡 If your company wants to collaborate efficiently, cut costs, and drive innovation, it\u0026rsquo;s time to adopt Delta Sharing! 🚀 Instead of relying on transfers, copies, and replications, the data is accessible wherever and whenever it\u0026rsquo;s needed, with full control and security.\n","permalink":"https://julianamarialop.github.io/en/articles/azure-databricks-delta-sharing-and-the-data-buffet/","summary":"Imagine a world where sharing data between companies is as simple as opening a delivery app and ordering your favorite meal. Instead of complicated processes, slow transfers, and worries…","title":"Azure Databricks Delta Sharing and the Data Buffet: Pick, Serve Yourself, and Analyze!"},{"content":"If the ancient Egyptians could build an artificial intelligence, they would probably call on their gods to make the system wiser, faster and more efficient. Just as Ra lit up the world, Thoth recorded knowledge, Anubis judged the truth and Isis brought agility and balance, a well-structured LLM (Large Language Model) needs to combine these elements to work properly.\nIn practice, building an LLM on Databricks isn\u0026rsquo;t just about training a language model; it\u0026rsquo;s about creating a complete solution that can retrieve information in real time (RAG), adapt to context (Fine-Tuning) and run efficiently (MosaicML).\nIf you want to understand how these technologies can transform your business (and why Databricks is the ideal platform for it), keep reading.\nThe Traditional LLM: Ra without the Book of Thoth In Egyptian mythology, Ra was the Sun god, responsible for crossing the sky every day in his barque, lighting up the world and bringing knowledge. But even as a powerful god, he had a limitation: he only knew what he saw during his daily journey. If something happened in the underworld or in distant lands, he had no way of knowing.\nThe same goes for a traditional LLM. It\u0026rsquo;s trained on a huge amount of data, but once it\u0026rsquo;s done, it can\u0026rsquo;t learn anything else on its own. If something new happens in the world (a change in a law, say, or a new product hitting the market), it may give an outdated answer or even make up false data (hallucination).\nIn a business context, that means a chatbot that answers questions incorrectly, a contract analysis system that ignores recent clauses, or a virtual assistant that can\u0026rsquo;t explain the company\u0026rsquo;s new policy.\nTo solve this problem, we need something that lets the LLM fetch up-to-date information before answering. And that \u0026ldquo;something\u0026rdquo; has a name: RAG.\nLLM with RAG: Thoth and the Book of Wisdom In Egyptian mythology, Thoth was the god of wisdom, writing and accounting. He was responsible for recording everything that happened and owned the Book of Thoth, which held all the knowledge in the universe.\nIf Ra could consult this book before answering questions, he wouldn\u0026rsquo;t depend on his own memory alone; he could access external information to provide accurate, up-to-date answers.\nThat\u0026rsquo;s exactly what RAG (Retrieval-Augmented Generation) does.\nWith RAG, the LLM can access databases, internal documents, APIs and Data Lakes to complement its answer. It doesn\u0026rsquo;t depend only on what it learned during training; it can fetch new data in real time.\n💡 Practical example: Imagine a bank wants to use a chatbot to answer customer questions about investments. If the interest rate changes today, a traditional LLM may keep giving the old answer. But with RAG, it looks up the correct information before answering, making sure the customer gets up-to-date, reliable data.\nHow do you implement RAG in Databricks? Databricks is ideal for this approach because it:\nLets you store documents and structured tables in a Delta Lake. Makes it easy to build vector databases using libraries such as ChromaDB. Offers scalable APIs for data retrieval. In Databricks, we can set up a RAG pipeline like this:\nfrom langchain.document_loaders import CSVLoader from langchain.vectorstores import Chroma from langchain.embeddings import HuggingFaceEmbeddings # Carregar documentos empresariais loader = CSVLoader(\u0026#34;dbfs:/mnt/datalake/documentos.csv\u0026#34;) documents = loader.load() # Criar embeddings para busca eficiente embeddings = HuggingFaceEmbeddings(model_name=\u0026#34;sentence-transformers/all-MiniLM-L6-v2\u0026#34;) vectorstore = Chroma.from_documents(documents, embeddings) # Criar um retriever para buscar informações relevantes retriever = vectorstore.as_retriever() Now the LLM can consult the \u0026ldquo;Book of Thoth\u0026rdquo; whenever it needs to, ensuring smarter, more reliable answers.\nFine-Tuning: The Judgment of Anubis and Osiris The ancient Egyptians believed that, upon death, the soul went through the Judgment of Osiris. Anubis, the god of the dead, weighed the heart of the deceased against the feather of the goddess Maat, symbol of truth and justice. If the heart was light, the soul could move on to eternal life. If it was heavy, it would be devoured by Ammit, the devourer of souls.\nFine-Tuning plays this role for an LLM. It tunes the model so its answers are evaluated and refined until they reach an acceptable level of accuracy.\nIn Databricks, this can be done with clusters optimized for deep learning, speeding up the process of tuning the model to meet the specific needs of the business.\nfrom transformers import AutoModelForCausalLM, TrainingArguments, Trainer model_name = \u0026#34;mistralai/Mistral-7B\u0026#34; model = AutoModelForCausalLM.from_pretrained(model_name) training_args = TrainingArguments( output_dir=\u0026#34;./results\u0026#34;, per_device_train_batch_size=2, num_train_epochs=1 ) trainer = Trainer( model=model, args=training_args, train_dataset=meus_dados_treinamento ) trainer.train() Now the model has passed the judgment of Anubis and Osiris and is ready to provide reliable answers.\nMosaicML: The Wings of Isis for Productization Just having a trained model isn\u0026rsquo;t enough. It has to be fast and efficient. MosaicML is like the wings of Isis, letting the LLM run without consuming absurd amounts of computing power.\nWith MosaicML on Databricks, we can:\nCut inference costs with optimizations such as quantization and pruning. Speed up response time without compromising quality. Scale the model to thousands of users without freezing up. In Databricks, the optimized model can be saved and deployed with MLflow:\nimport mlflow mlflow.set_experiment(\u0026#34;/Users/seu_usuario/RAG_experiment\u0026#34;) with mlflow.start_run(): mlflow.transformers.log_model(model, artifact_path=\u0026#34;llm_produtizado\u0026#34;) Now we have an LLM that not only answers well, but does it with speed and efficiency.\nCreating a God of Artificial Intelligence Throughout Egyptian mythology, the gods didn\u0026rsquo;t act alone. Each one had an essential role in keeping the world running in balance. To create a truly efficient LLM, we need to combine these divine forces: the light of knowledge, access to infinite wisdom, careful evaluation of answers and the speed to operate effectively.\nJust as Egyptian priests combined different deities to solve the challenges of the ancient world, we bring together RAG, Fine-Tuning and MosaicML to build a robust, reliable and agile model. Below is the full analogy:\n_Comparison of mythological elements_\nConclusion Building an LLM on Databricks with RAG, Fine-Tuning and MosaicML isn\u0026rsquo;t just a technical challenge; it\u0026rsquo;s a way to take artificial intelligence to a new level of usefulness and efficiency.\nImagine the impact of a system that doesn\u0026rsquo;t just answer questions, but looks up the right information, learns from the company and runs in an optimized way. This kind of solution can transform companies by cutting operating costs, improving the user experience and increasing the reliability of the information provided.\nIf in Ancient Egypt knowledge was passed down through papyri and temples, today we have tools like Databricks and MosaicML to process billions of data points in seconds. The principle, however, remains the same: knowledge must be accurate, reliable and efficient.\nAnd so, by combining these technologies, we create a true god of artificial intelligence, able to bring light, wisdom, justice and speed to the age of AI.\n","permalink":"https://julianamarialop.github.io/en/articles/egyptian-gods-and-ai-build-an-llm-on-azure-databricks-rag-fine-tuning-mosaicml/","summary":"If the ancient Egyptians could build an artificial intelligence, they would probably call on their gods to make the system wiser, faster and more efficient. Just as Ra lit up the world, Thoth recorded…","title":"The Egyptian Gods and Artificial Intelligence: How to Build an LLM on Azure Databricks with RAG, Fine-Tuning and MosaicML"},{"content":"These days, data-driven decision-making is at the heart of every company. However, the effectiveness of those decisions depends directly on the quality of the data used. Inconsistent, incomplete, or incorrect data can compromise analyses, produce misleading insights and, in the worst-case scenario, lead to disastrous decisions.\nThis is where Databricks DQX comes in: a library built to ensure data quality in pipelines created in Databricks , using custom, automated rules to validate, monitor, and fix quality issues before they contaminate more sensitive layers. If we picture the data pipeline as a multiverse of parallel dimensions that has to be protected from anomalies, DQX is like Doctor Strange, using his powers to keep order and make sure only quality data moves forward through the right dimensions.\nWhat Is Databricks DQX? Databricks DQX is a data quality management tool designed to be used in Databricks pipelines built on Apache Spark. It lets you define validation rules for DataFrames and streams, providing a programmatic approach to checking data integrity, completeness, and consistency.\nDQX supports different types of rules, such as field format validation, null values, duplicate values, and much more. These rules can be configured to raise alerts or to completely block invalid data from propagating. On top of that, it offers detailed reports and logs that help teams understand and fix problems quickly.\nWhy Is Data Quality Critical in Analytics Projects? In a data project, the layers of a Data Lake are usually split into bronze, silver, and gold:\nBronze Layer: Where raw data is stored without any processing or validation. Silver Layer: Where data goes through cleansing and standardization. Gold Layer: Where structured, high-quality data is consumed by BI tools and analytical models. If invalid data passes from the bronze layer to the silver layer, it can contaminate the entire value chain, hurting the efficiency of downstream analyses and compromising the performance of machine learning models. Low-quality data can introduce bias, produce inaccurate predictions, and lead to wrong decisions. That makes it essential to adopt good validation and cleansing practices right from the early stages of the pipeline., it can contaminate the entire chain, causing serious problems further down the line. So applying strict quality rules between these layers is fundamental. The earlier quality problems are identified and fixed, the lower the cost of rework, keeping errors from spreading into the most valuable layers, such as the gold layer. This also improves the efficiency of analytical processes, making sure decisions are always based on consistent, reliable data.\nHow to Use DQX Between the Bronze and Silver Layers The image below shows how DQX fits into the data pipeline in a Data Lakehouse. The process starts with the bronze layer, where raw data is stored. DQX runs a data profiling process, analyzing the data and generating candidate quality rules. Next, the quality rules are applied, separating valid data from invalid data. Data that fails validation is sent to a quarantine dataset, where it can be monitored and reviewed. Corrected or enriched data can be reintegrated into the pipeline after curation, making sure only consistent information moves on to the silver layer. In the gold layer, the data is ready to be consumed by BI tools and analytical models.\n_DQX Quality Checking_\nLet\u0026rsquo;s imagine we\u0026rsquo;re dealing with financial data. In the bronze layer, we have bank transaction records, and we need to make sure only valid transactions move on to the silver layer.\nLet\u0026rsquo;s imagine we\u0026rsquo;re dealing with financial data. In the bronze layer, we have bank transaction records, and we need to make sure only valid transactions move on to the silver layer.\nThe first step is to define the quality rules we want to apply. Some common rules might include:\nRequired fields cannot be null. The transaction amount must be positive. The date format must be consistent. With DQX, you can create rules like this:\n_Example notebook using DQX_\nThis process ensures that only records that meet every rule are promoted to the silver layer. Invalid data can be stored separately for auditing and correction.\nCheck out the full notebook here!\nHow to Install and Configure Databricks DQX Installing DQX is simple and can be done directly with pip:\npip install databricks-labs-dqx Once it\u0026rsquo;s installed, you can import the library and start defining your quality rules, as shown in the previous example.\nFor more details, you can check the official documentation on GitHub.\nConclusion Databricks DQX is much more than a simple data validation tool: it\u0026rsquo;s the true master of quality, making sure only consistent data moves through the layers of the Data Lake. Just as Doctor Strange protects the multiverse from interdimensional threats, DQX protects the data pipeline, keeping incorrect information from contaminating your analyses.\nIf you want to build robust pipelines and make sure your decisions are always based on high-quality data, DQX is the ideal solution. After all, as Doctor Strange might say: \u0026ldquo;This data line is protected\u0026rdquo;: only quality data may cross.\n","permalink":"https://julianamarialop.github.io/en/articles/azure-databricks-dqx-the-doctor-strange-of-your-data-pipeline/","summary":"These days, data-driven decision-making is at the heart of every company. However, the effectiveness of those decisions depends directly on the quality of the data used. Inconsistent, incomplete, or…","title":"Azure Databricks DQX: The Doctor Strange of Your Data Pipeline, Protecting the Multiverse of Quality!"},{"content":"Introduction - Smart Cooperation: The Foundation of Databricks Clusters Databricks is a robust platform for processing and analyzing large volumes of data, supporting workloads that range from data engineering to data science and business analytics. One of the main components of Databricks is the cluster, a collection of virtual machines configured to run code in parallel, distributed fashion using Apache Spark.\nA good way to understand how a cluster works is to compare it to a beehive. Just as a hive is made up of hundreds or thousands of bees working together to keep it productive and organized, a cluster is made up of several nodes (virtual machines) that cooperate to process large volumes of data efficiently.\nTraditionally, with All-Purpose Compute, users could create custom clusters by configuring aspects such as the Spark version, the node types (driver and workers), and the number of nodes in the cluster. It was also possible to install additional libraries to support specific workloads, which offered a lot of flexibility. This method was widely used as the default for a long time.\nHowever, Databricks has evolved and now offers new possibilities, including the serverless compute option (Serverless Compute), which brings a more automated and scalable approach to handling varied demands. Just as worker bees can quickly adapt to the needs of the hive, Databricks adapts to workloads with different types of clusters.\nTypes of Clusters in Databricks All-Purpose Compute Clusters Jobs Clusters Serverless Clusters High Concurrency Clusters Photon Clusters A Closer Look at Each Cluster Type 1. All-Purpose Compute Clusters General-purpose (All-Purpose) clusters are designed to be highly configurable, letting users define details such as:\nApache Spark version Instance type (driver and worker nodes) Number of nodes in the cluster In this setup, the driver node acts as the queen bee, coordinating all the activity in the hive, while the worker nodes are the worker bees that carry out the data processing tasks. Efficient communication between them ensures that data is processed quickly and accurately.\n_All-Purpose Clusters_\nAdvantages:\nFull flexibility to customize the environment as needed. Allows installing additional libraries to support specific workloads. Recommended use cases:\nData development and exploration. Running interactive notebooks. Work that requires a highly customized environment. 2. Jobs Clusters Jobs Clusters are created automatically to run a specific task (\u0026ldquo;job\u0026rdquo;) and are terminated as soon as the task finishes.\nJust as some bees can be sent on specific missions, like collecting nectar from a distant source, job clusters are created to carry out a one-off task and are then shut down, saving resources.\n_Job Clusters_\nAdvantages:\nLower costs thanks to automatic creation and termination. Quick, simple setup for running scheduled workloads. Recommended use cases:\nRunning scheduled ETL pipelines. Batch data processing. Running recurring tasks. 3. Serverless Clusters Serverless Clusters remove the need for manual configuration, scaling automatically according to the workload.\nJust like in a hive where the number of worker bees can vary depending on the season or the hive\u0026rsquo;s needs, serverless clusters adjust automatically to demand, ensuring efficiency and avoiding wasted resources.\n_Serverless Compute_\nAdvantages:\nAutomatic scaling and simplified management. Shorter startup times, improving productivity. Ideal for intermittent workloads, since you pay only for actual usage. The warehouses used in these clusters can be configured in different sizes, such as Small, Medium, Large, and X-Large, depending on the processing demand. In addition, autoscaling lets the system dynamically adjust the allocated resources to the workload, ensuring optimal performance and cost control.\n_Cluster sizes for SQL Warehouses_\nRecommended use cases:\nRunning ad hoc workloads. SQL queries and data processing without complex configuration. Real-time data analysis. 4. High Concurrency Clusters High Concurrency Clusters are designed to serve multiple users at the same time, supporting the parallel execution of multiple tasks.\nJust as different bees work simultaneously on various tasks inside the hive, high concurrency clusters let multiple users run their code at the same time without compromising performance.\nAdvantages:\nSupports multiple concurrent users without significant performance degradation. Ideal for collaborative environments. Recommended use cases:\nCollaborative analysis. Interactive dashboards with many concurrent users. Business Intelligence platforms integrated with Databricks. 5. Photon Clusters Photon is a next-generation execution engine designed to significantly improve task performance, especially in typical data warehouse workloads. It is fully compatible with the Apache Spark DataFrame and SQL APIs and has been optimized for operations such as:\nUPDATE DELETE MERGE INSERT CREATE TABLE AS Advantages:\nSignificant reduction in task execution time. Lower operating cost thanks to improved efficiency. Ideal for workloads involving frequent changes to large volumes of data. Recommended use cases:\nLarge-scale data processing. Data warehouse workloads that demand high performance. Pipelines that run complex data modification operations. The Cost of Each Cluster Type All-Purpose Compute Clusters: They have high costs because the cluster has to stay running during development. They\u0026rsquo;re recommended only for interactive tasks and development. Jobs Clusters: More economical, since they\u0026rsquo;re created and terminated automatically. The cost is proportional to the tasks\u0026rsquo; run time. Serverless Clusters: The cost is based on actual usage, making them a highly efficient option for sporadic workloads. High Concurrency Clusters: They have an intermediate cost, suited to collaborative environments with many concurrent users. Photon Clusters: Although their costs are similar to All-Purpose Clusters, Photon\u0026rsquo;s improved efficiency can lead to a significant reduction in overall costs, especially for intensive workloads. Cost Monitoring To keep costs under control in Databricks, especially when using serverless compute, you can monitor usage by querying the system tables (system.billing.usage), which contain detailed information about users and workloads related to costs.\nAlternatively, you can import a billing dashboard directly into the account console. With these tools, you can track serverless costs and usage, getting detailed reports that provide information about:\nSQL costs DLT (Delta Live Tables) costs Job costs Machine learning model costs These reports give you a comprehensive, detailed view, helping you optimize resource usage and cut unnecessary spending. Below is an example query for this kind of monitoring:\nSELECIONE t1.workspace_id, SOMA(t1.usage_quantity * list_prices.pricing. default ) como list_cost DE system.billing.usage t1 INNER JOIN system.billing.list_prices em t1.cloud = list_prices.cloud e t1.sku_name = list_prices.sku_name e t1.usage_start_time \u0026gt;= list_prices.price_start_time e (t1.usage_end_time \u0026lt;= list_prices.price_end_time ou list_prices.price_end_time é nulo) ONDE t1.sku_name COMO \u0026#39;%SERVERLESS%\u0026#39; AGRUPAR POR t1.workspace_id Conclusion - Efficiency Inspired by Nature: Databricks Clusters in Action Just like in a beehive, where every member plays a key role in the smooth functioning of the whole, a Databricks cluster is made up of several nodes working together to process large volumes of data efficiently. Databricks offers a variety of cluster options to serve different types of workloads, from development and pipeline execution to collaborative analysis and ad hoc processing. Each cluster type has its own advantages and suits specific scenarios, letting organizations choose the best solution for their needs.\nIn addition, the ability to monitor costs and usage in detail ensures greater financial and operational control. With these features, Databricks stands out as a scalable, efficient solution for data projects across many market segments.\n","permalink":"https://julianamarialop.github.io/en/articles/like-a-well-organized-beehive-exploring-azure-databricks-clusters/","summary":"Databricks is a robust platform for processing and analyzing large volumes of data, supporting workloads that range from data engineering to data science and business analytics. One of the…","title":"Like a Well-Organized Beehive: Exploring Azure Databricks Clusters"},{"content":"Before the Battle: Meet the Data Heroes In recent years, the need for robust platforms to analyze large volumes of data has become clear for companies looking to become more data-driven. Two of the most talked-about solutions today are Databricks and WatsonX.data from IBM , each offering a distinct approach to meeting these demands.\nIn this analysis, we\u0026rsquo;ll use a superhero analogy to illustrate the differences between these tools. We\u0026rsquo;ll compare Databricks to Iron Man and WatsonX.data to Captain America.\nDatabricks - The Iron Man of Data Platforms Like Tony Stark, Databricks is a highly technological and innovative solution. Founded in 2013 by the creators of Apache Spark, Databricks has established itself as a leader in data lakes and machine learning. Its proposition is to offer a unified environment for data engineering, data science, and data analytics, always with cutting-edge technology and constant updates. Its big differentiator is the implementation of the \u0026ldquo;Lakehouse\u0026rdquo; concept, which combines the advantages of data lakes and data warehouses.\nJust like Iron Man\u0026rsquo;s armor, Databricks is constantly evolving, offering new features and integrations that make it extremely versatile for different kinds of missions.\nWatsonX.data - The Captain America of Data Analytics Part of IBM\u0026rsquo;s WatsonX portfolio, it represents the classic, reliable solution, just like Steve Rogers. Launched in 2023, this solution was designed to provide a robust, secure platform with a focus on governance and compliance. IBM has always been known for secure, stable solutions, and WatsonX.data carries on that tradition, ensuring companies can fully trust the platform to handle sensitive data.\nJust as Captain America leads by example and through steadiness, WatsonX offers a solid, stable foundation for companies that prioritize security and governance.\nComparison Criteria To make things easier to follow, let\u0026rsquo;s break this comparison down into the topics below:\nArchitecture and Integration Performance and Scalability Governance and Security Ease of Use and Development Ecosystem and Tools Total Cost of Ownership (TCO) 1. Architecture and Integration Databricks (Iron Man): Databricks stands out for its Lakehouse architecture, which lets you work with structured and unstructured data in a single repository. It integrates natively with several cloud solutions, such as Azure, AWS, and Google Cloud, giving you flexibility in choosing a cloud provider. Just as Iron Man connects to all kinds of technology, Databricks offers robust integration with modern tools.\nPros:\nUnified architecture for transactional and analytical data. Strong integration with machine learning tools. Broad support for distributed workloads. Cons:\nInitial setup can be complex for beginners. Costs can scale quickly depending on data volume. WatsonX.data (Captain America): Its architecture is geared toward companies that already use other IBM solutions, offering strong integration with the Watson ecosystem and other enterprise tools. The platform also has native support for workloads in hybrid and multi-cloud environments. Just as Captain America is known for his loyalty and steadiness, this tool stands out for secure, reliable integration.\nPros:\nStrong integration with IBM legacy solutions. Optimized for workloads in hybrid environments. Focus on enterprise security. Cons:\nLess support for third-party tools compared to Databricks. May require a larger upfront investment for companies unfamiliar with the IBM ecosystem. 2. Performance and Scalability Databricks (Iron Man): Databricks uses the Apache Spark engine for distributed processing, delivering high performance on analytical and machine learning workloads. The platform also offers optimizations such as Delta Lake, which improves the performance of transactional queries.\nPros:\nHigh performance in distributed processing. Support for large data volumes without losing efficiency. Cons:\nRequires manual optimization for more complex workloads. WatsonX.data (Captain America): It was designed to deliver high performance on complex analytical queries, with optimizations specific to workloads in hybrid environments. Like Captain America, who stands out for his consistency and endurance, it delivers consistent performance.\nPros:\nConsistent performance in hybrid environments. Native support for AI workloads. Cons:\nMay not be as efficient as Databricks for machine learning workloads. 3. Governance and Security Databricks (Iron Man): Databricks offers robust governance features through Unity Catalog, which enables centralized metadata management and role-based access control.\nPros:\nUnity Catalog makes data governance at scale easier. Strong support for GDPR and other regulations. Cons:\nMay require additional configuration for highly regulated environments. WatsonX.data (Captain America): IBM has always been known for its focus on enterprise security, and it continues that tradition. The platform offers end-to-end encryption and fine-grained access control.\nPros:\nA highly secure solution. Native support for enterprise compliance standards. Cons:\nCan be complex to configure in non-IBM environments. 4. Ease of Use and Development Databricks (Iron Man): Databricks offers a friendly, highly collaborative interface that lets teams of data engineers, data scientists, and analysts work together efficiently. Support for interactive notebooks makes development more intuitive.\nPros:\nCollaborative interface with notebook support. Extensive documentation and an active community. Cons:\nMay involve an initial learning curve, especially for people without Spark experience. WatsonX.data (Captain America): It offers an interface focused on ease of use for companies that already use other IBM solutions. In addition, native integration with Watson tools makes it easier to develop AI solutions.\nPros:\nAn intuitive interface for users of the IBM ecosystem. Strong support for AI solutions. Cons:\nLess flexibility for customization compared to Databricks. 5. Ecosystem and Tools Databricks (Iron Man): Databricks has an extensive ecosystem, with support for multiple programming languages (Python, R, Scala, SQL) and integration with a wide range of third-party tools, such as Power BI, Tableau, and Looker.\nPros:\nSupport for multiple programming languages. Robust integration with BI tools. Cons:\nSome integrations may require additional configuration. WatsonX.data (Captain America): WatsonX.data benefits from the IBM ecosystem, offering native integration with solutions like Watson AI, SPSS, and Cognos Analytics.\nPros:\nA well-established ecosystem for enterprise solutions. Strong integration with predictive analytics tools. Cons:\nA smaller variety of supported tools compared to Databricks. 6. Total Cost of Ownership (TCO) Databricks (Iron Man): Databricks\u0026rsquo; total cost of ownership can vary significantly depending on usage and scale. Although the platform offers great flexibility, costs can climb quickly with complex workloads.\nPros:\nA usage-based cost model that allows for scalability. Flexibility to adjust resources as demand changes. Cons:\nCan be expensive for continuous, high-scale workloads. WatsonX.data (Captain America): It has a licensing model that favors companies already using other IBM solutions. Costs can be more predictable for those companies, but less competitive for new customers.\nPros:\nA predictable licensing model. Benefits for existing IBM customers. Cons:\nMay be less competitive on upfront cost for new customers. Choose the Right Hero for Your Mission Databricks and WatsonX.data are powerful solutions for large-scale data analytics, each with its own strengths and weaknesses. Databricks, like Iron Man, shines through innovation, flexibility, and the ability to adapt quickly. WatsonX.data, like Captain America, stands out for stability, security, and reliability in enterprise environments. The ideal choice will depend on each organization\u0026rsquo;s specific needs.\n","permalink":"https://julianamarialop.github.io/en/articles/azure-databricks-and-watsonx-data-in-the-heroes-arena/","summary":"In recent years, the need for robust platforms to analyze large volumes of data has become clear for companies looking to become more data-driven. Two of the most talked-about solutions…","title":"Azure Databricks and WatsonX.data in the Heroes' Arena: Who Comes Out on Top?"},{"content":"After writing my previous article on migrating from SAS to Databricks as a cake recipe, I enjoyed the experience so much that I decided to repeat the idea, but this time with COBOL! Since COBOL is a language as old as those family recipes handed down from generation to generation, I thought it would be more fun to make a pavê recipe (pavê is a classic Brazilian layered dessert, a cousin of the trifle). After all, COBOL and pavê are classics that stand the test of time!\nHave you ever heard of COBOL? If you haven\u0026rsquo;t, don\u0026rsquo;t worry, I\u0026rsquo;ll explain! COBOL (\u0026ldquo;Common Business-Oriented Language\u0026rdquo;) is one of the oldest programming languages and is still very important in the corporate world. Created in 1959, it was designed to be simple and efficient for business solutions, and it ended up becoming the main choice for large-scale systems, especially in sectors such as banking, insurance and government.\nDespite its age, COBOL still runs many critical systems around the world. Think of systems that process bank transactions, payments and financial records: there is a good chance a COBOL program is behind them. But over time, companies need to modernize these solutions to keep up with new technologies and demands.\nIn this article, I\u0026rsquo;ll show you how to modernize COBOL processes in a really fun format: a pavê recipe! We\u0026rsquo;ll go step by step, including a hands-on JCL example and how to use Databricks to give all of it a modern touch. Shall we make this pavê?\n🍰 Ingredients for our recipe: Survey of the existing COBOL processes: Essential tools: Specialized team: 👩🍳 Directions: Step 1 - Understanding and preparing the environment Before modernizing, it\u0026rsquo;s important to understand the current process well.\nGather the input and output files: Understand the JCL flow: Check the COBOL program: 👩🍳 Directions: Step 2 - Hands-on JCL example with PCICDIT and VSAM Now let\u0026rsquo;s make the pavê with a hands-on example.\nJCL example: //JOBNAME JOB (ACCT),\u0026#39;COBOL MIGRATION\u0026#39;,CLASS=A,MSGCLASS=X //STEP1 EXEC PGM=PCICDIT //INFILE DD DSN=INPUT.VSAM.FILE,DISP=SHR //OUTFILE DD DSN=OUTPUT.FLAT.FILE,DISP=(NEW,CATLG,DELETE), // SPACE=(CYL,(5,5),RLSE), // DCB=(RECFM=FB,LRECL=80,BLKSIZE=800) //SYSPRINT DD SYSOUT=* //SYSIN DD * COMMAND=COPY INPUT=INFILE OUTPUT=OUTFILE /* JCL explanation: //JOBNAME JOB: Defines the job name, the account, and the execution and message classes. //STEP1 EXEC PGM=PCICDIT: Runs the PCICDIT program. //INFILE DD DSN=INPUT.VSAM.FILE: Specifies the input file, which is a VSAM file. //OUTFILE DD DSN=OUTPUT.FLAT.FILE: Defines the output file as a flat file, with space and record format specifications. //SYSPRINT DD SYSOUT=*: Sends the log output to the console. //SYSIN DD: Holds the PCICDIT commands, which specify that the input file will be copied to the output file. 👩🍳 Directions: Step 3 - Converting the VSAM file with Databricks With the VSAM file data already identified, we can create a Databricks notebook to convert that data into a modern format such as Delta Lake. Below is an example of PySpark code that performs this conversion:\nPlain PySpark code example: from pyspark.sql import SparkSession # Criando a sessão Spark spark = SparkSession.builder.appName(\u0026#34;VSAM_to_Delta\u0026#34;).getOrCreate() # Lendo o arquivo VSAM (simulado como um arquivo sequencial) vsam_df = spark.read.format(\u0026#34;csv\u0026#34;).option(\u0026#34;header\u0026#34;, \u0026#34;false\u0026#34;).load(\u0026#34;/mnt/input/vsam-file.txt\u0026#34;) # Definindo o esquema do arquivo VSAM vsam_df = vsam_df.withColumnRenamed(\u0026#34;_c0\u0026#34;, \u0026#34;key\u0026#34;).withColumnRenamed(\u0026#34;_c1\u0026#34;, \u0026#34;value\u0026#34;) # Gravando no formato Delta vsam_df.write.format(\u0026#34;delta\u0026#34;).mode(\u0026#34;overwrite\u0026#34;).save(\u0026#34;/mnt/output/delta-vsam\u0026#34;) print(\u0026#34;Conversão concluída com sucesso!\u0026#34;) Code explanation: We create a SparkSession: it is required to use Spark\u0026rsquo;s features. Reading the VSAM file: here we take a simulated text file as input. We define the DataFrame schema: renaming the columns to reflect the VSAM structure. Writing in Delta format: the file is saved to a directory in Delta Lake format. Using Databricks Assistant: You can use Databricks Assistant to automatically generate a notebook similar to this one. Just give it a clear prompt, such as:\nSuggested prompt: \u0026ldquo;Create a PySpark notebook to read a VSAM file, convert it to a DataFrame and write it in Delta format.\u0026rdquo;\nDatabricks Assistant will understand the request and generate base code that you can adjust to your needs.\nI have a gift for you: Download the notebook I created here!\nConclusion By following this pavê recipe, you\u0026rsquo;ll be able to modernize your COBOL processes in a structured way, ensuring business continuity and paving the way for new technology solutions. What about you, have you ever taken part in a modernization like this? Share your experiences in the comments!\nDid you like this content? Leave a like and share it with someone who may be facing this challenge!\n#COBOL #JCL #VSAM #Mainframe #Modernization #DigitalTransformation #DataEngineering\n","permalink":"https://julianamarialop.github.io/en/articles/modernize-cobol-processes-azure-databricks-pave-recipe-jcl-vsam/","summary":"After writing my previous article on migrating from SAS to Databricks as a cake recipe, I enjoyed the experience so much that I decided to repeat the idea, but this time with COBOL! Since COBOL is a language…","title":"How to Modernize COBOL Processes with Azure Databricks: A Pavê Recipe with a JCL Example Using PCICDIT and a VSAM Input File"},{"content":"Migrating a well-established platform like SAS to Databricks can be challenging, but when we follow a well-structured recipe, everything becomes simpler and more efficient. In this recipe, we\u0026rsquo;ll split the process into two main steps: table migration and SAS project migration. And, as a grand finale, there\u0026rsquo;s a special gift at the end to make your journey even easier!\n🍰 Ingredients for our migration recipe: Understanding the current environment:\nSurvey every table, process and project in SAS that needs to be migrated. Identify the dependencies between tables and projects. The right tools:\nDatabricks (with support for the Delta Lake format). Connectors for the databases in use. The Databricks Assistant (it\u0026rsquo;ll be our sous-chef in this kitchen!). A prepared team:\nSolutions Architect. Data Engineers. Data Scientists. 👩🍳 Directions: Step 1 - Table migration Just like baking a cake, we start with the basic ingredients: the tables. In this step, the goal is to migrate the tables from SAS to Databricks using the Delta format, ensuring scalable and efficient storage.\nSteps:\nSurvey all the tables to be migrated:\nList the existing tables in SAS and identify their characteristics (schema, data volume, update frequency).\nUse the parameterized notebook:\nTo make the migration easier, I prepared a special notebook that automatically migrates tables from SAS to the Delta format in Databricks.\nThis notebook is fully parameterized and lets you easily configure connections, table names and target directories.\nValidate the migration:\nCompare the migrated data with the original data in SAS to ensure consistency and integrity.\n👩🍳 Directions: Step 2 - SAS project migration Now that the tables are ready, it\u0026rsquo;s time to migrate the SAS projects. This step is like adding the frosting to the cake: this is where the ETL processes and SAS code get converted and adapted to Databricks.\nSteps:\nList the SAS projects:\nIdentify every SAS process and piece of code that needs to be migrated. Prioritize the most critical ones.\nUse the Databricks Assistant to convert the code:\nTake the existing SAS code and paste it into the Databricks Assistant.\nAsk the Assistant to convert the code to SQL or PySpark, depending on your needs.\nThis feature lets you quickly turn SAS code into a format Databricks understands, drastically reducing migration time.\nDeploy and test the migrated projects:\nRun the converted code and validate the results.\nMake sure the new processes are producing the same results expected in SAS.\n🎂 Grand Finale: A special gift! To help even more on this migration journey, I prepared a ready-to-use, fully parameterized notebook to migrate SAS tables to the Delta format. This notebook lets you:\nQuickly configure the SAS data source and the destination in Databricks. Run the migration automatically and efficiently. Make sure every table is in Delta format, ready to be used in the new projects. Want the notebook? Just copy the Python code below!\nIf you prefer, download the notebook.\n# Cell 1: Criar widgets de comando para os parâmetros de entrada dbutils.widgets.text(\u0026#34;sas_file_dir\u0026#34;, \u0026#34;/caminho/para/seu/diretorio\u0026#34;, \u0026#34;SAS File Directory\u0026#34;) dbutils.widgets.text(\u0026#34;sas_file_name\u0026#34;, \u0026#34;\u0026#34;, \u0026#34;SAS File Name (optional)\u0026#34;) dbutils.widgets.text(\u0026#34;catalog\u0026#34;, \u0026#34;seu_catalogo\u0026#34;, \u0026#34;Catalog\u0026#34;) dbutils.widgets.text(\u0026#34;schema\u0026#34;, \u0026#34;seu_schema\u0026#34;, \u0026#34;Schema\u0026#34;) dbutils.widgets.text(\u0026#34;table_name\u0026#34;, \u0026#34;\u0026#34;, \u0026#34;Table Name (optional)\u0026#34;) # Cell 2: Obter os valores dos widgets sas_file_dir = dbutils.widgets.get(\u0026#34;sas_file_dir\u0026#34;) sas_file_name = dbutils.widgets.get(\u0026#34;sas_file_name\u0026#34;) catalog = dbutils.widgets.get(\u0026#34;catalog\u0026#34;) schema = dbutils.widgets.get(\u0026#34;schema\u0026#34;) table_name = dbutils.widgets.get(\u0026#34;table_name\u0026#34;) # Função para carregar e salvar um arquivo SAS como tabela Delta def load_and_save_sas_file(sas_file_path, catalog, schema, table_name): # Carregar o arquivo SAS em um DataFrame df = spark.read.format(\u0026#34;com.github.saurfang.sas.spark\u0026#34;) \\ .load(sas_file_path) # Exibir o DataFrame carregado display(df) # Especificar o caminho Delta delta_path = f\u0026#34;/mnt/delta/{catalog}/{schema}/{table_name}\u0026#34; # Escrever o DataFrame como uma tabela Delta df.write.format(\u0026#34;delta\u0026#34;) \\ .mode(\u0026#34;overwrite\u0026#34;) \\ .save(delta_path) # Criar a tabela Delta no catálogo especificado spark.sql(f\u0026#34;\u0026#34;\u0026#34; CREATE TABLE {catalog}.{schema}.{table_name} USING DELTA LOCATION \u0026#39;{delta_path}\u0026#39; \u0026#34;\u0026#34;\u0026#34;) # Verificar se um nome de arquivo específico foi fornecido if sas_file_name: # Construir o caminho completo do arquivo SAS sas_file_path = f\u0026#34;{sas_file_dir}/{sas_file_name}\u0026#34; # Usar o nome da tabela fornecido ou derivar do nome do arquivo table_name = table_name if table_name else sas_file_name.split(\u0026#34;.\u0026#34;)[0] # Carregar e salvar o arquivo SAS load_and_save_sas_file(sas_file_path, catalog, schema, table_name) else: # Listar todos os arquivos SAS no diretório files = dbutils.fs.ls(sas_file_dir) sas_files = [f.path for f in files if f.path.endswith(\u0026#34;.sas7bdat\u0026#34;)] # Carregar e salvar cada arquivo SAS for sas_file_path in sas_files: # Derivar o nome da tabela do nome do arquivo table_name = sas_file_path.split(\u0026#34;/\u0026#34;)[-1].split(\u0026#34;.\u0026#34;)[0] load_and_save_sas_file(sas_file_path, catalog, schema, table_name) Conclusion By following this recipe, you\u0026rsquo;ll have a smooth and efficient migration, with a modern, scalable platform ready to meet your business\u0026rsquo;s new demands. Remember: the secret to a good migration is planning each step carefully and having the right tools at hand.\nWhat about you, have you taken part in a migration like this? What were the main challenges? Share in the comments! Let\u0026rsquo;s swap experiences!\n","permalink":"https://julianamarialop.github.io/en/articles/how-to-migrate-from-sas-to-azure-databricks-a-foolproof-two-step-recipe/","summary":"Migrating a well-established platform like SAS to Databricks can be challenging, but when we follow a well-structured recipe, everything becomes simpler and more efficient. In this recipe, we\u0026rsquo;ll split the process into…","title":"How to migrate from SAS to Azure Databricks: A foolproof two-step recipe"},{"content":"How about comparing the mistakes that sabotage your data pipelines to the villains who keep getting in the way of the superheroes\u0026rsquo; mission? Picture your pipeline as the hero of the story, tasked with carrying the data to its final destination, where it will turn into valuable insights that save the day. But, as in every good story, there are villains along the way ready to derail the mission. Let\u0026rsquo;s meet the 5 villains and learn how to defeat them with your secret weapons: good practices and the Databricks platform.\nVillain 1: Ingestion Chaos, the Master of Disorder Villain profile: This villain shows up when you have to deal with multiple data sources in varied formats (JSON, CSV, Parquet, XML, etc.) and no standardization. It throws the pipeline into disarray, which makes maintenance hard and raises the chance of errors during data ingestion and transformation.\nSigns it\u0026rsquo;s attacking: Inconsistent data ingestion across different sources. Heavy manual effort to handle data in different formats. Frequent reprocessing due to failures in the ingestion pipeline. How to beat it in Databricks: Adopt the Medallion Architecture: Bronze Layer: store the raw data exactly as it was received. Silver Layer: normalize the data, standardizing formats, fixing inconsistencies and deduplicating. Gold Layer: apply the final transformations, producing data ready for analytical consumption. Use Delta Lake for reliable ingestion: Use Delta Tables to store each data layer with support for versioning and transactional control. This guarantees consistency, even when failures happen. Automate ingestion with Auto Loader: Databricks Auto Loader automatically detects new files and ingests them incrementally, reducing the need for manual intervention. Villain 2: The Silent One, the Invisible Saboteur Villain profile: The Silent One works unnoticed. It causes pipeline failures or slow processing without you noticing in time to fix them. As a result, reports end up stale or inconsistent, and nobody knows why.\nSigns it\u0026rsquo;s attacking: Jobs that fail without alerts. Delayed reports due to unidentified failures. No visibility into pipeline status. How to beat it in Databricks: Implement job monitoring: Set up Databricks Job Monitoring to track the execution of every job. It gives you a detailed view of runs, processing times and failures. Configure automatic alerts: Use Databricks Alerts to send email or Slack notifications whenever a failure occurs or when a job\u0026rsquo;s run time exceeds a defined threshold. Use monitoring dashboards: Build dashboards in Databricks that show critical metrics, such as the average run time of each pipeline stage, job status and recent failures. Villain 3: The Complex Trickster, the Lord of Giant Tasks Villain profile: This villain loves turning simple tasks into giant blocks of code. It creates monolithic, complex scripts that mix several transformations into a single step, making the pipeline hard to debug and maintain.\nSigns it\u0026rsquo;s attacking: Pipelines that are hard to understand and maintain. Any failure requires reprocessing the whole pipeline. Difficulty debugging errors because of code complexity. How to beat it in Databricks: Break transformations into smaller steps: Create modular notebooks, where each notebook handles a specific part of the pipeline. This makes the code more organized and easier to maintain. Use Delta Tables to save intermediate states: After each transformation step, save the data to an intermediate Delta Table. If a failure occurs, you can restart the pipeline from the last successful step instead of reprocessing everything. Implement Checkpoints: Use Spark Checkpoints to save the processing state while long-running jobs execute. This improves pipeline resilience and makes recovery easier when failures happen. Villain 4: The Inefficient Giant, the Growth Monster Villain profile: This villain is the terror of scalability. It shows up when data volume grows but your pipeline wasn\u0026rsquo;t designed to handle that growth. The result? Extreme slowness, frequent failures and high infrastructure costs.\nSigns it\u0026rsquo;s attacking: Exponential growth in pipeline run times. Very high infrastructure costs. Frequent failures in pipelines that handle large data volumes. How to beat it in Databricks: Enable Auto Scaling: Configure clusters with Auto Scaling, letting Databricks automatically adjust the number of nodes according to demand. This ensures efficiency and saves resources. Optimize reads and writes with efficient formats: Use optimized storage formats such as Parquet and Delta Lake, which allow faster reads and writes on large data volumes. Partition your data: When working with large datasets, always use proper partitioning to avoid reading unnecessary data. Villain 5: Mutant Data, the Destabilizer Villain profile: This villain loves corrupting data along the pipeline, introducing inconsistent or null values, or values outside the expected standard. It makes reports and predictive models deliver wrong results.\nSigns it\u0026rsquo;s attacking: Inconsistent data in reports. Machine learning models with low accuracy due to poor-quality data. Constant rework to fix errors after processing. How to beat it in Databricks: Define quality rules with Delta Expectations: Use Delta Expectations to create validation rules, such as required fields and value limits. If the data doesn\u0026rsquo;t meet the criteria, Databricks flags the error before it moves on to the next step. Implement automated data quality tests: Automate data validation with libraries such as Deequ or testing frameworks integrated with Databricks. Set up tests that check formats, ranges and critical values. Run periodic audits: Schedule audit jobs that review the quality of data already processed and stored, ensuring historical data stays consistent. Conclusion: Assemble Your League of Heroes and Rule the Data Game With these villains defeated, your pipelines will be ready to deliver reliable, high-quality data that drives strategic decisions. Using Databricks tools and good practices, you don\u0026rsquo;t just defeat the villains, you also turn your pipelines into true superheroes, able to take on any data challenge.\nAre you ready to lead your own league of data engineering superheroes?\nThanks for reading!\n","permalink":"https://julianamarialop.github.io/en/articles/turn-your-pipelines-into-superheroes-5-villains-azure-databricks/","summary":"How about comparing the mistakes that sabotage your data pipelines to the villains who keep getting in the way of the superheroes\u0026rsquo; mission? Picture your pipeline as the hero of the story, tasked with carrying the data to its final destination…","title":"Turn Your Pipelines into Superheroes: The 5 Villains That Sabotage Your Data Journey and How to Beat Them with Azure Databricks"},{"content":"With the rise of cloud data storage, working with data has become far more accessible, in terms of both time and cost. Tools such as Databricks and Microsoft Fabric have helped break down those barriers, offering innovative features such as zero-copy cloning and scalable warehouses that enable rapid prototyping.\nThese reductions in storage and processing costs have made design adjustments less impactful and easier to handle than in the past. That\u0026rsquo;s why many data engineers end up skipping the dimensional modeling phase and going straight to transformations, adjusting the model as needed.\nHowever, this approach only works well at the start of a project. As data is loaded and dependencies are built around the core tables, changing the data model can become so expensive and complex that changes become unfeasible.\nThis situation can be compared to the movie The Lost World: just as in the film, where exploring the \u0026ldquo;lost world\u0026rdquo; brings the value of something forgotten back to light, dimensional modeling is a practice that may seem obsolete but is essential for long-term success. Rediscovering and applying its fundamentals can be the difference between a sustainable project and one full of obstacles.\nEven with storage and compute costs falling steadily, the principle that 20% of effort on the initial design saves 80% of rework later still holds firm. Investing in modeling from the start is a strategy that pays off.\nIn short, no matter how much technology evolves, the fundamentals of dimensional modeling remain indispensable. Like the \u0026ldquo;lost world,\u0026rdquo; these principles are ready to be rediscovered and appreciated. 😊\nPrerequisites Before starting the actual work, it\u0026rsquo;s important to set some guidelines right at the beginning to make sure the project stays on the right track.\nGood practices such as creating clear naming conventions for tables and columns and defining who will be responsible for deliverables are well-known strategies to reduce complexity and keep the project flowing smoothly. However, other, less obvious considerations often end up neglected.\nData modeling, by its nature, is based on patterns: most of the time, there\u0026rsquo;s no need to \u0026ldquo;reinvent the wheel.\u0026rdquo; So, both to make communication easier and to save time, it\u0026rsquo;s essential to be familiar with widely used terms and patterns.\nThat covers everything from the fundamentals (such as facts, dimensions and measures) to more advanced concepts (such as type II SCDs, upserts and degenerate dimensions), plus any specific conventions the team adopts along the way.\nTo keep the key concepts fresh in your mind, it\u0026rsquo;s worth keeping a copy of Ralph Kimball\u0026rsquo;s classic The Data Warehouse Toolkit close at hand. If you prefer, you can also check his website, which offers a handy summary of the main terms and concepts.\nA single source of truth Dimensional modeling, as we\u0026rsquo;ll see later, is a collaborative and iterative process. That means people from different areas of the organization need to work together to create a solution that not only meets business requirements but is also efficient and sustainable in the long run.\nWith so many people involved in the design, changes and adjustments along the way are inevitable. That\u0026rsquo;s why it\u0026rsquo;s essential to use a tool that makes this collaboration easy throughout the project lifecycle, enabling real-time work, dynamic sharing and change tracking, and avoiding outdated documents or knowledge silos.\nIn this article, I\u0026rsquo;ll use SqlDBM, an online modeling tool, to illustrate the process. SqlDBM is flexible enough to start with an initial, \u0026ldquo;whiteboard\u0026rdquo;-style design. Unlike traditional diagramming tools (such as Lucidchart or Visio), SqlDBM also generates well-formatted DDL code, specific to the chosen database, as the model evolves, saving rework.\nWith SqlDBM, the map really does become the territory. 😊\nThe process In this article, we\u0026rsquo;ll explore the sales process of a retail company, walking through the main steps of dimensional modeling.\n1. Choose the business process The first step is to choose a single business process to model. It may seem simple, but this decision deserves attention. Trying to build the whole data warehouse at once is a common mistake: the ideal is to build one business process at a time.\nThe choice should be guided by the company\u0026rsquo;s priorities, considering relevance and urgency relative to other processes. The BI team can help with technical estimates, but business needs should be the deciding factor.\nAt this point, the BI team should meet with business experts and carry out an initial data survey. This helps create a high-level technical proposal, useful for setting the budget and schedule.\n2. Define the grain After selecting the business process, it\u0026rsquo;s time to define the grain, that is, the lowest level of detail at which the data will be analyzed. Once again, this is a decision driven by business goals.\nFor example, at an online retailer, we can track extremely specific details, such as the device ID and operating system for each order. That data may be useful for marketing, but it may not add value to overall sales monitoring.\nIf the company decides that sales should be analyzed \u0026ldquo;daily by product type,\u0026rdquo; that will be the minimum grain. It defines what each row of the fact table represents and should be treated as a commitment to stakeholders, since later changes may require significant rework.\nEven when defining a specific granularity, always collect the source data at the most detailed level possible, following the ELT (extract, load, transform) pattern. That way, if tomorrow the analysis needs more detail, such as \u0026ldquo;hourly by product ID,\u0026rdquo; you\u0026rsquo;ll already have the data you need.\n3. Identify the dimensions With the grain defined, the next step is to identify the dimensions that will support the analysis. Some key dimensions may already have been identified while choosing the business process. Now it\u0026rsquo;s time to dig deeper.\nDuring this step, dimensions shared across different business processes may also emerge, laying the foundation for the company\u0026rsquo;s bus matrix.\nEven if the initial focus is a single process (such as retail sales), the bus matrix is a valuable tool for mapping related processes and highlighting common dimensions. This helps keep the design consistent and scalable for future expansion.\n_Sample bus matrix_\nA dimension is usually represented by a noun, such as store, employee or vehicle. These nouns come with attributes, such as name, description or address, which help enrich and contextualize reports.\nAfter identifying the existing dimensions, we can start sketching a basic whiteboard. This draft will serve as the starting point for adding more detail in the next steps of the process.\n_Identifying dimensions in SqlDBM through logical modeling_\n4. Identify the dimension relationships After identifying the dimensions, the next step is to understand how they connect and interact with one another.\nFor example:\nAre promotions applied at the store level, the product level or both? Is an employee tied to a specific store, or can they work at several? These questions guide the technical design, but the answers depend directly on the knowledge of the business experts. That\u0026rsquo;s why it\u0026rsquo;s essential to keep those experts involved throughout the modeling process, making sure functional questions are cleared up as soon as they arise.\nSo, are promotions applied at the product level? 🤔\n_product-level promotion_\nSo, are promotions applied at the product level? Or do they cover the whole store? 🤔\n_store-level promotion_\nLogical modeling offers a simplified way to visualize the design, allowing rapid prototyping before moving on to physical design. This approach makes the process more accessible, letting non-technical team members take an active part in the discussions.\nIn this example, we\u0026rsquo;ll go with the latter option: applying promotions at the store level. 🎯\n5. Identify the facts The next step combines the dimensions already identified (what, when) with quantitative measures (how much, how many), resulting in the model\u0026rsquo;s facts.\nFor example:\nCustomer A bought X units of product B at a price of Y dollars, served by employee D, at store E, using promotion F, on date G. This combination of dimensions and quantitative measures forms the fact table for the retail sales process. It will be the heart of the analysis, connecting business details to the metrics that really matter.\n_the retail sales business process fact table with associated dimensions_\nWith this fact table, we can answer any business question related to retail sales using a combination of filters and aggregations.\nIn addition, we can expand the diagram to include important technical details, such as primary and foreign keys, making sure the model is functional and efficient.\nFor example, if the company wants to know monthly sales by store for product B, just apply the filters and sum the relevant values in the fact table. Easy, right? 😊\nSELECT store_id, date_trunc(\u0026#39; MONTH \u0026#39;, datekey) como sale_month, sum(price) como Monthly_sales_usd FROM fct_retail_sales WHERE productkey = \u0026#39;B\u0026#39; GROUP BY store_id, datekey This can also be done directly within the project itself, taking advantage of the fact table\u0026rsquo;s capabilities to filter and aggregate the data as needed.\nSetting a very granular level of detail for the fact table allows flexibility: we can always aggregate up to simpler levels, such as monthly or weekly summaries. The opposite, however, isn\u0026rsquo;t possible. For example, if the fact table were built at the monthly level, getting daily detail wouldn\u0026rsquo;t be feasible. This initial choice is crucial to keeping the model flexible.\n6. Deployment With the facts, measures, grains and relationships defined and agreed upon, it\u0026rsquo;s finally time to go to the database and create the physical objects.\nAt this stage, it\u0026rsquo;s essential to make sure every technical and functional detail documented during the modeling exercise is followed, such as:\nColumn lengths; Data types; Table properties. This care ensures the model meets expectations and works perfectly in practice. 🎯\nIf you\u0026rsquo;re following along with SqlDBM, you can generate the required DDL directly from the diagram (remember the idea that \u0026ldquo;the map becomes the territory\u0026rdquo;?).\nIf you choose to use Excel, you can also build formulas to generate the SQL from the details you\u0026rsquo;ve entered.\nWhichever tool you choose, the most important thing is to make sure the design is always aligned and in sync with the technical details. This avoids inconsistencies and ensures a functional, efficient model.\nConclusion After walking through the steps of the dimensional modeling process, it\u0026rsquo;s clear that it\u0026rsquo;s an iterative, collaborative effort, where changes in requirements and design decisions are practically inevitable.\nTo keep the project flowing smoothly, it\u0026rsquo;s essential to use tools and methods that handle those changes well, keeping everyone involved aligned and in sync.\nOnce the instruments and processes are defined, modeling becomes a repeatable, pattern-based exercise. So choose your tools wisely, model and keep iterating.\nThank you for following this article to the end! I hope it helped clarify the dimensional modeling process and how to apply it in practice. If you enjoyed the content, I invite you to explore other articles and materials I share about the world of data and analytics. Feel free to leave your comments, questions or suggestions: your participation is always very welcome! 😊\n","permalink":"https://julianamarialop.github.io/en/articles/the-lost-world-rescuing-lost-art-of-dimensional-modeling-practical-guide/","summary":"With the rise of cloud data storage, working with data has become far more accessible, in terms of both time and cost. Tools such as Databricks and Microsoft Fabric have helped break down those…","title":"The Lost World - Rescuing the Lost Art of Dimensional Modeling: A Practical Guide"},{"content":"Introduction Imagine being able to talk to your data as if you were chatting with a coworker or, better yet, as if you were dealing with the genie of a magic lamp. That\u0026rsquo;s exactly what AI/BI Genie makes possible! Just like the legendary genie who grants wishes, Genie makes life easier for business users, letting them \u0026ldquo;rub the lamp\u0026rdquo; of their data to get answers and insights through natural language, the same way we speak every day.\nGenie is an innovative solution built by Databricks to make data access and analysis magically simple, through natural language interactions. Launched in 2024, AI/BI Genie is part of Databricks\u0026rsquo; advances in generative artificial intelligence (GenAI) applied to data analysis. Like Aladdin\u0026rsquo;s genie, it\u0026rsquo;s here to democratize the use of data, letting anyone in the organization ask complex questions without writing a single line of code.\nThe secret behind Genie is Unity Catalog, which works as the magic \u0026ldquo;map\u0026rdquo; for understanding the structure of your data. That includes everything from where the data comes from (lineage), through documentation, to the history of queries already run. With that information, Genie can answer a wide variety of business questions quickly and reliably.\nJust like in the classic story, the Databricks Genie has no limit on wishes. It combines the simplicity of natural language with the reliability of generative artificial intelligence (GenAI), making access to data and to artificial intelligence more democratic than ever, all under the unified governance of the Databricks Data Intelligence Platform.\nSetting up and using AI/BI Genie Genie spaces are prepared by data analysts to ensure a smoother, more accurate experience. The process starts with selecting specific tables from Unity Catalog, focusing on a data set that makes sense for the organization. Next, the analysts expose the metadata of those tables and add instructions carrying business-specific information, such as business rules and concepts that matter in the user\u0026rsquo;s context. To make things even easier, the analysts also add sample questions and answers, helping business users take their first steps in a guided, safe way.\n_Flow for creating and using Genie spaces_\nIn the image above, you can see the flow for creating and using Genie spaces, divided into three main parts: the data team, the Genie space and the business users. It illustrates the whole cycle, from the data team setting up the space to the end users\u0026rsquo; interactions, highlighting the feedback loop and the use of natural language for questions and answers. This flow shows how Genie is fed by trusted data and practical examples, all designed to deliver insights in a simple, accessible way.\nContinuing the explanation, below are the best practices for creating and using Genie spaces, listed and explained to guide the ideal setup of this analytics environment.\nChoosing the data sets To start using Genie efficiently, it\u0026rsquo;s best to work with a small set of tables (between 5 and 10), with a limited number of columns (fewer than 50), focused on a single topic. But why start this way? Because the more coherent and semantically connected your data set is, the better Genie performs.\nWhat does a coherent, semantically connected data set mean? It means the relationships between tables should reflect real connections or logical associations. For example, the tables should be clearly related, as they are in a real-world database, and have primary key and foreign key constraints, so Genie understands how the tables are connected and can perform joins correctly.\nEvery table and column should contribute to the overall topic of the data set. If there are columns that aren\u0026rsquo;t aligned with the purpose of the set, it\u0026rsquo;s important to create views to clean up the data and make it more coherent and organized.\nIn general, the data used in Genie spaces comes from business-level tables in the gold layer, similar to the data you\u0026rsquo;d use in a BI tool to build dashboards. This is often done with a Kimball-style star schema, a multidimensional model that makes data easier to understand and analyze.\nFinally, make sure every user who interacts with the Genie space has, at a minimum, SELECT privileges on the data being used. Since data access is governed by Unity Catalog, missing privileges will result in error messages for users.\nClear Documentation, Accurate Answers: Setting AI/BI Genie Up for Success For Genie to work effectively, it\u0026rsquo;s essential that your data is well documented and annotated. Documentation is fundamental because it brings business context to the data, making it easier to understand and avoiding misinterpretation, both by users and by Genie itself. Put simply: the clearer and more detailed your annotations, the better the answers Genie can give.\nYou can use artificial intelligence tools to generate part of this documentation automatically, saving time and reducing manual work. Even so, it\u0026rsquo;s important to review and adjust the descriptions so they match the specific use case and your domain knowledge. Well-written documentation doesn\u0026rsquo;t just make things easier to use; it also ensures Genie\u0026rsquo;s answers are more accurate and relevant to your business goals.\nFrom Natural Language to SQL: How to Prepare Genie for Efficient Queries While preparing the Genie space, it\u0026rsquo;s essential to test it to check the quality of the answers and make sure it\u0026rsquo;s delivering the expected results. That includes rephrasing the sample questions and adjusting the instructions until Genie provides the correct, desired answers.\nGenie converts natural language questions into SQL, which means it needs a SQL warehouse to run the queries. For best results, we recommend a serverless SQL warehouse, since it offers faster startup times and intelligent workload management, letting queries be processed quickly and cost-effectively.\nRemember that users who interact with the space need \u0026ldquo;CAN USE\u0026rdquo; access to the designated SQL warehouse, so they can run queries without any problems.\nImproving answer quality To improve the quality of Genie\u0026rsquo;s answers, it\u0026rsquo;s essential to turn all the business context you have about the data into information Genie can use effectively. Today there are three ways to do this:\nInstructions: Add extra semantics in natural language so Genie better understands the concepts and relationships specific to your business. SQL examples: Provide sample SQL queries that guide Genie on how to query your data and get the right information. This helps align the answers with what you expect in terms of business logic. Trusted Assets: Create predefined queries that establish a \u0026ldquo;single version of the truth\u0026rdquo;, ensuring results are consistent and reliable across every interaction. These practices don\u0026rsquo;t just increase answer accuracy; they also keep Genie aligned with business goals, delivering more relevant results to users.\nExample: transfer your business language to Genie Instructions let you provide additional information to help Genie understand your business\u0026rsquo;s specific language. They guide the language model to pick up company jargon or concepts specific to a given domain.\nFor example, at Databricks our fiscal year starts in February, unlike most companies, where it starts in January. So this detail needs to be added as an instruction for Genie, ensuring it aggregates the numbers correctly:\nThe fiscal year starts in February. Another option is to instruct Genie on how to apply filter logic or look up values in columns, especially when some of them are case-sensitive:\nAlways convert strings to lowercase and use the \u0026ldquo;like\u0026rdquo; operator when applying filters. Countries in the \u0026ldquo;country_code\u0026rdquo; column are stored as two characters (e.g., US, AT). You can also use \u0026ldquo;one-shot learning\u0026rdquo; techniques to teach Genie how to handle specific columns and extract information from them:\nThe \u0026ldquo;People_Name\u0026rdquo; column is in the format \u0026ldquo;FirstName MiddleName, LastName\u0026rdquo;. If the column content is \u0026ldquo;Francis Ford, Coppola\u0026rdquo;, then FirstName = Francis, MiddleName = Ford, and LastName = Coppola. Make sure your instructions are clear and to the point. The same goes for the data: the more coherent and simple the instructions, the better the results Genie delivers.\nTrusted Assets: predefined queries for specific questions To better understand the difference between Trusted Assets and SQL Examples, let\u0026rsquo;s use a simple analogy: imagine you\u0026rsquo;re teaching a student to solve multiplication problems.\nFirst, you teach basic multiplication, like 2 * 3, which they usually memorize. Later, you teach tools for solving more complex multiplication, like 123*64. The student tends to memorize recurring multiplications, like 8 * 8 = 64, but also learns the tools to work out more complex ones independently.\nTrusted Assets are user-defined queries that run for specific questions. Genie runs the predefined query and returns the result with a \u0026ldquo;trusted asset\u0026rdquo; label. The association between Trusted Assets and specific questions is made through function comments. In our analogy, this would be like memorizing the result of 8 * 8 without having to redo the calculation.\n1. Create a Trusted Asset Here\u0026rsquo;s another example: suppose you need to convert beer prices from euros (EUR) to US dollars (USD). You can create a predefined query that performs this conversion and associate it with a specific question, ensuring Genie always gives the right answer with the \u0026ldquo;trusted asset\u0026rdquo; seal, without recalculating every time.\nSELECT o.year, o.beer_price, (o.beer_price * e.EUR2USD) AS beer_price_usd FROM main.default.oktoberfest_gold o JOIN main.default.eur_usd_conversion e ON o.year = e.Year WHERE o.year = 2005 These Trusted Assets ensure consistency and accuracy in the answers to recurring questions, saving time and giving users greater reliability.\n2. Register the generalized query (adding the filter as a parameter) as a Function To make the query more flexible and adaptable to different scenarios, you can register it as a SQL function, letting the filters be passed as parameters. This makes it easy to reuse the query in many situations while keeping the result consistent and reliable.\nLet\u0026rsquo;s use the example of converting beer prices from euros to dollars, but now with a filter as a parameter. That way, the user can choose to filter by a specific type of beer, for example:\nCREATE OR REPLACE FUNCTION convert_beer_price_to_usd(beer_filter STRING) RETURNS TABLE\u0026lt;beer_name STRING, price_eur DECIMAL(10,2), price_usd DECIMAL(10,2)\u0026gt; AS SELECT beer_name, price_eur, (price_eur * exchange_rate_usd) AS price_usd FROM beer_prices WHERE beer_name LIKE CONCAT(\u0026#39;%\u0026#39;, beer_filter, \u0026#39;%\u0026#39;); In this example, the convert_beer_price_to_usd function accepts a parameter called beer_filter, which can be used to look up specific beers in the beer_name column. That way, Genie can run the query adjusted to the filter provided by the user, making the process more dynamic and efficient.\nThe Trusted Assets associated with this function will let Genie provide more accurate and faster answers to specific questions about beer prices, keeping the answers consistent.\n3. Add the registered function as a Trusted Asset To make the beer price conversion function a Trusted Asset in Genie, you need to register it as one. This ensures Genie can use the function to answer specific questions quickly and consistently, flagging the result with the \u0026ldquo;Trusted Asset\u0026rdquo; seal.\n4. Retrieve trusted results for questions After adding the beer price conversion function as a Trusted Asset, you can use Genie to retrieve the trusted results when a specific question is asked. This ensures Genie will use the predefined query, returning accurate results labeled as \u0026ldquo;trusted\u0026rdquo;.\nFor example, when you ask a question like:\n\u0026ldquo;What is the price of Guinness beer converted to USD?\u0026rdquo;\nGenie will automatically trigger the Trusted Asset associated with the question, running the convert_beer_price_to_usd function with the appropriate parameter:\nSELECT * FROM convert_beer_price_to_usd(\u0026#39;Guinness\u0026#39;); The result will be a table that includes the beer name, the original price in euros and the price converted to US dollars:\nGenie will indicate that this is a result from a Trusted Asset, showing the corresponding label to the user. This not only provides the right answer but also reinforces the reliability of the information, ensuring it\u0026rsquo;s based on a previously validated query.\nThis approach lets users get faster, more accurate answers, since Genie doesn\u0026rsquo;t need to build a new query from scratch every time, taking advantage of the logic already established in the function registered as a Trusted Asset.\nConclusion Genie stands out for its ability to replace complex, dashboard-specific queries, offering a more interactive and flexible solution for business users. While traditional dashboards are useful for tracking the main KPIs and answering standard questions, they can become limiting when it comes to exploring deeper, more detailed questions. To meet that need, Genie offers a faster, more effective approach, letting users ask more complex questions, all based on the same semantic data, enriched by Trusted Assets and the right instructions.\nAdopting this strategy not only frees up the data team\u0026rsquo;s time but also empowers business users to get more complete insights, increasing autonomy and analytical power within the organization. In short, dashboards are best suited for the 10% of recurring day-to-day questions, while Genie is ideal for digging into more complex questions that dashboards can\u0026rsquo;t answer.\nTo ensure the success of an AI/BI Genie space, it\u0026rsquo;s essential to follow a few key components:\nData: use a clean, coherent, well-documented data set specific to the topic, ensuring accurate and clear information. Instructions: convey your business-specific information clearly, using natural language to guide Genie. Predefined Queries: use Trusted Assets to establish a single source of truth and use SQL examples to guide Genie\u0026rsquo;s answers, improving accuracy for common questions. Feedback: monitor the questions and their ratings, adjusting the Genie space based on user comments and the examples provided. By following these components, you\u0026rsquo;ll be able to optimize your use of Genie, boosting insights and making data analysis more complete and effective.\nReferences https://docs.databricks.com/en/genie/index.html\nhttps://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-alter-table-add-constraint.html\nhttps://www.databricks.com/glossary/medallion-architecture\nhttps://www.databricks.com/glossary/star-schema\nhttps://www.databricks.com/blog/onboarding-your-new-aibi-genie\n","permalink":"https://julianamarialop.github.io/en/articles/rub-the-lamp-and-get-insights-a-guide-to-ai-bi-genie-on-azure-databricks/","summary":"Imagine being able to talk to your data as if you were chatting with a coworker or, better yet, as if you were dealing with the genie of a magic lamp. That\u0026rsquo;s exactly what AI/BI Genie makes possible!…","title":"Rub the Lamp and Get Insights: A Guide to AI/BI Genie on Azure Databricks"},{"content":"In the data world, building a bronze layer, which organizes raw data into an accessible structure for later transformations, has traditionally involved a lot of coding. Recently, however, I explored a new approach using Databricks Assistant AI that let me build a bronze layer without writing a single line of code.\nDatabricks Assistant AI is an artificial intelligence tool built into the Databricks environment, designed to simplify the development and automation of data pipelines. It works by interpreting natural language commands, letting users set up complex tasks such as data ingestion, transformations and validations without writing code by hand. It\u0026rsquo;s an ideal solution for professionals who want more speed and efficiency in their data projects, cutting the time spent coding and minimizing common development mistakes.\nHere\u0026rsquo;s the step by step of how I managed to automate this process using notebooks and artificial intelligence.\nCreating the Notebook To start, I created a notebook called 01_Tabelas_Bronze_GenAI with the comment: \u0026ldquo;Notebook for creating the bronze layer.\u0026rdquo; This notebook was the foundation for building the bronze layer automatically, with no manual coding. Each step was set up simply, using Databricks Assistant AI to handle the logic behind the actions.\nThere are 2 ways to access the Assistant: through the notebook cell, or in the top right corner right after the name of your Databricks service (in my case \u0026ldquo;jml-databricks\u0026rdquo;). I chose to use it per cell to better explain building the Bronze layer step by step.\nOnce it\u0026rsquo;s created, let\u0026rsquo;s have some fun!\nStep 1: Creating Parameters First, the essential parameters for running the process were defined. These parameters control the file paths and the source format of the data.\nName: par_path Type: Text Comment: Path of the data source folder Name: file_format Type: Text Comment: Format of the source file (CSV, JSON, etc.) Name: file_name Type: Text Comment: Name of the source file With these parameters, you can make sure the pipeline adapts to different data sources.\nCrie um notebook Nome: 01_Tabelas_Bronze_GenAI Comentário: Notebook de criação da camada bronze. Ação: Cada passo abaixo é uma célula no notebook. -------------------------------------------------------------------------------------------- Passo 1 - Criação de Paramêtros Nome: par_path Tipo: Texto Comentário: Caminho da pasta Nome: file_format Tipo: Texto Comentário: Nome do Arquivo de Origem Nome: file_name Tipo: Texto Comentário: Formato do Arquivo de Origem -------------------------------------------------------------------------------------------- To generate the code, click \u0026ldquo;Toggle Assistant\u0026rdquo;, paste the prompt above and click Generate.\n_Loading information into the prompt_\nNotice that Databricks will create your Python code; if everything looks right, hit \u0026ldquo;Accept\u0026rdquo;.\n_Accepting the code suggestion_\nNow just run it!\n_Code executed_\nThis run generated 3 widgets.\nStep 2: Automatic Parameter Validation With the parameters configured, the next step was to make sure they were read and validated correctly. Databricks Assistant AI automatically sets up the code to:\nGet all parameters from the widgets. Print the values for debugging purposes. Check whether the parameters are defined and, if not, display a message saying they are required, ending the notebook run. This validation step is crucial to avoid problems with incorrect or incomplete data.\nPasso 2 - Paramêtros de execução automático Obter os parâmetros de todos os widgets. Imprima os valores desses parâmetros para fins de depuração. Verifique se todos os parâmetros estão definidos. Se não estiver, mostre uma mensagem informando que ele é obrigatório e encerre a execução do notebook com um código de saída 1.\u0026#34; _Step 2 prompt_\n_Step 2 code executed_\nTo run this code I added the parameters:\npar_path: /mnt/jmldatalake/movies/dataset/imdb/ file_format: csv file_name: name_basics.csv Step 3: Creating the Database With the parameters validated, the next action was to create the database where the bronze layer would be stored. Through Databricks Assistant AI, I set up the following command:\nDatabase Name: movies_bronze Storage Path: /mnt/jmldatalake/movies_bronze/database Comment: Bronze layer database If the database already existed, the notebook wouldn\u0026rsquo;t recreate it, keeping the process efficient.\nPasso 3 - Criar Banco de Dados Ação: Crie uma base de dados (se não existir) Nome: movies_bronze Caminho do Armazenamento: /mnt/jmldatalake/movies_bronze/database Comentário: Banco de Dados da camada Bronze _Step 3 code generated_\nBronze layer database created!\nStep 4: Defining the Table Name To name the bronze layer table, the source file name was used as the basis, with the following transformations applied automatically:\nRemoving the file extension. Replacing \u0026ldquo;-\u0026rdquo; characters with \u0026ldquo;_\u0026rdquo;. Adding the tb_raw_ prefix to the file name. For example, if the original file were movies-2024.csv, the table name would become tb_raw_movies_2024.\nPasso 4 - Definição do nome da tabela Condição: Utilize a variável que contenha Nome do Arquivo de Origem Ação: 1. Retire a extensão do Nome do Arquivo de Origem 2. Substitua \u0026#34;-\u0026#34; por \u0026#34;_\u0026#34; 3. Adicione o prefixo \u0026#34;tb_raw_\u0026#34; ao Nome do Arquivo de Origem _Step 4 prompt_\n_Code Executed_\nStep 5: Reading Files The next step was reading the raw data files. Using the source path and the file name, Databricks Assistant AI automated reading the data with the following settings:\nHeader: True InferSchema: False Delimiter: Tab (\\t) After reading, the number of ingested records was printed automatically for verification.\nPasso 5 - Leitura de Arquivos Crie uma variável que una o caminho de origem do arquivo e o nome do arquivo de origem, retire os 5 últimos caracteres e adicione * Ação: Leia o Arquivo de acordo com a variável criada acima com os parâmetros abaixo: Cabeçalho: Verdadeiro InferSchema: Falso Delimitado: \\t Após isso, imprima na tela a quantidade de registros ingeridos na tabela. _Code executed in step 5_\nStep 6: Writing Data in Delta Format Finally, the data was saved in Delta format, which offers high performance and versioning capability. The write was configured with the following options:\nOverwrite Mode: True Overwrite Schema: True Multi Line: False That way, the data was overwritten, keeping the bronze layer consistent. After the write, the number of records in the target was also printed, confirming the operation succeeded.\nPasso 6 - Gravação de Arquivos Salve o dataframe criado no passo anterior em formato Delta utilizando os parâmetros abaixo: Modo Overwrite: Verdadeiro Substituir Schema: Verdadeiro Multi Linhas: False Após isso, imprima na tela a quantidade de registros ingeridos na tabela destino. _Code executed for step 6_\nConclusion With the help of Databricks Assistant AI, I was able to automate building a bronze layer quickly and efficiently, without writing a single line of code. What used to be a manual, error-prone task became a simple, controlled and auditable process.\nThis approach not only saved time but also brought more security and scalability to the data pipeline. I\u0026rsquo;m excited to see how artificial intelligence will keep transforming the development of data solutions and simplifying processes that, until recently, required deep technical knowledge.\nI hope I\u0026rsquo;ve inspired you to use the Assistant, because it\u0026rsquo;s really worth it!!\nIn the next article we\u0026rsquo;ll tackle the Silver Layer!\nSee you then!\n","permalink":"https://julianamarialop.github.io/en/articles/from-zero-to-bronze-build-a-notebook-without-coding-databricks-assistant/","summary":"In the data world, building a bronze layer, which organizes raw data into an accessible structure for later transformations, has traditionally involved a lot of coding. Recently, however, I explored a…","title":"From Zero to Bronze: How to Build a Notebook Without Coding Using Azure Databricks Assistant AI"},{"content":"In recent years, Databricks has established itself as a powerful platform, much like the Force that guides the Jedi on their missions, helping companies accelerate their data and analytics projects. Just as the union between the Force and the Jedi creates harmony, Databricks brings the data warehouse and the data lake together in one balanced, powerful solution: the lakehouse.\nBut how, exactly, can we structure and configure a Databricks platform efficiently so it\u0026rsquo;s as precise as a lightsaber in the hands of a Jedi? Just as the Jedi follow their code of honor, we also need a set of best practices to make sure Databricks runs in perfect harmony, protecting our data as if it were the Resistance\u0026rsquo;s greatest secret.\nIn this article, we\u0026rsquo;ll set off on our galactic journey and explore the main pillars that make up a well-architected Databricks platform. From data governance (our organizing \u0026ldquo;Force\u0026rdquo;) to cost optimization (the fuel that keeps the ride across the galaxy smooth), we\u0026rsquo;ll cover everything you need to master this powerful tool.\nSo, ready for the mission? May governance be with you! 🌌\nTo kick off our best practices journey, let\u0026rsquo;s break the topics down as follows:\n1. Data Governance The first step toward a solid architecture is making sure the data truly adds value to the business and supports the company\u0026rsquo;s strategy. Data governance involves overseeing policies and processes to ensure data is reliable, secure, and used efficiently.\nData governance in Databricks goes beyond simply storing and accessing information. It involves a set of best practices to make sure data is always secure, high quality, and compliant with the organization\u0026rsquo;s policies and regulations. Let\u0026rsquo;s look at how to apply these concepts clearly and effectively:\n🔍 1.1. Data Classification and Tagging First of all, it\u0026rsquo;s essential to classify and understand the sensitivity and importance of the data. This helps you identify data that requires more protection, such as confidential or restricted data, and also define who owns the data within the company.\nData Classification: Categorize data into levels such as public, confidential, or restricted. Also define the data owner, that is, who is responsible for it within the organization. Tags and Comments: Use consistent tags to mark tables with important information, such as the data owner, its origin, and its sensitivity level. This makes it easier to manage and monitor data usage. Adding comments to tables and columns also helps document important details, and they can even be generated automatically with the help of AI. Data Masking: For test or development environments, implement data masking techniques that protect sensitive information, making sure that data isn\u0026rsquo;t improperly exposed. ✅ 1.2. Data Quality Data quality is essential to make sure the decisions based on it are reliable.\nData Validation: Automate consistency checks, using features such as ADD CONSTRAINT, to make sure data always complies with the established rules. Error Handling: Implement robust mechanisms to identify and fix data errors quickly. Automated Tests: Add automated tests to your processes to make sure everything works as expected. Standardization: Use standardized data formats, such as consistent naming conventions, unified date formats, and clear units of measurement, avoiding confusion and discrepancies. 🗂 1.3. Unity Catalog Unity Catalog is a powerful tool for managing and organizing data securely and efficiently.\nData Segregation: Use catalogs to segment data by team, business unit, or specific environment. Access Control: Implement Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC) to make sure each user has access only to the data they need for their job. Fine-Grained Permissions: Apply detailed permissions at the database, table, row, and column level, enforcing the principle of \u0026ldquo;least privilege.\u0026rdquo; Audit Logs: Enable audit logging to track access to and changes made to the data, making monitoring and regulatory compliance easier. 🛡️ 1.4. Data Policies Finally, it\u0026rsquo;s essential that governance practices are established and followed rigorously.\nData Retention Policies: Define and enforce clear policies for the data lifecycle, using features such as Databricks VACUUM to ensure compliance with legal requirements. Regulatory Compliance: Make sure governance practices are aligned with regulations such as GDPR, CCPA, HIPAA, among others. Third-Party Tools: Take advantage of Databricks integrations with catalog and governance tools, such as Azure Purview, to strengthen your data control capabilities. Policies and Procedures: Create clear policies and procedures that define responsibilities, roles, and processes. Regular Reports: Produce periodic reports on governance metrics, such as access logs, data quality, and compliance status. Data Lineage: Use data lineage tools to trace where data comes from and how it is transformed along the pipelines, ensuring full transparency and control. 2. Interoperability and Usability A good lakehouse should be able to interact easily with both users and other systems. This means its architecture must support multiple tools and technologies, ensuring a smooth, frictionless experience for everyone involved.\n📋 2.1. General Managing a Databricks platform requires best practices to ensure security, efficiency, and ongoing maintenance. Here are some guidelines that can help you structure your environment in an organized, secure way:\nService Principals: When setting up accounts that need to interact with Databricks, such as accounts used by Terraform, it\u0026rsquo;s essential to use Service Principals. This gives each service its own account, making auditing and access control easier. That way, you\u0026rsquo;ll know exactly who has access and which permissions are in effect. Infrastructure as Code (IaC): To automate the deployment and maintenance of Databricks, use Infrastructure as Code (IaC). The Databricks provider for Terraform is a great solution for this, since it lets you manage the environment in a repeatable, secure way. Secrets Management: Make sure all secrets and credentials used to access other services are stored securely. Use Databricks Secret Scopes, which can be integrated with tools such as Azure KeyVault, to make sure this information is protected from unauthorized access. CI/CD: Deploy changes to the Databricks environment through CI/CD (Continuous Integration/Continuous Delivery) pipelines. All source code should be stored in a Git repository, enabling version control, change tracking, and safer development. 🔄 2.2. Data Sharing Data sharing in Databricks requires following best practices to ensure security and privacy, especially when multiple partners or business units are involved:\nDelta Sharing: When it comes to sharing data securely, Delta Sharing is an ideal solution. It lets you share data with external partners or business units while keeping control over access. For extra security, set up IP access lists to restrict who can access the data, and implement token rotation to keep control tight. Databricks Clean Rooms: Use Databricks Clean Rooms when you need collaborative analysis across organizations without sharing the raw data. This ensures data privacy is preserved, while also complying with privacy regulations such as GDPR. Marketplace: If you need access to commercial or open data, consider the Databricks Marketplace, which offers a wide range of data to meet different business needs. Lakehouse Federation: Data federation is useful when you need to access information from several systems without migrating the data to a central location. However, consider the impact on query performance and the data ingress/egress costs when adopting this approach. 3. Operational Excellence A cutting-edge architecture is no use if operations can\u0026rsquo;t keep up. Operational excellence covers all the processes that keep the Databricks platform running efficiently and stably in a production environment, minimizing downtime and ensuring continuity.\n🚀 Processing (Clusters) Cluster management in Databricks is essential for good performance, security, and cost optimization. Here are some recommended practices:\n3.1. General Cluster Policies: Define cluster policies to limit size and resources, or even to create \u0026ldquo;T-shirt-size clusters\u0026rdquo; (small, medium, large), tailored to the needs of different workloads. Databricks Runtime: Databricks Runtime is constantly evolving, with improvements in usability, performance, and security. Whenever possible, use the latest version to make sure you have access to the best features. Restart Long-Running Clusters: Clusters that run for long periods can miss critical security and operating system updates. Make sure they\u0026rsquo;re restarted periodically to avoid performance and security issues. Use SSD Volumes: Clusters with workers that use SSD volumes are automatically configured with disk caching, speeding up queries and improving the performance of your environment. Cluster Pre-warming: If your queries are slow because of cluster cold starts, consider using Databricks pools, which let you pre-warm clusters to cut down wait times. ⚙️ Orchestration (Jobs) Configuring and managing jobs in Databricks can be optimized to ensure efficiency and scalability.\n3.2. Job Configuration Job Clusters: When configuring jobs, use job-specific clusters, sized correctly for the workload. Using autoscaling lets you handle variations in demand efficiently. Job Retries: Set an appropriate number of retries for critical jobs, taking transient failures into account. To avoid overload, implement exponential backoff logic in the code, adjusting the retry frequency when errors occur. Service Principals: Make sure jobs are configured with a Service Principal as owner, which avoids continuity problems if the job\u0026rsquo;s original creator leaves the team. Using Parameters: In multi-stage jobs, use parameters to share information between stages, making integration and process control easier. Event-Driven Jobs: Whenever possible, choose event-driven jobs, which are more efficient in terms of time and cost. For one-off ingestions, consider using the COPY INTO command as an alternative. 3.3. Code Management Version Control: Store your code in a version control system such as Git. Use Databricks Repos to sync notebooks and files with the repository, making it easier to manage and track changes. Modular Code: Write modular, reusable code so it can be shared across different jobs and notebooks, promoting efficiency and standardization. Automated Tests: Implement unit, integration, and E2E tests to make sure the code works as expected, from the simplest functions to the most complex flows. Databricks Asset Bundles (DAB): Use Databricks Asset Bundles to make developing and deploying jobs easier, ensuring consistency and automation. CI/CD Pipelines: Set up CI/CD pipelines to automate workflows in Databricks, using tools such as Azure DevOps, GitLab, or GitHub Actions. 3.4. Error Handling and Logging Error Handling: Implement robust error handling with try-catch blocks, making sure exceptions are handled properly without abrupt interruptions. Logging: Use structured logging to record events, metrics, and errors. Databricks integrates with logging tools such as Azure Log Analytics, which makes monitoring and diagnosis easier. 🧑💻 People and Processes Beyond the technical side, it\u0026rsquo;s essential to set up a dedicated team and create well-documented, standardized processes.\n3.5. General Operations Team: Put together an operations team dedicated to the Lakehouse, responsible for upholding best practices, ensuring security, and monitoring the tools in use. Managing Service Limits: When designing the architecture, take into account the service limits and quotas of Databricks, the cloud provider, and Unity Catalog, to avoid disruptions from exceeding limits. Naming Standards: Consider creating a document with naming standards for Databricks objects, such as catalogs, which can follow conventions like \u0026lt;sigla_do_ambiente\u0026gt;-\u0026lt;nome_do_projeto\u0026gt;. This standardization can be enforced through CI/CD pipelines. 3.6. Collaboration and Documentation Complete Documentation: Fully document your workflows, code, and configurations. Use features such as comments, Markdown, and visualizations inside notebooks to make sure everything is clear to the team. Code Reviews: Implement a code review process to ensure quality and promote knowledge sharing among team members. Resource Tagging: Use consistent tags to mark resources according to company policies. This makes monitoring, cost management, and finding important information easier. 4. Security, Privacy, and Compliance Protecting data and systems is essential. That\u0026rsquo;s why a good Databricks setup must have robust mechanisms for security, privacy, and compliance with standards and regulations. Keeping the environment safe from threats is a priority for protecting the value of the data and customers\u0026rsquo; trust.\n🛡️ 4.1. General Security Security Analysis Tool (SAT): Databricks offers the Security Analysis Tool (SAT), which helps you check whether your workspaces follow security best practices. This tool is available on GitHub. Multi-Factor Authentication (MFA): Make sure every account that uses Databricks is set up with multi-factor authentication (MFA), using an appropriate identity provider. For Azure Databricks, the default identity provider is Entra, while for GCP or AWS you can configure SSO. Role-Based Access Control (RBAC): Use Azure AD groups and Databricks SCIM to manage permissions based on the principle of least privilege, making sure each user has only the access they need. Service Principals: When providing automated or programmatic access to Databricks, use Azure AD Service Principals, making sure these identities have the minimum permissions needed to do their jobs. Enhanced Security and Compliance: If your company needs to meet compliance standards such as PCI-DSS or HIPAA, consider using the Enhanced Security and Compliance add-on to make sure the specific requirements are met. 👨💻 4.2. Code Management Secrets: All secrets, such as access credentials, should be stored in secret scopes with the appropriate permissions to protect them from unauthorized access. DBFS Root: Avoid storing data in the DBFS root. That\u0026rsquo;s because when data is stored in the root, every user can access it, compromising security. Static Code Analysis (IaC): Consider using tools such as checkov, tfsec, and terrascan to run static analysis on Infrastructure as Code (IaC), identifying vulnerabilities and compliance issues. Static Code Analysis (Python): For Python code, use tools such as Pylint to check code quality and security. Code Reviews: Hold regular code reviews to make sure notebooks and jobs follow coding best practices and that no confidential information is exposed. Dependency Management: Keep dependencies up to date, applying patches regularly. Tools such as Snyk help identify vulnerabilities in third-party libraries and tools, which is essential for maintaining operational security. 🌐 4.3. Network Security (Azure) Deploy Databricks in a VNet: Use Azure VNet injection to deploy Databricks inside a private virtual network (VNet), making sure your cluster and workspace are isolated from the public internet. Private Endpoints: Configure Azure Private Link to create private endpoints, enabling secure communication between Databricks and other Azure services, such as storage accounts and databases. Network Security Groups (NSGs): Use NSGs to control inbound and outbound traffic on the Databricks subnets, making sure only the necessary traffic is allowed. Secure Cluster Connectivity: Configure clusters to run with secure connectivity, that is, with no public IP, preventing direct access from the internet. IP Access Lists: Consider implementing IP access lists to restrict workspace access to allowed IP addresses only. 🔐 4.4. Data Protection (Azure) Encryption at Rest: Make sure all data stored in services such as Azure Data Lake Storage (ADLS) is encrypted at rest, using Azure storage service encryption. Encryption in Transit: Enable HTTPS and SSL/TLS to encrypt data in transit, securing communications between Databricks and external systems. Managed Identity for Azure Resources: Use managed identities to give Databricks secure access to Azure Key Vault and other resources, without having to expose credentials. Customer-Managed Keys (CMK): Configure CMKs to manage data encryption in Databricks, implementing key rotation and usage monitoring to detect unauthorized access. 📚 4.5. Documentation and Training Disaster Recovery (DR) Plan Documentation: Document your disaster recovery (DR) strategy, including backup procedures, recovery steps, and assignment of responsibilities. Runbooks: Create runbooks with step-by-step instructions for disaster recovery scenarios, making sure your team knows how to act quickly in case of incidents. Knowledge Sharing: Promote knowledge sharing about the DR plan across the whole organization, making sure everyone is prepared. 💾 4.6. Operational Security Backups: Use cross-region storage to protect your critical data, or implement backup jobs that write the data to a local region or storage account. Git: Store all Infrastructure as Code (IaC), cluster configuration, jobs, and notebooks in Git repositories. This makes recovery easier in case of incidents and promotes version control. Disaster Recovery (DR) Plan: Develop and regularly test a disaster recovery plan, ensuring business continuity in case of serious incidents. ⚙️ 4.7. Workspace Configuration Credential Passthrough: Starting with Databricks Runtime 15.0, Credential Passthrough will be deprecated. Instead, use Unity Catalog (UC) to manage identities and permissions more efficiently. Environment Isolation: Set up different workspaces for development, stage, and production environments, or for different business units. This isolation enables better resource management and security. Data Separation: Logically and physically separate your confidential data from non-confidential data, making sure sensitive information is protected. 5. Performance Efficiency With demand for data analytics and processing growing exponentially, performance efficiency is crucial. Your platform must be able to adapt to changes in workload without compromising the speed and quality of results.\n🔎 Queries Query optimization in Databricks is essential for efficient performance and for avoiding unnecessary bottlenecks. Here are some recommended practices to maximize the performance of your queries:\n5.1. General Liquid Clustering: Configure liquid clustering on the columns most often used in queries. This technique is preferable to Z-ordering or partitioning, since it improves lookup performance without the risk of over-partitioning. Efficient File Formats: Use file formats such as Parquet or ORC, which offer better performance and data compression. For even more efficiency, store the data in managed tables in the lakehouse. Bigger Clusters: When your workload grows linearly, plan for bigger clusters. A bigger cluster usually processes the same load faster and, in many cases, doesn\u0026rsquo;t cost more, it just runs faster. Data Skipping: Databricks automatically collects statistics on the first 32 columns defined (including nested columns). To improve query performance, make sure the most frequently queried columns are within those first 32 columns. Avoid Partitioning: With the arrival of liquid clustering, partitioning has become less necessary. In many cases, partitioning the data can lead to over-partitioning, where each partition should be at least 1 GB, which can hurt performance. 5.2. Performance Predicate Pushdown: Use predicate pushdown to filter data right at the start of query execution. This reduces the amount of data moved around and improves overall performance. When reading from an RDBMS, combine options such as partitionColumn, lowerBound, and upperBound to parallelize the read operation. Broadcast Joins: For smaller tables, use broadcast joins, sending the smaller table to all executors. This improves join performance in complex queries. Skew Handling: Identify and manage skewed data (skew) to make sure data partitions are distributed evenly. Techniques such as salting can help mitigate data skew. Persistence (Cache): Use the .persist command to store subquery results and data in formats other than Parquet. However, avoid overusing persistence unless you know exactly what impact it will have on performance. Speculative Execution: Turn on speculative execution to re-run slow tasks on other nodes, which helps mitigate the impact of long-running tasks and improves total execution time. Compaction: Use the OPTIMIZE command to manually optimize tables in Delta Lake. For continuous optimization, enable AUTO OPTIMIZE and AUTO COMPACT when writing to tables, improving performance over the long run. ANALYZE TABLE: The ANALYZE TABLE statement collects statistics about tables within a specified schema, helping improve query performance. 💻 Code Well-structured, optimized code can significantly improve query performance and the overall efficiency of the system.\n5.3. Code Avoid Wide Transformations: Minimize the use of wide transformations, such as groupBy and join, which require shuffling large amounts of data across the network. When these operations are necessary, optimize them to reduce the impact on performance. Prefer DataFrames and Datasets: Use DataFrames and Datasets instead of RDDs, since they offer automatic optimizations and are easier to use. Take advantage of the Catalyst optimizer and the Tungsten engine to get better performance from your operations. Code Profiling: Profile your code regularly to identify and resolve performance bottlenecks. Use tools such as the Spark UI to gain insight into execution and improve efficiency. MERGE Statements: When possible, use partition pruning during MERGE operations, optimizing how these operations run. 5.4. Timestamps Timestamps: In Databricks, timestamps are stored as floating-point numbers, representing the number of seconds (and fractions of a second) since the Unix epoch. Notebooks and Time Zones: The value of timestamps displayed in notebooks is adjusted to the current session\u0026rsquo;s time zone, which is usually UTC. If the data in the file doesn\u0026rsquo;t contain a time zone indicator, timestamp parsing may depend on the session\u0026rsquo;s time zone. SQL Warehouses: In SQL warehouses, the default time zone is also UTC, unless configured otherwise. However, there is no time zone offset or indicator. 6. Cost Optimization Finally, a good architecture can\u0026rsquo;t forget about cost optimization. Managing resources smartly and effectively ensures the company maximizes its return on investment, delivering value without waste.\nControlling costs in Databricks is essential to make sure you\u0026rsquo;re getting the most out of the platform without going over budget. Here are some practices to help you manage costs effectively:\n💰 6.1. Cost Monitoring Regular Monitoring: It\u0026rsquo;s essential to constantly monitor usage and costs in Databricks. Use the Cost Analysis available in Databricks or third-party tools to get a clear view of your consumption. In the Databricks account console, there\u0026rsquo;s a dedicated page for analyzing DBU (Databricks Units) usage, although it doesn\u0026rsquo;t include VM or storage costs. Budgets and Notifications: Databricks offers a budget setting feature, where you can set up notifications to be alerted if usage exceeds the set amount, helping keep spending under control. 💰 6.2 Optimized Clusters Cost-Efficient Cluster Configurations: Use cluster configurations that offer a good cost-benefit ratio. For non-critical workloads, consider using spot instances, which are cheaper but can be interrupted at any time. To ensure resilience, implement a retry policy when using these instances. Auto-Termination Policies: Configure auto-termination policies for idle clusters, keeping unnecessary clusters from running and racking up costs. Graviton Clusters (AWS): If you\u0026rsquo;re running on AWS, consider using Graviton clusters, which offer a better price/performance ratio than other instance types. 💰 6.3 Photon and Query Optimization Photon Runtime: The Databricks Photon Runtime offers a significant performance boost compared with traditional Spark, while also being more cost-efficient. It\u0026rsquo;s ideal if you\u0026rsquo;re looking to optimize performance without raising spending. Query Optimization: Regularly review and optimize your Spark queries. Use the Spark Query Profile and analyze the execution plans to identify performance bottlenecks. Making queries more efficient reduces execution time and, as a result, costs. 💰 6.4 Serverless Instances Serverless Options: When appropriate, consider using serverless SQL warehouses, workflows, and notebooks. These options can be more economical, since you pay only for the time you use, without having to keep dedicated clusters running. 🔄 7. Reliability Failures happen. But the question is: how does your platform react to them? The reliability of a Databricks system involves the ability to recover quickly from failures and keep operating without affecting critical business processes.\n📊 7.1. Monitoring and Logging Keeping a Databricks environment running efficiently requires a solid monitoring and logging strategy. Here are some recommended practices to ensure the integrity, performance, and security of your environment:\nEnable Audit Logs: Turn on audit logs in Databricks to capture detailed information about activity, such as login events, data access, and administrative actions. These logs help keep a history of operations and can be configured with audit filters to prevent data leakage between different environments. Azure Monitor: Use Azure Monitor to track the health and performance of your Databricks environment, making sure operations are running as expected and spotting potential problems before they get worse. Integration with Monitoring Tools: When needed, integrate Databricks with monitoring tools such as Datadog, Prometheus, Splunk, or Azure Monitor itself, to improve visibility into the environment and the tracking of critical metrics. Cluster Monitoring: Regularly monitor your clusters, checking their health, performance, and logs. Set up automatic alerts for important metrics, such as CPU usage, memory, or job duration, to avoid unexpected outages. Integrating with platforms such as Slack or PagerDuty can be useful for getting immediate notifications. Resource Management: Monitor cluster resource usage with tools such as Ganglia or the Databricks Cluster UI, making sure resources are being used efficiently. Auto Loader Monitoring: Auto Loader lets you inspect the state of a data stream through a SQL API and the Spark Streaming Query Listener, providing detailed insight into the files processed and the state of queries in real time. Job Monitoring: Use the Databricks Jobs interface to monitor job status and performance. You can also implement custom monitoring for specific metrics, making bottlenecks easier to spot. Delta Live Tables Monitoring: Every pipeline in Delta Live Tables produces an event log that includes audit logs, data quality checks, pipeline progress, and data lineage information. This makes it easier to keep track of pipelines continuously. Streaming Monitoring: With the Structured Streaming interface, you can monitor input rates, processing rates, and latency for your streaming queries. The StreamingQuery.progress method provides detailed reports in JSON format for deeper analysis. ML and AI Monitoring: For monitoring Machine Learning and AI models, use inference tables that continuously record model inputs and outputs in a Delta table. This lets you monitor, debug, and optimize models with the help of SQL queries, notebooks, and Lakehouse monitoring tools. Security and Cost Monitoring: Security monitoring should be implemented to ensure compliance and privacy. On top of that, monitor and control costs regularly to avoid budget surprises. 📦 7.2. Other Important Aspects Default Workspace Storage: Avoid using the workspace\u0026rsquo;s default storage for production data. Instead, create dedicated storage locations, configured and tuned to your specific needs, ensuring greater security and flexibility. Resilience to Availability Zone (AZ) Failures: Databricks is already resilient to AZ failures by default. Make sure clusters are configured for Auto-Scaling and have a retry policy for jobs. Control plane functionality should be restored about 15 minutes after an AZ failure. Regional Failure: Although an active-active multi-region deployment in Databricks is usually not very financially viable, it\u0026rsquo;s recommended to use Infrastructure as Code (IaC) to create a secondary deployment in a backup region. Use replicated cloud storage to ensure continuity of operations. 🛠️ 7.3. Infrastructure on Azure GRS (Geo-Redundant Storage): In most cases, GRS (Geo-Redundant Storage) offers an economical storage option while still guaranteeing 16 nines of durability. However, if regional recovery time is critical, consider using GZRS (Zone-Redundant Storage). That wraps up our galactic journey through Databricks, exploring the main pillars for setting up this powerful platform strategically and efficiently.\nJust as a Jedi masters the Force, mastering Databricks requires best practices that keep data governance, cost optimization, and performance in balance. These guidelines are like the Jedi code, offering guidance but always flexible enough to adapt to the needs and realities of each organization.\nMay the power of data be with you as you build a successful platform!\nThank you for reading, and be sure to follow the other articles in the newsletter.\nReferences Databricks SAT: https://github.com/databricks-industry-solutions/security-análise-t\nData Lakehouse: https://learn.microsoft.com/en-us/azure/databricks/lakehouse-arch\nBest practices: https://learn.microsoft.com/en-us/azure/databricks/data-governan\nProtection: https://www.databricks.com/blog/data-exfiltration-protection-with-\nCluster connectivity: https://learn.microsoft.com/en-us/azure/databricks/security/netw\nDatabricks budgets API: https://www.databricks.com/blog/best-practices-cost-management-databricks#:~:text=Warehouse%20creation%20permissions.-,Monitoring%20usage,-Along%20with%20controlling\nDatabricks Photon: https://www.databricks.com/product/photon#:~:text=no%20rewrite%20required.-,Por que%20Photon%3F,-Desempenho%20da%20consulta\n","permalink":"https://julianamarialop.github.io/en/articles/may-the-analytical-force-be-with-you-exploring-databricks-star-wars-style/","summary":"In recent years, Databricks has established itself as a powerful platform, much like the Force that guides the Jedi on their missions, helping companies accelerate their data and analytics projects. Just as the union between the…","title":"May the Analytical Force Be with You: Exploring Databricks Star Wars Style"},{"content":"An overview and high-level comparison of Databricks and Snowflake, their strengths and their weaknesses. Databricks and Snowflake are two big names in cloud data solutions. Both platforms have been key to helping companies generate value from their internal and external data assets. Each platform has its own distinct advantages and features, and their offerings have overlapped more and more, leaving many people unsure which solution best fits their business needs.\nTo start a meaningful comparison, we have to understand the history and core competency of each offering. To help with that, let\u0026rsquo;s go over the main differences between Snowflake and Databricks, including price, performance, integration, security and the best use cases, so each user can match them to their own needs.\nDatabricks vs. Snowflake: What are the main differences? The first thing to understand about the two platforms is what they are and what problem they set out to solve.\nDatabricks is a unified, cloud-based data analytics platform for building, deploying and sharing data analytics solutions at scale. Databricks aims to provide a unified interface where users can store data and run jobs in interactive, shareable workspaces. These workspaces hold notebooks through which every compute function is built to run on cloud-based machines.\nSnowflake, on the other hand, is a fully managed, cloud-based SaaS data warehouse. While Databricks was originally designed to unify data pipelines, Snowflake was designed to be the easiest data warehouse solution to manage. While Databricks\u0026rsquo; target market is data scientists and data engineers, Snowflake\u0026rsquo;s target market is typically data analysts: people who are highly proficient in SQL queries and data analysis, but not so interested in complex computations or machine learning workflows.\nOver time, Databricks and Snowflake have competed more and more, as each one hopes to expand its offering into an all-in-one cloud data platform. New products such as Snowflake\u0026rsquo;s Snowpark (which offers Python functionality) and Databricks\u0026rsquo; DBSQL (its serverless data warehouse) have made it increasingly hard to tell the two offerings apart.\nFor now, most would agree that Snowflake tends to be the dominant name for easy-to-use cloud data warehouse solutions, and Databricks is the winner for cloud-based machine learning and data science workflows.\nDatabricks vs Snowflake: Data storage Right now, Snowflake has the edge for querying structured data, and Databricks has the edge for the raw and unstructured data that ML needs. In the near future, I believe Databricks\u0026rsquo; data lakehouse platform will be the dominant and most comprehensive market solution for all data management.\nOne of the biggest differences between Snowflake and Databricks is how they store and access data. Both lead the industry in speed and scale. The biggest difference between the two is the data warehouse vs. data lakehouse architecture, and the storage of unstructured vs. structured data.\nSnowflake At its core, Snowflake is a cloud data warehouse. It stores structured data in a closed, proprietary format for fast, seamless querying and transformation. Its proprietary format delivers high speed and reliability at the cost of flexibility. More recently, Snowflake has been allowing data ingestion and storage in additional formats (such as Apache Iceberg), but the vast majority of its customers\u0026rsquo; data still lives in its own format.\nSnowflake uses a multi-cluster shared disk architecture, in which compute resources share the same storage device but keep their own CPU and memory. To achieve this, Snowflake ingests, optimizes and compresses data into a cloud object storage layer, such as Amazon S3 or Google Cloud Storage. The data there is organized in a columnar format and split into micro-partitions of 50 to 500 MB. These micro-partitions store metadata, which helps a lot with speed. Interestingly, Snowflake\u0026rsquo;s own internal storage file format isn\u0026rsquo;t open source, which keeps most customers locked in.\nTo work efficiently, Snowflake uses several layers to give cloud processing workloads an enterprise experience. Snowflake maintains a cloud services layer that handles enterprise authentication and access control.\nFor execution, Snowflake uses virtual warehouses, which are abstractions over regular cloud instances (such as EC2). These warehouses query data from a separate data storage layer, effectively separating storage and compute. This separation of compute and storage makes Snowflake infinitely scalable and lets users run concurrent queries on the same data with reasonable isolation.\n_Snowflake Architecture Layers_\nSnowflake can run on all three major cloud service providers.\nBottom line: Snowflake\u0026rsquo;s architecture enables fast, reliable queries over structured data, at scale. It appeals to those who want simple ways to manage their jobs\u0026rsquo; resource requirements (through T-shirt-sized warehouse options). It\u0026rsquo;s aimed mainly at people proficient in SQL, but it lacks the flexibility to easily handle raw and unstructured data.\nDatabricks One of Databricks\u0026rsquo; selling points is that it uses an open source storage layer known as Delta Lake, which aims to combine the flexibility of cloud data lakes with the reliability and unified structure of a data warehouse, without the challenges that come with vendor lock-in. Databricks pioneered this hybrid structure, called the \u0026ldquo;data lakehouse\u0026rdquo;, as a cost-effective way for data scientists, data engineers and analysts to work on the same data, regardless of structure or format.\n_Databricks Lakehouse_\nThe Databricks data lakehouse works by using three layers to allow raw and unstructured data to be stored, but it also stores metadata (such as a structured schema) for warehouse-like capabilities on structured data. Notably, this data lakehouse provides support for ACID transactions, automatic schema enforcement (which validates DataFrame and table compatibility before writes) and end-to-end streaming for real-time data ingestion: some of the most desirable advances for data lake systems.\nBottom line: Lakehouses bring the speed, reliability and fast query performance of data warehouses to the flexibility of a Data Lake.\nDatabricks vs Snowflake Scalability Snowflake and Databricks keep battling for dominance over enterprise workloads. Although both have proven to be industry leaders in this capability, the biggest practical difference between the two lies in their resource management capabilities.\nSnowflake Snowflake offers compute resources as a serverless offering. That means users don\u0026rsquo;t need to select, install, configure or manage any software or hardware. Instead, Snowflake uses a series of virtual warehouses (independent compute resources containing memory and CPU) to run queries. This separation of memory and compute resources lets Snowflake scale infinitely without slowing down, and multiple users can query the same single segment of data at the same time.\nSnowflake uses a simple \u0026ldquo;T-shirt size\u0026rdquo; scaling model for its virtual warehouses, with 10 sizes, each with twice the compute power of the previous size. The largest is the 6XL, which has 512 nodes. Since warehouses don\u0026rsquo;t share compute resources or store data, if one goes down, it can be replaced in minutes without affecting any of the others.\n_A diagram of the virtual nodes associated with each warehouse size_\nMost notably, Snowflake\u0026rsquo;s multi-cluster warehouses provide a \u0026ldquo;maximized\u0026rdquo; mode and an \u0026ldquo;auto-scale\u0026rdquo; mode, which give it the ability to dynamically shut down unused clusters, saving money.\nDatabricks Databricks started with a much more \u0026ldquo;open\u0026rdquo;, traditional infrastructure, where basically all compute runs inside a user\u0026rsquo;s cloud VPC. This is the complete opposite of the \u0026ldquo;serverless\u0026rdquo; model, where compute runs inside Databricks\u0026rsquo; VPC, since all cluster settings are exposed to end users. This has its pros and cons: the main advantage is that users can hyper-optimize their clusters to improve performance, but the downside is that it can be painful to use or require an expert to maintain.\nMore recently, Databricks has been moving toward the \u0026ldquo;serverless\u0026rdquo; model with Databricks SQL Serverless, and will probably extend this model to other products, such as notebooks. The pros and cons flip here: the pro is that users don\u0026rsquo;t have to worry about cluster settings; the con is that users have no access to or visibility into the underlying infrastructure and can\u0026rsquo;t customize the clusters to fit their needs.\nSince Databricks is currently in a \u0026ldquo;transition\u0026rdquo; period between classic and \u0026ldquo;serverless\u0026rdquo; offerings, its scalability really depends on the use case people choose.\nAn important note is that Databricks has a diverse set of compute use cases, from SQL warehouses, Jobs, All Purpose Compute and Delta Live Tables to streaming, each with slightly different compute settings and use cases. For example, SQL warehouses can be used as a shared resource, where multiple queries can be sent to the warehouse at any time by multiple users. Jobs are more singular: a notebook runs on a cluster and then shuts down (jobs can also be shared now, but that\u0026rsquo;s less common).\nBottom line: When it comes to scaling to large workflows, both Snowflake and Databricks can handle the workload. However, Databricks is better able to boost and tune performance on large data volumes, which ultimately saves money.\nDatabricks vs Snowflake: Cost Both Databricks and Snowflake are sold as pay-as-you-go models. In other words, the more compute you reserve/request, the more you pay. On both Databricks and Snowflake, users can and will pay for the resources they request, whether or not those resources are actually needed or ideal for running the job.\nAnother big difference between the two services is that Snowflake runs and charges for the entire compute engine (warehouses and cloud instances), while Databricks runs and charges only for managing the compute, so users still have to pay a separate bill to the cloud provider. It\u0026rsquo;s worth noting that Databricks\u0026rsquo; new serverless product mimics Snowflake\u0026rsquo;s operating model. Databricks works with compute/time units called Databricks Units (or DBUs) per second, and Snowflake uses a system of Snowflake credits.\nAs a formula, it breaks down like this:\nDatabricks (classic compute) = Data storage + Databricks service cost (DBUs) + Cloud compute cost (virtual machine instances) Snowflake = Data storage (average daily volume of bytes stored in Snowflake) + Compute (number of virtual warehouses used) Both Databricks and Snowflake offer pricing tiers and discounts based on company size, and both let you save money by buying units or credits up front.\nDatabricks has more price variation, since prices differ by workload type, with certain types of compute costing 5x more per compute hour than simple jobs.\nA big cost advantage Databricks has is that it lets users take advantage of Spot instances on their cloud provider, which can translate into significant savings. Snowflake hides all of this, and the end user has no way to benefit from Spot instances.\nBottom line: There\u0026rsquo;s no definitive answer as to which service is \u0026ldquo;cheaper\u0026rdquo;, because it really depends on how much of the service or platform you use and for what kinds of tasks. However, the control and introspection features Databricks provides are pretty much unmatched in the Snowflake ecosystem. That gives Databricks a significant advantage when optimizing for large compute workloads.\nDatabricks vs Snowflake: Ease of use All things being equal, Snowflake is widely considered the \u0026ldquo;easier\u0026rdquo; cloud solution of the two to learn. It has an intuitive SQL interface and, as a serverless experience, doesn\u0026rsquo;t require users to manage any virtual or on-premises hardware resources. In addition, as a managed service, using Snowflake requires no installation, maintenance, upgrades or fine-tuning of the platform. Everything is handled by Snowflake.\nSnowflake also has automated features such as auto-scaling and auto-suspend to help start and stop clusters without fine-tuning. Although Databricks also has autoscaling and autosuspend, it was designed for a more technical user, and there\u0026rsquo;s more involved in fine-tuning its clusters.\nBottom line: Although the Databricks user interface has a steeper learning curve than Snowflake\u0026rsquo;s, it offers more advanced control and customization, which makes this a trade-off that depends heavily on how complex you intend your operations to be.\nDatabricks vs. Snowflake: Ecosystem and Integration Databricks and Snowflake are becoming the abstractions on top of the Cloud Vendors for data compute workloads. As such, both connect to a wide range of vendors, tools and products.\nOn the vendor side, both Databricks and Snowflake provide Marketplaces that let other prevailing tools and technologies be co-deployed. There are also community-built and community-contributed resources, such as the Databricks Airflow Operators / Snowflake Airflow Operators.\nOverall, though, Databricks\u0026rsquo; ecosystem is typically more \u0026ldquo;open\u0026rdquo; than Snowflake\u0026rsquo;s, since Databricks still runs in a user\u0026rsquo;s cloud VPC. That means users can still install custom libraries or even introspect low-level cluster data. That access isn\u0026rsquo;t possible in Snowflake, so integrating with your favorite tools can be harder. Databricks also tends to be generally more developer/integration-friendly than Snowflake for exactly this reason.\nDatabricks vs Snowflake: Which one is better? Both Databricks and Snowflake have a solid reputation in the business and data community. Although both are cloud-based platforms, Snowflake is more optimized for data warehousing, data manipulation and querying, while Databricks is optimized for machine learning and heavy data science.\nBroken down by component, here\u0026rsquo;s a list of advantages for each:\n_Databricks vs Snowflake comparison_\nWith that, we conclude that if you want to integrate structured data into an existing ETL pipeline using structured data and tools such as Tableau, Looker and Power BI, Snowflake may be the right option for you. If, instead, you\u0026rsquo;re looking for a unified analytics workspace where you build compute pipelines, Databricks may be the right choice for you.\nThanks for reading! See you next time!\nReference Links https://docs.databricks.com/en/introduction/index.html#etl-and-data-engineering\nhttps://www.snowflake.com/en/data-cloud/snowpark/\nhttps://www.informatica.com/resources/articles/what-is-a-cloud-data-warehouse.html\nhttps://www.databricks.com/glossary/acid-transactions#:~:text=ACID%20is%20an%20acronym%20that,operations%20are%20called%20transactional%20systems.\nhttps://docs.snowflake.com/en/user-guide/warehouses-multicluster\nhttps://www.snowflake.com/blog/industry-benchmarks-and-competing-with-integrity/\nhttps://www.gartner.com/reviews/market/cloud-database-management-systems/vendor/snowflake/product/snowflake-data-cloud/alternatives\n","permalink":"https://julianamarialop.github.io/en/articles/clash-of-the-titans-databricks-vs-snowflake-in-the-mortal-data-battle/","summary":"Databricks and Snowflake are two big names in cloud data solutions. Both platforms have been key to helping companies generate value from their internal and external data assets. Each platform…","title":"Clash of the Titans: Databricks vs. Snowflake in the Mortal Data Battle"},{"content":"Azure Synapse is Microsoft\u0026rsquo;s Data Warehouse solution on the Azure cloud. However, it has been facing internal competition from Databricks for some time and, since last year, from Microsoft Fabric as well.\nMicrosoft Synapse and Databricks are well-known solutions for building a Data Warehouse or Data Lakehouse, both offered by Microsoft on its Azure cloud. Since last year, Microsoft has introduced Microsoft Fabric , which competes with both but is also integrated with them. Although Databricks is a more standalone solution, this article focuses more on whether Fabric is effectively the successor to Synapse.\nUsers and companies may now be asking themselves which service to choose. With the introduction of Fabric, it becomes increasingly important to understand the difference between it and Azure Synapse Analytics. Synapse is Microsoft\u0026rsquo;s cloud tool for data processing, well tested over the years. Constant improvements have brought countless features and bug fixes, offering many options for preparing data, and it is widely used to build ETL pipelines.\n_Synapse vs Fabric_\nFabric, on the other hand, aims to be the complete solution for data. Parts of Synapse are integrated into Fabric, with Azure Synapse Analytics features being gradually adopted. Both products coexist, and Fabric Synapse includes the creation of Data Lakehouses and Warehouses. Data storage, which is not provided in Azure Synapse Analytics, happens, for example, in an Azure Datalake. Technologically, in Fabric, storage is just an extension of an Azure Datalake, called OneLake, managed automatically by Fabric.\nNow you may be wondering why you would choose Azure Synapse when Fabric offers more features, and companies using Synapse may be asking whether they need a major migration to the Fabric project. There won\u0026rsquo;t be many updates and new features for Azure Synapse, while Microsoft Fabric, on the other hand, is the newer product, still has some features in preview, and is being developed quickly. Microsoft plans to ease the migration from Synapse to Fabric, making the move from Azure Synapse Analytics to Microsoft Fabric simpler in the future, which should reassure Synapse users.\nIn general, Fabric is the tool for the future and the right choice, especially for long-term planning. By standardizing multiple functions, Fabric offers the most comprehensive solution.\nSources and further reading Microsoft, Azure Synapse Analytics - https://azure.microsoft.com/en-us/products/synapse-analytics#:~:text=Azure%20Synapse%20Analytics%20is%20an,log%20and%20time%20series%20analytics.\nMicrosoft, Introducing Microsoft Fabric: Data analytics for the era of AI - https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/\nMicrosoft, Microsoft Fabric explained for existing Synapse users - https://blog.fabric.microsoft.com/en-us/blog/microsoft-fabric-explained-for-existing-synapse-users/\n","permalink":"https://julianamarialop.github.io/en/articles/data-analytics-on-scorched-earth-the-decline-of-azure-synapse/","summary":"Azure Synapse is Microsoft\u0026rsquo;s Data Warehouse solution on the Azure cloud. However, it has been facing internal competition from Databricks for some time and, since last year, from Microsoft Fabric as well.","title":"Data Analytics on Scorched Earth: The Decline of Azure Synapse?"},{"content":"Introduction Databricks Unity Catalog makes it easier for your team to understand its data. It automatically generates information about how data is used, based on cluster consumption logs. This reflects how customers use the Data Intelligence Platform, offering excellent discoverability and context for optimizations.\nIn this article I want to teach you how to use lineage data to audit data usage through the Unity Catalog System Tables. I\u0026rsquo;ll start with an overview of the lineage tables and then show how to analyze data usage to understand who is accessing which data, from where and when.\nOverview With Databricks Unity Catalog, we get lineage information as our users work. Dependency relationships are tracked while users prepare and organize data. Metadata is generated automatically, giving your data immediate context and meaning.\nThat context speeds up the work of key users:\nData analysts and data scientists: They can understand where data comes from directly, reducing confusion about which table or metric to use for a model or report. This speeds up delivery and increases trust in the data. Data governance: Owners, stewards, and security and privacy officers can use table access and usage information to understand how often data is used and track access patterns. Lineage is available in the Catalog interface, where it\u0026rsquo;s expressed as visual graphs showing dependency relationships. The image below shows the lineage for three views generated from the system tables. Table and column lineage are also available:\n_Unity Catalog lineage_\nWe can also query lineage directly in the system tables through Notebooks or the SQL Editor. We can query table lineage to see upstream/downstream tables and analyze data usage, such as who accesses the data, when and how. The example below shows every access to the billing usage table in the last seven days:\nSELECT * FROM system.access.table_lineage WHERE source_table_full_name = \u0026#39;system.billing.usage\u0026#39; AND datediff(now(), event_date) \u0026lt; 7 If you want a quick view of a table\u0026rsquo;s activity over the last 30 days, you can open the table insights directly in the user interface. The UC Insights tab provides information about recent activity, users, queries and more.\nOverview of the lineage tables The lineage system tables in Databricks let you query lineage data at the table and column level in the system.lineage.table_lineage and system.lineage.column_lineage tables.\nBoth lineage system tables follow the same schema, with two additions for column-level lineage, as shown below:\n_Table of Names and Columns_\nTo determine whether the event was a read or a write, you can check the source_type and target_type fields.\nRead only: The source type is not null, but the target type is null. Write only: The target type is not null, but the source type is null. Read and write: Both the source and target types are not null. In the rest of the article, I\u0026rsquo;ll explore some use cases highlighting data access auditing and data object popularity, demonstrate techniques for analyzing your lineage data, and we\u0026rsquo;ll build a dashboard you can customize to get insights.\nIn this article we have four sets of code: a notebook for exploring the lineage table data and three dashboards that show some of the notebook\u0026rsquo;s results. All of them can be used right away in any workspace where Unity Catalog and the system tables are enabled.\nData Exploration: Setting Up the Analysis At the beginning of the notebook, we set variables for running the subsequent queries. Please adjust the values to match your environment.\nDECLARE OR REPLACE VARIABLE catalog_val STRING; DECLARE OR REPLACE VARIABLE target_catalog_val STRING; DECLARE OR REPLACE VARIABLE schema_val STRING; DECLARE OR REPLACE VARIABLE table_val STRING; DECLARE OR REPLACE VARIABLE column_val STRING; DECLARE OR REPLACE VARIABLE email_val STRING; DECLARE OR REPLACE VARIABLE table_full_name_val STRING; SET VARIABLE catalog_val = \u0026#39;system\u0026#39;; SET VARIABLE target_catalog_val = \u0026#39;pdavis\u0026#39;; SET VARIABLE schema_val = \u0026#39;billing\u0026#39;; SET VARIABLE table_val = \u0026#39;usage\u0026#39;; SET VARIABLE column_val = \u0026#39;usage_quantity\u0026#39;; SET VARIABLE table_full_name_val = concat(catalog_val, \u0026#39;.\u0026#39;, schema_val, \u0026#39;.\u0026#39;, table_val) SET VARIABLE email_val = \u0026#39;demo@databricks.com\u0026#39;; Table Access by User: See which users accessed a table and which Databricks interface they used in the last 7 days:\nUsers: Identify the users who accessed the table. Interfaces: Check whether they used notebooks, jobs or other Databricks interfaces to access the table. This will help you understand how different users access and use tables over a one-week period.\nSELECT mask(created_by) as created_by, -- Masking function - nice feature entity_type, source_type, COUNT(distinct event_time) as access_count, MIN(event_date) as first_access_date, MAX(event_date) as last_access_date FROM system.access.table_lineage WHERE source_table_catalog = catalog_val AND source_table_schema = schema_val AND source_table_name = table_val AND datediff(now(), event_date) \u0026lt; 7 AND entity_type IS NOT NULL AND source_type IS NOT NULL GROUP BY ALL -- Nice feature ORDER BY ALL -- Nice feature Here we get a result with the email addresses (created_by), interfaces (entity_type), first and last access dates, and the table access count.\nI limited the total period to one week to minimize resource usage during exploration, and I use a masking function to hide the email addresses in the created_by field for this article.\nTable access within a specific catalog and schema:\nSELECT source_table_name, entity_type, created_by, source_type, COUNT(distinct event_time) as access_count, MIN(event_date) as first_access_date, MAX(event_date) as last_access_date FROM system.access.table_lineage WHERE source_table_catalog = catalog_val AND source_table_schema = schema_val AND datediff(now(), event_date) \u0026lt; 30 GROUP BY ALL ORDER BY ALL Here we see every table accessed within a specific catalog and schema in the last 30 days, who accessed it and the first and last access dates.\nTable Access by Specific Users Which tables a specific user accessed in the system catalog in the last 90 days:\nSELECT source_table_catalog, source_table_schema, source_table_name, entity_type, source_type, mask(created_by) as created_by, COUNT(distinct event_time) as access_count, MIN(event_date) as first_access_date, MAX(event_date) as last_access_date FROM system.access.table_lineage WHERE created_by = email_val AND datediff(now(), event_date) \u0026lt; 90 and entity_type is not NULL and source_table_catalog = \u0026#39;system\u0026#39; GROUP BY ALL ORDER BY ALL Object Lineage For a single object, which objects are immediately upstream (before it) and downstream (after it)?\nwith downstream AS ( select distinct target_table_catalog as table_catalog, target_table_schema as table_schema, target_table_name as table_name, \u0026#39;downstream\u0026#39; as direction, CASE WHEN tbl.table_catalog is null then \u0026#39;no\u0026#39; else \u0026#39;yes\u0026#39; end as current from system.access.table_lineage tl left join system.information_schema.tables tbl on tl.target_table_full_name = concat(tbl.table_catalog, \u0026#39;.\u0026#39;, tbl.table_schema, \u0026#39;.\u0026#39;, tbl.table_name) where source_table_full_name = table_full_name_val AND target_table_full_name is not null order by current desc, table_catalog, table_schema, table_name ) , upstream AS ( select distinct source_table_catalog as table_catalog, source_table_schema as table_schema, source_table_name as table_name, \u0026#39;upstream\u0026#39; as direction, CASE WHEN tbl.table_catalog is null then \u0026#39;no\u0026#39; else \u0026#39;yes\u0026#39; end as current from system.access.table_lineage tl left join system.information_schema.tables tbl on tl.source_table_full_name = concat(tbl.table_catalog, \u0026#39;.\u0026#39;, tbl.table_schema, \u0026#39;.\u0026#39;, tbl.table_name) where target_table_full_name = table_full_name_val AND source_table_full_name is not null order by current desc, table_catalog, table_schema, table_name ) select * from upstream UNION ALL select * from downstream Shall we put it all together? Dashboards The following dashboards will help you start analyzing the Unity Catalog lineage tables. Users are encouraged to try the dashboards and then customize them to whatever is most useful in their environment.\nWe have three dashboards with the following features:\nTable lineage Column lineage Tags and column lineage Table lineage Putting it all together, I built a dashboard to show:\nTable accesses by date Columns in use in the table Who accessed the table and through which interfaces Related upstream/downstream tables (including deprecated dependencies) By importing the dashboard and entering any three-level table name, you can review the same information about tables (as long as you have permission to see the table\u0026rsquo;s metadata).\n_Table Lineage Dashboard_\nColumn lineage I added a dashboard to review column lineage information. This dashboard shows:\nThe count of tables with the same column name How many objects depend on that column Total accesses to the column during the specified period A chart of accesses over time A table showing how often the column is used downstream A table showing who accessed the column, from which interface, how many accesses were made, and the first and last access dates within the period It also shows tables that have the same column name and their usage frequency counts.\n_Column Lineage Dashboard_\nTags and Lineage Dashboard Finally, I built a dashboard to review tags in the system:\nTag name and value heat map: Helps you understand how people use tags and reduce potential tag noise across the company. List of tagged column names: Shows the columns that were tagged with a given name or value (masked in this example). Total accesses to tagged columns: Shows how many times the tagged columns were accessed during the specified period. Lists of tagged Catalogs, Schemas and Tables: Displays the objects that were tagged (masked for the example). List of tagged columns accessed: Shows who accessed these columns and how (again, masked for the example). _Tags and Lineage Dashboard_\nCurrent Limitations The Lineage System Tables are currently in Public Preview. As the tables move toward GA, columns, schemas and retained data may change.\nNot every entity type is trackable at the moment; the entity type will be NULL for untracked types.\nConclusion In this article, we explored how Databricks Unity Catalog simplifies auditing and understanding data usage. Using the Unity Catalog System Tables, we can track and analyze data lineage. This helps us see clearly who is accessing which data, from where and when, which improves data governance and the efficiency of data analysts\u0026rsquo; and data scientists\u0026rsquo; work.\nWe showed how to query lineage directly in the system tables, either through Notebooks or the SQL Editor. This enables a detailed analysis of data access and usage. We also presented practical query examples for monitoring table access, data object popularity and access patterns.\nThe dashboards we showed are valuable tools for visualization and analysis, offering insights into table access, the use of specific columns and the use of tags in the system. Even while in Public Preview, the Lineage System Tables already provide significant support for data governance and process optimization on the Databricks platform.\nWith these capabilities, teams can make better-informed decisions, increase trust in their data and make better use of the available resources. Unity Catalog, with its lineage capabilities, is essential to any modern data management strategy.\nI hope this article helps you implement your governance!\nThanks for reading!\n","permalink":"https://julianamarialop.github.io/en/articles/demystifying-databricks-unity-catalog-simplifying-data-governance-and-auditing/","summary":"Databricks Unity Catalog makes it easier for your team to understand its data. It automatically generates information about how data is used, based on cluster consumption logs. This reflects how…","title":"Demystifying Databricks Unity Catalog: Simplifying Data Governance and Auditing"},{"content":"There are several ways to pause, resume, or scale a Microsoft Fabric Capacity. Note that this applies only to F-SKUs in Azure, since Premium Capacity (P-SKU) can only be scaled.\nYou can do this with a Fabric Data Factory pipeline, so you can orchestrate one or several capacities and workloads.\nThis assumes you already understand the concepts of Fabric capacities, workspaces, and workloads, and that you have good hands-on knowledge of Azure. You\u0026rsquo;ll also need a Fabric Capacity, ideally one that is always on.\nGeneral approach: Provision a Fabric Capacity in Azure Create a Service Principal in Azure Grant the appropriate permissions (RBAC) to the Service Principal Create a Fabric Data Factory pipeline that runs a web activity to post to the relevant API (not formally supported at the moment) This lets you build a Pipeline that can be called from anywhere in Fabric to pause, resume, or scale a Capacity.\nTo provision a capacity, choose Microsoft Fabric from the list of services and specify:\nSubscription and Resource Group Capacity Name and Region (note that this can be anywhere in the world, but it may be best to start where your Power BI tenant already lives (check under Help and Support in Power BI and look for \u0026ldquo;About Power BI\u0026rdquo;) Size (F2-F2048) A Fabric Capacity Administrator Service Principal: Create an SPN using whichever method you prefer. My favorite is the Azure CLI.\nPermissions: Make sure your service principal has contributor rights on the Fabric Capacity.\nCreate a Data Pipeline: Go to the Workspace in Microsoft Fabric where you plan to create the Pipeline. Create a new pipeline ManageCapacityPauseResume.\nAdd a few parameters to make your pipeline more flexible, and fill them in accordingly.\nAdd a Web Activity to the canvas. You\u0026rsquo;ll need to specify the service principal you created earlier.\nFor the Base URL, add https://management.azure.com, and for the token audience URI, add https://management.azure.com;https://management.core.windows.net/.\nFor the relative URL in the Web Activity, add the following expression:\n@concat(‘/subscriptions/’,pipeline().parameters.subscription_id,’/resourceGroups/’,pipeline().parameters.resourcegroup,’/providers/Microsoft.Fabric/capacities/’,pipeline().parameters.capacities,’/’,pipeline().parameters.action,’?api-version=2022–07–01-preview’) Note that some users have reported issues with the single quotes when pasting the expression above.\nChoose the POST method and put {} in the body.\nNotice the action parameter, which can be \u0026ldquo;suspend\u0026rdquo; or \u0026ldquo;resume\u0026rdquo; to pause or resume as needed.\nNow you can save and run the pipeline to test it. If your capacity is running, run a \u0026ldquo;suspend\u0026rdquo;. Notice how the Azure portal shows the capacity as paused.\nNote that if the capacity isn\u0026rsquo;t running and you try to pause it, the activity will fail. To deal with that, you can handle that specific error and fail the pipeline only if it occurs, as in the image below.\nTo scale a capacity, use a pipeline similar to the one described above. Change the expression in the Web Activity to:\n@concat(‘/subscriptions/’,pipeline().parameters.subscription_id,’/resourceGroups/’,pipeline().parameters.resourcegroup,’/providers/Microsoft.Fabric/capacities/’,pipeline().parameters.capacities,’?api-version=2022–07–01-preview’) Use the PATCH method.\nYou\u0026rsquo;ll also need to add a new parameter for the SKU value.\nChange the Body to:\n@json(concat(‘{“sku”:{“name”:”’,pipeline().parameters.sku,’”,”tier”:”Fabric”}}’)) Additional Topics You can pass variables and arrays between pipelines, as well as schedule them. This means that if you have development workspaces that live on a specific capacity or capacities, you can schedule the pause pipeline to run daily at a set time to save costs, or to scale capacities as needed.\nFinal Thoughts Although the APIs used here for Fabric capacities are not yet formally documented at the time of writing, using Fabric pipelines is a useful orchestration alternative to the more common use of Logic Apps for this purpose. It means you can use a Fabric SaaS solution to manage Fabric capacities.\n","permalink":"https://julianamarialop.github.io/en/articles/microsoft-fabric-pause-resume-scale-capacity-with-data-factory-pipeline/","summary":"There are several ways to pause, resume, or scale a Microsoft Fabric Capacity. Note that this applies only to F-SKUs in Azure, since Premium Capacity (P-SKU) can only be scaled.","title":"Microsoft Fabric: Pause, Resume, and Scale a Capacity with a Fabric Data Factory Pipeline"},{"content":"Microsoft Fabric is the complete solution for all of your company\u0026rsquo;s analytics needs, unifying the whole process in a single platform. From ingesting data into the data lake to delivering information to business users, everything can be done in Microsoft Fabric. With the ability to integrate data from many sources, such as Amazon AWS and Azure, it brings all the information together in one place.\nAs a SaaS (Software as a Service), Microsoft Fabric combines the power of Power BI with many capabilities from Synapse and other areas of Azure, with no need to install or configure software.\nOne of the main features of Microsoft Fabric is the separation of storage and compute, which lets different compute workloads run over the same datasets. OneLake is the storage layer built on ADLS (Azure Data Lake Storage) Gen2, providing a single place for an organization to store all of its data, both structured and unstructured. This eliminates data silos and simplifies security, governance and data discovery, allowing every user and application to access the data they need.\nData Mesh is a domain-oriented decentralization of data access. The Data Mesh architecture shifts responsibility for data access from technology teams to business teams.\nThe four principles of Data Mesh are:\nDomain ownership: Domain teams take responsibility for their data, owning both analytical and operational data. Data as a product: Each team defines not only the data it owns, but also the data it produces and consumes from other teams. Domain teams are responsible for providing high-quality data that meets the needs of other domains. Self-service data infrastructure: A dedicated team provides domain-agnostic capabilities, tools and systems, enabling domain teams to consume and create data products in an integrated way. Federated data governance: Promotes standardization and interoperability of all data products across the entire Data Mesh, creating a data ecosystem that follows organizational rules and industry regulations. Adopting the Data Mesh pattern lets business groups work independently with multiple data lakes organized around business domains.\nData is organized by domain and control sits directly over the data, providing easy access and complete governance at the data level rather than at the application level.\nEnabling Data Mesh with OneLake in Microsoft Fabric OneLake offers a true data mesh as a service.\nAn organization can have many data domains (SALES, FINANCE, MARKETING, HUMAN RESOURCES, etc.). A single dataset can be used across different domains, clouds and engines. A single data product can span multiple domains. \u0026ldquo;Shortcuts\u0026rdquo; provide the connections between domains, allowing data to be virtualized into a single data product.\nEvery engine can access the same data without needing to import or export it. You can choose the right engine for the right job. The engines work with data optimized in Delta Parquet as the native format.\nThat\u0026rsquo;s all for today, folks! Thanks for reading!\n","permalink":"https://julianamarialop.github.io/en/articles/exploring-new-horizons-transform-your-organization-data-mesh-fabric/","summary":"Microsoft Fabric is the complete solution for all of your company\u0026rsquo;s analytics needs, unifying the whole process in a single platform. From ingesting data into the data lake to delivering information to…","title":"Exploring New Horizons: Transform Your Organization with Data Mesh on Microsoft Fabric"},{"content":"This article reflects my personal experiences and points of view, not the official position of Microsoft or Databricks. In addition, although this post describes potential scenarios, it does not necessarily reflect Fabric\u0026rsquo;s roadmap or intentions. Not every option mentioned may become operational in the future.\nIn the dynamic, ever-evolving technology landscape, integrating different tools and platforms becomes essential to maximize efficiency and innovation. In the article \u0026ldquo;Connected Future: Unity Catalog and Microsoft Fabric Working Together,\u0026rdquo; we\u0026rsquo;ll explore how combining Unity Catalog, a robust data management solution from Databricks, with Microsoft Fabric, a powerful cloud and data application platform from Microsoft, is transforming the way organizations manage, analyze, and use their data. This collaboration promises not only to optimize data storage and governance, but also to speed up the development of insights and strategic decision-making, creating a more connected and intelligent future for companies in every industry.\nThe integration scenarios can essentially be viewed based on the entry point, Unity Catalog and Fabric:\nAccessing Unity Catalog from Fabric (Fabric → Unity Catalog): This capability could let users seamlessly access Unity Catalog\u0026rsquo;s catalog, schemas, and tables from within Fabric.\nUsing Fabric from Unity Catalog (DBX/Unity Catalog → Fabric): This capability could give users the ability to access and use OneLake directly from Unity Catalog and run federated queries against a SQL endpoint or the Fabric Data Warehouse.\nLet\u0026rsquo;s explore these scenarios in more detail.\nFabric → Unity Catalog Using Unity Catalog from Fabric\nIf you\u0026rsquo;re in Fabric, here are some options for accessing Unity Catalog tables from Fabric. You can also read/write directly from Fabric Spark to ADLS Gen2.\nCurrent options\nToday, users have two options for creating shortcuts to Unity Catalog tables: manual or semi-automatic, the latter being achievable through a notebook. With the semi-automatic method, users can bring external UC Delta tables into OneLake by creating shortcuts. They specify the catalog and schema names to sync, which generates shortcuts for the tables in those schemas inside the Fabric lakehouse.\nSee additional instructions on running the utility notebook.\n# configuration dbx_workspace = \u0026#34;\u0026lt;databricks_workspace_url\u0026gt;\u0026#34; dbx_token = \u0026#34;\u0026lt;pat_token\u0026gt;\u0026#34; dbx_uc_catalog = \u0026#34;catalog_example\u0026#34; dbx_uc_schemas = \u0026#39;[\u0026#34;schema1\u0026#34;, \u0026#34;schema2\u0026#34;]\u0026#39; fab_workspace_id = \u0026#34;\u0026lt;workspace_id\u0026gt;\u0026#34; fab_lakehouse_id = \u0026#34;\u0026lt;lakehouse_id\u0026gt;\u0026#34; fab_shortcut_connection_id = \u0026#34;\u0026lt;connection_id\u0026gt;\u0026#34; fab_consider_dbx_uc_table_changes = True # sync UC tables to lakehouse sc.addPyFile(\u0026#39;https://raw.githubusercontent.com/microsoft/fabric-samples/main/docs-samples/onelake/unity-catalog/util.py\u0026#39;) from util import * databricks_config = { \u0026#39;dbx_workspace\u0026#39;: dbx_workspace, \u0026#39;dbx_token\u0026#39;: dbx_token, \u0026#39;dbx_uc_catalog\u0026#39;: dbx_uc_catalog, \u0026#39;dbx_uc_schemas\u0026#39;: json.loads(dbx_uc_schemas) } fabric_config = { \u0026#39;workspace_id\u0026#39;: fab_workspace_id, \u0026#39;lakehouse_id\u0026#39;: fab_lakehouse_id, \u0026#39;shortcut_connection_id\u0026#39;: fab_shortcut_connection_id, \u0026#34;consider_dbx_uc_table_changes\u0026#34;: fab_consider_dbx_uc_table_changes } sync_dbx_uc_tables_to_onelake(databricks_config, fabric_config) Potential future options\nNative Unity Catalog item in Fabric: similar to migrating Hive Metastore metadata to the Fabric lakehouse, Unity Catalog metadata could be synced with the Fabric lakehouse, enabling access to Unity Catalog tables. A preview of this scenario was demonstrated at FabCon, showing how users could directly access and query Unity Catalog tables using the Fabric interface.\nUnity Catalog shortcut in Fabric: similar to Dataverse shortcuts, the OneLake shortcut user experience could potentially support creating shortcuts to Unity Catalog tables.\nDatabricks Unity Catalog → Fabric Using Fabric and OneLake from Databricks Unity Catalog\nUnity Catalog offers different ways to connect to and take advantage of cloud object storage connections (for example, ADLS Gen2), as well as to connect to external data systems to run federated queries (for example, Azure Synapse).\nCurrent options\nToday, users can use OneLake from Unity Catalog-enabled clusters as follows: (i) read/write to OneLake using Service Principal (SPN) based authentication, and (ii) read/write to OneLake using mount points with SPN authentication.\n# r/w using spn workspace_name = \u0026#34;\u0026lt;workspace_name\u0026gt;\u0026#34; lakehouse_name = \u0026#34;\u0026lt;lakehouse_name\u0026gt;\u0026#34; tenant_id = \u0026#34;\u0026lt;tenant_id\u0026gt;\u0026#34; service_principal_id = \u0026#34;\u0026lt;service_principal_id\u0026gt;\u0026#34; service_principal_password = \u0026#34;\u0026lt;service_principal_password\u0026gt;\u0026#34; spark.conf.set(\u0026#34;fs.azure.account.auth.type\u0026#34;, \u0026#34;OAuth\u0026#34;) spark.conf.set(\u0026#34;fs.azure.account.oauth.provider.type\u0026#34;, \u0026#34;org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider\u0026#34;) spark.conf.set(\u0026#34;fs.azure.account.oauth2.client.id\u0026#34;, service_principal_id) spark.conf.set(\u0026#34;fs.azure.account.oauth2.client.secret\u0026#34;, service_principal_password) spark.conf.set(\u0026#34;fs.azure.account.oauth2.client.endpoint\u0026#34;, f\u0026#34;https://login.microsoftonline.com/{tenant_id}/oauth2/token\u0026#34;) # read df = spark.read.format(\u0026#34;parquet\u0026#34;).load(f\u0026#34;abfss://{workspace_name}@onelake.dfs.fabric.microsoft.com/{lakehouse_name}.Lakehouse/Files/data\u0026#34;) df.show(10) # write df.write.format(\u0026#34;delta\u0026#34;).mode(\u0026#34;overwrite\u0026#34;).save(f\u0026#34;abfss://{workspace_name}@onelake.dfs.fabric.microsoft.com/{lakehouse_name}.Lakehouse/Tables/dbx_delta_spn\u0026#34;) # mount with spn workspace_id = \u0026#34;\u0026lt;workspace_id\u0026gt;\u0026#34; lakehouse_id = \u0026#34;\u0026lt;lakehouse_id\u0026gt;\u0026#34; tenant_id = \u0026#34;\u0026lt;tenant_id\u0026gt;\u0026#34; service_principal_id = \u0026#34;\u0026lt;service_principal_id\u0026gt;\u0026#34; service_principal_password = \u0026#34;\u0026lt;service_principal_password\u0026gt;\u0026#34; configs = { \u0026#34;fs.azure.account.auth.type\u0026#34;: \u0026#34;OAuth\u0026#34;, \u0026#34;fs.azure.account.oauth.provider.type\u0026#34;: \u0026#34;org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider\u0026#34;, \u0026#34;fs.azure.account.oauth2.client.id\u0026#34;: service_principal_id, \u0026#34;fs.azure.account.oauth2.client.secret\u0026#34;: service_principal_password, \u0026#34;fs.azure.account.oauth2.client.endpoint\u0026#34;: f\u0026#34;https://login.microsoftonline.com/{tenant_id}/oauth2/token\u0026#34; } mount_point = \u0026#34;/mnt/onelake-fabric\u0026#34; dbutils.fs.mount( source = f\u0026#34;abfss://{workspace_id}@onelake.dfs.fabric.microsoft.com\u0026#34;, mount_point = mount_point, extra_configs = configs ) # read df = spark.read.format(\u0026#34;parquet\u0026#34;).load(f\u0026#34;/mnt/onelake-fabric/{lakehouse_id}/Files/data\u0026#34;) df.show(10) # write df.write.format(\u0026#34;delta\u0026#34;).mode(\u0026#34;overwrite\u0026#34;).save(f\u0026#34;/mnt/onelake-fabric/{lakehouse_id}/Tables/dbx_delta_mount_spn\u0026#34;) Note: Creating an external table using the OneLake abfss path or the mount path will currently result in an exception in Unity Catalog. Right now, you can\u0026rsquo;t register an external table in Unity Catalog with OneLake as the underlying storage. This could lead to potential future scenarios.\nINVALID_PARAMETER_VALUE: Missing cloud file system scheme\nFailed to acquire a SAS token for list. Invalid Azure Path\nPotential future options\nSimilar to ADLS Gen2 and Azure Synapse, different options could exist in the future:\nOneLake as the default managed storage: Databricks has started rolling out automatic Unity Catalog enablement, meaning a Unity Catalog metastore provisioned automatically with Databricks-managed storage (for example, ADLS Gen2). However, users can also create user-managed storage at the metastore level when creating the Unity Catalog metastore, pointing to OneLake in this case. That isn\u0026rsquo;t possible yet.\nOneLake as an external location: External locations are used to define managed storage locations for catalogs and schemas, and to define locations for external tables and external volumes. For example, if users are working with external tables in Spark, OneLake could be used as an external location.\nOneLake for Volumes: Volumes represent a logical volume of storage in a cloud object storage location, adding governance over non-tabular datasets. External and managed volumes could exist using OneLake, for example, just as they do for ADLS Gen2.\nFederated Lakehouse: Read-only access to data in a SQL endpoint or the Fabric Data Warehouse using Unity Catalog foreign catalogs could be a future option. Current Azure Synapse and SQL authentication is based on username/password and SPN isn\u0026rsquo;t supported yet, so this option isn\u0026rsquo;t possible yet. Foreign catalogs don\u0026rsquo;t support object storage right now, so it\u0026rsquo;s still unclear whether a foreign catalog for OneLake/Lakehouse will be possible.\n","permalink":"https://julianamarialop.github.io/en/articles/connected-future-unity-catalog-and-microsoft-fabric-working-together/","summary":"This article reflects my personal experiences and points of view, not the official position of Microsoft or Databricks. In addition, although this post describes potential scenarios, it does not necessarily reflect the…","title":"Connected Future: Unity Catalog and Microsoft Fabric Working Together"},{"content":"We\u0026rsquo;ve reached the end of the \u0026ldquo;Integrating Azure Databricks and Microsoft Fabric\u0026rdquo; series. Throughout this journey, we explored how these two powerful tools can be combined to optimize data analytics, storage, and visualization processes. If you haven\u0026rsquo;t read the previous articles yet, start here.\nThe next design consideration focuses on the importance of using Microsoft Fabric and the V-Order feature. This feature is a write-time optimization for the parquet file format that enables fast data reads in Microsoft Fabric compute engines such as Power BI.\nBoth Databricks and Microsoft chose to adopt Delta Lake, an open-source columnar file format. However, Microsoft added an extra V-Order compression layer that delivers up to 50% more compression. V-Order is fully compatible with the open-source parquet format; every parquet engine can read it as regular parquet files.\nIt\u0026rsquo;s worth noting that you can apply V-Ordering to tables that don\u0026rsquo;t have it by using Fabric\u0026rsquo;s maintenance feature.\nV-Order brings notable benefits to Microsoft Fabric, especially for components like Power BI and SQL endpoints. For example, it lets Power BI connect directly to real-time data using Direct Lake mode while keeping data queries highly performant. Since there\u0026rsquo;s no import process, changes in the data source show up in Power BI instantly, with no waiting for a refresh.\nIt\u0026rsquo;s essential to point out that using V-Order optimized tables is, for now, exclusive to Microsoft Fabric. Databricks hasn\u0026rsquo;t implemented this feature yet. So, until it does, you\u0026rsquo;ll need to use a service inside Microsoft Fabric to take advantage of V-Order optimized tables.\nIt\u0026rsquo;s also worth noting that, if V-Order optimization isn\u0026rsquo;t essential, the Databricks processing step between the Silver and Gold stages can still be relevant. Although this may seem redundant, it\u0026rsquo;s a viable option for continuing to process data with Databricks.\nAnother significant reason organizations choose this design is transactional consistency across multiple tables. Maintaining that consistency, especially in the Gold stage, is crucial. Currently, Spark only supports transactions on individual tables. So, if there are data inconsistencies across tables, they have to be resolved through compensating measures. For example, you can commit inserts to several tables, or to none of them if an error occurs. If you\u0026rsquo;re changing the details of a purchase order that affect three tables, you can group those changes into a single transaction. That means that when you query those tables, they\u0026rsquo;ll have all the changes or none of them. This concern for integrity highlights the importance of an environment that can manage complex transactions across multiple tables. Microsoft Fabric Warehouse is the only platform that supports this on top of Delta Lake. To learn more, click here.\nIn the updated architecture, shown in the image above, Synapse Engineering now acts as the processing engine from the Silver stage to Gold. This approach ensures that all tables are V-Order optimized. In addition, Synapse Warehouse was added for use cases that require transactional capabilities. However, these architectural changes require data engineers to navigate across different data processing services. So it\u0026rsquo;s essential to provide clear guidance to every team. For example, you can set guidelines for the Bronze and Silver stages using native Databricks features, such as ingestion tracking with AutoLoader and validations with Delta Live Tables to ensure data quality. And, for the Gold stage, focus on building integration logic specific to consumption, exclusively with Microsoft Fabric.\nConclusion Integrating Azure Databricks with Microsoft Fabric offers organizations a wide range of benefits and possibilities. Combining the flexibility and scalability of Azure Databricks with the simplicity and intuitive features of Microsoft Fabric can considerably improve how data is used and managed across every layer. There are several architectural design options, from enhancing a Databricks-centric architecture with a Microsoft Fabric layer to incorporating a OneLake gold layer into the architecture to improve performance and security.\nIn addition, the introduction of V-Order optimization in Microsoft Fabric and the use of additional components can significantly simplify data processing and increase its efficiency. However, these combinations or integrations require careful consideration, since they may involve moving between services and balancing flexibility, data security, and isolation.\nAnd that wraps up our series of articles exploring the integration between Databricks and Microsoft Fabric! Thanks for reading!\n","permalink":"https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-4-v-order/","summary":"We\u0026rsquo;ve reached the end of the \u0026ldquo;Integrating Azure Databricks and Microsoft Fabric\u0026rdquo; series. Throughout this journey, we explored how these two powerful tools can be combined to optimize data analytics processes…","title":"Integrating Azure Databricks and Microsoft Fabric: A Meeting of Giants! - Part IV - Extending with V-ORDERED"},{"content":"Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.\nWe\u0026rsquo;ve reached the third part of our series on integrating two of the most powerful data tools in the Microsoft world: Microsoft Fabric and Databricks. In Part I, we explored the foundations of this integration and covered the basic concepts.\nIf you haven\u0026rsquo;t read the first and second parts yet, I strongly recommend doing so before moving on, because many of the concepts covered here build on what was discussed earlier. Get ready to take your skills and knowledge to a new level and make the most of the combined power of Azure Databricks and Microsoft Fabric. Here are the links: Part I and Part II\nThe second architecture shown below modifies the design pattern referenced in the previous article of this series by adding a gold layer in OneLake to the architecture. This is possible thanks to the Azure Blob Filesystem (ABFS) driver in Azure Databricks, which supports both ADLS and OneLake.\nWithin this architecture, the workflow and the data processing steps (ingestion, processing, validation and enrichment) remain essentially the same, all managed by Azure Databricks. The main change is that the data for consumption is now more tightly integrated with Microsoft Fabric, because Databricks writes the data to a Gold layer stored in OneLake. You may wonder whether this is a recommended practice and what the benefits are.\nIt\u0026rsquo;s crucial to mention that this form of integration is not officially supported by Databricks, which has some implications for data management, covered below.\nDatabricks distinguishes between two types of tables: managed tables and external tables. Managed tables are created by default and administered by Unity Catalog, which manages their lifecycle and file layout. Manipulating the files of these tables directly with external tools is not recommended. External tables, on the other hand, store data outside the managed storage location specified for the metastore, catalog or schema.\nAccording to the documentation, all tables created by writing directly to OneLake must be classified as external tables, since the data is managed outside the scope of the metastore. As a result, these tables must be administered somewhere else, such as inside Fabric. The motivation for this approach may include:\nFirst, storing data in OneLake can improve performance within Microsoft Fabric. OneLake tables are optimized for performance, especially for queries involving joins and aggregations. By contrast, queries that read data from ADLS Gen2 through shortcuts may perform more slowly.\nSecond, managing data in OneLake makes it easier to apply security measures within Microsoft Fabric. For example, OneLake tables can be protected with role-based access control (RBAC), simplifying data access management. If ADLS Gen2 were used, you would have to deal with the permissions of the ADLS Gen2 storage account, a more complex task.\nThird, OneLake tables can be governed by policies, making compliant use easier. This is an advantage when (externally) sharing tables with domains located elsewhere.\nBeyond just reading data, it may be worth considering generating new data within Microsoft Fabric. An upcoming feature may draw attention: soon, Fabric users will be able to access data items, such as lakehouses, through Unity Catalog in Azure Databricks. Although the data stays in OneLake, you\u0026rsquo;ll be able to access and view its lineage and other metadata directly in Azure Databricks. This improvement will make it easier to read Fabric data from Databricks. For example, if you plan to use Azure Databricks\u0026rsquo; Mosaic AI, you\u0026rsquo;ll be able to do so by reading data from Microsoft Fabric. The likely technology for this is Lakehouse Federation.\nIn conclusion, the strategy of integrating and processing all data within Databricks, while the consumption layer is managed in Fabric, gives organizations the convenience of using the best features of each application. This approach ensures optimal performance and security when handling data.\nSee you in the next and final article of the series!!\nSubscribe to the newsletter to follow this journey.. See you soon!\n","permalink":"https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-iii-onelake-gold-layer/","summary":"Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.","title":"Integrating Azure Databricks and Microsoft Fabric: A Meeting of Giants! - Part III - Adding the OneLake Gold Layer"},{"content":"Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.\nWelcome to the second part of our series on integrating two of the most powerful data tools in the Microsoft ecosystem: Microsoft Fabric and Databricks. In Part I, we explored the foundations of this integration and covered the basic concepts. Now, in this Part II, we\u0026rsquo;re going to dig even deeper.\nIf you haven\u0026rsquo;t read the first part yet, I strongly recommend doing so before moving on, since many of the concepts covered here build on what we discussed earlier. Get ready to take your skills and knowledge to a new level and make the most of the combined power of Azure Databricks and Microsoft Fabric.\nAn effective way to improve an Azure Databricks based architecture is to add a reporting and analytics layer. The traditional Azure Databricks Medallion Lakehouse architecture uses services such as Azure Data Lake Storage (ADLS) Gen2, Azure Data Factory, and Azure Databricks itself for end-to-end management of data ingestion, processing, validation, and enrichment. PowerBI is usually used for reporting and delivering analytical insights.\nExpanding this architecture to include Microsoft Fabric can significantly enhance self-service capabilities and improve the experience for business users. Think of Microsoft Fabric as an evolution of PowerBI, offering a new set of features that make data analysis more engaging and efficient.\n_Reference architecture_\nNew Microsoft Fabric Features Microsoft recently introduced the \u0026ldquo;shortcut\u0026rdquo; feature in Microsoft Fabric. This feature acts as a lightweight data virtualization mechanism, letting you read data from multiple sources without duplicating it. For example, when using PowerBI, you can access the data directly, without having to copy it or import it into PowerBI.\nFor Databricks-based architectures, we can use the ADLS Gen2 shortcut feature, since Databricks writes all of its data to ADLS. However, there are a few important considerations:\nFabric Lakehouse: Shortcuts require a Fabric Lakehouse. Make sure to create one if you don\u0026rsquo;t have it yet.\nDelta Lake Format: Shortcuts only work with tables in Delta Lake format.\nExternal Tables: Use shortcuts on external tables whenever possible, instead of tables managed by Databricks.\nFolder Limitation: Each shortcut can only reference a single Delta folder. So, to access data from multiple folders, create individual shortcuts for each one.\nRead-Only Access: Use a read-only approach to access Delta files in ADLS, avoiding direct manipulation of files in those directories.\nManual Shortcut Creation: Shortcuts can be created manually through the Fabric interface or programmatically using the REST API.\nAdvanced Integration with Unity Catalog To make the integration between Databricks and Microsoft Fabric even easier, Microsoft announced exciting developments at the Microsoft Build 2024 Conference. Soon it will be possible to integrate Azure Databricks Unity Catalog with Fabric. From the Fabric portal, you\u0026rsquo;ll be able to create and configure a new Unity Catalog item, and all managed tables can be upgraded to shortcuts. This integration will dramatically simplify bringing Azure Databricks data together in Fabric, enabling seamless operations across all Fabric workloads.\nSo, the expanded architecture combining Databricks with Microsoft Fabric is a popular choice among customers who are happy with Databricks. These organizations have already invested significantly in building a Lakehouse with Databricks and plan to keep using it. Microsoft Fabric recognizes the strength and versatility of the Lakehouse approach with the Delta format, allowing an optimized data consumption layer to be added on top. This lets organizations augment their existing Databricks-centric setup with an additional layer specifically designed to make data consumption easier.\nThat\u0026rsquo;s all for today! Part III is coming soon, where we\u0026rsquo;ll look at how, in a Databricks-enabled architecture, we can incorporate a OneLake gold layer. See you then!!\n","permalink":"https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-part-2-reporting-layer/","summary":"Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.","title":"Integrating Azure Databricks and Microsoft Fabric: A Meeting of Giants! - Part II - Adding a Reporting and Analytics Layer"},{"content":"Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks .\nIn this first article, we\u0026rsquo;ll tackle a crucial topic that often comes up in conversations with our clients: integrating Microsoft Azure Databricks with Microsoft Fabric. Both platforms are leaders in their respective fields. Microsoft Azure Databricks is outstanding at scaling data engineering, data science and machine learning workloads. Microsoft Fabric, in turn, stands out for its simplicity and self-service features, making it ideal for a wide range of data applications. The question we\u0026rsquo;re often asked is: how do you integrate these two powerful tools efficiently?\nIntegration Approaches Today there are several options to consider for this integration (keeping in mind that Microsoft Fabric is constantly being updated, which means we\u0026rsquo;ll soon have even more possibilities).\nAdding Reporting and Analytics Layers One way to improve a Databricks-enabled architecture is by adding a reporting and analytics layer. This enables more effective visualization of the processed data.\nIntegration with OneLake Another approach is to complement the architecture by incorporating a OneLake gold layer. We can also consider having Microsoft Azure Databricks write all of its data directly to OneLake. Although this practice isn\u0026rsquo;t generally recommended, it\u0026rsquo;s important to discuss its aspects to understand when it can be useful.\nExtending with V-ORDERED We can also extend Microsoft Azure Databricks by adding a V-ORDERED-enabled consumption layer. This can improve data processing efficiency in both Microsoft Azure Databricks and Microsoft Fabric, adding extra components that can make a difference in performance.\nExploring the Options In the upcoming articles of this series, we\u0026rsquo;ll explore the options available today in detail. We\u0026rsquo;ll look at the nuances, weigh the pros and cons and go through the relevant documentation. Before that, it\u0026rsquo;s essential to understand why so many organizations choose to use these two powerful tools together.\nLet\u0026rsquo;s dive into these strategies and find out how to get the most out of combining Microsoft Azure Databricks and Microsoft Fabric.\nBut before wrapping up, I\u0026rsquo;d like to answer the following question: Why combine the two platforms? Organizations choose to integrate Microsoft Azure Databricks and Microsoft Fabric because of the unique benefits this combination delivers.\nMicrosoft Azure Databricks is a powerful and versatile platform designed for data processing, analytics and data science. Widely adopted by companies of all sizes, Microsoft Azure Databricks, created by the developers of Spark, is highly valued for its ability to manage complex workloads at large scale. With a long track record of success, it has established itself as a reliable and efficient tool for data engineers.\nMicrosoft Fabric, on the other hand, stands out for its simplicity and ease of use. Launched in 2023 and evolved from Microsoft Power BI, it offers a smooth transition for users who are used to Microsoft Power BI. Its intuitive interface, built-in self-service features and seamless integration with Microsoft Office 365 make Microsoft Fabric especially attractive for the corporate environment. Designed to democratize access to data and remove barriers to use, it presents itself as an accessible and efficient solution.\nSo combining Microsoft Azure Databricks with Microsoft Fabric offers a robust, complete solution that meets both the technical and the business needs of organizations, which explains its growing popularity.\nNow that we understand why this combination is so often chosen, let\u0026rsquo;s get to the main question: how do you integrate these two services effectively?\nSee you in the next article of the series!!\nSubscribe to the newsletter to follow this journey. See you soon!\n","permalink":"https://julianamarialop.github.io/en/articles/integrating-azure-databricks-and-microsoft-fabric-a-meeting-of-giants-part-1/","summary":"Disclaimer: this article reflects my personal experiences and views, not an official position of Microsoft Fabric or Databricks.","title":"Integrating Azure Databricks and Microsoft Fabric: A Meeting of Giants! - Part I"},{"content":"Today\u0026rsquo;s article is about a subject I love and that never goes out of style. SQL!!\nIf you\u0026rsquo;ve been involved in Data Analysis for a while, you\u0026rsquo;re probably already familiar with basic commands such as SELECT, INSERT, UPDATE and DELETE, etc. Although they are the most widely used, it\u0026rsquo;s also good to know the queries below for deeper data analysis.\n1. Window Functions Window functions are used for calculations over a set of rows related to the current row. Let\u0026rsquo;s look at an example where we calculate a running total of sales using the SUM( ) function with the OVER( ) clause. Imagine we have a sales data table called ‘Sales_Data’ that records sales amounts on various dates. We want to calculate the running total of sales for each date, meaning the total sales up to and including each date.\nSELECT Date, Sales_Amount, SUM(Sales_Amount) OVER (ORDER BY Date) AS Running_Total FROM Sales_Data; In this example, the SUM(Sales_Amount) OVER (ORDER BY Date) function calculates the running total of sales up to the current date for each row in the \u0026lsquo;Sales_Data\u0026rsquo; table.\nWindow functions can be used for many tasks, such as calculating running totals, moving averages, rankings and much more, without collapsing the result set into a single row per group.\n2. Common Table Expressions (CTEs) CTEs (Common Table Expressions) provide a way to create temporary result sets that can be referenced within a query. They improve readability and simplify complex queries. Here\u0026rsquo;s how we can use a CTE to calculate the total revenue for each product category.\nWITH category_revenue AS ( SELECT category, SUM(revenue) AS total_revenue FROM sales GROUP BY category ) SELECT * FROM category_revenue; The query defines a CTE called ‘category_revenue’. It calculates the total revenue for each category by summing the revenue from the sales table and grouping the results by the category column. The main query selects all columns from the ‘category_revenue’ CTE, effectively displaying the calculated total revenue for each category.\n3. Recursive Queries Recursive queries let you traverse hierarchical data structures, such as org charts or bills of materials. Suppose we have a table representing employee relationships and we want to find all the subordinates of a given manager.\nWITH RECURSIVE subordinates AS ( SELECT employee_id, name, manager_id FROM employees WHERE manager_id = \u0026#39;manager_id_of_interest\u0026#39; UNION ALL SELECT e.employee_id, e.name, e.manager_id FROM employees e JOIN subordinates s ON e.manager_id = s.employee_id ) SELECT * FROM subordinates; This recursive CTE finds every employee who reports directly or indirectly to a specific manager, \u0026lsquo;manager_id_of_interest\u0026rsquo;. It starts with the employees who report directly to the manager and then recursively finds their subordinates, building up the hierarchy.\n4. Pivot Tables Pivot tables turn rows into columns, summarizing data in a tabular format. Let\u0026rsquo;s say we have a table containing sales data and we want to pivot the data to show the total sales of each product across different months.\nSELECT product, SUM(CASE WHEN month = \u0026#39;Jan\u0026#39; THEN sales ELSE 0 END) AS Jan, SUM(CASE WHEN month = \u0026#39;Feb\u0026#39; THEN sales ELSE 0 END) AS Feb, SUM(CASE WHEN month = \u0026#39;Mar\u0026#39; THEN sales ELSE 0 END) AS Mar FROM sales_data GROUP BY product; This query aggregates the sales data for each product by month using conditional aggregation. It sums the sales for January, February and March separately for each product, producing a table that shows total sales per product for those months.\n5.Analytic Functions Analytic functions calculate aggregate values based on a group of rows. For example, we can use the ROW_NUMBER() function to assign a unique row number to each record in a dataset.\nSELECT customer_id, order_id, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date) AS order_rank FROM orders; This query assigns a unique rank to each order per customer based on the order date, using the ROW_NUMBER() window function. The result shows the sequence of orders placed by each customer.\n6. Unpivot Unpivoting is the opposite of pivoting: columns are turned into rows. Let\u0026rsquo;s say we have a table with sales data aggregated by month and we want to \u0026ldquo;unpivot\u0026rdquo; it to analyze trends over time.\nSELECT product, month, sales FROM sales_data UNPIVOT ( sales FOR month IN (sales_jan AS \u0026#39;Jan\u0026#39;, sales_feb AS \u0026#39;Feb\u0026#39;, sales_mar AS \u0026#39;Mar\u0026#39;) ) AS unpivoted_sales; This query turns the monthly sales columns into rows, making it easier to analyze trends over time by product. Each row represents a product\u0026rsquo;s sales in a specific month.\n7. Conditional Aggregation Conditional aggregation involves applying aggregate functions conditionally based on specified criteria. For example, we might want to calculate the average sales amount only for orders placed by repeat customers.\nSELECT customer_id, AVG(CASE WHEN order_count \u0026gt; 1 THEN order_total ELSE NULL END) AS avg_sales_repeat_customers FROM ( SELECT customer_id, COUNT(*) AS order_count, SUM(order_total) AS order_total FROM orders GROUP BY customer_id ) AS customer_orders; This query calculates the average order total for customers who have placed more than one order. It aggregates the order count and the total order value for each customer, then calculates the average for repeat customers.\n8. Date Functions Date functions in SQL let you manipulate and extract date-related information. For example, we can use the DATE_TRUNC() function to group sales data by month.\nSELECT DATE_TRUNC(\u0026#39;month\u0026#39;, order_date) AS month, SUM(sales_amount) AS total_sales FROM sales GROUP BY DATE_TRUNC(\u0026#39;month\u0026#39;, order_date); This output shows the total sales amount (total_sales) aggregated for each month as month, where each month is represented by its first day (for example, 2023-01-01 for January). Total sales are summed for each respective month.\n9. MERGE or UPSERT Statement MERGE statements (also known as UPSERT or ON DUPLICATE KEY UPDATE) let you insert, update or delete records in a target table based on the results of a join with a source table. Let\u0026rsquo;s say we want to synchronize two tables containing customer data.\nMERGE INTO customers_target t USING customers_source s ON t.customer_id = s.customer_id WHEN MATCHED THEN UPDATE SET t.name = s.name, t.email = s.email WHEN NOT MATCHED THEN INSERT (customer_id, name, email) VALUES (s.customer_id, s.name, s.email); The MERGE statement updates the customers_target table based on the customers_source table. If a customer_id in customers_source matches one in customers_target, the name and email are updated. If there\u0026rsquo;s no match, a new row is inserted.\n10. CASE Statements CASE statements allow conditional logic inside SQL queries. For example, we can use a CASE statement to categorize customers based on their total purchase amount.\nSELECT customer_id, CASE WHEN total_purchase_amount \u0026gt;= 1000 THEN \u0026#39;Platinum\u0026#39; WHEN total_purchase_amount \u0026gt;= 500 THEN \u0026#39;Gold\u0026#39; ELSE \u0026#39;Silver\u0026#39; END AS customer_category FROM ( SELECT customer_id, SUM(order_total) AS total_purchase_amount FROM orders GROUP BY customer_id ) AS customer_purchases; The query sorts customers into categories based on their total purchase amount. Customers with a total purchase amount of $1000 or more are labeled \u0026lsquo;Platinum\u0026rsquo;, those with $500 to $999 are labeled \u0026lsquo;Gold\u0026rsquo; and those with less than $500 are labeled \u0026lsquo;Silver\u0026rsquo;.\n**Explanation:**\n**Customer 1:** Total purchases = 200 + 300 = 500. Classified as \u0026lsquo;Gold\u0026rsquo;.\n**Customer 2:** Total purchases = 800. Classified as \u0026lsquo;Gold\u0026rsquo;.\n**Customer 3:** Total purchases = 150 + 400 = 550. Classified as \u0026lsquo;Silver\u0026rsquo;.\n**Customer 4:** Total purchases = 1200. Classified as \u0026lsquo;Platinum\u0026rsquo;.\n11. String Functions String functions in SQL let you manipulate text data. For example, we can use the CONCAT() function to concatenate first and last names.\nSELECT CONCAT(first_name, \u0026#39; \u0026#39;, last_name) AS full_name FROM employees; Let\u0026rsquo;s look at a sample dataset and explain the output.\n**Sample data in the employees table:**\n| first_name | last_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;|\n| John | Doe |\n| Jane | Smith |\n| Alice | Johnson |\n| Bob | Brown |\n**Output:**\n| full_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\n| John Doe |\n| Jane Smith |\n| Alice Johnson |\n| Bob Brown |\nThe query concatenates the first_name and last_name columns from the employees table with a space between them, creating a full_name for each employee.\n**Explanation:**\n**John Doe:** The first_name (\u0026ldquo;John\u0026rdquo;) and last_name (\u0026ldquo;Doe\u0026rdquo;) columns are concatenated with a space between them, resulting in \u0026ldquo;John Doe\u0026rdquo;.\n**Jane Smith:** The first_name (\u0026ldquo;Jane\u0026rdquo;) and last_name (\u0026ldquo;Smith\u0026rdquo;) columns are concatenated, resulting in \u0026ldquo;Jane Smith\u0026rdquo;.\n**Alice Johnson:** The first_name (\u0026ldquo;Alice\u0026rdquo;) and last_name (\u0026ldquo;Johnson\u0026rdquo;) columns are concatenated, resulting in \u0026ldquo;Alice Johnson\u0026rdquo;.\n**Bob Brown:** The first_name (\u0026ldquo;Bob\u0026rdquo;) and last_name (\u0026ldquo;Brown\u0026rdquo;) columns are concatenated, resulting in \u0026ldquo;Bob Brown\u0026rdquo;.\n12. Grouping Sets Grouping sets allow data to be aggregated at multiple levels of granularity in a single query. Let\u0026rsquo;s say we want to calculate total sales revenue by month and year.\nSELECT YEAR(order_date) AS year, MONTH(order_date) AS month, SUM(sales_amount) AS total_revenue FROM sales GROUP BY GROUPING SETS ( (YEAR(order_date), MONTH(order_date)), YEAR(order_date), MONTH(order_date) ); **Sample data in the sales table:**\n| order_date | sales_amount |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;|\n| 2023-01-15 | 1000 |\n| 2023-01-20 | 1500 |\n| 2023-02-10 | 2000 |\n| 2023-03-05 | 2500 |\n| 2024-01-10 | 3000 |\n| 2024-01-20 | 3500 |\n| 2024-02-25 | 4000 |\n**Output:**\n| year | month | total_revenue |\n|\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;-|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\n| 2023 | 1 | 2500 |\n| 2023 | 2 | 2000 |\n| 2023 | 3 | 2500 |\n| 2024 | 1 | 6500 |\n| 2024 | 2 | 4000 |\n| 2023 | NULL | 7000 |\n| 2024 | NULL | 10500 |\n| NULL | 1 | 9000 |\n| NULL | 2 | 6000 |\n| NULL | 3 | 2500 |\n**Explanation:**\n**Grouping by year and month:** 2023-01: 1000 + 1500 = 2500\n2023-02: 2000\n2023-03: 2500\n2024-01: 3000 + 3500 = 6500\n2024-02: 4000\n**Grouping by year:** 2023: 2500 (Jan) + 2000 (Feb) + 2500 (Mar) = 7000\n2024: 6500 (Jan) + 4000 (Feb) = 10500\n**Grouping by month:** January (all years): 2500 (2023) + 6500 (2024) = 9000\nFebruary (all years): 2000 (2023) + 4000 (2024) = 6000\nMarch (all years): 2500\nThis result provides subtotals for each month of each year, grand totals for each year and grand totals for each month across all years.\n13. Cross Joins Cross joins (CROSS JOIN) produce the Cartesian product of two tables, resulting in a combination of every row from each table. For example, we can use a CROSS JOIN to generate every possible combination of products and customers.\nSELECT p.product_id, p.product_name, c.customer_id, c.customer_name FROM products p CROSS JOIN customers c; Let\u0026rsquo;s look at a sample dataset for the products and customers tables.\n**products table:**\n| product_id | product_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;|\n| 1 | Product A |\n| 2 | Product B |\n**customers table:**\n| customer_id | customer_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\n| 101 | Customer X |\n| 102 | Customer Y |\n**Output:**\n| product_id | product_name | customer_id | customer_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\n| 1 | Product A | 101 | Customer X |\n| 1 | Product A | 102 | Customer Y |\n| 2 | Product B | 101 | Customer X |\n| 2 | Product B | 102 | Customer Y |\nThe query performs a CROSS JOIN between the products and customers tables, resulting in a Cartesian product. This means each product is paired with each customer, generating every possible combination of products and customers.\n**Explanation:**\n**Product A with Customer X:** Combination of product_id 1 and customer_id 101.\n**Product A with Customer Y:** Combination of product_id 1 and customer_id 102.\n**Product B with Customer X:** Combination of product_id 2 and customer_id 101.\n**Product B with Customer Y:** Combination of product_id 2 and customer_id 102.\nThe result is a complete set of combinations between products and customers, demonstrating the Cartesian product of the two tables.\n14. Derived Tables Inline views (also known as derived tables) let you create temporary result sets within a SQL query. Let\u0026rsquo;s say we want to find customers who made purchases above the average order value.\nSELECT customer_id, order_total FROM ( SELECT customer_id, SUM(order_total) AS order_total FROM orders GROUP BY customer_id ) AS customer_orders WHERE order_total \u0026gt; ( SELECT AVG(order_total) FROM orders ); **orders table:**\n| customer_id | order_total |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\n| 1 | 100 |\n| 1 | 200 |\n| 2 | 500 |\n| 3 | 300 |\n| 3 | 200 |\n| 4 | 700 |\n**Calculating the order total for each customer:**\n| customer_id | order_total |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\n| 1 | 300 |\n| 2 | 500 |\n| 3 | 500 |\n| 4 | 700 |\n**Calculating the average order value:**\nTo calculate the average order value, we add up all the order values and divide by the total number of orders:\n(100 + 200 + 500 + 300 + 200 + 700) / 6 = 2000 / 6 ≈ 333.33\n**Filtering customers with total orders above the average:**\n| customer_id | order_total |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-|\n| 2 | 500 |\n| 3 | 500 |\n| 4 | 700 |\n**Explanation:**\n**Calculating the order total for each customer:** The subquery groups orders by customer_id and sums the order_total values for each customer.\n**Calculating the average order value:** Another subquery calculates the average of the order values across the entire orders table.\n**Filtering customers:** The outer query filters the customers whose order totals are greater than the average order value.\nThis way, the query finishes by displaying the customers whose order totals exceed the average, producing the correct output.\n15. Set Operators Set operators such as UNION, INTERSECT and EXCEPT let you combine the results of two or more queries. For example, we can use the UNION operator to merge the results of two queries into a single result set.\nSELECT product_id, product_name FROM products UNION SELECT product_id, product_name FROM archived_products; This query combines the results of the products and archived_products tables, eliminating any duplicate entries, to create a unified list of product IDs and names. The UNION operator ensures each product appears only once in the final result.\n**Sample data:**\n**products table:**\n| product_id | product_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;|\n| 1 | Chocolate Bar |\n| 2 | Dark Chocolate |\n| 3 | Milk Chocolate |\n**archived_products table:**\n| product_id | product_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;|\n| 3 | Milk Chocolate |\n| 4 | White Chocolate |\n| 5 | Almond Chocolate|\n**Output:**\n| product_id | product_name |\n|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;|\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;|\n| 1 | Chocolate Bar |\n| 2 | Dark Chocolate |\n| 3 | Milk Chocolate |\n| 4 | White Chocolate |\n| 5 | Almond Chocolate|\nThe query merges the results of the products and archived_products tables, eliminating the duplicates (in this case, Milk Chocolate with product_id 3), creating a unified list of products.\nThat\u0026rsquo;s all for today! If you enjoyed this article, help spread the word and subscribe to our newsletter for more interesting content. See you next time! ","permalink":"https://julianamarialop.github.io/en/articles/advanced-sql-queries-you-should-know/","summary":"Today\u0026rsquo;s article is about a subject I love and that never goes out of style. SQL!!","title":"Advanced SQL Queries You Should Know"},{"content":"Shall we learn how to calculate the cost of SQL Warehouse instances in Microsoft Azure Databricks the easy way? Then follow along with this article.\nSQL Warehouse in Databricks is a tool that lets us query and explore data efficiently.\nCurrently, Microsoft Azure Databricks offers three types of SQL Warehouse:\nSQL Warehouse Classic: This type of SQL Warehouse uses our Azure subscription account. It supports Photon, but not Predictive IO or Intelligent Workload Management. SQL Warehouse Pro: Like Classic, it uses our Azure subscription account, but it supports both Photon and Predictive IO. However, it does not support Intelligent Workload Management. SQL Warehouse Serverless: This type uses the Azure Databricks serverless architecture and supports every Databricks SQL performance feature, including Photon, Predictive IO and Intelligent Workload Management. It runs directly in our Azure Databricks account. The main difference between SQL Warehouse Classic, Pro and Serverless is where the compute layer lives and which features are supported. While Classic and Pro use our Azure subscription account, Serverless runs in the Microsoft Azure Databricks account and supports more features.\nDIY Method This section is aimed at developers or technical folks who want to understand the logic behind cost analysis and perhaps build their own tools or scripts to calculate the cost of Microsoft Azure Databricks SQL Warehouse instances.\nDIY Method: SQL Warehouse Classic To calculate the cost of a Microsoft Azure Databricks SQL Warehouse Classic or Pro, we need to consider the following components:\nSQL compute hours: The time the SQL Warehouse was running. Databricks Unit (DBU) hours: Compute units used by the SQL Warehouse, billed per hour. Storage costs: Costs associated with data storage, if applicable. Bandwidth costs: Costs associated with data transfer, if applicable. Getting the list of classic SQL Warehouse instances from the Databricks API The first step is to retrieve the list of SQL Warehouse instances using the Microsoft Azure Databricks API: /api/2.0/sql/warehouses.\nAfter running the API call, we\u0026rsquo;ll get a JSON response with all the Microsoft Azure Databricks SQL Warehouses.\nHere\u0026rsquo;s an example of the SQL Warehouse Classic JSON:\n\u0026#34;warehouses\u0026#34;:[ { \u0026#34;id\u0026#34;:\u0026#34;2b1623d985c31e5d\u0026#34;, \u0026#34;name\u0026#34;:\u0026#34;Classic Warehouse\u0026#34;, \u0026#34;size\u0026#34;:\u0026#34;XXSMALL\u0026#34;, \u0026#34;cluster_size\u0026#34;:\u0026#34;2X-Small\u0026#34;, \u0026#34;min_num_clusters\u0026#34;:1, \u0026#34;max_num_clusters\u0026#34;:1, \u0026#34;auto_stop_mins\u0026#34;:45, \u0026#34;auto_resume\u0026#34;:true, \u0026#34;creator_name\u0026#34;:\u0026#34;julianamarialopes@hotmail.com\u0026#34;, \u0026#34;creator_id\u0026#34;:1036471208901988, \u0026#34;tags\u0026#34;:{ }, \u0026#34;spot_instance_policy\u0026#34;:\u0026#34;COST_OPTIMIZED\u0026#34;, \u0026#34;enable_photon\u0026#34;:true, \u0026#34;channel\u0026#34;:{ \u0026#34;name\u0026#34;:\u0026#34;CHANNEL_NAME_CURRENT\u0026#34; }, \u0026#34;enable_serverless_compute\u0026#34;:false, \u0026#34;warehouse_type\u0026#34;:\u0026#34;CLASSIC\u0026#34;, \u0026#34;num_clusters\u0026#34;:0, \u0026#34;num_active_sessions\u0026#34;:0, \u0026#34;state\u0026#34;:\u0026#34;STOPPED\u0026#34;, \u0026#34;jdbc_url\u0026#34;:\u0026#34;jdbc:spark://adb-xxxxxxxx.x.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/2b1623d985c31e5d;\u0026#34;, \u0026#34;odbc_params\u0026#34;:{ \u0026#34;hostname\u0026#34;:\u0026#34;adb-xxxxxxxx.x.azuredatabricks.net\u0026#34;, \u0026#34;path\u0026#34;:\u0026#34;/sql/1.0/warehouses/2b1623d985c31e5d\u0026#34;, \u0026#34;protocol\u0026#34;:\u0026#34;https\u0026#34;, \u0026#34;port\u0026#34;:443 } } } ] Getting the cost of SQL Warehouse Classic resources from the Microsoft API To calculate SQL Warehouse Classic costs, we need to combine the usage data with the cost information.\nWe\u0026rsquo;ll use the Microsoft Generate Cost Details Report API to get all the data from the Microsoft Azure subscription where our Microsoft Azure Databricks clusters are running.\nHere are some important notes when using this API:\nFiltering the data: We must filter the Databricks-related data and remove any data related to other services or data with no associated cost. Subscription type: We need to adjust the results based on the Azure subscription type (pay-as-you-go or enterprise), since the data format may differ. Data limits: The API lets you extract data for a period of at most one month and no older than 13 months. When extracting data from the Microsoft Azure API, we select the following columns from the report:\nSqlEndpointId: The SQL Warehouse instance ID. ProductName: Description of the Azure resource consumed. MeterName: Label identifying the type of service or resource being billed. CostInBillingCurrency: Cost of the Azure resource consumed. We\u0026rsquo;ll get data similar to this (formatted for easy reading):\nIn SQL Warehouse Classic instances, the Product equals Azure Databricks - Premium - SQL Analytics and the MeterName equals Premium SQL Analytics DBU.\nNext, we add up all the records to calculate the final cost of the SQL Warehouse instance.\nDIY Method - SQL Warehouse Pro To calculate the cost of a Microsoft Azure Databricks SQL Warehouse Pro, we need to take the following components into account:\nSQL compute hours: the time during which the SQL Warehouse was running. Databricks Unit (DBU) hours: the compute units used by the SQL Warehouse, which are billed per hour. Storage Costs: Costs associated with data storage, if applicable. Bandwidth costs: Costs associated with bandwidth transfer, if applicable. After running the API call, we\u0026rsquo;ll get a JSON response with all the Microsoft Azure Databricks SQL Warehouses.\nThis is the SQL Warehouse Pro JSON:\n\u0026#34;warehouses\u0026#34;:[ { \u0026#34;id\u0026#34;:\u0026#34;aedd502a567f273a\u0026#34;, \u0026#34;name\u0026#34;:\u0026#34;Starter Warehouse\u0026#34;, \u0026#34;size\u0026#34;:\u0026#34;XXSMALL\u0026#34;, \u0026#34;cluster_size\u0026#34;:\u0026#34;2X-Small\u0026#34;, \u0026#34;min_num_clusters\u0026#34;:1, \u0026#34;max_num_clusters\u0026#34;:1, \u0026#34;auto_stop_mins\u0026#34;:10, \u0026#34;auto_resume\u0026#34;:true, \u0026#34;creator_name\u0026#34;:\u0026#34;julianamarialopes@hotmail.com\u0026#34;, \u0026#34;creator_id\u0026#34;:1036471128901988, \u0026#34;tags\u0026#34;:{ }, \u0026#34;spot_instance_policy\u0026#34;:\u0026#34;COST_OPTIMIZED\u0026#34;, \u0026#34;enable_photon\u0026#34;:true, \u0026#34;channel\u0026#34;:{ }, \u0026#34;enable_serverless_compute\u0026#34;:false, \u0026#34;warehouse_type\u0026#34;:\u0026#34;PRO\u0026#34;, \u0026#34;num_clusters\u0026#34;:0, \u0026#34;num_active_sessions\u0026#34;:0, \u0026#34;state\u0026#34;:\u0026#34;STOPPED\u0026#34;, \u0026#34;jdbc_url\u0026#34;:\u0026#34;jdbc:spark://adb-xxxxxxxx.x.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/aedd502a567f273a;\u0026#34;, \u0026#34;odbc_params\u0026#34;:{ \u0026#34;hostname\u0026#34;:\u0026#34;adb-xxxxxxxx.x.azuredatabricks.net\u0026#34;, \u0026#34;path\u0026#34;:\u0026#34;/sql/1.0/warehouses/aedd502a567f273a\u0026#34;, \u0026#34;protocol\u0026#34;:\u0026#34;https\u0026#34;, \u0026#34;port\u0026#34;:443 } } } ] Very similar to the previous example.\nGetting the cost of SQL Warehouse PRO resources from the Microsoft API To calculate SQL Warehouse Pro costs, we need to combine the retrieved usage data with the cost information to calculate the total cost.\nWe\u0026rsquo;ll use the Microsoft Generate Cost Details Report API to get all the data from the Azure subscription where our Databricks clusters are running.\nImportant notes when using the API:\nWhen we get the cost details data from a Microsoft Azure subscription, we must filter the Databricks-related data and remove any data related to other services and data with no associated cost. We need to adjust the results based on the Microsoft Azure subscription type (pay-as-you-go or enterprise) because the output is different. The API only allows data to be extracted for one month or less and no older than 13 months. When extracting the data from the Microsoft API, we\u0026rsquo;ll need to select the following columns from the report:\nSqlEndpointId: the SQL Warehouse instance ID ProductName: the description of the Azure resource consumed MeterName: the label identifying the type of service or resource being billed CostInBillingCurrency: the cost of the Azure resource consumed BillingCurrency: the billing currency code (USD, EUR, etc.) 1.2.3. Calculating the cost of SQL Warehouse PRO instances After retrieving the data from Microsoft Azure Databricks and from the Microsoft Azure APIs, the final step is to filter the Azure cost data by SQL endpoint ID and match it to the SQL endpoint ID from the Microsoft Azure Databricks API.\nIn SQL Warehouse Pro instances, the Product equals Azure Databricks Regional - Premium - SQL Compute Pro and the MeterName equals Premium SQL Compute Pro DBU.\nNext, we add up all the records to calculate the final cost of the SQL Warehouse instance.\n1.3. DIY Method: SQL Warehouse Serverless Microsoft Azure Databricks SQL Warehouse serverless instances run inside our Microsoft Azure Databricks account, so we\u0026rsquo;re billed only in Databricks Unit (DBU) hours.\nAfter running the API call, we\u0026rsquo;ll get a JSON with all the Microsoft Azure Databricks SQL Warehouses.\nThis is the SQL Warehouse Serverless JSON:\n\u0026#34;warehouses\u0026#34;:[ { \u0026#34;id\u0026#34;:\u0026#34;e5e7201bde7c3a43\u0026#34;, \u0026#34;name\u0026#34;:\u0026#34;Serveless Warehouse\u0026#34;, \u0026#34;size\u0026#34;:\u0026#34;XXSMALL\u0026#34;, \u0026#34;cluster_size\u0026#34;:\u0026#34;2X-Small\u0026#34;, \u0026#34;min_num_clusters\u0026#34;:1, \u0026#34;max_num_clusters\u0026#34;:1, \u0026#34;auto_stop_mins\u0026#34;:10, \u0026#34;auto_resume\u0026#34;:true, \u0026#34;creator_name\u0026#34;:\u0026#34;julianamarialopes@hotmail.com\u0026#34;, \u0026#34;creator_id\u0026#34;:1036471128901988, \u0026#34;tags\u0026#34;:{ }, \u0026#34;spot_instance_policy\u0026#34;:\u0026#34;COST_OPTIMIZED\u0026#34;, \u0026#34;enable_photon\u0026#34;:true, \u0026#34;enable_serverless_compute\u0026#34;:true, \u0026#34;warehouse_type\u0026#34;:\u0026#34;PRO\u0026#34;, \u0026#34;num_clusters\u0026#34;:0, \u0026#34;num_active_sessions\u0026#34;:0, \u0026#34;state\u0026#34;:\u0026#34;STOPPED\u0026#34;, \u0026#34;jdbc_url\u0026#34;:\u0026#34;jdbc:spark://adb-xxxxxxxx.x.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/e5e7201bde7c3a43;\u0026#34;, \u0026#34;odbc_params\u0026#34;:{ \u0026#34;hostname\u0026#34;:\u0026#34;adb-xxxxxxxx.x.azuredatabricks.net\u0026#34;, \u0026#34;path\u0026#34;:\u0026#34;/sql/1.0/warehouses/e5e7201bde7c3a43\u0026#34;, \u0026#34;protocol\u0026#34;:\u0026#34;https\u0026#34;, \u0026#34;port\u0026#34;:443 } } } ] 1.3.2. Getting the cost of SQL Warehouse serverless resources from the Microsoft API We need to combine the retrieved usage data with the cost information to calculate SQL Warehouse Serverless costs.\nWe\u0026rsquo;ll use the Microsoft Generate Cost Details Report API to get all the data from the Azure subscription where our Microsoft Azure Databricks clusters are running.\nImportant notes when using the API:\nWhen we get the cost details data from a Microsoft Azure subscription, we must filter the data related to Microsoft Azure Databricks and remove any data related to other services and data with no associated cost. We need to adjust the results based on the Microsoft Azure subscription type (pay-as-you-go or enterprise) because the output is different. The API only allows data to be extracted for one month or less and no older than 13 months. When extracting the data from the Microsoft API, we\u0026rsquo;ll need to select the following columns from the report:\nSqlEndpointId: the SQL Warehouse instance ID ProductName: the description of the Azure resource consumed MeterName: the label identifying the type of service or resource being billed CostInBillingCurrency: the cost of the Azure resource consumed BillingCurrency: the billing currency code (USD, EUR, etc.) 1.3.3. Calculating the cost of SQL Warehouse serverless instances After retrieving the data from Microsoft Azure Databricks and from the Microsoft Azure APIs, the final step is to filter the cost data by SQL endpoint ID and match it to the SQL endpoint ID from the Microsoft Azure Databricks API.\nIn SQL Warehouse Serverless instances, the Product equals Azure Databricks Regional - Premium - Serverless SQL and the MeterName equals Premium Serverless SQL DBU.\nNext, we add up all the records to calculate the final cost of the SQL Warehouse instance.\nAnd that\u0026rsquo;s it, folks. If you enjoyed this article, please show your support 👏. Thanks for reading!\n","permalink":"https://julianamarialop.github.io/en/articles/how-to-calculate-real-cost-azure-databricks-sql-warehouse-instances/","summary":"Shall we learn how to calculate the cost of SQL Warehouse instances in Microsoft Azure Databricks the easy way? Then follow along with this article.","title":"How to Calculate the Real Cost of Azure Databricks SQL Warehouse Instances"},{"content":"Introduction In the dynamic landscape of information storage, the methodologies used to structure data play a crucial role in uncovering valuable insights for informed decision-making. Since the foundational theories of the 1990s and 2000s, proposed by experts such as Inmon (Inmon WH, Building the Data Warehouse, 1990), Kimball (Kimball R, The Data Warehouse Toolkit, 1996), and later Linstedt (Linstedt D, Data Vault Series 1: Data Vault Overview, 2002), a range of modeling techniques for traditional data storage have evolved and been widely discussed. In this article, we\u0026rsquo;ll cover Dimensional Modeling. We\u0026rsquo;ll explore the strengths and challenges of this approach and discuss best practices for implementing it on Databricks.\nTo design a dimensional data model, you need a deep understanding of the business requirements and a good knowledge of the data sources. The most common implementation of Dimensional Modeling is the Star Schema, widely adopted as the presentation layer in most data warehouses over the past decades. This method organizes data in a denormalized way around measurable events, known as Facts, and the contextual details surrounding those events, called Dimensions.\nThe success of dimensional modeling: Why it became the Gold standard Dimensional Modeling was introduced to optimize the data model for analysis in the logical layer of Relational Database Management Systems (RDBMS) without having to redesign the physical layer. While the logical and physical layers of an RDBMS were designed specifically for Online Transaction Processing (OLTP), enabling efficient row-oriented data entry and guaranteeing ACID properties (Atomicity, Consistency, Isolation, Durability) on normalized data, Dimensional Modeling focuses on online analytical processing (OLAP). This approach allows historical data to be processed and aggregated with better performance.\nBenefits of Dimensional Modeling Simplified Data Usability: Dimensional modeling makes data easier to understand by structuring it into fact and dimension tables. This makes it easier for end users to map real-world processes to the data model. It also supports efficient data aggregation, serving as a semantic layer for Business Intelligence (BI) tools.\nQuery Performance: One of the main advantages of dimensional modeling is optimizing query performance without sacrificing the depth of historical analysis. The Star schema, a common implementation of this modeling, denormalizes data into granular business facts and dimensions, significantly improving query performance and data aggregation.\nScalability and Consistency: Dimensional models are highly scalable, accommodating growing data volumes and adapting to constantly changing business requirements. The star schema allows adjustments to dimensions and facts and makes it easier to manage slowly changing dimensions and integrate incremental data. Other approaches, such as the Snowflake schema or the use of surrogate keys in dimension tables, help reduce data redundancy.\nWith these benefits, Dimensional Modeling established itself as the gold standard for data analysis in data warehousing environments, offering a robust and flexible structure that meets modern business needs.\nBut it\u0026rsquo;s not all sunshine and roses\u0026hellip; Dimensional Modeling reached a high level of optimization in the logical layer, effectively balancing redundancy and query performance. This was essential when storage and compute resources were expensive and row-oriented databases couldn\u0026rsquo;t adequately handle analytical processing. As data technologies advanced, new considerations about traditional dimensional modeling emerged.\nOperational and Design Limitations: Dimensional Modeling requires a significant upfront investment in schema design, plus ongoing maintenance of the data pipelines, usually managed by ETL tools. This approach carries operational overhead and faces design challenges, such as the complexity of managing slowly changing dimensions and fact-to-fact joins. In the past, these challenges were accepted as necessary trade-offs to improve query performance. However, with modern data technologies, these problems can be mitigated, reducing the need for complex schemas and operational overhead.\nTechnological Evolution in Data Storage and Processing: Modern data storage technologies provide flexibility and scalability by decoupling storage and compute. These technologies use massively parallel processing (MPP) and physical columnar storage, optimizing the aggregation and analysis of historical data by default. With storage costs dropping significantly over time, the downsides of denormalization have been minimized. Advances such as data compression at the storage layer and clustering overcome the complexity of keeping data normalized.\nWith these technological advances, it\u0026rsquo;s possible to reassess the need for complex schema designs and explore alternatives that offer simplicity and operational efficiency.\nAlright, if you choose to go with a multidimensional model, here are some practical recommendations for Databricks\u0026hellip; Given Databricks\u0026rsquo; advanced Data Warehousing capabilities, strictly adhering to Dimensional Modeling, for example a Star Schema, is no longer a necessity; it is, however, possible and very well supported. Here we\u0026rsquo;ll look at several Databricks technologies that make it possible to implement and optimize the Dimensional Modeling technique.\nACID Properties: Databricks Delta Lake supports ACID transactions on Delta tables, simplifying the maintenance and quality of dimensional models. Data layers : The Star Schema can be deployed in a Gold Layer of the medallion architecture on Databricks to boost analysis and fast decision-making. ELT pipelines : The pipeline that transforms transactional data into dimensional data models is supported by Databricks SQL and Delta Live Tables (DLT) . Relational constraints : Unlike typical data lakes, Databricks supports schema enhancement , such as relational constraints, for example primary keys, foreign keys, and identity columns in Databricks SQL as surrogate keys, plus enforced CHECK constraints for data quality. Physical and virtual constraints can exist as meta-objects in Unity Catalog . Unified governance : every data model, dimension table, fact table, and their relationships are registered centrally in Unity Catalog. With Unity Catalog as the unified governance layer, Dimensions and Facts can be discovered and shared across organizations without having to duplicate them. Optimization : Databricks supports Liquid Clustering , which incrementally optimizes the data layout without rewriting the data, and which can be applied to fact tables as well as dimension tables. I hope these simple tips help you implement your model. Good luck!\n","permalink":"https://julianamarialop.github.io/en/articles/databricks-sql-and-dimensional-modeling-optimizing-your-data-warehouse/","summary":"In the dynamic landscape of information storage, the methodologies used to structure data play a crucial role in uncovering valuable insights for informed decision-making. Since the…","title":"Databricks SQL and Dimensional Modeling: Optimizing Your Data Warehouse"},{"content":"Effective management of data pipelines is key to ensuring continuous development, testing, and deployment processes. Integrating Git with Microsoft Fabric workspaces offers a structured and efficient approach to managing these pipelines, fostering collaboration and version control in a more organized and secure way.\nIn this article, we\u0026rsquo;ll take a detailed look at best practices for using Git-enabled Microsoft Fabric workspaces. We\u0026rsquo;ll cover essential aspects such as initial setup, code organization, branching and merge strategies, as well as automation and monitoring.\nThe integration between Git and Microsoft Fabric lets development teams work collaboratively, keeping a detailed history of changes and making it easier to track and roll back changes when needed. We\u0026rsquo;ll also discuss how to set up CI/CD (Continuous Integration and Continuous Delivery) pipelines using Azure DevOps to make sure code is tested and deployed consistently and reliably.\nThe diagram below describes the integration between Azure DevOps and Microsoft Fabric, illustrating the flow from development environments to production environments. This visual representation helps you understand how code changes are managed and promoted through the different stages, from initial development all the way to final deployment in production.\n_Example of a Git + Fabric Workspace setup_\nAzure DevOps (Git)\n• Main Branch: The main branch for stable releases.\n• Development Branch: The main branch for ongoing development.\n• Feature branches: for developing specific features.\nMicrosoft Fabric (workspaces)\nDeployment pipeline: manages promotion to different environments (development, staging, production) Development workspaces: individual developer environments. How to Manage Complex Projects in Microsoft Fabric Managing large or complex projects in Microsoft Fabric can be challenging, but applying best practices can make the process easier. Here are some of the key lessons that can help you maintain these projects efficiently.\nBranching Strategies Microsoft Fabric developers, who work with pipelines, notebooks, reports, data warehouses, lakehouses, and much more, need to master Git. I understand the learning curve can be steep, especially for data engineers who started out as analysts or DBAs, where Git wasn\u0026rsquo;t an essential skill. However, as you work on large, complex projects in Microsoft Fabric, Git becomes essential for efficient collaboration.\nFeature Branches: Each developer should create a branch off the development branch to work on their own feature or bug fix. Those using Fabric natively can attach that branch to their own personal workspace, such as \u0026ldquo;MJ\u0026rsquo;s Project A Workspace\u0026rdquo;. It\u0026rsquo;s worth noting that, currently, you can\u0026rsquo;t enable Git on \u0026ldquo;My workspace\u0026rdquo;, so you need to provision a new workspace for this purpose. This also applies to developers working on Power BI reports using Power BI Desktop, which makes this approach independent of the Fabric workspace.\nDevelopment Branch: Use this branch to integrate new features and run tests before merging into the main branch. Implementing branch policies is crucial to prevent developers from accidentally pushing or making changes without going through a pull request process. This process ensures that each developer\u0026rsquo;s work is reviewed before it is deployed.\nMain Branch: This should be the source of truth for production-ready code. Only stable, tested code should be merged here. Tagging releases is a good practice for separating different versions, making it easier to roll back changes if needed.\nIt\u0026rsquo;s worth pointing out that the main branch is not Git-enabled in the Fabric workspace; only the development branch is associated with a workspace. We\u0026rsquo;ll go into this in more detail in the next section.\nGit Integration in Microsoft Fabric Microsoft Fabric workspaces, including Power BI ones, support Git integrations. Currently, this integration is available only with Azure DevOps, but GitHub support is expected to be added soon.\nTo set up the integration, just open the workspace settings and go to the \u0026ldquo;Git integration\u0026rdquo; tab. From there, connect the workspace to Azure DevOps. This integration makes it easier to manage code and collaborate among developers, making the development process more robust and efficient.\nConclusion Implementing these best practices will help you get the most out of Git-enabled Microsoft Fabric workspaces. This not only improves the efficiency of data pipeline management, but also strengthens collaboration and version control within development teams.\n","permalink":"https://julianamarialop.github.io/en/articles/best-practices-for-using-data-fabric-workspaces-with-git/","summary":"Effective management of data pipelines is key to ensuring continuous development, testing, and deployment processes. Integrating Git with Microsoft Fabric workspaces offers a…","title":"Best Practices for Using Data Fabric Workspaces + Git"},{"content":"Following up on the previous article, \u0026ldquo;Databricks Lakehouse Federation: An Alternative to Zero ETL?\u0026rdquo;, let\u0026rsquo;s explore some practical examples of combining data from different sources.\nBefore we start building the environment, let\u0026rsquo;s identify and name 2 main components:\nConnection: A connection to the database, using a username and password or a service account Foreign Catalog: A catalog in Unity Catalog that points to the source database; it will contain all the schemas and tables from your source Our environment will consist of 4 different connections, and in the same SQL query we\u0026rsquo;ll combine information from several data sources in real time.\nTo create the connections, go to the \u0026ldquo;connection management\u0026rdquo; tab in your environment.\nTo illustrate this article, the connections have already been created, but to create a new one, just click Create Connection:\nGive your connection a name and choose from the available data sources:\nEach connection type has its own particularities, so pay attention to the settings for each environment.\nOnce the connection is created, we\u0026rsquo;ll create the foreign catalog: just go to the catalogs home screen and click Create catalog.\nWhen creating it, select the Foreign type and a list of the connections you\u0026rsquo;ve already created will appear. Important: the Database field needs to have the same name as your Database at the source.\nAfter it\u0026rsquo;s created, you can explore your catalog and view the source tables as if they were inside the Databricks environment. However, it\u0026rsquo;s important to remember that any query run against these tables will be sent to the sources in real time. So it\u0026rsquo;s crucial to be careful not to overload your data sources.\nIt\u0026rsquo;s worth noting that while browsing the catalog, each source type has its own specific navigation structure. In the case of PostgreSQL, for example, we browse by schemas. After selecting the schemas you want, you can see the available tables.\nBelow are the tables from our example.\nLet\u0026rsquo;s look at the same thing in SQL Server, BigQuery, and Databricks.\nBigQuery: List of Datasets\nBigQuery tables:\nDatabricks: Pointing to external workspaces (this is different from Delta Sharing)\nNotice that here I can see all the schemas inside the catalog referenced in the connection.\nDatabricks tables:\nSo far we\u0026rsquo;ve explored four distinct data sources, all containing three tables in common: Autor (Author), Editora (Publisher), and Livro (Book). Now we\u0026rsquo;ll run a few queries that combine information from these sources.\nIn a simple example, I queried the \u0026ldquo;Livro\u0026rdquo; table in all four sources without needing to replicate data between them. There was no need to apply transformation rules or implement CDC to reflect changes originating in the data sources.\nNow let\u0026rsquo;s combine the 4 sources in the same query:\nIn the same query, we\u0026rsquo;re querying different sources, each with its own schemas, without needing to replicate any data into our central repository (Lake), thereby avoiding all the complexity associated with ETL, CDC, SCD, and other processes.\nAnd what about JOINs, do they work? Let\u0026rsquo;s imagine, for example, comparing all the books to check whether the titles are consistent across every source.\nOf course, every case is unique, and generalizations aren\u0026rsquo;t recommended. It\u0026rsquo;s essential to study your specific needs and assess whether the Zero ETL concept fits your scenario. Always take into account the particularities of your data sources, such as querying a replica so you don\u0026rsquo;t overload the production environment, among other important points.\nIt\u0026rsquo;s important to note that excessive use of JOINs can hurt performance and undermine the ability to perform pushdowns. So it\u0026rsquo;s recommended to use JOINs sparingly and consider alternatives where possible.\nSummary\nDatabricks\u0026rsquo; Lakehouse Federation feature applies to use cases such as the following:\nWhen you don\u0026rsquo;t want to ingest data into the Databricks environment. When you want queries to take advantage of the compute processing performed in the external database system. When you want to use the data governance benefits offered by Unity Catalog, such as fine-grained access control, data lineage, and searchability, centralizing these capabilities in a single place while saving on storage and processing. As highlighted earlier in the analysis of strengths and weaknesses, although there are savings in storage (avoiding data duplication between the source and the Lakehouse) and in cluster processing (delegating heavy processing to the source), it\u0026rsquo;s essential to examine each use case individually. You need to make sure you don\u0026rsquo;t overload the data sources with frequent, intensive queries, which may not be optimized because of pushdown limitations.\nI\u0026rsquo;m currently testing Lakehouse Federation in a KPI validation scenario. For example, the entire ETL/ELT process runs into the Lakehouse, with its respective modeling and transformations. To check data integrity against the source, I\u0026rsquo;m using Lakehouse Federation to query the source on a spot basis and validate that the data matches.\nThis approach has proven quite useful and efficient for these scenarios, including for exploring data before starting the ELT processes into the Lakehouse.\nTo learn more, I recommend reading all the available documentation carefully and testing extensively. It\u0026rsquo;s essential to understand the specific use case and assess whether Lakehouse Federation is right for your particular scenario.\nReferences\nhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/\n","permalink":"https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-exploring-practical-examples/","summary":"Following up on the previous article, \u0026ldquo;Databricks Lakehouse Federation: An Alternative to Zero ETL?\u0026rdquo;, let\u0026rsquo;s explore some practical examples of combining data from different sources.","title":"Databricks Lakehouse Federation: Exploring Practical Examples"},{"content":"Released in December 2023, the Lakehouse Federation feature is currently in Public Preview, but it has enormous potential. In today\u0026rsquo;s article, I\u0026rsquo;ll explore some interesting perspectives, such as the ability to combine information across Databricks, BigQuery, SQL Server, and PostgreSQL in a single SQL query.\nWhat is lakehouse federation? Databricks lakehouse federation is a query federation feature that lets users and systems run queries against multiple data sources without having to migrate all the data into a unified system.\nHere you eliminate the need to build complex ETLs, apply CDC, SCD, and other techniques, while also running the risk of the data drifting from the source, which can be very useful in a number of scenarios.\nWe can read federation, in its literal sense, as a way to democratize access to data from many sources through a single platform, delegating the processing work to the sources. In this context, federation makes it possible to speed up data exploration, avoid storage duplication, and gain other benefits.\n_Data Federation_\nHow does it work? Databricks Lakehouse Federation uses Unity Catalog to manage distributed queries. To use Databricks Lakehouse Federation, you need to configure read connections using the drivers included natively in the clusters, such as Pro SQL Warehouses, Serverless SQL Warehouses, and Databricks Runtime \u0026gt;=13.1. The data sources currently supported include:\nMySQL PostgreSQL Amazon RedShift Snowflake SQL Server Azure Synapse Google BigQuery Databricks (with other workspaces) In essence, Databricks sends queries to run on the source databases using native drivers like JDBC and returns the resulting data to Databricks in real time.\nAdvantages and Disadvantages Agility: It lets you run queries on data coming from multiple sources (listed above) without having to migrate it, which can save time and resources.\nScalability: It can be used to connect to data sources of any size, allowing processing to be delegated to the source.\nGovernance: It uses Unity Catalog to manage data governance, ensuring the security and reliability of your data.\nPotentially lower performance than traditional ETL methods, because the data isn\u0026rsquo;t moved into a unified system. This can result in network latency and possible performance issues, since not every filter and function can be sent to the source, which is known as \u0026ldquo;pushdowns.\u0026rdquo;\nWhen you point queries at production OLTP environments, there\u0026rsquo;s a risk of overloading the sources\u0026rsquo; performance, unlike more targeted ETLs or using CDC for data collection.\nLimitations It\u0026rsquo;s important to be aware of the limitations when using Lakehouse Federation:\nRead-only connections: The connections established are strictly for reading data. Pushdowns: Not every type of pushdown is supported, and that compatibility varies by data source. Data types: You need to pay attention to the mapping tables between the source and Databricks to ensure data type consistency. Case sensitivity: The system doesn\u0026rsquo;t support case-sensitive object names, so names are normalized to lowercase. This can cause problems depending on the data source. Source-specific limitations: Each data source may have its own limitations that need to be considered when implementing and using Lakehouse Federation. In upcoming articles, we\u0026rsquo;ll take a hands-on look at Lakehouse Federation!\nReferences\nhttps://learn.microsoft.com/pt-br/azure/databricks/query-federation/\nhttps://www.databricks.com/resources/webinar/query-federation-lakehouse\n","permalink":"https://julianamarialop.github.io/en/articles/databricks-lakehouse-federation-an-alternative-to-zero-etl/","summary":"Released in December 2023, the Lakehouse Federation feature is currently in Public Preview, but it has enormous potential. In today\u0026rsquo;s article, I\u0026rsquo;ll explore some interesting perspectives, such as the ability to…","title":"Databricks Lakehouse Federation: An Alternative to Zero ETL?"},{"content":"Today\u0026rsquo;s topic is Data Factory + DevOps. Shall we take a look?\nCI is the practice of automatically testing every code change, while CD deploys those changes to staging or production systems. In Azure Data Factory, CI/CD involves moving pipelines between different environments.\nHere\u0026rsquo;s how it works:\n1️⃣ Developers create a feature branch to make changes and debug pipeline runs.\n2️⃣ The changes are reviewed through pull requests and merged into the main branch.\n3️⃣ The changes are then published to the development factory.\n4️⃣ When they\u0026rsquo;re ready, the changes are deployed to a test or UAT factory using Azure Pipelines.\n5️⃣ After verification, the changes can be deployed to the production factory.\nBest practices for CI/CD in Azure Data Factory:\n✅ Configure Git integration only for the development factory.\n✅ Use pre- and post-deployment scripts for tasks such as stopping/restarting triggers.\n✅ Consider using a shared factory for integration runtimes across all stages.\n✅ Carefully manage the deployment of private endpoints to avoid conflicts.\n✅ Keep separate key vaults for different environments and use consistent secret names.\n✅ Avoid spaces in resource names; use \u0026lsquo;_\u0026rsquo; or \u0026lsquo;-\u0026rsquo; instead.\n✅ Be careful when changing the repository to avoid errors.\n✅ Use exposure control and feature flags to manage logic based on the environment.\n❌ Remember that selectively publishing resources is not recommended because of dependencies.\nTake care, and see you in the next article!\n","permalink":"https://julianamarialop.github.io/en/articles/continuous-integration-and-delivery-cicd-in-azure-data-factory/","summary":"CI is the practice of automatically testing every code change, while CD deploys those changes to staging or production systems. In Azure Data Factory, CI/CD involves moving pipelines between…","title":"🚀 Continuous Integration and Continuous Delivery (CI/CD) in Azure Data Factory! 🔁✨"},{"content":"Many of the clients I work with implement a Medallion architecture to logically organize their data in a Lakehouse. As data flows, it moves through several stages or layers. The best-known design, as seen below, uses a Bronze, Silver and Gold layer. Hence the term \u0026ldquo;medallion\u0026rdquo;.\nAlthough the three-layer design is common and widely recognized, I\u0026rsquo;ve noticed a lot of discussion about the scope, purpose and best practices of each of these layers. I\u0026rsquo;ve also observed that there\u0026rsquo;s a big difference between theory and practice.\nData Platform Strategy The most important consideration when layering your architecture is determining how your data platform will be used. A centralized, shared data platform will have a different structure from a federated, multi-platform setup used by several domains. The layers also vary depending on whether you align your platforms with the source-system side or the consumption side of your architecture. A source-aligned platform is usually easier to standardize in terms of layers and structure than a consumer-aligned platform, because of the diversity of data uses on the consumption side.\nWith these considerations in mind, let\u0026rsquo;s explore layer by layer. For each layer, I\u0026rsquo;ll start with some abstract, high-level goals and then dig deeper with practical observations.\nLanding Area An optional layer often seen in organizations implementing a data platform is a landing area or landing zone. This is an intermediate location where data from various source systems is stored before being moved into the Bronze layer. This layer is often needed in situations where it\u0026rsquo;s hard to extract data from the target source system, such as when working with external clients or SaaS vendors. In those situations, there\u0026rsquo;s a dependency, or sometimes the data is delivered in a non-preferred format or structure.\nThe design of a landing zone varies from organization to organization. Often it\u0026rsquo;s simply a blob storage account. In other cases, the landing zone is part of the data lake services, such as a specific container, bucket or folder where data is ingested. Data in landing zones tends to be highly diverse in terms of file formats, which can include CSV, JSON, XML, Parquet, Delta and others.\nBronze Layer The Bronze layer is usually a reservoir that stores data in its natural, original state. It contains unvalidated data (there\u0026rsquo;s no need to define schemas first). In this layer, data is obtained using full loads or delta loads. Data stored in the Bronze layer usually has the following characteristics:\nIt keeps the raw state of the data source in its \u0026ldquo;as is\u0026rdquo; structure. The data is immutable (read-only). It\u0026rsquo;s managed using range-partitioned tables, for example using a YYYYMMDD or date/time folder structure. It keeps the complete (unprocessed) history of each dataset in an efficient storage format, such as Parquet or Delta. For transactional data, it can be appended incrementally and grow over time. It provides the ability to recreate any state of a given data system. It can be a combination of streaming and batch transactions. It can include additional metadata, such as schema information, source file names or a record of when the data was processed. A common question is: \u0026ldquo;What\u0026rsquo;s the best file format? Should I use Delta or Parquet?\u0026rdquo; Delta is faster, but since the data already has versioning or history through a folder structure, I don\u0026rsquo;t see any compelling benefit in keeping a transaction log or applying versioning. Data in the Bronze layer is usually new data or is being appended. So if you prefer to use Parquet, that\u0026rsquo;s fine. Alternatively, you can use Delta to stay aligned with all the other layers.\nSome people argue that data in the Bronze layer can be useful for ad hoc queries or analysis by business users. In my experience working with clients, I rarely see raw data used as input for running ad hoc queries or analysis. Working with raw data requires a deep understanding of how the source system was designed and the crafting of complex business logic that is encapsulated in the data itself. This data usually contains many small tables and is hard to secure. In short, the Bronze layer is an intermediate layer and serves as the entry point for other layers, accessed mainly by technical teams.\nSilver Layer The Silver layer provides a refined structure for the data that has been ingested. It represents a validated, enriched version of the data that can be trusted for downstream workloads, both operational and analytical. The Silver layer usually has the following characteristics:\nIt uses data quality rules for data validation and processing. It normally contains only functional data, filtering out technical or irrelevant data coming from the Bronze layer. It applies historization, usually using type 2 or type 4 slowly changing dimension (SCD) techniques. This involves adding columns such as start, end and current. It stores data in an efficient storage format, preferably Delta or Parquet. It uses versioning to roll back processing errors. It handles missing data and standardizes empty or dirty fields. It\u0026rsquo;s often enriched with reference and/or master data. The data is often organized around specific subject areas. The data is usually aligned and organized with the source system. For the Silver layer, some points to watch include:\nThe Silver layer can act as a temporary storage layer in some cases, where older data can be deleted or storage accounts can be created on demand. This depends on the intended use of the data. If you don\u0026rsquo;t plan to use the data in its original context for operational reporting or operational analytics, the Silver layer can be used temporarily. However, if you plan to retain history and use this data for operational reporting and analytics, it\u0026rsquo;s recommended to make the Silver layer persistent.\nData in the Silver layer can be queried. So, in terms of data modeling, it\u0026rsquo;s recommended to follow a more denormalized data model. That\u0026rsquo;s because this design makes better use of distributed, column-based storage that is separated from compute. Although it\u0026rsquo;s possible to implement a more normalized design, there are no convincing arguments for doing so, since the Delta layer already offers isolation and protection, such as schema merging to handle changes and the history available in the Bronze layer. So adding extra complexity and reducing performance is generally not recommended.\nAnother discussion that comes up is whether there should be data joining or integration across applications and source systems in the Silver layer. The answer depends on the scenario. If you plan to use the Silver layer for operational reporting or operational analytics, it\u0026rsquo;s generally recommended not to combine or integrate data across source systems, to avoid unnecessary coupling between applications. However, if you want a more isolated design, data integration across sources can happen in a higher layer.\nThe same argument mentioned above also applies when you align your Lakehouses with the source-system side of your architecture. If you plan to build data products and strongly want to align data ownership, it\u0026rsquo;s not advisable for your engineers to join data across applications from other domains. That would create unnecessary coupling points.\nAs for enrichments, such as calculations, if you plan to support operational reporting that requires enrichments, it\u0026rsquo;s recommended to enrich the data in the Silver layer. This may result in some extra calibration when combining data at a later stage in the Gold layer, but it takes advantage of the benefits of flexibility.\nGold Layer Data in the Gold layer, according to the principles of a Lakehouse architecture, is usually organized into \u0026ldquo;project-specific\u0026rdquo; databases ready for consumption. In this context, data ownership can be considered changed, since the data is no longer aligned between source and system. Instead, it has been integrated and combined with other data.\nIn the Gold layer, depending on the use cases, a more denormalized, read-optimized data model with fewer joins is recommended. So a Kimball-style star schema may be appropriate. In addition, expect the following characteristics:\nGold tables represent data that has been transformed for consumption or use cases. The data is stored in an efficient storage format, preferably Delta. The Gold layer uses versioning to roll back processing errors. Historization is applied only for the set of use cases or consumers, so the Gold layer can be a selection or aggregation of data found in the Silver layer. In the Gold layer, complex business rules are applied, which includes post-processing activities, calculations, enrichments and use-case-specific optimizations. The data is highly governed and well documented. The Gold layer is usually the most complex, because its design depends on the scope of the architecture. In the simplest scenario, where your Lakehouses are aligned only with the source-system side, the data in the Gold layer will represent \u0026ldquo;data product\u0026rdquo; data. This data is generic and can be used for wide distribution across several domains after distribution. In this case, the data is expected to meet the precise needs of analytical consumers after distribution to another platform, which may also be a Lakehouse. So the data is modeled with a structure specific to the use case.\nIf the Lakehouse\u0026rsquo;s scope is larger and covers both sides, additional layers will be needed. Some organizations call these additional layers workspace or presentation layers. In this design, the data in the Gold layer is more generic and serves as an integration layer from which data marts or subsets can be populated. Some companies also use these layers to share data with other platforms or teams, making selections and/or pre-filtering for specific use cases.\nI hope you found this article useful. To everyone building a next-generation data platform, enjoy the journey!\n","permalink":"https://julianamarialop.github.io/en/articles/databricks-best-practices-managing-bronze-silver-gold-medallion/","summary":"Many of the clients I work with implement a Medallion architecture to logically organize their data in a Lakehouse. As data flows, it moves through several stages or layers. The best-known design…","title":"Databricks: Best Practices for Managing Bronze, Silver and Gold (Medallion Architecture)"},{"content":"Hello everyone, this week we\u0026rsquo;ll go over some best practices for a data ingestion pipeline:\n💡 Build dynamic pipelines with Metadata-Driven Ingestion Patterns:\n✅ ADF generates code that uses metadata to read data sources and tables/directories into the lakehouse.\n✅ Speed up onboarding of new data sources by adding metadata to the solution framework.\n💡 Ingest data using Auto Loader or directly into Delta Lake:\n✅ Auto Loader efficiently processes new data files in ADLS Gen2, with schema inference and evolution as the data changes.\n✅ The ADF Delta Lake connector automatically lands data in ADLS Gen2 in the Delta Lake file format.\n💡 Run Azure Databricks Jobs with ease:\n✅ Use ADF web activities and the Azure Databricks Jobs API to run Databricks jobs and Delta Live Tables pipelines.\n✅ Take advantage of the latest job features, such as cluster reuse, parameter passing, and repair and rerun.\n💡 Make the most of Pools + Job Clusters:\n✅ ADF can use Azure Databricks pools to create job clusters for notebook activity runs.\n✅ Enjoy workload isolation, lower pricing, auto-termination, fault tolerance, and faster job cluster creation.\n💡 Ensure secure authentication with ADF Managed Identity:\n✅ Authenticate ADF to Azure Databricks using managed identity authentication.\n✅ It provides a more secure authentication technique and removes the need to manage personal access tokens.\nI hope you enjoyed the read!\n","permalink":"https://julianamarialop.github.io/en/articles/azure-data-factory-and-azure-databricks-best-practices/","summary":"Hello everyone, this week we\u0026rsquo;ll go over some best practices for a data ingestion pipeline:","title":"🔵✨ Azure Data Factory and Azure Databricks Best Practices ✨🔵"},{"content":"Traditional large-scale data analytics solutions rely on data warehouses and SQL queries to store and retrieve data. However, the rise of big data, characterized by large #volumes , #variety, and #velocity of new data, along with affordable storage and cloud-based distributed computing, introduced a new approach: the data lake. Unlike data warehouses, data lakes store information as files without a fixed schema.\nTo bridge the gap between data warehouses and data lakes, data engineers and analysts are increasingly turning to a hybrid solution called the data lakehouse. Microsoft Fabric offers a powerful lakehouse solution that combines the scalability of file storage in OneLake (built on Azure Data Lake Store Gen2) with a relational metadata layer based on the open-source Delta Lake table format. This lets you store data in files inside your data lake and apply a relational schema to enable SQL queries.\nWith Microsoft Fabric, you get the advantages of both data warehouses and data lakes, letting you define tables and views using Delta Lake\u0026rsquo;s schema capabilities and query them using familiar SQL semantics.\nSetting Up Your Workspace in Microsoft Fabric Setting up a workspace in Microsoft Fabric is the first step to unlocking its data capabilities. By following these simple instructions, you\u0026rsquo;ll be ready to work with data seamlessly:\nGo to Microsoft Fabric at https://app.fabric.microsoft.com and sign in. Go to the menu bar on the left side and click the Workspaces option (you\u0026rsquo;ll recognize it by the icon that looks like 🗇). Create a new workspace, choosing a name that suits your needs. Make sure you select a licensing mode that includes Fabric capacity, such as Trial, Premium, or Fabric. Once your new workspace is created, it will open, looking empty and ready for you to start tapping into its potential. Creating a Data Lakehouse in Microsoft Fabric Now that you\u0026rsquo;ve set up your workspace, it\u0026rsquo;s time to dive into the Data Engineering experience in the portal and build a data lakehouse to hold your valuable data files. Follow these steps to get started:\nIn the Power BI portal, find the bottom-left corner and switch to the Data Engineering experience. The Data Engineering home page presents a variety of tiles that let you easily create commonly used data engineering assets. On the Data Engineering home page, start creating a new Lakehouse by giving it a name that fits its purpose.\nAfter a short wait, a new data lake will be created, opening up exciting possibilities for exploration.\nNotice the Lakehouse explorer pane on the left side, which lets you browse the tables and files that live in your lakehouse. The Tables folder holds tables that can be queried using SQL semantics. These tables in the Microsoft Fabric lakehouse follow the open-source Delta Lake file format, widely used in Apache Spark.\nThe Files folder contains the data files stored in OneLake storage, designated specifically for your data lake. This folder also lets you create shortcuts that reference externally stored data.\nAt the moment, your lakehouse contains no tables or files, giving you a clean slate to begin your data exploration journey.\nIngesting Data into Your Microsoft Fabric Lakehouse To enrich your lakehouse with valuable data, Microsoft Fabric offers several data ingestion methods. While pipelines and dataflows provide advanced options for copying data from external sources, one of the simplest approaches is to upload files or folders directly from your local computer. Follow these steps to upload data seamlessly to your lakehouse:\nDownload the sales.csv file from the following source: Kaggle sales.csv file URL. Save the file as sales.csv on your local computer.\nGo back to the browser tab where your data lake is open. In the Lakehouse Explorer pane, go to the Files folder and click the ellipsis menu (\u0026hellip;). From the drop-down menu, select \u0026ldquo;New subfolder\u0026rdquo; to create a subfolder called \u0026ldquo;dados\u0026rdquo;.\nFrom the ellipsis menu of the newly created data folder, choose \u0026ldquo;Upload\u0026rdquo; and select \u0026ldquo;Upload file\u0026rdquo;. Then upload the sales.csv file from your local computer. Once the upload is complete, go to the Files/dados folder and check that the sales.csv file was uploaded successfully. It should be visible in the folder contents. To view the contents of the uploaded file, just select the sales.csv file. By following these steps, you can easily bring data into your Microsoft Fabric lakehouse, letting you explore and analyze the uploaded sales.csv file and unlock its insights.\nTurning File Data into Queryable Tables To make the uploaded sales data more usable, Microsoft Fabric offers the ability to load the data from a file into a table. This lets data analysts and engineers use SQL queries for efficient data exploration. Follow these steps to load the file data into a table seamlessly:\nStart by going to the home page and selecting the Files/Dados folder. Here you\u0026rsquo;ll find the sales.csv file you uploaded earlier.\nOpen the ellipsis menu (\u0026hellip;) for the sales.csv file and choose \u0026ldquo;Load to tables\u0026rdquo; from the options provided.\nIn the Load to table dialog box, give the table a suitable name, such as \u0026ldquo;vendas\u0026rdquo;, and confirm the load operation. Now wait patiently for the table creation and data load process to finish.\nIn the Lakehouse explorer pane, find the newly created \u0026ldquo;vendas\u0026rdquo; table to get visibility into its data.\nTo explore the underlying files associated with the sales table, open the table\u0026rsquo;s ellipsis menu (\u0026hellip;) and select \u0026ldquo;View files\u0026rdquo;. It\u0026rsquo;s worth noting that the files of a Delta table are stored in Parquet format, including a subfolder called \u0026ldquo;_delta_log\u0026rdquo; that keeps the transactional details applied to the table.\nBy following these steps, you can turn the uploaded sales data into a queryable table, opening up a world of possibilities for data analysis and extracting valuable insights in Microsoft Fabric.\nUnlocking Data Exploration with SQL Once you\u0026rsquo;ve created a lakehouse and defined tables in it, a powerful SQL endpoint is generated automatically. This endpoint lets you query your tables effortlessly using SQL SELECT statements, enabling seamless data exploration. Follow these steps to take advantage of the SQL endpoint in your Microsoft Fabric lakehouse:\nIn the top-right corner of the Lakehouse page, find the option that toggles between Lakehouse and SQL endpoint. Click it to switch to SQL endpoint mode. Wait a moment for the SQL query endpoint to start up. Soon you\u0026rsquo;ll see a visual interface that lets you query tables in your lakehouse. The interface opens up interesting possibilities for data exploration, as shown here. To start querying, use the \u0026ldquo;New SQL query\u0026rdquo; button, which opens a query editor. SELECIONE CÓDIGO DO PRODUTO, AVG(QUANTITYORDERED) AS AvgQuantityOrdered, AVG(PRICEEACH) AS AvgPrice DE sales_data_sample AGRUPAMENTO POR CÓDIGO DE PRODUTO; In the query editor, enter the SQL query you want. For example, calculate the average quantity ordered and the average price for each product code. To run the query and view the results, just click the \u0026ldquo;▷ Run\u0026rdquo; button. By following these steps, you can tap into the potential of the SQL endpoint in your Microsoft Fabric lakehouse, running SQL queries effortlessly and getting valuable insights from your data.\nBuilding Powerful Reports with Power BI Inside the Microsoft Fabric lakehouse, the tables you define automatically become part of a default dataset, forming the foundation for reporting and analytics with Power BI. Follow these steps to take advantage of default datasets and build insightful reports:\nAt the bottom of the SQL Endpoint page, find and select the \u0026ldquo;Model\u0026rdquo; tab. This tab reveals the schema of the data model associated with the dataset. On the menu ribbon, go to the \u0026ldquo;Reporting\u0026rdquo; tab and click \u0026ldquo;New report\u0026rdquo;. This opens a new browser tab dedicated to building your report. In the Data pane on the right, expand the \u0026ldquo;vendas\u0026rdquo; table. Select the fields you want for your report, such as \u0026ldquo;PRODUCTLINE\u0026rdquo; and \u0026ldquo;SALES\u0026rdquo;. A table visualization is automatically added to the report, showing the selected data.\nTo make the most of the workspace, hide the Data and Filters panes, creating more room for designing the report. Make sure the table visualization is selected and, in the Visualizations pane, customize the visualization by converting it into a clustered bar chart. Adjust the size and look of the chart as you like.\nTo save your progress, open the File menu and select \u0026ldquo;Save\u0026rdquo;. Save the report as \u0026ldquo;Product Sales Report\u0026rdquo; in the workspace you created earlier.\nClose the browser tab containing the report to return to your lakehouse\u0026rsquo;s SQL endpoint. In the center-left menu bar, select your workspace to confirm it now includes the following items:\nYour data lake\nThe SQL endpoint for your data lake\nA default dataset representing the tables in your data lake\nThe \u0026ldquo;Product Sales Report\u0026rdquo; report\nBy following these steps, you can use the default datasets in the Microsoft Fabric Lakehouse to build visually appealing, insightful reports with Power BI. This lets you gain valuable business intelligence and make data-driven decisions with ease.\nClean Up Resources After successfully creating a lakehouse and importing data into it, it\u0026rsquo;s essential to make sure resources are managed efficiently. This section walks you through the process of cleaning up your resources. Let\u0026rsquo;s dive in:\nIf you\u0026rsquo;ve finished exploring the data lake and no longer need the workspace created for this exercise, it\u0026rsquo;s time to remove it.\nOn the left side of the interface, find and select the icon representing your workspace. This shows all the items contained in the workspace.\nOn the toolbar, open the ellipsis menu (\u0026hellip;) and click \u0026ldquo;Workspace settings\u0026rdquo;.\nIn the \u0026ldquo;Other\u0026rdquo; section of the settings, find and select the \u0026ldquo;Remove this workspace\u0026rdquo; option.\nBy following these steps, you can organize your resources effectively by deleting the workspace associated with the exercise. This ensures efficient use of resources in your Microsoft Fabric Lakehouse environment. After successfully creating a lakehouse and importing data into it, it\u0026rsquo;s essential to make sure resources are managed efficiently. This section walks you through the process of cleaning up your resources. Let\u0026rsquo;s dive in:\nIf you\u0026rsquo;ve finished exploring the data lake and no longer need the workspace created for this exercise, it\u0026rsquo;s time to remove it.\nOn the left side of the interface, find and select the icon representing your workspace. This shows all the items contained in the workspace.\nOn the toolbar, open the ellipsis menu (\u0026hellip;) and click \u0026ldquo;Workspace settings\u0026rdquo;.\nIn the \u0026ldquo;Other\u0026rdquo; section of the settings, find and select the \u0026ldquo;Remove this workspace\u0026rdquo; option.\nBy following these steps, you can organize your resources effectively by deleting the workspace associated with the exercise. This ensures efficient use of resources in your Microsoft Fabric Lakehouse environment.\nPhew, that was a long read, but I hope it\u0026rsquo;s a big help!!!\nTake care!\n","permalink":"https://julianamarialop.github.io/en/articles/hands-on-with-microsoft-fabric-a-beginners-guide-part-1/","summary":"Traditional large-scale data analytics solutions rely on data warehouses and SQL queries to store and retrieve data. However, the rise of big data, characterized by large #volumes …","title":"Hands-on with Microsoft Fabric: A Beginner's Guide (Part 1)"},{"content":"Hi everyone, in this article I\u0026rsquo;ll talk a bit about these data platforms.\nFirst, let\u0026rsquo;s start with some definitions:\n🔹 Databricks: Azure Databricks is an analytics platform based on Apache Spark and optimized for Microsoft Azure. It offers streamlined workflows and an interactive workspace for collaboration between data scientists, data engineers, and business analysts. With runtimes optimized for machine learning and GPU support, Databricks is ideal for machine learning development. It also integrates with Azure ML and provides rigorous version control and CI/CD capabilities.\n🔹 Synapse Analytics: Azure Synapse is a limitless analytics service that combines enterprise data warehousing and Big Data analytics. It lets you query data at scale using either serverless on-demand or provisioned resources. Synapse brings these two worlds together with a unified experience for data ingestion, preparation, management, and delivery, meeting immediate BI and machine learning needs.\n💡 When to Use Databricks and Synapse Analytics 💡 ✅ Machine Learning Development: If your focus is machine learning, Databricks is the preferred choice. It offers ML-optimized runtimes, GPU-enabled clusters, and a managed version of MLflow. You can also use AzureML from within Databricks and benefit from tight version control and CI/CD integration across full environments.\n✅ Ad-hoc Data Lake Discovery: Both Synapse and Databricks are well suited for ad-hoc data lake discovery. Databricks lets you query data using Python, Scala, or R after mounting the data lake in your workspace. Synapse, on the other hand, provides on-demand SQL or Spark to query data in your data lake. Pick the tool or interface that matches your preferences and expertise.\n✅ Real-Time Transformations: For real-time transformations, Databricks is the recommended option. It offers Spark Structured Streaming with advanced features such as Z-order clustering and join optimizations. Databricks Autoloader enables incremental loading. While Synapse can ingest real-time data using Stream Analytics, it currently doesn\u0026rsquo;t fully support Delta and isn\u0026rsquo;t fully focused on real-time transformations.\n✅ SQL Analytics and Data Warehousing: If you need comprehensive SQL analytics and data warehousing capabilities, Synapse is the right choice. It provides a complete data warehousing experience with relational data models, stored procedures, and a full standard T-SQL environment. Synapse brings together the best SQL technologies, including columnar indexing.\n✅ Self-Service Reporting and BI: For self-service reporting and BI, Synapse takes the lead. Synapse lets you use Power BI directly from Synapse Studio. Its SQL pool (SQL DWH) is widely recognized in enterprise data warehousing.\nBy understanding the differences between Azure Synapse and Databricks, you can make informed decisions about which platform fits your specific needs. Choose the right tool for your data analytics and unlock the full potential of your data! ✨💼\n","permalink":"https://julianamarialop.github.io/en/articles/key-differences-between-azure-synapse-and-databricks/","summary":"Hi everyone, in this article I\u0026rsquo;ll talk a bit about these data platforms.","title":"🚀 Key Differences Between Azure Synapse and Databricks"},{"content":"Wouldn\u0026rsquo;t it be a dream to have a column in your Bronze table (or Landing, whatever you prefer to call it) telling you which file each record came from?\nHere\u0026rsquo;s how to do it:\nBelow we have a set of JSON files to load.\nLet\u0026rsquo;s read the files into a dataframe.\nNow, let\u0026rsquo;s save the dataframe as a Delta table.\nNow here\u0026rsquo;s my question: how do we know which file each person came from?\nThe secret is the input_file_name() function. You can use it with either SQL or Python, and it returns the name of the file that was read. In the example below, I\u0026rsquo;m using the function in the select on my Bronze table in Delta format.\nHowever, if we use this function after the table has been created, what we get back are the parquet files, not the original source files.\nTo solve this, let\u0026rsquo;s add the \u0026ldquo;nomeArquivo\u0026rdquo; (file name) column when we create the Delta table.\nNow we\u0026rsquo;re talking!!\nThat\u0026rsquo;s it for today, folks. I hope this was useful.\nSee you next time.\n","permalink":"https://julianamarialop.github.io/en/articles/databricks-tip-track-data-source-by-file-input-file-name/","summary":"Wouldn\u0026rsquo;t it be a dream to have a column in your Bronze table (or Landing, whatever you prefer to call it) telling you which file each record came from?","title":"Databricks Tip: How to Track Where Each Record Came From, File by File - input_file_name()"},{"content":"In May 2023, Microsoft announced Microsoft Fabric. This new solution extends the integration promise made in Azure Synapse Analytics to cover every analytics workload, and it can be used by everyone from data engineers to knowledge-focused business professionals. Microsoft Fabric combines Power BI, Data Factory and Data Lake into a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to cut costs and implementation time while enabling advanced data science capabilities.\nBut now what? Where does this leave the companies that invested in the Azure platform? Microsoft Fabric is the natural successor to Synapse. However, organizations that made a significant commitment to Azure Synapse Analytics may be disappointed to find that there is no automatic upgrade path for their workloads.\nDepending on the workload, different degrees of effort will be needed to adapt them to run on Microsoft Fabric. For some organizations, the costs of that migration effort (and the risks inherent in any migration) may act as a barrier.\nThis article aims to help people who are familiar with Azure Synapse Analytics understand Fabric and lay out the cost/benefit of a future migration. We\u0026rsquo;ll describe how Synapse features map to their Fabric equivalents, highlighting the main differences along the way.\nFor companies that choose to migrate, some important factors will need to be considered. For example, Spark-based workloads offer a more direct migration path than SQL-based ones. However, there are significant gaps where Synapse features do not carry over to Fabric (we\u0026rsquo;ll go into detail later in this article). That complexity makes the decision more challenging. My recommendation is for organizations to invest in a \u0026ldquo;technical spike\u0026rdquo; to assess how Fabric can support their most common Synapse workloads, in order to gain valuable insight into the migration challenges and the long-term opportunities the platform offers.\nWhat are the main gaps for a migration? The following list maps each Azure Synapse versus Fabric feature, with the main issues to be resolved:\nSQL Serverless (Synapse) versus SQL Endpoint (Fabric): OPENROWSET syntax is not supported, which means SQL cannot be used to query files in the Data Lake. However, structured data placed in the OneLake \u0026ldquo;Tables\u0026rdquo; area can be queried via SQL in the \u0026ldquo;Default Warehouse\u0026rdquo;, so this provides functionality similar to OPENROWSET. Apache Spark Pools (Synapse) versus Managed Spark Pools (Fabric): Fabric is SaaS, so there is no need to create and manage Spark pools. You will be able to choose which version of the Spark environment you want to use and load specific Python packages into your environment, including the option to do so dynamically inside a notebook. Performance improvements mean the underlying Spark environment \u0026ldquo;spins up\u0026rdquo; in seconds rather than minutes. At last we have a worthy competitor to Databricks. Spark Notebooks (Synapse) versus Notebooks (Fabric): A number of new features have been added, such as the ability to add comments to notebooks and co-editing (several users can open and edit a Notebook at the same time), as well as Data Wrangler, with new utilities available in Notebooks to let data be explored. Synapse Studio versus Power BI Interfaces: The user experience is now organized around specific personas, \u0026ldquo;Data Engineering\u0026rdquo;, \u0026ldquo;Data Science\u0026rdquo;, \u0026ldquo;Data Warehousing\u0026rdquo; and \u0026ldquo;Real-time Analytics\u0026rdquo;, so you still have Power BI and Data Factory as standalone tools. Pipelines: Some pipeline actions were removed in Fabric. For example, integration with Machine Learning resources is now done through notebooks. Let\u0026rsquo;s talk about improvements.. Looking at the broader Microsoft ecosystem, Fabric offers \u0026ldquo;out-of-the-box\u0026rdquo; integration with capabilities that would otherwise require creating and configuring additional Azure resources to integrate with Synapse, for example:\nAzure Machine Learning: there is no need to create an Azure Machine Learning instance to register your machine learning models and log experiments, since Fabric provides an MLFlow endpoint by default. Power BI: although there was integration between Synapse and Power BI, it is greatly improved with Fabric. Datasets can be easily created directly in OneLake, and models and measures can be built from the Fabric UX. In addition, a default dataset is created in every Lakehouse in Fabric, simplifying the process even further. Finally, the way datasets are presented in Fabric, not just as an input to Power BI but as a \u0026ldquo;data product\u0026rdquo; that can be consumed by many other means, feels like a big step forward. Commercial model and costs Microsoft Fabric adopts a different commercial model from Azure Synapse Analytics. For small organizations (with smaller workloads and early in their data journey), the \u0026ldquo;pay per query\u0026rdquo; model of SQL Serverless and the \u0026ldquo;pay per minute\u0026rdquo; nature of Spark Pools are differentiators compared with other cloud vendors.\nFabric largely abandons the \u0026ldquo;pay for what you use\u0026rdquo; approach. It adopts a capacity-based approach. This will force organizations to commit to a minimum monthly spend on the platform. Although lower tiers are available that will allow organizations with smaller workloads to adopt Fabric, the main concern is that this may result in an impact on Azure costs.\nConclusions Organizations currently using Azure Synapse Analytics should evaluate Microsoft Fabric and determine how it fits into their technology roadmap. The main factors to consider include:\nCost impact: how does the move from PaaS (Synapse) to SaaS (Fabric) affect cost? For example, can you save by eliminating the need to manage Azure resources? Pros and cons of SaaS: as a SaaS platform, Fabric takes an opinionated approach to deploying, configuring and implementing technologies over which you have more control with the PaaS nature of Synapse (and the broader Azure resources that usually come with it). Vendor lock-in: does moving to Fabric make it harder to switch to a different vendor in the future? Is this a significant issue for your company? Time to value: do the developer experience and productivity features in Fabric (for example, the new notebook experience) open up opportunities to simplify the end-to-end development lifecycle? Minimizing technical debt: will Fabric simplify your codebase and therefore reduce maintenance effort? For example, direct connectivity from Power BI to Delta tables in OneLake means a SQL layer (and the associated scripts to create the SQL views) no longer needs to be maintained. Azure costs: Fabric adopts a different billing model from Synapse, so how will this affect your monthly Azure OpEx? New features: looking beyond a \u0026ldquo;like for like\u0026rdquo; migration of existing functionality, what opportunities do Fabric\u0026rsquo;s new features offer, and do they unlock use cases that are sitting in your backlog? Strategy: how does Fabric fit into your long-term data and analytics strategy? If you believe there are opportunities, as a next step I recommend making a small, targeted investment in a \u0026ldquo;technical spike\u0026rdquo; to assess how well Fabric can support your data and analytics vision.\nReferences\nhttps://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/\n","permalink":"https://julianamarialop.github.io/en/articles/azure-synapse-analytics-vs-microsoft-fabric-another-migration-project/","summary":"In May 2023, Microsoft announced Microsoft Fabric. This new solution extends the integration promise made in Azure Synapse Analytics to cover every analytics workload, and it can be used…","title":"Azure Synapse Analytics vs Microsoft Fabric: How About Another Migration Project?"},{"content":"This article describes some special techniques for optimizing SQL queries Time goes by, new data ingestion techniques, languages and tools come along, but one thing is a universal truth. SQL never goes out of style!\nIn this guide, I\u0026rsquo;ve included several suggestions for optimizing SQL statements. I hope it\u0026rsquo;s useful for everyone.\nLet\u0026rsquo;s go!\n1. Try not to use SELECT * to query SQL; select specific fields instead. Bad example:\nSELECT * FROM funcionário; Good example:\nSELECT id,nome FROM funcionário; Rationale:\nBy using only the fields we need, we save resources and reduce network overhead. With * no index on the table is used 2. If you know the query has only one result, it\u0026rsquo;s recommended to use LIMIT 1 Suppose there\u0026rsquo;s an employee table and you want to find a person named Maria.\nCREATE TABLE employee ( id int (11) NOT NULL, nome varchar (50) DEFAULT NULL, idade int (2) DEFAULT NULL, data datetime DEFAULT NULL, sex int (1) DEFAULT NULL, PRIMARY KEY (id)); Bad example:\nSELECT id,nome FROM funcionário WHERE LOWER(name) = \u0026#39;Maria\u0026#39;; Good example:\nSELECT id,nome FROM funcionário WHERE LOWER(name) = \u0026#39;Maria\u0026#39; LIMIT 1; SELECT TOP 1 id,nome FROM funcionário WHERE LOWER(name) = \u0026#39;Maria\u0026#39;; Rationale :\nAfter adding LIMIT 1, once a matching record is found, it won\u0026rsquo;t keep scanning, and efficiency improves considerably. 3. Try to avoid using the OR condition to join conditions Create a new user table with an index on the idx_userId column\nCREATE TABLE user ( id int (11) NOT NULL AUTO_INCREMENT, userId int (11) NOT NULL, age int (11) NOT NULL, name varchar (255) NOT NULL, PRIMARY KEY (id), KEY idx_userId (userId)) Suppose you now need to query users with userid 1 or who are 18 years old; it\u0026rsquo;s easy to end up with the following SQL.\nBad example:\nSELECT * FROM usuário WHERE userid = 1 OR age = 18; Good example:\n// Use union all SELECT * FROM usuário WHERE userid = 1 UNION ALL SELECT * FROM usuário WHERE idade = 18; // Ou escreva dois SQL separados SELECT * FROM usuário WHERE userid = 1; SELECT * FROM usuário WHERE idade = 18; Rationale :\nUsing OR can invalidate the index and therefore requires a full table scan. 4. Optimize your LIKE statement In day-to-day development, if you use fuzzy keyword queries, it\u0026rsquo;s easy to reach for LIKE, but it will probably invalidate your index.\nBad example:\nSELECT userId,nome FROM usuário WHERE userId LIKE \u0026#39;%123\u0026#39;; Good example:\nSELECT userId,nome FROM usuário WHERE userId LIKE \u0026#39;123%\u0026#39;; Rationale :\nWhen % is used at the beginning, the query scans the whole table; when it\u0026rsquo;s used at the end, the search only touches values starting with 1. 5. You should avoid using the ! = or \u0026lt;\u0026gt; operator in the WHERE clause as much as possible; otherwise the engine will stop using the index and perform a full table scan Bad example:\nSELECT idade,nome FROM usuário WHERE idade \u0026lt;\u0026gt; 18; Good example:\n// Você pode considerar duas gravações sql separadas ou UNION ALL SELECT idade, nome do usuário WHERE idade \u0026lt; 18; SELECT idade, nome do usuário WHERE idade \u0026gt; 18; Reason : using ! = or \u0026lt;\u0026gt; will probably invalidate the index\n6. Use the DISTINCT keyword carefully The DISTINCT keyword is usually used to filter out duplicate records and return unique ones. When used to query one field or a few fields, it has an optimizing effect on the query.\nHowever, when there are many fields, it greatly reduces query efficiency.\nBad example:\nSELECT DISTINCT * FROM usuário; Good example:\nSELECT DISTINCT nome FROM usuário; Rationale :\nthe CPU time and occupancy time of a statement with distinct are higher than those of a statement without it. Because, when querying many fields, if you use distinct, the database engine will compare the data and filter out the duplicates. However, this comparison and filtering process consumes system resources and CPU time. 7. Remove redundant and duplicate indexes Bad example:\nKEY idx_userId (userId) KEY idx_userId_age (userId,age) Good example:\n// Exclua o índice userId, porque o índice combinado (A, B) é equivalente à criação dos índices (A) e (A, B) KEY idx_userId_age (userId,age) Rationale :\nDuplicate indexes have to be maintained, and the optimizer also has to consider them one by one when optimizing queries, which hurts performance. 8. If the data volume is large, optimize your DELETE Avoid modifying or deleting a lot of data at once, because it will cause high CPU usage, which will affect other people\u0026rsquo;s access to the database.\nBad example:\n// Excluir 100.000 ou mais 1 milhão de cada vez DELETE FROM usuário WHERE id \u0026lt; 100000; Good example:\n// Excluir em lotes DELETE FROM usuário WHERE \u0026lt; 500; DELETE FROM produto WHERE id \u0026gt; = 500 AND id \u0026lt; 1000 ； Rationale :\nwhen deleting a lot of data at once, you may hit a lock wait timeout exceeded error, so it\u0026rsquo;s recommended to work in batches. 9. Consider using default values instead of NULL in the where clause Good example:\nSELECT * FROM usuário WHERE idade IS NOT NULL; Good example:\nSELECT * FROM usuário WHERE idade \u0026gt; 0; // Defina 0 como padrão Rationale:\nIf you replace the null value with a default value, it usually makes indexing possible and, at the same time, the expression will be relatively clear. 10. Try replacing UNION with UNION ALL If there are no duplicate records in the search results, it\u0026rsquo;s recommended to replace union with union all.\nBad example:\nSELECT * FROM usuário WHERE userid = 1 UNION SELECT * FROM usuário WHERE idade = 10 Good example:\nSELECT * FROM usuário WHERE userid = 1 UNION ALL SELECT * FROM usuário WHERE idade = 10 Rationale :\nIf you use UNION, regardless of whether the search results repeat, it will try to merge and sort them before producing the final results. If the search results have no duplicate records, use UNION ALL instead of union, which will increase efficiency. 11. Use numeric fields as much as possible. If fields contain only numeric information, try not to design them as VARCHAR. Bad example:\nking_id varchar(20）NOT NULL; Good example:\nking_id int(20）NOT NULL; Rationale :\ncompared with numeric fields, character types reduce query and join performance and increase storage overhead. 12. Use VARCHAR/NVARCHAR instead of CHAR/NCHAR whenever possible Bad example:\ndeptName char(100) DEFAULT NULL Good example:\ndeptName varchar(100) DEFAULT NULL Rationale :\nFirst, since a variable-length field takes up little storage space, storage space can be saved. Second, for queries, searching a relatively small field is more efficient. Bonus: Use EXPLAIN to analyze your SQL plan When writing SQL in day-to-day development, try to build a habit. Use EXPLAIN to analyze the SQL you\u0026rsquo;ve written, especially the index.\nEXPLAIN SELECT * FROM user WHERE userid = 10086 AND age =18; Yes, we\u0026rsquo;ve reached the end. I hope you enjoyed it and got an idea of how to optimize and speed up your SQL queries.\nBe sure to check out the other articles I\u0026rsquo;ve written! The links are here at the bottom of the page..\nThank you and see you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/how-to-optimize-sql-queries-a-pocket-guide-to-data-tuning/","summary":"Time goes by, new data ingestion techniques, languages and tools come along, but one thing is a universal truth. SQL never goes out of style!","title":"How to Optimize SQL Queries: A Pocket Guide to Data \"Tuning\""},{"content":"This week (June 2020) a GCP update came out that lets you use the EXECUTE IMMEDIATE command in BigQuery. For more information, see the documentation.\nLet\u0026rsquo;s say we want to find the number of confirmed COVID cases over the last three days in several Canadian provinces. There\u0026rsquo;s a public BigQuery dataset we can query like this:\nSELECT * FROM `bigquery-public-data`.covid19_jhu_csse.confirmed_cases WHERE country_region LIKE \u0026#39;Canada\u0026#39; We get:\nThere\u0026rsquo;s one column for every date. How do we find the last three days for which there is data? Notice the format \u0026ldquo;_1_22_20, _1_23_20,\u0026hellip;.\u0026rdquo;\nFinding Columns We can use INFORMATION_SCHEMA to get the list of columns and find the last three days using:\nSELECT column_name, parse_date(\u0026#39;_%m_%d_%y\u0026#39;, column_name) AS date FROM `bigquery-public-data`.covid19_jhu_csse.INFORMATION_SCHEMA.COLUMNS WHERE table_name = \u0026#39;confirmed_cases\u0026#39; AND STARTS_WITH(column_name, \u0026#39;_\u0026#39;) ORDER BY date DESC LIMIT 3 Returning\nBuilding a dynamic SQL statement You can run a dynamic SQL statement using EXECUTE IMMEDIATE. For example, suppose we have a variable holding the column name _5_18_20; this is how to use it to run a SELECT statement:\nDECLARE col_0 STRING; SET col_0 = \u0026#39;_5_18_20\u0026#39;; EXECUTE IMMEDIATE format(\u0026#34;\u0026#34;\u0026#34; SELECT country_region, province_state, %s AS cases_day0 FROM `bigquery-public-data`.covid19_jhu_csse.confirmed_cases WHERE country_region LIKE \u0026#39;Canada\u0026#39; ORDER BY cases_day0 DESC \u0026#34;\u0026#34;\u0026#34;, col_0); Look at the query above. First of all, because I\u0026rsquo;m declaring a variable and so on, this is a BigQuery script in which each statement ends with a semicolon.\nI\u0026rsquo;m using BigQuery\u0026rsquo;s string format function to build the statement I want to run. Since I\u0026rsquo;m passing a string, I specify %s in the format string and pass in col_0.\nThe result comes in two steps:\nwith the result of the second step being:\nScripting the last 3 days We can combine the three ideas above (INFORMATION_SCHEMA, scripting and EXECUTE IMMEDIATE) to get the data for the last three days.\nDECLARE columns ARRAY\u0026lt;STRUCT\u0026lt;column_name STRING, date DATE\u0026gt;\u0026gt;; SET columns = ( WITH all_date_columns AS ( SELECT column_name, parse_date(\u0026#39;_%m_%d_%y\u0026#39;, column_name) AS date FROM `bigquery-public-data`.covid19_jhu_csse.INFORMATION_SCHEMA.COLUMNS WHERE table_name = \u0026#39;confirmed_cases\u0026#39; AND STARTS_WITH(column_name, \u0026#39;_\u0026#39;) ) SELECT ARRAY_AGG(STRUCT(column_name, date) ORDER BY date DESC LIMIT 3) AS columns FROM all_date_columns ); EXECUTE IMMEDIATE format(\u0026#34;\u0026#34;\u0026#34; SELECT country_region, province_state, %s AS cases_day0, \u0026#39;%t\u0026#39; AS date_day0, %s AS cases_day1, \u0026#39;%t\u0026#39; AS date_day1, %s AS cases_day2, \u0026#39;%t\u0026#39; AS date_day2 FROM `bigquery-public-data`.covid19_jhu_csse.confirmed_cases WHERE country_region LIKE \u0026#39;Canada\u0026#39; ORDER BY cases_day0 DESC \u0026#34;\u0026#34;\u0026#34;, columns[OFFSET(0)].column_name, columns[OFFSET(0)].date, columns[OFFSET(1)].column_name, columns[OFFSET(1)].date, columns[OFFSET(2)].column_name, columns[OFFSET(2)].date ); The steps:\nDeclare columns as an array variable that will store the column name and date for the three most recent days Set columns to the result of the query that gets the 3 days. Notice that I\u0026rsquo;m doing an ARRAY_AGG so the full result set is stored in a single variable. Format the query. Notice that I\u0026rsquo;m using % t to represent a timestamp (see the String format documentation for details) and passing six parameters. The result looks like this:\nUsing EXECUTE IMMEDIATE Instead of using String format, you can run named variables like this:\nEXECUTE IMMEDIATE \u0026#34;\u0026#34;\u0026#34; SELECT country_region, province_state, _5_18_20 AS cases FROM `bigquery-public-data`.covid19_jhu_csse.confirmed_cases WHERE country_region LIKE @country ORDER BY cases DESC LIMIT 3 \u0026#34;\u0026#34;\u0026#34; USING \u0026#39;Canada\u0026#39; AS country; You can also run positional variables using question marks:\nEXECUTE IMMEDIATE \u0026#34;\u0026#34;\u0026#34; SELECT country_region, province_state, _5_18_20 AS cases FROM `bigquery-public-data`.covid19_jhu_csse.confirmed_cases WHERE country_region LIKE ? ORDER BY cases DESC LIMIT ? \u0026#34;\u0026#34;\u0026#34; USING \u0026#39;Canada\u0026#39;, 3; The USING clause gets tricky in some situations. For example, the following doesn\u0026rsquo;t work:\nEXECUTE IMMEDIATE \u0026#34;\u0026#34;\u0026#34; SELECT country_region, province_state, ? AS cases FROM `bigquery-public-data`.covid19_jhu_csse.confirmed_cases WHERE country_region LIKE ? ORDER BY cases DESC LIMIT ? \u0026#34;\u0026#34;\u0026#34; USING \u0026#39;_5_18_20\u0026#39;, \u0026#39;Canada\u0026#39;, 3; -- ISSO NÃO FUNCIONA !!! That\u0026rsquo;s because the first parameter is interpreted as:\n\u0026#39;_5_18_20\u0026#39; AS cases So you can\u0026rsquo;t pass a column name through USING. That\u0026rsquo;s why I recommend using String FORMAT () to build the query to be executed immediately, because it doesn\u0026rsquo;t have these problems.\nThe PIVOT() feature is coming soon!!!\nThank you and see you soon!\n","permalink":"https://julianamarialop.github.io/en/articles/how-to-use-dynamic-sql-in-bigquery-with-execute-immediate/","summary":"This week (June 2020) a GCP update came out that lets you use the EXECUTE IMMEDIATE command in BigQuery. For more information, see the documentation.","title":"How to Use Dynamic SQL in BigQuery with EXECUTE IMMEDIATE"},{"content":"Last week, I wrote an article about data pipelines (check it out here) covering the main characteristics of a robust and optimized build, ideally using a Big Data environment.\nContinuing on that theme, let\u0026rsquo;s look at the main components for building a highly scalable environment.\nThe figure below shows an architecture that uses open source technologies to implement every stage of the big data pipeline. The preparation and compute stages are often merged to optimize compute costs.\nThe main components of big data architecture and technology are the following:\nHTTP / MQTT endpoints for data ingestion and for serving results. There are several frameworks and technologies for this. Pub/sub message queue for ingesting high-volume streaming data. Kafka is currently the most common choice in on-premises environments. High-volume, low-cost data storage for the data lake (and data warehouse): Hadoop HDFS or cloud blob storage such as AWS S3 or Azure Blob. Query and data catalog infrastructure to turn a data lake into a data warehouse. Apache Hive is a popular query language option. MapReduce batch compute engine for high-throughput processing, for example Hadoop MapReduce and Apache Spark. Data streaming, for example Apache Storm and Apache Flink. Apache Beam has also emerged as an option for data flows. Machine learning frameworks for data science and ML. Scikit-Learn, TensorFlow and PyTorch are popular options for building and training models. Deployment orchestration options include Hadoop YARN and Kubernetes / Kubeflow. Big Data Architecture in the Cloud With the rise of computing as a service, you can stand up your environment faster and get more out of your money. Several components in the architecture can be replaced by equivalent services from the cloud provider.\nTypical big data pipeline architectures on Amazon Web Services, Microsoft Azure and Google Cloud Platform (GCP) are shown below. Each one maps closely to the general big data architecture discussed in the previous section. You can use them as a reference to shortlist the technologies that fit your needs.\nAWS Big Data Environment Azure Big Data Environment Google Big Data Environment Conclusion This article covered the main components for deploying a Big Data environment. There are basically two main kinds of \u0026ldquo;infrastructure\u0026rdquo;, on-premises and cloud, and either way it starts with adopting storage, processing and queuing services.\nChoosing the best environment depends on a few pillars to analyze, such as cost, the complexity of the service or technology to be deployed, and the infrastructure the company already has.\nThank you, and see you next time.\n","permalink":"https://julianamarialop.github.io/en/articles/big-data-architecture-choose-the-right-plumbing-for-your-pipelines/","summary":"Last week, I wrote an article about data pipelines (check it out here) covering the main characteristics of a robust and optimized build, ideally using a…","title":"Big Data Architecture: Choose the Right Plumbing for Your Pipelines"},{"content":"Introduction - What are data pipelines? Pipeline is an English term that can be translated into Portuguese as “tubagem” or “canalização” (piping or plumbing). In our language, the concept is used to refer to a computing architecture. These virtual pipes are created to segment data and, in doing so, increase the throughput of a digital system.\nA data flow pipeline is a series of components, or data flow blocks, and each one performs a specific task that contributes to a larger goal.\nIts set of resources brings together the end-to-end operation of collecting the data, transforming it, training a model, delivering insights, and applying the model when and where action needs to be taken to reach the business goal.\nScalable, efficient data pipelines are as important to the success of analytics, data science and machine learning as an army\u0026rsquo;s front-line strategies are to winning a war.\nRoughly 50% of the effort in designing this orchestra goes into preparing data for analysis and Machine Learning. The remaining effort is split: 25% goes into making model insights and inferences easily consumable at scale, and 25% into training.\nA big data pipeline ties everything together. It is the railroad on which heavy, fast freight cars run. Long-term success depends on building and deploying the services correctly.\nPerspectives - A View by Area There are three stakeholders involved in building data analytics or machine learning applications: data scientists, engineers and business managers.\nFrom the data science perspective, the goal is to find the most robust and computationally cheapest model for a given problem using the available data.\nFrom the engineering point of view, the goal is to build things others can rely on; to innovate by creating new things or finding better ways to build existing things that run 24x7 without much human intervention.\nAs for the business perspective, the goal is to add value for customers; science and engineering are means to that end.\nLooking more closely through the engineering lens, we can name the following pillars as the main characteristics for success:\nAccessibility: data easily accessible to data scientists for hypothesis evaluation and model experimentation, preferably through a query language. Scalability: the ability to scale as the volume of ingested data grows, while keeping cost low. Efficiency: data and machine learning results are ready within the specified latency to meet business objectives. Monitoring: automatic alerts on the health of the data and the pipeline, needed for a proactive response to potential business risks. Building Stages A data pipeline has five stages, grouped into three phases in its construction:\nPhase I - Data engineering: collection, ingestion, preparation (~ 50% of the effort) Phase II - Analytics / Machine Learning: computation (~ 25% of the effort) Phase III - Delivery: presentation (~ 25% of the effort) Collection: Data sources (mobile apps, websites, web applications, microservices, IoT devices, etc.) are instrumented to collect relevant data.\nIngestion: The instrumented sources pump data into various entry points (HTTP, MQTT, message queues, etc.). There may also be jobs that import data from services such as Google Analytics. Data can come in two forms: blobs and streams. All of this data is collected in a Data Lake.\nPreparation: This is the extract, transform, load (ETL) operation that cleans, conforms, shapes, transforms and catalogs the data blobs and streams in the data lake, preparing the data for ML and storing it in a Data Warehouse.\nComputation: this is where analytics, data science and machine learning happen. Computation can be a combination of batch and stream processing. Models and insights (structured data and streams) are stored back in the Data Warehouse.\nPresentation: information is delivered through dashboards, emails, text messages and push notifications. ML model inferences are exposed as microservices.\nConclusion As you have seen, a data pipeline is like an “orchestra” that performs defined tasks sequentially or in parallel within a given period of time.\nThe more aligned your pipeline is with the expectations of data scientists and business managers, the easier it will be to obtain insights that deliver positive results for the company.\nPipeline development also needs to be thought out and designed in an optimized way, preferably on a robust Big Data environment.\nNow think about it: at your company, are the orchestration processes and data loads robust and fast enough to meet your business demand?\nThank you and see you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/data-pipelines-is-your-company-plumbing-clean-water-or-sewage/","summary":"Pipeline is an English term that can be translated into Portuguese as “tubagem” or “canalização” (piping or plumbing). In our language, the concept is used to refer to a computing architecture. These virtual pipes are created…","title":"Data Pipelines - Is Your Company's Plumbing Carrying Clean Water or Sewage?"},{"content":"For some years now, companies all over the world have been tapping into the immense value Big Data can add to their business. Implementing Big Data ecosystems is a reality in both the global and the Brazilian market. With ever-growing demand, it has become very common to see plenty of openings for Data Engineers, Architects and Data Analysts on the LinkedIn timeline.\nWith so much demand popping up, many professionals are looking for knowledge and to keep up to date. With so many courses, tools and different ecosystems, where do you start?\nThere are two ways to use Big Data: set it up on your company\u0026rsquo;s premises (On Premise) or use a provider that offers a Big Data platform in the cloud (Cloud). In the past, companies only had the option of installing it locally, but that is no longer the case.\nSmall and midsize companies have taken a liking to cloud platforms, mostly because subscriptions are easy to acquire and processing services are quick to start. Today Microsoft, with the Azure ecosystem, holds the largest share of the Brazilian market, followed by Amazon. Of course, here I am only sharing my personal opinion based on my experience; I\u0026rsquo;ll leave a source to back it up: https://cio.com.br/azure-passa-aws-como-infraestrutura-de-nuvem-favorita-dos-cios/\nSo, after all, where do I start studying? Big Data On Premise or Cloud?\nHere is my answer: Start studying based on the direction you want to give your career! Let me explain\u0026hellip;\nIf you are an IT professional whose goal is to work at a large company, such as telecoms, traditional banks, government and others, you want a permanent position as an employee, and you come from a Business Intelligence and Analytics career, start with On Premise. Begin with the concepts of Hadoop and Spark. There is wonderful material that Fábio Jardim compiled on GitHub. https://github.com/fabiogjardim/bigdata_docker\nNow, if your goals are different, if you want to work in the market as an independent consultant and keep up with the trends of the big vendors, I recommend starting by creating an Azure account (portal.azure.com) and studying the content Microsoft itself provides through the Learn platform (https://docs.microsoft.com/pt-br/learn/?WT.mc_id=home_homepage-azureportal-learn)\nI hope this helps! I\u0026rsquo;m available for any questions from my fellow professionals.\nBest regards!\n","permalink":"https://julianamarialop.github.io/en/articles/big-data-on-premise-vs-cloud-computing-market-view-study-guide/","summary":"For some years now, companies all over the world have been tapping into the immense value Big Data can add to their business. Implementing Big Data ecosystems is a reality in both the global and the Brazilian market. With ever…","title":"Big Data: On Premise vs Cloud Computing - A Market View and a Starter Study Guide"},{"content":"Imagine a team made up of a SQL developer, a Python professional and a Scala contributor. They all work together to ingest data into your Datalake, your Datawarehouse or simply a Relational Database, using Spark clusters. Wouldn\u0026rsquo;t it be great if they could all work on a unified platform that understands every programming language in the same script or block? That platform exists!\nDatabricks is an analytics platform based on Apache Spark. Designed with the founders of Apache Spark, Databricks gives us streamlined workflows and an interactive workspace that enables collaboration between data scientists, data engineers and business analysts.\nWith support for Python, Scala, R and SQL, plus deep learning libraries and frameworks such as TensorFlow, Pytorch and Scikit-learn, using Databricks removes all the hassle and complexity of getting a Spark cluster. Its notebooks deliver a seamless experience with no management headaches, thanks to integration with the leading cloud providers, including Amazon AWS and Microsoft Azure.\nIn this tutorial we\u0026rsquo;ll walk through the main steps to get started with Databricks on Azure. The first part covers setting up and launching the environment. The second part covers the steps to get a notebook up and running. The last part gives you a few basic queries to check that everything is working properly.\nHow to create your first cluster Create a subscription at portal.azure.com. Under \u0026ldquo;Create a resource\u0026rdquo;, search for the term \u0026ldquo;Databricks\u0026rdquo;\nNext, launch the \u0026ldquo;Workspace\u0026rdquo; you created.\nNow you\u0026rsquo;re in the Databricks Workspace\nThe next step is to create a cluster that will run the source code in your notebooks.\nYou can adjust the cluster size according to the price you want to pay. Note that the cluster will shut down automatically after a period of inactivity.\nCreating the cluster can take several minutes. Meanwhile, we can create our first notebook and attach it to that cluster.\nNow what? Python, Scala or SQL? Which one should I use in my notebook.. This is the main question every new developer asks. If you\u0026rsquo;re familiar with Python (Python is the data engineers\u0026rsquo; language of choice), you can stick with Python, since you can do almost everything with it.\nHowever, Scala is Spark\u0026rsquo;s native language. Since Spark itself is written in Scala, you\u0026rsquo;ll find 80% of the examples, libraries and StackOverflow discussions in Scala.\nThe good news is that you don\u0026rsquo;t have to choose in a Databricks notebook, because you can mix both languages to keep development simple with Python.\nThe Python library for working with Spark is called PySpark.\nCreating your first notebook.. On the workspace home page, click \u0026ldquo;New Notebook\u0026rdquo;\nYou can also create your notebook in a specific folder\nSelect the default language for your notebook. In this example I\u0026rsquo;ll use Python\nOnce the notebook is created, let\u0026rsquo;s attach the cluster to run the code.\nType some Python or Scala code. For Scala, you need to add \u0026ldquo;%scala\u0026rdquo; on the first line, since the default language we chose is Python:\nTo run the code, you can use the shortcut CTRL + ENTER or SHIFT + ENTER. If the cluster isn\u0026rsquo;t running, a prompt will ask you to confirm starting it. Now we\u0026rsquo;re ready for the next step: data ingestion. In the example below we fetch the CSV file and then read it with \u0026ldquo;spark.read.csv\u0026rdquo;\n%python # Use the Spark CSV datasource with options specifying: # - First line of file is a header # - Automatically infer the schema of the data data = spark.read.csv(\u0026#34;/databricks-datasets/samples/population-vs-price/data_geo.csv\u0026#34;, header=\u0026#34;true\u0026#34;, inferSchema=\u0026#34;true\u0026#34;) data.cache() # Cache data for faster reuse data ​= data.dropna() # drop rows with missing values OK, got it.. But I\u0026rsquo;m a SQL Developer and I don\u0026rsquo;t know Python.. No problem, we can do the same task with SQL commands.. Let\u0026rsquo;s create the \u0026ldquo;data_geo\u0026rdquo; table..\n%python # Register table so it is accessible via SQL Context data.createOrReplaceTempView(\u0026#34;data_geo\u0026#34;) And now for the query\n%sql select `State Code`, `2015 median sales price` from data_geo As we can see, there are many ways to develop in Databricks, which makes it a very interesting platform. Another advantage is the price, which is much cheaper than traditional tools such as Integration Services in the Cloud, for example..\nIn the next article, I\u0026rsquo;ll focus on Databricks SQL functions and Databricks Delta.\nI hope this short explanation was helpful.. Thank you!\n","permalink":"https://julianamarialop.github.io/en/articles/databricks-the-swiss-army-knife-of-your-big-data-platform/","summary":"Imagine a team made up of a SQL developer, a Python professional and a Scala contributor. They all work together to ingest data into your Datalake, your Datawarehouse or simply…","title":"Databricks: The Swiss Army Knife of Your Big Data Platform"},{"content":"Increasingly popular and widely used, Apache Hive is a data warehouse system for Hadoop. Hive enables data summarization, querying, and analysis. Hive queries are written in HiveQL, a query language similar to SQL. Hive lets you project structure onto large volumes of unstructured data.\nOnce you\u0026rsquo;ve defined the structure, you can use HiveQL to query the data without any knowledge of Java or MapReduce.\nIn this article, we\u0026rsquo;ll cover the main commands for using Hive. Take a look:\nMain Queries Object and Structure Queries For more information and the complete documentation, visit https://cwiki.apache.org/confluence/display/Hive/LanguageManual\nI hope this post has added to your learning. See you next time!\n","permalink":"https://julianamarialop.github.io/en/articles/afraid-of-bees-tips-and-tricks-for-apache-hive/","summary":"Increasingly popular and widely used, Apache Hive is a data warehouse system for Hadoop. Hive enables data summarization, querying, and analysis. Hive queries are written in HiveQL, which is a language…","title":"Afraid of Bees? Tips and Tricks for Apache Hive"},{"content":"In this article, I\u0026rsquo;ll walk through the main Hadoop operations for working with HDFS through shell commands. To try them out, you can run these commands on one of the VMs from Cloudera, Hortonworks, HDInsight, or on your own cluster setup if you have one.\n1 - Create a directory in HDFS hadoop fs -mkdir \u0026lt;pastas\u0026gt; Exemplo: hadoop fs -mkdir /home/user/diretorio_1 /home/user/diretorio_2 2 - List the contents of a directory hadoop fs -ls \u0026lt;pastas\u0026gt;  Examplo: hadoop fs -ls /home/user/diretorio_1 3 - Upload and download a file in HDFS Upload - Copy a single file or several files from the local file system to Hadoop\nhadoop fs -put \u0026lt;pasta de origem\u0026gt; ... \u0026lt;pasta de destino HDFS\u0026gt; Exemplo: hadoop fs -put /home/user/ArquivoExemplo.txt /user/HDFS/diretorio_1/ Download - Copy files to local directories\nhadoop fs -get \u0026lt;pasta de Origem HDFS\u0026gt; \u0026lt;pasta de destino\u0026gt; Exemplo: hadoop fs -get /user/HDFS/diretorio_1/ArquivoExemplo.txt /home/ 4. View the contents of a file hadoop fs -cat \u0026lt;pastas e nome de arquivo\u0026gt; Exemplo: hadoop fs -cat /user/HDFS/diretorio_1/ArquivoExemplo.txt 5. Copy files between HDFS directories hadoop fs -cp \u0026lt;pasta de origem\u0026gt; \u0026lt;pasta de destino\u0026gt; Exemplo: hadoop fs -cp /user/HDFS/diretorio_1/ArquivoExemplo.txt /user/HDFS/dir_2 This command also lets you copy multiple files; to do that, use only the folder structure in the source reference.\n6. Move files hadoop fs -mv \u0026lt;pasta de origem\u0026gt; \u0026lt;pasta de destino\u0026gt; Exemplo: hadoop fs -mv /user/HDFS/diretorio_1/Arquivo.txt /user/HDFS/dir_2 7. Remove a file or directory from HDFS hadoop fs -rm \u0026lt;pasta ou arquivo\u0026gt; Exemplo: hadoop fs -rm /user/HDFS/diretorio_1/ArquivoExemplo.txt 8. Show the last lines of a file hadoop fs -tail \u0026lt;nome de arquivo\u0026gt; Exemplo: hadoop fs -tail /user/HDFS/diretorio_1/ArquivoExemplo.txt I hope this post has added to your learning and helps you in your day-to-day work handling files.\nSee also:\n","permalink":"https://julianamarialop.github.io/en/articles/taming-the-elephant-key-hadoop-commands-for-managing-hdfs/","summary":"In this article, I\u0026rsquo;ll walk through the main Hadoop operations for working with HDFS through shell commands. To try them out, you can run these commands on one of the VMs from Cloudera, Hortonworks, HDInsight, or…","title":"Taming the Elephant - Key Hadoop Commands for Managing HDFS"},{"content":"Over the years, data warehouses have been the main tool for company decision-making and analysis. These include conventional OLAP and ETL tools supplied by countless hardware and software vendors, DW appliances built by combining databases and servers, in-memory databases and virtualization tools.\nBut how do you accommodate and process an ever-growing amount of data at an acceptable speed for analysis and for finding information that\u0026rsquo;s crucial to the business?\nIn Brazil, we\u0026rsquo;re still at a very early stage when it comes to Big Data. I believe that companies will soon start their projects to \u0026ldquo;migrate\u0026rdquo; conventional DW databases to HDFS.\nIn my opinion, the BI professional of today who invests in learning Hadoop, Spark and Fast Data loading methods will be one step ahead in the job market.\nThe Hadoop ecosystem, which is evolving alongside large data environments, also has a set of technologies for building a data warehouse. For example, Apache Hive is a data warehouse option within the Hadoop ecosystem that can process large data sets stored in distributed storage environments using SQL.\nSo, is there a recipe or a magic formula for migrating the data? Unfortunately not, but we can follow a few guidelines.\n1 - Denormalize Denormalization is commonly used to improve performance when implementing a Star Schema model on top of the Hadoop distributed file system (HDFS). Unlike an RDBMS, Hadoop tolerates the data overlap caused by denormalization thanks to relatively cheap hardware and software.\nTherefore, the decision to denormalize data sets that change frequently should be made carefully, weighing the trade-off between query performance and deployment. The following is an example of the denormalization method that can be used to build a Hadoop-based data warehouse.\n1.1 - Merge Dimensions Several dimensions with a small number of rows are bundled and turned into a single dimension data set, so the number of fact data sets and joins can be reduced. It\u0026rsquo;s hard to optimize the system with a large number of joins between dimension data sets and fact data sets when using Hadoop.\nA small number of large files is better than a large number of small files for optimal performance with Hadoop; one large dimension is processed faster than several small dimensions.\n1.2 - Merge Facts with Dimensions Dimension and fact data sets that are queried together are organized into a single set by applying the join during the deployment process instead of at query time on the user side.\nThe data sets are joined at data load time to reduce the number of joins. It\u0026rsquo;s similar to the dimension merging process, except that the merge is fact-to-dimension.\n1.3 - Generate Historical and Current Data Sets Together for Loading Since it\u0026rsquo;s usually faster to regenerate the entire data set than to update a single event, given how HDFS works, the data sets needed for the loading process are appended according to their temporal and business characteristics.\nIt\u0026rsquo;s better to reload all the current content together with the history than to run multiple updates.\n2 - Organize Fact tables in a \u0026ldquo;Columnar\u0026rdquo; way It may sound strange, but it\u0026rsquo;s much better to have hundreds of columns than millions of rows. Especially when users build a data warehouse that extracts and uses part of a dataset\u0026rsquo;s columns, a columnar format that allows partial column extraction greatly improves query performance.\nAs an example, we can have a Sales fact organized by \u0026ldquo;Month/Year\u0026rdquo; that expands its columns with each new period.\n3 - Compress the Data Compression helps improve processing performance by reducing not only the size of the stored data but also the amount of disk I/O. It\u0026rsquo;s better to use a compression codec that supports splittable processing so the nodes can be processed in parallel in a distributed environment.\nIf compression isn\u0026rsquo;t feasible, additional splittable file storage formats, such as Avro, are needed. As with file storage, different compression methods can be used for each partition within a single data set.\n4 - Partition the Data Data sets are partitioned according to data query patterns, so results can be found without scanning the entire data set. Partitioning helps improve performance, since it reduces the objects a query has to scan and reduces I/O.\nOnce a data set is split into partitions according to some criterion, such as dates that reflect user query patterns, only one daily partition needs to be scanned when extracting data for a given date, which makes it far more efficient than a full scan of the whole data set.\nThe partitioning method is similar to building a data warehouse based on an RDBMS. One difference is that, unlike an RDBMS that defines how a data set is partitioned, Hadoop creates several partitioning strategies according to a user\u0026rsquo;s query patterns.\nHadoop copies the original dataset and creates several data sets, and each data set is strategically partitioned. In other words, Hadoop uses much more disk space, copying and multiplying a data set to improve performance.\n5 - Bucketing Bucketing is a process by which a dataset is split into buckets, groups of clustered data, using the values of a given column. Each block can have several column values as criteria, but a given value among the criteria column\u0026rsquo;s values is included in only one block.\nAnother benefit of bucketing is that it makes data sets small enough to use a map-side join, which loads that block into memory for fast processing.\n6 - Index the Data In Apache Hive, you can generate indexes that play roles similar to RDBMS indexes. Hive supports compact indexes, bitmap indexes and so on. It\u0026rsquo;s important to first analyze user query patterns in order to generate indexes that reflect those patterns (as in an RDBMS indexing strategy).\nWhen using an index, statistics should be collected regularly to help the optimizer find the best route.\nConclusion The fundamental value of a data warehouse is the same, even when it\u0026rsquo;s built as an application within a Hadoop ecosystem. Fortunately, data warehouse building methodology can be used without much change when building Hadoop-based databases.\nAs long as the characteristics of Hadoop ecosystems are reflected in your physical modeling process, the data warehouse will be ready for the era of big data, where structured and unstructured data will be integrated.\nReferences\nRalph Kimball and Matt Brandwein – Hadoop 101 for EDW Professionals Hadoop 101 for EDW Professionals – Dr. Ralph Kimball Answers Your Questions Ralph Kimball and Eli Collins – EDW 101 for Hadoop Professionals Ralph Kimball – Newly Emerging Best Practices for Big Data Ralph Kimball – The Evolving Role of the Enterprise Data Warehouse in the Era of Big Data Analytics Josh Wills - What Comes After The Star Schema? See also:\nNoSQL databases: what do you mean there\u0026rsquo;s no Query?\nWhat you need to know about Big Data so you don\u0026rsquo;t embarrass yourself at the coffee machine?\n","permalink":"https://julianamarialop.github.io/en/articles/a-dw-on-hadoop-how-to-model-your-star-schema-for-hdfs/","summary":"Over the years, data warehouses have been the main tool for company decision-making and analysis. These include conventional OLAP and ETL tools supplied by countless hardware and…","title":"A DW on Hadoop? How to model your Star Schema for HDFS"},{"content":"Continuing the series of articles on Big Data, let\u0026rsquo;s talk a bit about non-relational databases.\nPersonally, when I first started hearing the term \u0026ldquo;NoSQL\u0026rdquo;, my head would tie itself in knots. \u0026ldquo;NOSQL databases, what do you mean there\u0026rsquo;s no query?\u0026rdquo;, \u0026ldquo;How do you store the data?\u0026rdquo;, \u0026ldquo;How do you retrieve the information?\u0026rdquo;.\nTo start building our understanding, I\u0026rsquo;ll go back to the topic of MapReduce covered in the previous article. As mentioned, in Hadoop the way to check, for example, how many times a given term showed up on social media is to use the Reduce framework.\nWhen a job starts on a node, the \u0026lt;key,value\u0026gt; pairs are transferred in chunks to the mapping nodes. A new mapping instance is created for each input record. These pairs are collected locally on the mapping node through a function on a reduce node.\nSo, how do you store these Key/Value pairs? With NoSQL databases. Basically, the big advantage of this kind of storage is how flexibly the data is structured. Notice below that we no longer have tables, rows and columns.\nFine, but how do you run a simple query on this database if it can\u0026rsquo;t be done with SQL? That\u0026rsquo;s exactly where programming languages come into the picture. Python, Java, R and so many others help us search and cross-reference this data.\nYes, I know, this image is pretty scary and makes \u0026ldquo;Selects\u0026rdquo; look like child\u0026rsquo;s play. But don\u0026rsquo;t panic, there are already IDEs on the market that help you build these queries.\nSo what happens to the names Rows, Tables and Columns? See below.\nKeep in mind that Key/Value databases are the simplest in the NoSQL category; there are other types based on Columns, Documents and Graphs.\nColumn-Based (Column Stores): Hbase, Cassandra, Hypertable, Accumulo, Amazon SimpleDB, Cloudata, Cloudera, SciDB, HPCC, Stratosphere;\nDocument-Based (Document Stores): MongoDB, CouchDB, BigCouch, RavenDB, Clusterpoint Server, ThruDB, TerraStore, RaptorDB, JasDB, SisoDB, SDB, SchemaFreeDB, djondb;\nGraph-Based (Graph-Based Stores): Neo4J, Infinite Graph, Sones, InfoGrid, HyperGraphDB, DEX, Trinity, AllegroGraph, BrightStarDB, BigData, Meronymy, OpenLink Virtuoso, VertexDB, FlockDB;\nKey-Value-Based (Key-Value Stores): Dynamo, Azure Table Storage, Couchbase Server, Riak, Redis, LevelDB, Chordless, GenieDB, Scalaris, Tokyo, Cabinet/Tyrant, GT.M, Scalien, Berkeley DB, Voldemort, Dynomite, KAI, MemcacheDB, Faircom C-Tree, HamsterDB, STSdb, Tarantool/Box, Maxtable, Pincaster, RaptorDB, TIBCO Active Spaces, allegro-C, nessDB, HyperDex, Mnesia, LightCloud, Hibari, BangDB.\nIf you missed the beginning of this series and want to learn more about Big Data, also read What You Need to Know About Big Data So You Don\u0026rsquo;t Embarrass Yourself at the Coffee Machine\nIn the next post I\u0026rsquo;ll talk about Apache Spark.\nThank you.\n","permalink":"https://julianamarialop.github.io/en/articles/nosql-databases-what-do-you-mean-there-is-no-query/","summary":"Continuing the series of articles on Big Data, let\u0026rsquo;s talk a bit about non-relational databases.","title":"NoSQL Databases: What Do You Mean There's No Query?"},{"content":"Picture this: a director at the company you work for starts asking the following questions:\n\u0026ldquo;How many times is my company\u0026rsquo;s name mentioned on Twitter?\u0026rdquo; or \u0026ldquo;What are the biggest questions or complaints from customers on our Facebook page?\u0026rdquo;\nAs an IT analyst, how would you organize this data coming from social networks and produce a report or dashboard that makes sense to the leadership team?\nIt\u0026rsquo;s precisely to answer questions like these that Big Data platforms are becoming more and more famous and talked about.\nBut what is Big Data? These days the acronyms Hadoop, Hive, Spark, NoSQL and so many others are all the rage. But do you really understand what all this \u0026ldquo;alphabet soup\u0026rdquo; means?\nI\u0026rsquo;m also trying to understand what this whole ecosystem means, and in an attempt to organize my ideas and studies on such a fascinating subject, I\u0026rsquo;m starting a series of articles.\nWhat exactly is this pile of acronyms that everyone is talking about and chasing as if it were the last oil well discovered on the planet?\nAs almost always happens, creativity shows up in the face of a problem that needs solving. What do we do with this immense amount of data generated every second by computers and phones?\nPhotos, videos, posts: how can I use this information to better understand my customer\u0026rsquo;s profile and make more money? How do I organize, store and cross-reference these different forms of data?\nWith that in mind, brilliant minds started developing organization and storage models, the famous MapReduce and NoSQL Databases. The first ecosystem to gain fame, which you\u0026rsquo;ve surely heard of, was Hadoop.\nWritten in Java, the goal of this framework, broadly speaking, is to map information as if it were a dictionary, count which items are similar, and store the data in reduced form in a database built for that purpose (NoSQL). For each type of data there are different databases. I\u0026rsquo;ll go into more detail on this in the next article.\nTo keep it simple, Hadoop is the whole thing, the ecosystem. It has a MapReduce layer called YARN that stores data in the databases designed for each type of data. Cassandra, Hive, HBase: those are the ones.\nYou\u0026rsquo;re probably wondering, \u0026ldquo;Imagine how powerful a machine you need to run this enormous volume!\u0026rdquo; Well, that\u0026rsquo;s why a file system was created, HDFS, which splits everything into nodes and clusters. In other words, the amount of processing and hardware can grow or shrink as needed.\nHadoop has been around for more than 10 years and has proven to be the best solution for processing large data sets. MapReduce is a great solution for single-pass computations, but not very efficient for use cases that require computations and algorithms with multiple iterations\nTo solve this other data complexity problem, Spark enters the scene.\nI hope this post has contributed to your learning and helps you put together the pieces of today\u0026rsquo;s puzzle.\nReferences:\nhttp://spark.apache.org/\nhttp://hadoop.apache.org/\n","permalink":"https://julianamarialop.github.io/en/articles/what-you-need-to-know-about-big-data-so-you-dont-embarrass-yourself/","summary":"Picture this: a director at the company you work for starts asking the following questions:","title":"What you need to know about Big Data so you don't embarrass yourself at the coffee machine?"},{"content":"Technology executive with more than 20 years of experience leading Artificial Intelligence, Data and Analytics initiatives, working strategically on solution design, enterprise architecture and platform modernization at large organizations such as NET, Claro, Telefônica, C\u0026amp;A, Nestlé, UHG, Avon and Natura.\nOver her career she has built deep experience in data ecosystems, analytical environments, Big Data, Data Warehouse and scalable cloud architectures, connecting business vision, innovation and technical excellence. In recent years she has expanded her work in Azure Cloud, Databricks and AI-driven solutions, helping companies adopt modern platforms that speed up decisions, increase efficiency and create competitive advantage.\nShe has a strong track record in technical leadership, solution architecture and engineering team management, working in consulting firms and on highly complex projects. Her approach combines technical depth with an executive view, positioning AI, Data and Analytics as central pillars for transformation, automation, scalability and value creation.\nShe is also active in developing professionals and strengthening the technology ecosystem, with academic experience at FIAP, specialized training programs and active participation in the Microsoft community. She holds certifications such as Azure Data Engineer Associate and Azure Database Administrator Associate, and was recognized as a Microsoft Certified Trainer for 4 consecutive years.\nShe mentors professionals on career development and keeps an active presence on LinkedIn, where she shares her views on architecture, modern platforms, artificial intelligence and the future of Data and Analytics in organizations.\nInterests: Artificial Intelligence · Data \u0026amp; Analytics · Executive Technology Leadership · Solution Architecture · Azure Cloud · Databricks · Microsoft Fabric · AI Agents · Data Engineering · Cloud Architecture\n","permalink":"https://julianamarialop.github.io/en/about/","summary":"Juliana Maria Lopes: technology executive in AI, Data and Analytics.","title":"About"},{"content":"What does an artificial intelligence agent cost? The most common estimate multiplies the number of requests by the price of one model call. That method underestimates cost, because an agent does not execute a single call: it executes several iterations before it answers, and each iteration processes more text than the previous one.\nAn agent\u0026rsquo;s cost is determined by its architecture: how many iterations it executes, which model each step uses, and how much context is reprocessed on every iteration. This page presents that reasoning in four steps, with a single example: an online store\u0026rsquo;s customer service assistant answering the question \u0026ldquo;Where is my order?\u0026rdquo;. Each step introduces one concept, and the values can be changed to observe their effect on cost.\n","permalink":"https://julianamarialop.github.io/en/agent-cost/","summary":"An estimate of what an AI agent costs, derived from its architecture rather than its request volume, using the prices the vendors themselves publish.","title":"Agent Cost Calculator"}]